EvalPort export/import helpers for bridging evaluate's aggregate-only API to portable per-example results

[adhabnr-ux]

Motivation

I maintain EvalPort, an open, framework-agnostic JSON format for LLM evaluation test cases, graders, and results (schema-validated Suite/ResultSet documents — think "a portable file format eval tools can read/write instead of everyone inventing their own"). It already has independently-tested adapter packages for LangSmith, MLflow, Ragas, Braintrust, Arize Phoenix, Weave, and 16 other frameworks.

While building the evaluate one, I ran into a real, specific friction point that isn't a bug in evaluate — it's a genuine shape mismatch worth flagging directly. EvaluationModule.compute(predictions=..., references=...) always returns a whole-batch aggregate (confirmed against 0.4.6: exact_match, accuracy, f1, bleu all behave this way), which is exactly right for evaluate's own purpose. But anyone trying to export evaluate results into a per-example format (for storing in a dataset, diffing two model versions example-by-example, or feeding a downstream eval-tracking tool) has no supported path to get there — you either write your own per-example loop yourself, or you're stuck with only the aggregate.

I already solved this on my side (real per-example .compute() calls, one call per example, never interpolated from the aggregate — no fabricated numbers) and it works well enough that I'd like to check whether something like it belongs upstream, since I'd guess I'm not the only person who's hit this.

Feature description

A small, optional helper — something like evaluate.compute_per_example(metric, predictions, references, **kwargs) — that returns both the real whole-batch aggregate evaluate already computes today and a list of real per-example scores, obtained by calling the same metric's .compute() once per individual (prediction, reference) pair rather than once for the whole batch. No new metric logic, no new dependency — just a thin convenience wrapper around the existing .compute() call pattern, with a clear docstring caveat that per-example scoring isn't a meaningful concept for corpus-level statistics (classic BLEU being the standard example) the way it is for exact_match/accuracy/per-item f1.

If this is something the maintainers would want as a PR, I'm happy to open one against the API shape you'd prefer (a module-level function, a method on EvaluationModule, or something else) — I already have a tested reference implementation (compute_per_example) I built and validated for the standalone EvalPort adapter that I'd adapt to fit evaluate's own conventions rather than paste in as-is.

Code snippet demonstrating current vs. desired use

import evaluate

metric = evaluate.load("exact_match")
predictions = ["Paris", "5", "Berlin"]
references = ["Paris", "4", "Berlin"]

# Today: only the aggregate is available from the public API
metric.add_batch(predictions=predictions, references=references)
result = metric.compute()
# {"exact_match": 0.666...}  -- which examples failed? not recoverable from this alone

# Desired: an opt-in helper that also returns real per-example scores
aggregate, per_example = evaluate.compute_per_example("exact_match", predictions, references)
# aggregate == {"exact_match": 0.666...}          <- same real batch call as today
# per_example == [1.0, 0.0, 1.0]                  <- one real .compute() call per example, not interpolated

Additional context

For anyone curious what this looks like end-to-end (including the honesty caveat about corpus-level metrics), the standalone EvalPort adapter is public and tested: huggingface-evaluate-openeval-adapter (25/25 tests passing against the real installed evaluate package, README has a full "why this looks different" writeup). Not asking for an EvalPort integration itself — just flagging a real API gap I hit and offering to help close it if it's wanted upstream. Totally fine if this isn't a direction the maintainers want to take evaluate in; the adapter works standalone either way.