Maya Bennett
AI evaluation results have long been scattered across papers, leaderboards, and logs, making it difficult to trust or compare them. Now, the EvalEval Coalition and Hugging Face are closing that gap. Launched in February 2026, the EvalEval project (EEE) introduced a standardized JSON schema for reporting evaluation results. Hugging Face's Community Evals, also launched in February 2026, decentralizes how benchmark scores appear on the Hub. Together, they provide a unified system for users, researchers, and policymakers to verify and compare model evaluations.
Evaluation results are critical for measuring model capabilities, comparing models, and assessing safety, but they often vary wildly. For example, LLaMA 65B has been reported at both 63.7 and 48.8 on MMLU, depending on who ran the test and how. These discrepancies stem from unreported evaluation settings. EEE addresses this by defining a single JSON schema that records who ran the evaluation, which model was used, how it was accessed, generation settings, metric definitions, and optionally per-sample outputs in a companion JSONL file.
The schema, built with input from researchers and policy experts, accepts results from any source—harness logs, leaderboard scrapes, or paper numbers—and converts them into a consistent format. The GitHub repository includes converters, examples, and a contributor guide. Since its launch, the EEE datastore on Hugging Face has grown to roughly 229,000 evaluation results across over 22,000 models and 2,200 benchmarks, drawn from 31 different reporting formats. Reproducing those runs from scratch would cost hundreds of thousands of dollars, underscoring the value of preserving this data.
Now, with deeper integration, contributors can send EEE results to Hugging Face Community Evals using a converter that transforms EEE records into the YAML files Hugging Face expects. This eliminates the need to maintain results in two formats. First-party evaluators reporting their own models and third-party evaluators reporting on others can submit to both systems. When submitted via an organization's official Hugging Face account, results receive a verified checkmark on EvalEval, signaling authenticity.
How Hugging Face Community Evals Works with EvalEval
Hugging Face Community Evals operates on two fronts. First, a benchmark lives in a dataset repo registered with an eval.yaml file, which collects and displays a leaderboard of all reported scores. Second, a model's scores reside in .eval_results/*.yaml files within the model repo, appearing on the model card and feeding into the benchmark leaderboard. Scores come with badges indicating whether they are author-submitted, community-submitted, or independently verified. Anyone can submit a score via a pull request, and model authors can manage results on their repos.
When a result is sent to both EEE and Community Evals, it appears on the Hugging Face model page and the benchmark leaderboard, carrying a source badge linking back to the full EEE record. That record includes generation config, harness version, reproducibility notes, and instance-level data.
"The two destinations do different jobs toward the same goal. Hugging Face puts your result where people look at models, with a link back to the source. EEE keeps the full structured record that makes the result interpretable, and powers Eval Cards on top of it." — EvalEval Coalition
Cross-compatibility means the same evaluation appears both on model cards and in Eval Cards, which combine EEE run data with benchmark and model metadata into an interpretable record.
How It Works
Hugging Face stores eval scores in model repos as YAML files under .eval_results/. Required fields include the benchmark dataset, task, and value. A source block creates a backlink to EEE. For example:
- dataset: id: openai/gsm8k task_id: gsm8k value: 96.8 date: '2024-07-16' notes: '8-shot CoT' source: url: https://huggingface.co/datasets/evaleval/EEE_datastore/blob/main/flat/objects/<xx>/<yy>/<uuid>.json name: EvalEvalThe converter maps existing EEE records to these fields: source_data.hf_repo to dataset.id, evaluation_name to task_id, score_details.score to value, and evaluation_timestamp to date. It currently supports four official benchmarks: MMLU-Pro, GPQA, HLE, and GSM8K.
The converter does more than reshape data. It downloads an EEE datastore collection, checks object hashes, and identifies scores for supported benchmarks. Before writing anything, it audits existing .eval_results YAML files on the model's main branch and in open PRs, comparing by dataset and task. Scores already present are marked as such, conflicts are flagged, and missing model repos are noted. Everything else is marked ready.
Nothing is pushed without user approval. The tool writes local YAML previews and a review file, shows a report of ready and problematic entries, and only opens PRs after the user types "OPEN PRS" and enters a commit message. Reruns reuse cached results unless --force is passed.
Start Here
To get started, submit your full records to the EEE datastore. Using EEE requires only one additional step, which the converter largely automates. The integration ensures that evaluation results are not only visible but also interpretable, building trust in AI model comparisons. As the datastore grows, the coalition aims to expand support for more benchmarks and encourage broader adoption across the AI community.
