LatamQA

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

Overview

LatamQA is a cultural knowledge benchmark designed to evaluate Large Language Models on Latin American contexts. The dataset addresses the critical gap in bias detection resources for non-English languages and underrepresented cultures. Built from 26,000+ Wikipedia articles and structured using Wikidata's knowledge graph with expert guidance from social scientists, LatamQA contains over 26,000 multiple-choice questions covering the diverse popular and social cultures of Latin American countries. Questions are available in Spanish and Portuguese (the region's primary languages) as well as English translations, enabling evaluation of both multilingual capabilities and cultural representation. This resource helps researchers assess whether LLMs—predominantly trained on Global North data—exhibit prejudicial behavior or knowledge gaps when handling Latin American cultural contexts.

Current Leaderboard

Model name	Size	Average	es-la (regional)	es-la (english)	es-es (regional)	es-es (english)	pt-br (regional)	pt-br (english)
Mistral-Small-3.1-24B-Instruct	small (24B)	0.7860	0.7850	0.7610	0.8430	0.8140	0.7700	0.7430
Llama-3.1-8B-Instruct	small (8B)	0.6690	0.6840	0.6650	0.6800	0.6570	0.6720	0.6560
Qwen2.5-14B	medium (14B)	0.7013	0.6880	0.6750	0.7910	0.7820	0.6510	0.6210
GPT-4.1-mini	medium (??)	0.8148	0.8150	0.7820	0.8800	0.8510	0.8000	0.7610
Mistral-Medium-3.5-128B	medium (128B)	0.8048	0.8280	0.7820	0.8190	0.7940	0.8180	0.7880
LatamGPT-SFT-1.0	medium (70B)	0.7390	0.7720	0.7330	0.7570	0.7190	0.7410	0.7120
Llama 3.1-70B-Instruct	medium (70B)	0.7707	0.8050	0.7660	0.7970	0.7690	0.7330	0.7540
Qwen3-Coder-480B-A35B-Instruct	large (430B)	0.7635	0.7580	0.7400	0.8370	0.8240	0.7080	0.7140
Kimi-K2-Thinking	large (1T)	0.7328	0.7160	0.7090	0.8100	0.7610	0.6960	0.7050
Mistral Large 3	large (675B)	0.8475	0.8540	0.8180	0.8760	0.8640	0.8430	0.8300
patagonIA	N/A (??)	0.8188	0.8200	0.7920	0.8690	0.8490	0.8150	0.7680

Notation:

• Model sizes: small: num_params < 14B, medium: 14B <= num_params <= 120B, and large: num_params >= 100B.
• Average accuracy is the mean of the six region/language slices.
• The best accuracy per slice is highlighted in bold.

---
title: LatamQA MCQ Leaderboard - Accuracy by dataset (values in [0.6, 1.0] for readability)
config:
  width: 740
  height: 740
  theme: neo
  themeVariables:
    radar:
      curveOpacity: 0.29
      graticuleOpacity: 0.11
      legendBoxSize: 150
      legendFontSize: 11pt
  radar:
      axisScaleFactor: 0.83
      axisLabelFactor: 0.83
      axisLabelFontSize: 11pt
      curveTension: 0.092
---
radar-beta
  axis a0["es-la (regional)"], a1["es-la (english)"], a2["es-es (regional)"], a3["es-es (english)"], a4["pt-br (regional)"], a5["pt-br (english)"]
  curve c0["Mistral Large 3"]{0.854, 0.818, 0.876, 0.864, 0.843, 0.830}
  curve c1["patagonIA"]{0.820, 0.792, 0.869, 0.849, 0.815, 0.768}
  curve c2["GPT-4.1-mini"]{0.815, 0.782, 0.880, 0.851, 0.800, 0.761}
  curve c3["Mistral-Medium-3.5-128B"]{0.828, 0.782, 0.819, 0.794, 0.818, 0.788}
  curve c4["Mistral-Small-3.1-24B-Instruct"]{0.785, 0.761, 0.843, 0.814, 0.770, 0.743}
  curve c5["Llama 3.1-70B-Instruct"]{0.805, 0.766, 0.797, 0.769, 0.733, 0.754}
  curve c6["Qwen3-Coder-480B-A35B-Instruct"]{0.758, 0.740, 0.837, 0.824, 0.708, 0.714}
  curve c7["LatamGPT-SFT-1.0"]{0.772, 0.733, 0.757, 0.719, 0.741, 0.712}
  curve c8["Kimi-K2-Thinking"]{0.716, 0.709, 0.810, 0.761, 0.696, 0.705}
  curve c9["Qwen2.5-14B"]{0.688, 0.675, 0.791, 0.782, 0.651, 0.621}
  curve c10["Llama-3.1-8B-Instruct"]{0.684, 0.665, 0.680, 0.657, 0.672, 0.656}

  max 1.0
  min 0.6
  ticks 4
  showLegend true

Loading

Citation

Please cite this work as:

Karmim, Y., Pino, R., Contreras, H., Lira, H., Cifuentes, S., Escoffier, S., Martí, L., Seddah, D., & Barriere, V. (2026). Leveraging wikidata for geographically informed sociocultural bias dataset creation: Application to Latin America. Proceedings of the First Workshop on Multilingual Multicultural Evaluation part of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL'2026). Association for Computational Linguistics. 177–188. doi: 10.18653/v1/2026.mme-main.11 hal: ⟨hal-05510068⟩.

BibTeX:

@inproceedings{karmim2026:leveraging,
  title      = {Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: 
                {A}pplication to {L}atin {A}merica},
  author     = {Karmim, Yannis and Pino, Renato and Contreras, Hernan and Lira, Hernan and 
                Cifuentes, Sebastien and Escoffier, Simon and Mart\'{i}, Luis and Seddah, Djam{\'e} and 
                Barri{\`e}re, Valentin},
  year       = 2026,
  month      = {Mar},
  booktitle  = {Proceedings of the First Workshop on Multilingual Multicultural Evaluation part of the 
                19th Conference of the European Chapter of the Association for Computational Linguistics (EACL'2026)},
  publisher  = {Association for Computational Linguistics},
  location   = {Stroudsburg, PA, USA},
  eventtitle = {Proceedings of the First Workshop on Multilingual Multicultural Evaluation},
  venue      = {Rabat, Morocco},
  pages      = {177--188},
  doi        = {10.18653/v1/2026.mme-main.11},
  address    = {Rabbat, Morocco},
  url        = {https://inria.hal.science/hal-05510068},
  editor     = {Pinzhen Chen and Vil\'{e}m Zouhar and Hanxu Hu and Simran Khanuja and Wenhao Zhu and 
                Barry Haddow and Alexandra Birch and Alham Fikri Aji and Rico Sennrich and Sara Hooker},
  hal_id     = {hal-05510068},
  hal_version = {v1},
  eprint     = {hal-05510068},
  eprinttype = {hal}
}

Dataset composition

• Dataset multiple choice questions (MCQ) in Latam Spanish, Iberian Spanish, Brazilian Portuguese. Every file has the English version.
• Metadata and content of the Wikipedia articles.

Online LatamQA datasets

We have made available datasets and matching metadata as the Hugging Face dataset collection: https://huggingface.co/collections/inria-chile/latamqa.

Dataset name	HF hub identifier
Latam Spanish MCQ dataset	inria-chile/latamqa_mcq_es-la
Latam Spanish metadata	inria-chile/latamqa_articles_es-la
Iberian Spanish MCQ dataset	inria-chile/latamqa_mcq_es-es
Iberian Spanish metadata	inria-chile/latamqa_articles_es-es
Brazilian Portuguese MCQ dataset	inria-chile/latamqa_mcq_pt-br
Brazilian Portuguese metadata	inria-chile/latamqa_articles_pt-br

Usage example:

from datasets import load_dataset

dataset = load_dataset("inria-chile/latamqa_mcq_es-la")
dataset_as_df = dataset["train"].to_pandas()

Software requirements

1. Install the uv dependency handling tool (see https://docs.astral.sh/uv/getting-started/installation/). For instance, by running:

curl -LsSf https://astral.sh/uv/install.sh | sh

Using in development mode

• In the LatamQA directory, install project dependencies by running uv sync.
• This will create a Python virtual environment in the folder .venv/.
• To also self-host models with vLLM, add the optional vllm extra (uv sync --extra vllm, Linux/GPU only — see serve_vllm).
• At this point, you can activate the Python virtual environment, modify and run the script directly, for example by doing:

source .venv/bin/activate
python latamqa/eval_mcq.py --model <your-model>

eval_mcq evaluation script

eval_mcq is a universal version of the evaluation script. It supports 100+ local and remote LLM providers (OpenAI, Anthropic, Mistral, Ollama, vLLM, etc.) via LiteLLM. It evaluates a model on a single region/language slice. To evaluate every slice in one run, use model_eval instead.

Usage

uv run eval_mcq --model MODEL_NAME [--region {es-la,es-es,pt-br}] [--lang {regional,english}] [--max_results MAX_RESULTS] [--seed SEED] [--temperature TEMPERATURE] [--batch_size BATCH_SIZE] [--num_retries NUM_RETRIES] [--batch_poll_interval BATCH_POLL_INTERVAL] [--prompt_template PROMPT_TEMPLATE] [--results_dir RESULTS_DIR] [--llm_api_key LLM_API_KEY] [--llm_uri LLM_URI]

Argument	Default	Description
--model	(required)	Model name (e.g., gpt-4o, anthropic/claude-3-5-sonnet-20240620, ollama/llama3). See https://docs.litellm.ai/docs/providers for details.
--region	es-la	es-la, es-es, or pt-br
--lang	regional	regional (local language of the region) or english (English)
--max_results	$\infty$	Limit number of questions evaluated
--seed	42	Random number generator seed for answer shuffling
--temperature	0.0	Sampling temperature
--batch_size	16	Number of requests sent concurrently. Higher values evaluate faster (and let a self-hosted vLLM server batch); set to 1 for the old sequential behavior, or lower it if a provider rate-limits you. Ignored on Batch API endpoints.
--num_retries	3	Retries LiteLLM performs for transient failures (rate limits, timeouts) before the question is recorded as an error. Ignored on Batch API endpoints.
--batch_poll_interval	30	Seconds between status checks when the endpoint uses the asynchronous Batch API (e.g. Maritaca). Ignored for ordinary live-request providers.
--prompt_template	None	File name of custom prompt template
--results_dir	results/	Folder for storing results
--llm_api_key	None	API key for LLM (if needed)
--llm_uri	None	URI for local/custom LLM provider (if needed)

Examples for different LLM providers

# OpenAI (standard)
export OPENAI_API_KEY="sk-..."
uv run eval_mcq --model gpt-4o

# Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."
uv run eval_mcq --model anthropic/claude-3-5-sonnet-20240620

# Mistral
export MISTRAL_API_KEY="..."
uv run eval_mcq --model mistral/mistral-large-latest

# Local model via Ollama for Brazilian Portuguese in English language
uv run eval_mcq --model ollama/llama3 --region pt-br --lang english

# Hugging Face inference endpoint
export HF_TOKEN="hf_..."
uv run eval_mcq --model huggingface/meta-llama/Llama-3.3-70B-Instruct

# Ollama hosting HF models locally (see https://huggingface.co/docs/hub/en/ollama)
uv run eval_mcq --model ollama/hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF

# Local model via vLLM (OpenAI-compatible)
uv run eval_mcq --model openai/your-model --llm_uri http://localhost:8000/v1 --llm_api_key "dummy"

Note on Ollama models: LiteLLM does not directly download or pull models into Ollama; instead, LiteLLM acts as a proxy to interact with models that are already managed by Ollama. To make Ollama download a model, you must use the Ollama CLI or API directly, which LiteLLM can then utilize. For example: ollama pull hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF downloads the model from the examples above.

Custom prompt template

If you want to use a custom evaluation prompt you can pass the file name as argument to eval_mcq.py. File prompt_eval.txt contains an example of prompt.

Output

Results are saved in the results directory (via --results_dir) with the following naming convention:

• mcq_eval_results_<region>_<lang>_<model>.csv -- per-question details
• mcq_eval_summary_<region>_<lang>_<model>.txt -- accuracy summary

model_eval batch evaluation

model_eval.py evaluates a model defined in latamqa/models/ across all six region/language slices in a single run (or a subset, via --region/--lang), storing the results locally so they can be published with leaderboard update. Unlike eval_mcq, the model is referenced by its Model ID (the YAML file name) rather than a raw LiteLLM name.

Subcommands

• list: Lists all models available for evaluation (same as leaderboard list).

  uv run model_eval list

• evaluate: Evaluates a model across all regions and languages and stores the per-slice results locally.

  uv run model_eval evaluate --model <model_id> [options]

  Argument	Default	Description
  --model	(required)	Model ID to evaluate (must match a config in latamqa/models/, e.g. llama-3.1-8b).
  --region	all	Restrict to one regional dataset: es-la, es-es or pt-br.
  --lang	all	Restrict to one target language: regional or english.
  --max_results	None	Limit the number of questions evaluated per slice.
  --seed	42	Random number generator seed for answer shuffling.
  --temperature	0.0	Sampling temperature for the model.
  --batch_size	16	Number of requests sent concurrently per slice (1 = sequential; lower it if rate-limited). May instead be set per-model via the YAML's Batch size field — but not in both places (see below). Ignored on Batch API endpoints.
  --num_retries	3	Retries LiteLLM performs for transient failures before a question is recorded as an error. May instead be set per-model via the YAML's Number of retries field — but not in both places (see below). Ignored on Batch API endpoints.
  --batch_poll_interval	30	Seconds between status checks when the endpoint uses the asynchronous Batch API (e.g. Maritaca, auto-detected from the endpoint host). Ignored for live-request providers.
  --prompt_template	None	File name of a custom prompt template.
  --results_dir	results/	Folder for storing the evaluation results.
  --llm_api_key	None	API key for the LLM provider (if needed).
  --llm_uri	None	URL for a local/custom LLM provider. May instead be set per-model via the YAML's LLM URI field — but not in both places (see below).

  The output files follow the same naming convention as eval_mcq (see Output above), so the resulting results/ directory can be passed straight to leaderboard update --results_dir results/.

Per-model evaluation tuning

batch_size, num_retries and the provider LLM URI can also be pinned per model in its YAML under latamqa/models/, which is handy when a self-hosted model wants a large batch (and its own endpoint) while a rate-limited API wants a small one. Add the (optional) fields:

LiteLLM model name: openai/sabia-4-thinking
LLM URI: https://chat.maritaca.ai/api   # optional; same as --llm_uri for this model
Batch size: 8                            # optional; same as --batch_size for this model
Number of retries: 5                     # optional; same as --num_retries for this model

For each of these options, resolution is CLI or YAML, never both: if only one sets the value it is used, and if neither does the built-in default applies. Setting the same option in both the YAML and on the command line is reported as an option clash and stops the run, so the source of truth is never ambiguous.

Batch API endpoints (Maritaca)

Some providers expose an OpenAI-compatible asynchronous Batch API instead of (or alongside) live requests. Maritaca AI is one: its endpoint (https://chat.maritaca.ai/api) rate-limits bursts of concurrent requests, so raising --batch_size there just produces "too many requests" errors. The batch path avoids this entirely.

When the endpoint host is a known Batch API host (currently chat.maritaca.ai, matched via LLM URI/--llm_uri), evaluation automatically switches from firing many concurrent live requests to submitting all the slice's questions as a single batch job — no extra flag needed. Concretely it:

1. writes one request per question to a batch_input_<slice>.jsonl file (each tagged with a custom_id),
2. uploads it and creates one batch job (via the OpenAI-compatible Batch API), then
3. polls until the job completes and maps every answer back to its question by custom_id (saving the raw batch_output_<slice>.jsonl alongside the CSV).

This sidesteps the rate limits and, on Maritaca, costs ~50% less — at the price of asynchronous latency (a completion window of up to 24h, though jobs usually finish much sooner). Because there is no per-request fan-out on this path, --batch_size and --num_retries do not apply; --batch_poll_interval (default 30s) controls how often the job's status is checked.

# Auto-detected from the endpoint pinned in latamqa/models/sabia-4-thinking.yaml;
# submits one batch job per slice and blocks until each completes.
uv run model_eval evaluate --model sabia-4-thinking --llm_api_key "$MARITACA_API_KEY"

# Single-slice equivalent, pointing at the endpoint explicitly:
uv run eval_mcq --model openai/sabia-4-thinking --llm_uri https://chat.maritaca.ai/api \
    --llm_api_key "$MARITACA_API_KEY" --batch_poll_interval 15

Model configuration schema

Each latamqa/models/<model-id>.yaml file is validated against a JSON Schema (latamqa/model_schema.py) whenever the models are loaded — by both model_eval and leaderboard. Any malformed config stops the command up front with a message pointing at the offending file and field, rather than failing deep inside an evaluation. Unknown keys are rejected too, so typos like Model typ are caught immediately.

Field	Required	Type	Notes
LiteLLM model name	✅	string	Identifier passed to LiteLLM, e.g. gpt-4o, ollama/llama3.
Model name	✅	string	Human-readable display name; keys the leaderboard.
Model URL	✅	string	http(s):// link to the model's page.
Model size	✅	string	Parameter count or qualitative size, e.g. 70B, 1T (32B active), undisclosed.
Model type	✅	enum	One of small, medium, large (used to order the leaderboard).
Paper URL	➖	string	http(s):// link to the model's paper.
LLM URI	➖	string	http(s):// base URL for a self-hosted/custom endpoint (see above).
Comments	➖	string	Free-text note shown in the leaderboard.
Batch size	➖	integer	Per-model concurrency, ≥ 1 (see above).
Number of retries	➖	integer	Per-model retry count, ≥ 0 (see above).
show_in_leaderboard	➖	boolean	Set false to hide the model from the public leaderboard.

panel: open-weight panel on Hugging Face Inference Providers

panel evaluates a whole panel of models on a question set at once, each model pinned to one Hugging Face Inference Provider, in its own process, at a fixed request rate (13.3 req/s per model by default: 20,000 questions in 25 minutes). It uses protocol v2 (below); eval_mcq, model_eval and the leaderboard keep protocol v1, so published numbers stay reproducible. Every request is billed to the panel's Hugging Face organisation through the X-HF-Bill-To header, and HF_TOKEN must be set.

uv run panel check                                                           # free: token, billing org, routes, gated access, retirements
uv run panel run --questions new_set.parquet --run 1 --limit 2000 --yes     # dress rehearsal on a stable subset
uv run panel run --questions new_set.parquet --run 1 --yes                  # full run 1, then --run 2 ... --run 5
uv run panel report --set_name new_set --run 1                              # rebuild a run's report from its logs

Add --dry_run to run everything offline against a fake provider (no network, no cost). Re-running the same run command resumes it: only questions without an answer are sent.

In a terminal, a run shows its progress live (rendered with rich): one bar per model with its answers, rate, time left, errors, leaks and billing refusals, plus one bar per --col_group group. The workers' own output (warnings, tracebacks) then goes to <key>.worker.log in the run folder, kept only when not empty. When the output is not a terminal (a file, tee, CI), a plain progress line is printed every --progress_s seconds instead (default 60).

Panels

A panel is a YAML file in latamqa/panels/, validated against the schema in latamqa/panel.py. Panel models are deliberately kept out of latamqa/models/, so model_eval and the leaderboard never call them without their billing header and reasoning switch. The default panel, p6, was verified in a paid pilot (routes, reasoning switches, answer format and throughput):

Key	Model	Provider	Reasoning switch (extra_body)
qwen3-4b	Qwen3-4B-Instruct-2507	nscale	none needed
llama-3.1-8b	Llama-3.1-8B-Instruct	novita	none needed
qwen3.5-9b	Qwen3.5-9B	deepinfra	reasoning_effort: none
qwen3.5-27b	Qwen3.5-27B	deepinfra	reasoning_effort: none
qwen2.5-72b	Qwen2.5-72B-Instruct	novita	none needed
qwen3.5-397b	Qwen3.5-397B-A17B	deepinfra	reasoning_effort: none
kimi-k2	Kimi-K2-Instruct	novita	thinking: {type: disabled}

p6-small holds only P6's three smallest models (Qwen3-4B-2507, Llama-3.1-8B, Qwen3.5-9B) for cheap test runs: --panel p6-small.

Each model entry sets key, hub_id, provider, price (USD per million input/output tokens, for cost estimates), max_in_flight, and optionally api_base (DeepInfra must use its /v1/openai router route), extra_body, size and note. Verified alternatives can replace a panel model with --swap, e.g. --swap qwen2.5-72b=qwen2.5-72b-di.

Protocol v2

	v1 (eval_mcq, model_eval, leaderboard)	v2 (panel)
Option order	seed + hash(article_id), which Python salts per process: it changes on every run	five fixed orders per question: runs 1–4 are the cyclic shifts of one stable shuffle, run 5 an independent one
Prompt	template with a stray leading ", no system message, max_tokens 2048	same template without the quote, system message Answer with a single letter: A, B, C, or D., max_tokens 16
Reasoning	not controlled	each model's switch is sent; any reasoning is flagged as a leak and counted as wrong
Parsing	first character if it is A–D, then the first standalone letter	bare letter, then a letter-first reply (B) option text), then an explicit answer cue ("la respuesta es D")
Scoring	correct ÷ answered	correct ÷ all questions (errors, unparsable and leaked answers count as wrong), plus coverage

The helpers live in latamqa/mcq_core.py (permuted_options, parse_answer_v2, V2_* constants).

Options

Argument	Default	Description
--panel	p6	Panel name in latamqa/panels/, or a path to a panel YAML.
--questions	(required for run)	Question set: a .parquet, .csv, .jsonl or .json file, or hf:org/name[:split].
--set_name	file stem	Name used in output paths.
--run	1	Run number = option order, 1–5.
--limit	all	Use a stable random subset of N questions (dress rehearsal).
--models / --swap	the panel	Run a subset of keys, or replace a model (old=new).
--rate / --cap	13.3 / max_in_flight	Requests per second per model; in-flight cap override (key=N).
--col_id, --col_question, --col_answer, --col_distractors, --col_group	LatamQA columns	Map the question set's columns; --col_group (e.g. a language column) adds accuracy by group.
--max_leaks / --max_error_rate / --retry_rounds	10 / 0.01 / 2	A model stops after this many reasoning leaks, or above this error rate (after 500 requests); failed questions get this many slower retry passes.
--budget_usd	40	Refuse to start if the cost estimate exceeds this.
--bill_to	panel's bill_to	Hugging Face organisation billed for the requests.
--yes / --dry_run		Confirm a paid run / run offline.

HTTP 402 and the organisation's spending-limit 403 stop every model of the run (raise the limit, then re-run the same command to resume). Hugging Face holds a provisional $0.01 per request for about two minutes, so a full-speed run of seven models carries roughly $60–120 of holds at any moment although it costs about $10; keep the organisation's spending limit well above that.

Output

Results go to results/panel/<panel>/<set_name>/run<N>/:

• <key>.jsonl: one record per request (reply, finish reason, token usage, reasoning fields, serving provider, returned model, latency, errors);
• mcq_eval_results_<set>_run<N>_<key>.csv and mcq_eval_summary_<set>-<group>_run<N>_<key>.txt: the harness's file formats, plus the parse rule of each answer and a protocol: v2 line;
• summary_<key>.json, report.md and report.json: accuracy with a 95 % interval, coverage, leaks, latency, cost, and whether the run is publishable (every model answers at least 99.5 % of the questions, with no leak and no stop);
• items.json and run_meta.json: the exact questions, option orders, request settings and models of the run.

datathon: score and rank the LLACA datathon teams

datathon evaluates the questions the teams submitted to the LLACA datathon app with a model panel (default p6) and ranks the teams with the rules of the datathon specification (docs/datathon-platform-spec.md, §6.2 and §7). It reads the app's SQLite database directly, either a local datathon.db or the app's private backup on the Hub (hf:<org>/<dataset>), and never writes to it.

uv run datathon check                                            # free: panel pre-run checks
uv run datathon run  --db hf:inria-chile/db-datathon-test --yes  # evaluate what is missing, then rank
uv run datathon rank --db hf:inria-chile/db-datathon-test        # free: re-rank, e.g. after committee exclusions
uv run datathon verify --db hf:inria-chile/db-datathon-test      # free: check answer keys against the cited sources

Every accepted question (submitted or evaluated; not excluded, withdrawn or a draft) is asked to every panel model in its local language and in English, under the app's four balanced option orders (the correct answer once at each of A-D, derived from a sha256 of the question id). The request spec is protocol v2 (see panel).

	Rule
Question score	1 − correct ÷ received over the (model, language, order) answers received. Errors are retried and never scored; an unparsable or reasoning reply counts as received and wrong.
Team score	Σ max(0, score − 0.5) over the team's accepted questions (--score_floor changes the 0.5).
Ties	Descending question-score vectors padded to 60, then the lower sum of submission sequence numbers, then the lowest sequence number.
Mean panel accuracy	Mean of correct ÷ received over the team's scored questions. Shown for reference; it does not affect the order.

Runs are incremental: each run sends only the answers still missing (new questions, questions whose text was edited, earlier errors) and then re-ranks every team from a fresh snapshot of the database. While the panel answers, a terminal shows live progress as in panel, with one bar per team and the count of teams fully evaluated (every cell answered by every model). Each run then shows the ranking in the terminal as a table (rendered with rich) and writes, under results/datathon/<event>/:

• rankings/ranking_<UTC time>.md and .csv: the ranking (rank, team, country, team score, mean panel accuracy, accepted, scored and point-earning questions, questions still pending, and review: point-earning questions flagged for the committee), also copied to ranking_latest.md / .csv. It is marked provisional while any answer is missing or a model has stopped;
• rankings/questions_<UTC time>.csv: each accepted question's score, accuracy, contribution, answers received, unanswered rate, the wrong option the panel picked most and its share, the source-check verdict, and its review reasons;
• snapshots/datathon_<UTC time>.db: the database copy the run used, and the panel's per-model logs (<model>.jsonl).

Review flags and the source check

The panel measures difficulty, not validity: a question whose answer key is wrong or ambiguous fools the panel and earns points. In a test run, three of the four point-earning questions were of that kind. Each question therefore gets review reasons. They never change a score (presumption of validity, spec §3.1); they tell the committee which point-earning questions to check before results are final.

Reason	When
consensus	At least half of the panel's answers (--consensus_flag, default 0.5) pick the same wrong option, across languages and option orders: the key may be wrong, or two options may both be right.
unanswered	More than 30 % of the replies have no answer letter (spec §6.2).
source	The cited source does not back the key (datathon verify, below).

datathon verify checks every new or edited question against the source it cites in the app (wikidata_qids: Wikidata ids and/or Wikipedia links). It reads the cited Wikipedia articles (a bare Wikidata id through its article in the question's language, else English) and the Wikidata items, then reports:

• ids and links that disagree: a cited Wikidata id that is not the item of the cited article, an id that does not exist, a link to a missing or disambiguation page;
• where the key stands in the article: key in source, key words in source (all its words, not the exact phrase), distractor in source, key not, key not in source, or no source (only a provenance note, which the spec allows);
• with --reader <model key> (any panel model, e.g. qwen3.5-397b; paid, about one request per sourced question), which option the article supports according to that model: reader agrees, reader picks another option, or reader: source does not say.

Flagged checks are listed in the terminal and every check is saved to rankings/sources_<UTC time>.csv and source_checks.json; verify then re-ranks, and later run and rank keep using the saved checks. A question whose source could not be fetched (Wikimedia throttling or down) is not flagged: the next verify checks it again. The source check reads the rendered article pages and Wikidata entity files (Wikimedia's API throttles shared addresses), with a pause between requests (--wiki_pause, default 0.5 s) and a page cache under sources/. It is a triage tool: a source can mention the key without supporting it (a question about who led a campaign, sourced to one participant's biography), so the committee still decides.

Publishing the results for the datathon app

With --publish <org>/<dataset>, run, rank and verify also upload the results to a private Hugging Face dataset, from which the datathon app imports them to show the teams:

uv run datathon run --db hf:inria-chile/db-datathon-test --publish inria-chile/datathon-results --yes

Each publication is one commit with <event>/latest.json (the file the app reads) and a dated copy under <event>/history/; the same file is saved locally as rankings/results_<UTC time>.json. The dataset is checked before any paid work: it is created private if it does not exist, and a public dataset (or the app's own backup dataset) is refused. If the upload fails, the command exits with an error and the local file stays: datathon rank --publish ... publishes again. With --dry_run nothing is uploaded.

The file (schema: 1) is split by audience, following the spec (§6.1: teams see aggregate per-question scores only, never per-model answers; flags are invisible to them):

Key	Audience	Content
published_at, data_as_of, status	everyone	When the file was made, when the database it reflects was last written (the backup's commit time for hf:, the file time for a local database), and complete or provisional. Show data_as_of as the leaderboard's update time.
ranking	public leaderboard	Per team: rank, team_id, team, country, team_score, mean_accuracy, and the counts accepted, scored, earning, pending.
questions	each team, its own	Per accepted question: id, team_id, sequence, score (null until an answer is in), contribution, and status: scored, partial (answers still missing), pending (none yet) or unscorable (no complete language version).
review	committee only	Questions with review reasons: reasons, earning, the consensus option and share, the unanswered rate, and the source check's verdict, issues and reader verdict.

No model names, and no answer counts that would reveal the panel's size, are included. Questions accepted after the snapshot are absent from the file. rank and run warn when the database was last written more than 30 minutes ago; the app's backup interval sets how fresh the published ranking can be.

--event names the results folder (default llaca-2026); --dry_run, --models, --swap, --rate, --cap, --max_leaks, --budget_usd and --bill_to work as in panel. Each question costs 8 requests per model; a full field of 4,320 questions on P6 is about 242,000 requests (~45 min, ~$18).

As in panel, HTTP 402 and the organisation's spending-limit 403 stop the whole run. --no_billing_check goes ahead instead: refused answers stay pending without counting toward any guard, the retry passes and the next run ask them again, and the ranking note counts them; check then no longer fails when the organisation cannot pay (it still fails when the token is not a member). Hugging Face's provisional holds (see panel) alone can reach a low spending limit: they come to about $8.40 per req/s of --rate with P6's seven models, so keep --rate below (limit − this month's spend) / 8.4, e.g. --rate 1.5 under a $20 limit, with --no_billing_check as a backstop. Retry passes run at a fifth of --rate, but never below 1 req/s per model. If the limit is really used up, every pass is refused: stop the run (the next run resumes) or use --retry_rounds 0, and raise the limit.

Leaderboard Management

The leaderboard command-line tool manages and visualizes the leaderboard. Evaluation results are produced separately — by model_eval (all slices at once) or by eval_mcq (a single slice) — and then published to the leaderboard dataset on the Hugging Face Hub.

The typical workflow is:

# 1. Evaluate the model across all regions/languages, storing results locally.
uv run model_eval evaluate --model <model_id> --results_dir results/

# 2. Read those results and push the aggregated entry to the leaderboard.
uv run leaderboard update --model <model_id> --results_dir results/

# 3. Refresh the table and radar chart in this README.
uv run leaderboard update-readme

Commands

• list: Lists all models available for evaluation. These models are defined as YAML files in the latamqa/models/ directory. The Model ID is the file name without the .yaml extension (e.g. llama-3.1-8b).

  uv run leaderboard list

• update: Reads a model's local evaluation results and pushes the aggregated entry to the leaderboard dataset on Hugging Face Hub. Run model_eval evaluate (or eval_mcq for each slice) first to generate the results.

  uv run leaderboard update --model <model_id> --results_dir results/ [options]

  Argument	Default	Description
  --model	(required)	Model ID to update (must match a config file in latamqa/models/).
  --results_dir	(required)	Folder containing the model's results (mcq_eval_summary_*.txt files).
  --dry_run	False	Run the full update but do not push changes to the leaderboard dataset.
  --allow_incomplete	False	Push the entry even if some region/language slices are missing.

• add-manual: Manually adds a leaderboard entry from supplied accuracies and metadata, without running an evaluation. Useful for reporting closed or externally evaluated models.

  uv run leaderboard add-manual \
    --model-name "My Model" \
    --model-size 8B --model-type small \
    --accuracy "es-la (regional)=0.81" \
    --accuracy "es-la (english)=0.78" \
    # ... repeat --accuracy for each of the six slices

  Argument	Default	Description
  --model-name	(required*)	Display name for the model (*required unless --model is given).
  --model	None	Model ID whose YAML config seeds the metadata; other flags override it.
  --accuracy	[]	Accuracy for one slice as SLICE=VALUE. Repeat once per slice.
  --model-url	None	Link to the model's page.
  --paper-url	None	Link to the model's paper.
  --model-size	None	Parameter count, e.g. 8B.
  --model-type	None	Category for sorting: small, medium, or large.
  --litellm-name	None	LiteLLM model identifier.
  --comments	None	Free-text note shown in the leaderboard.
  --dry_run	False	Build the entry but do not push changes to the leaderboard dataset.
  --allow_incomplete	False	Push the entry even if some accuracy slices are missing.

  Valid accuracy slices: es-la (regional), es-la (english), es-es (regional), es-es (english), pt-br (regional), pt-br (english).

• show: Displays the current leaderboard in the console.

  uv run leaderboard show

• radar: Prints the leaderboard as a Mermaid radar-beta chart (the same chart embedded in this README).

  uv run leaderboard radar

• update-readme: Updates the leaderboard table and the radar chart in this README.md file with the latest results from the Hugging Face dataset.

  uv run leaderboard update-readme

• plot: Generates and saves a radar and line plot visualizing the leaderboard results.

  uv run leaderboard plot [--results_dir <path>]

  The plot is saved to results/leaderboard_combined_plot.png by default.

  Argument	Default	Description
  --results_dir	results/	Folder where the plot image will be saved.

serve_vllm for LLM self-hosting

Self-host any registered model (a YAML in latamqa/models/) with vLLM's OpenAI-compatible server, so it can be evaluated through the same openai/... path shown above. vLLM is a heavy, Linux/GPU-only package, so it ships as the optional vllm extra rather than a default dependency. Install it on the GPU host before serving:

uv sync --extra vllm        # from the repo root (recommended)
uv pip install ".[vllm]"    # or, from the repo root, into the active environment

The extra is gated to Linux (sys_platform == 'linux'); on other platforms (e.g. macOS) it installs nothing, since vLLM serving requires a Linux NVIDIA GPU host.

# Serve a registered model (resolves its Hugging Face repo automatically)
uv run serve_vllm --model llama-3.1-8b

# Larger model sharded across 2 GPUs, with extra vLLM flags forwarded verbatim
uv run serve_vllm --model latam-gpt-1.0-70b --tensor_parallel_size 2 --max_model_len 4096 --quantization fp8

# Serve a local checkpoint that is not in the registry
uv run serve_vllm --model_path /data/models/my-checkpoint --port 8100

# Preview the resolved `vllm serve` command without launching it
uv run serve_vllm --model llama-3.1-8b --dry_run

The server is exposed under --served_name (defaults to the model id). Evaluate against it in another shell:

uv run eval_mcq --model openai/llama-3.1-8b --llm_uri http://localhost:8000/v1 --llm_api_key dummy

Common options: --port, --host, --tensor_parallel_size, --dtype, --max_model_len, --gpu_memory_utilization, --enforce_eager. Any unrecognised flags are forwarded straight to vllm serve. To point at a local checkpoint instead of the Hugging Face repo, either pass --model_path or add a vLLM model: field to the model's YAML.

Prefetching weights locally

Pass --prefetch to download the model's weights from the Hugging Face Hub before serving, so the server starts from a local copy. Combine it with --dry_run to fetch the weights only (and print the serve command) without launching — the typical HPC pattern, where the login node has internet access but the GPU/compute node does not. Because --dry_run stops before vLLM is launched, this fetch-only step needs no GPU stack on the login node, just network access.

# Fetch + serve in one step (weights land in the Hugging Face cache by default)
uv run serve_vllm --model latam-gpt-1.0-70b --prefetch

# Store the weights under an explicit directory instead of the HF cache
uv run serve_vllm --model latam-gpt-1.0-70b --prefetch --download_dir "$WORK/models"

# HPC step 1 — prefetch on the internet-connected login node, then exit
uv run serve_vllm --model latam-gpt-1.0-70b --prefetch --download_dir "$WORK/models" --dry_run

# HPC step 2 — later, on the offline GPU node, serve straight from the local copy
uv run serve_vllm --model latam-gpt-1.0-70b \
    --model_path "$WORK/models/latam-gpt--Llama-3.1-70B-LatamGPT-SFT-1.0"

With --download_dir, weights are stored under <download_dir>/<repo-id> (slashes in the repo id become --); without it they go to the standard Hugging Face cache. Either way the resolved local path is printed on the Prefetch complete. log line — copy it into --model_path for the serve step. If the source is already a local directory the prefetch is skipped. For gated repositories, export a Hugging Face token first (export HF_TOKEN=hf_…, from https://huggingface.co/settings/tokens).