Issues

• MATH500 returns empty results with --limit (KeyError 'model_answer' in grading)

  agent-generatedbug
  marin-community/evalchemy#204 · Filter by author karansag opened on Oct 6, 2026

• [nightly-e2e] scheduled run failing

  agent-generatednightly-failure
  marin-community/evalchemy#202 · Filter by author github-actions opened on Oct 6, 2026
  5

• [GPQADiamond] --limit fails with KeyError: model_answers

  agent-generatedbug
  marin-community/evalchemy#201 · Filter by author penfever opened on Oct 6, 2026

• Define opt-in shared grading contracts for Evalchemy

  agent-generated
  marin-community/evalchemy#197 · Filter by author penfever opened on Oct 2, 2026

• Decide whether NUPA should score response formatting strictly

  agent-generated
  marin-community/evalchemy#190 · Filter by author rjpower opened on Sep 26, 2026

• [eval] Exclude unresolved endpoint failures from benchmark scores

  agent-generated
  marin-community/evalchemy#82 · Filter by author penfever opened on Aug 14, 2026
  1

• Distributed multi-seed / multi-eval: served path runs serially on ONE endpoint; no seed-sharding fan-out

  enhancement
  marin-community/evalchemy#35 · Filter by author penfever opened on Jul 19, 2026
  2