Skip to content

🔥 Dataset loaders read the official shapes only - #77

Open
ebigunso wants to merge 3 commits into
feature/2026-09-29/evals-cleanup-seal-prunefrom
feature/2026-09-29/evals-cleanup-official-loaders
Open

ebigunso wants to merge 3 commits into
feature/2026-09-29/evals-cleanup-seal-prunefrom
feature/2026-09-29/evals-cleanup-official-loaders

Conversation

@ebigunso

Copy link
Copy Markdown
Owner

Dataset loaders read the official shapes only (evals value-audit cleanup Task_5).

  • The LoCoMo and LongMemEval loaders no longer accept wrapper keys (data, samples, items, instances, questions), field aliases (qid, id, q, a, utterance, dialog, messages and others) or alternate session forms that no real dataset file uses. Their tests and module docs go with them.
  • Kept: the canonical optional LoCoMo question_id with its effective-ID collision check, and the existing handling of malformed optional annotations. The official LoCoMo file carries no question_id, which a later audit can weigh.
  • The benchmark-converted continuity fixtures are untouched; their runs are deferred to the library's v0.2 closeout.

Ten files, 657 fewer lines; production edits are only in the two loaders. Loading both official local files gives identical complete output at base and tip: LoCoMo 10 items and 51,846 values, LongMemEval 500 items and 1,062,049 values, with nothing normalized. Tests: 394 top-level plus 2 child runs, none excluded. evals-reviewer APPROVED with no findings.

Stacked on #76.

🤖 Generated with Claude Code

…-seal-prune' into feature/2026-09-29/evals-cleanup-official-loaders
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant