Skip to content

🧭 Benchmarks query at a time their data gives: LongMemEval question dates parsed, LoCoMo as of its last turn - #86

Open
ebigunso wants to merge 4 commits into
feature/2026-09-30/evals-obligation-bridgefrom
feature/2026-10-15/longmemeval-question-date
Open

ebigunso wants to merge 4 commits into
feature/2026-09-30/evals-obligation-bridgefrom
feature/2026-10-15/longmemeval-question-date

Conversation

@ebigunso

@ebigunso ebigunso commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Three harness fixes found by the benchmark-readiness audit, each Tier D approved (.agent-work/reviewer/longmemeval-question-date-review.md), library pin b3aba0b2.

  • The LongMemEval loader parses the official question_date (for example 2023/05/30 (Tue) 23:40) into an RFC3339 scene time at UTC, since the benchmark gives no offset; a date that does not parse is a loud error naming the question. The official file is unchanged.
  • The conventional LoCoMo pipeline queries as of the conversation's latest parseable source timestamp, inclusive, for every QA, instead of mapping the absent query date to 1970 and dropping every memory as later than the reference time.
  • An oversized LongMemEval turn (four official surfaces exceed the embedding provider's 8,192-token input limit) is split at the loader into consecutive observation parts of the same episode and speaker, at the last fitting paragraph, sentence, newline or whitespace boundary; a part is never whitespace-only; raw turn text, index and gold are preserved and the manifest records each part's boundary kind. The same policy is backported onto the historical harness (branch feature/2026-10-16/historical-longmemeval-split from f920d38, library 4c46991 unchanged) so the two arms of the closeout comparison split identically.

Rulings 82, 85, 87 in the library's .agent-work/orchestrator/v0-2-load-bearing-decisions.md. Stacked on #85; CI is red by construction on the stack.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant