Repository navigation
Add CPU Rust BM25 search task - #11
Merged
hanhainebula merged 4 commits intoOct 14, 2026
Merged
Conversation
Collaborator
|
I reviewed PR #11 at head 7d69eba, against base 46197a4. Static checks, both Compose configurations, and all 17 task-local tests passed. However, two reproducible grading issues should be fixed before merging:
The instruction and model-observation runtime hashes match the reviewed revision. The reported near-threshold model failure is consistent with the exact quality gate; it does not justify lowering the threshold. Please revalidate the affected layers after the fixes and make the available model source, command logs, and trajectory evidence accessible for maintainer review outside the public task package. Official asset publication, redistribution-rights review, and a passing merge-ready check remain separate merge prerequisites. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a CPU task requiring a self-built Rust BM25 index and searcher over all
8,841,823 MS MARCO Passage v1 texts. A separate verifier recompiles the source,
rebuilds the index and evaluates all 6,980 dev queries. MRR@10 must reach the
Pyserini 1.4.0 reference with k1=0.9, b=0.4: exactly
3236243/17589600 (0.18398616227770956). Above this floor, equal geometric weights
reward quality, warm search time and index size against a same-run reference.
Four CPUs, 16 GiB memory, Rust 1.85.1, offline generic crates; no existing
search engines or prebuilt indexes in the submission. Five fresh-process cold
samples include startup and the first response without query warmup; OS file
cache is not evicted. Warm throughput is measured with one request outstanding.
Cold latency and sampled RSS are reported diagnostics.
Author: MXueguang x93ma@edu.uwaterloo.ca. The reviewed structural precedent
is
tasks/sparse-retrieval-pruningat base46197a43f8cbb364e72b6d63d5db873ae7d2cac8. The structured instruction and proserequest remain semantically aligned.
Review fixes
Addresses the two grading findings
at current PR head
ed24b87949e6040f8a7d59cdc50a0fce7b7a15b7(runtime fix commit
a5ba0d63c70eb159d5aeef820624bde17e19be0b):persistent System V IPC, POSIX message queues, keyrings and extended-attribute
operations, including indirect io_uring access. Alternate syscall ABIs cannot
bypass it. Ordinary Rust threads and file-backed mmap remain supported.
and fails closed if inspection fails. The task instructions expose these limits.
verifies the repaired image blocks them while fresh processes still read the
persisted index. This includes thread/child inheritance, existing shared-memory
attachment, alternate ABIs and outside-index metadata reads.
The exact quality gate, reference parameters, scoring and measurement protocol
are unchanged. Historical model observations are preserved; replay evidence is
recorded separately with the final runtime and instruction hashes.
Validation
is
action_requiredwith no jobs started; maintainer action is pending.These new remote checks have not passed yet.
pass. Four negative submissions still return
invalid_submissionand zero.MRR 0.1839805908036567, reward 0. All 6,980 responses are byte-identical
to the original trial. No model regeneration or hidden-feedback code change.
MRR 0.18729482194023742, reward 0.364022408613. All 6,980
responses also match the previous passing run byte-for-byte..
See revalidation evidence for current timings, disk,
image/source hashes, regression details and final runtime fingerprints.
The original independent coding trial used hosted Codex collaboration with
fork_turns="none", not Harbor's pinned Codex CLI. Earlier Harbor Oracle evidencecovers artifact transfer and separate verification. Full hosted-model trajectory
and token/billing data were not exported. Available source, agent notes,
container command events and grader logs are in an independent maintainer
review bundle: prepared locally as
maintainer-evidence.tar.gz(4,971,555 bytes; SHA-256
f2f2b562262da765d102a965a501e65340d6b30e81ce0c85cfe3235681b576fb).Delivery to a maintainer-accessible location is pending; the local archive path
is not a shared download link.
Before merge
Official asset publication and redistribution-rights review remain outstanding.
The merge-ready check reports four expected errors for the two inputs' official
repository/path pins. Maintainers must publish approved inputs under
search-swe/Search-SWE/tasks/rust-bm25-search/and pin the merged immutable HFcommit. This update publishes neither assets nor images and does not merge.