Skip to content

Add CPU Rust BM25 search task - #11

Merged
hanhainebula merged 4 commits into
VectorSpaceLab:mainfrom
MXueguang:add-rust-bm25-search
Oct 14, 2026
Merged

hanhainebula merged 4 commits into
VectorSpaceLab:mainfrom
MXueguang:add-rust-bm25-search

Conversation

@MXueguang

@MXueguang MXueguang commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a CPU task requiring a self-built Rust BM25 index and searcher over all
8,841,823 MS MARCO Passage v1 texts. A separate verifier recompiles the source,
rebuilds the index and evaluates all 6,980 dev queries. MRR@10 must reach the
Pyserini 1.4.0 reference with k1=0.9, b=0.4: exactly
3236243/17589600 (0.18398616227770956). Above this floor, equal geometric weights
reward quality, warm search time and index size against a same-run reference.

Four CPUs, 16 GiB memory, Rust 1.85.1, offline generic crates; no existing
search engines or prebuilt indexes in the submission. Five fresh-process cold
samples include startup and the first response without query warmup; OS file
cache is not evicted. Warm throughput is measured with one request outstanding.
Cold latency and sampled RSS are reported diagnostics.

Author: MXueguang x93ma@edu.uwaterloo.ca. The reviewed structural precedent
is tasks/sparse-retrieval-pruning at base
46197a43f8cbb364e72b6d63d5db873ae7d2cac8. The structured instruction and prose
request remain semantically aligned.

Review fixes

Addresses the two grading findings
at current PR head ed24b87949e6040f8a7d59cdc50a0fce7b7a15b7
(runtime fix commit a5ba0d63c70eb159d5aeef820624bde17e19be0b):

  • Every compile/index/search process now inherits a seccomp filter blocking
    persistent System V IPC, POSIX message queues, keyrings and extended-attribute
    operations, including indirect io_uring access. Alternate syscall ABIs cannot
    bypass it. Ordinary Rust threads and file-backed mmap remain supported.
  • Index accounting rejects xattrs on files, subdirectories and the index root,
    and fails closed if inspection fails. The task instructions expose these limits.
  • A container regression reproduces both bypasses with the old image, then
    verifies the repaired image blocks them while fresh processes still read the
    persisted index. This includes thread/child inheritance, existing shared-memory
    attachment, alternate ABIs and outside-index metadata reads.

The exact quality gate, reference parameters, scoring and measurement protocol
are unchanged. Historical model observations are preserved; replay evidence is
recorded separately with the final runtime and instruction hashes.

Validation

  • Local review-stage task/release checks and both Compose configurations pass.
  • The new GitHub Actions run
    is action_required with no jobs started; maintainer action is pending.
    These new remote checks have not passed yet.
  • Task tests: 19 passed. Repository tests: 169 passed, 2 optional skips.
  • Rebuilt verifier; original isolation probe and new persistent-state regression
    pass. Four negative submissions still return invalid_submission and zero.
  • Frozen GPT-5.6 Sol / max submission: complete final-verifier replay, exit 0,
    MRR 0.1839805908036567, reward 0. All 6,980 responses are byte-identical
    to the original trial. No model regeneration or hidden-feedback code change.
  • Frozen known-good author source: complete final-verifier replay, exit 0,
    MRR 0.18729482194023742, reward 0.364022408613. All 6,980
    responses also match the previous passing run byte-for-byte..
  • Both full replays rebuild and measure Pyserini and reproduce its exact MRR.
    See revalidation evidence for current timings, disk,
    image/source hashes, regression details and final runtime fingerprints.

The original independent coding trial used hosted Codex collaboration with
fork_turns="none", not Harbor's pinned Codex CLI. Earlier Harbor Oracle evidence
covers artifact transfer and separate verification. Full hosted-model trajectory
and token/billing data were not exported. Available source, agent notes,
container command events and grader logs are in an independent maintainer
review bundle: prepared locally as maintainer-evidence.tar.gz
(4,971,555 bytes; SHA-256 f2f2b562262da765d102a965a501e65340d6b30e81ce0c85cfe3235681b576fb).
Delivery to a maintainer-accessible location is pending; the local archive path
is not a shared download link.

Before merge

Official asset publication and redistribution-rights review remain outstanding.
The merge-ready check reports four expected errors for the two inputs' official
repository/path pins. Maintainers must publish approved inputs under
search-swe/Search-SWE/tasks/rust-bm25-search/ and pin the merged immutable HF
commit. This update publishes neither assets nor images and does not merge.

  • Actual author identity and contribution history preserved; maintainer edits enabled.
  • Task package contains no model/author solution, credentials, restored data or raw trial logs.
  • Review regressions, exact quality gate and frozen-source revalidation recorded.
  • Maintainer review of final runtime and available model evidence.
  • Terms review, official input publication and passing merge-ready check.
  • Merge commit preserving contributor history.

@hanhainebula

Copy link
Copy Markdown
Collaborator

I reviewed PR #11 at head 7d69eba, against base 46197a4. Static checks, both Compose configurations, and all 17 task-local tests passed. However, two reproducible grading issues should be fixed before merging:

  1. Persistent shared memory can bypass index-size accounting. In tests/isolate.py:107, Landlock restricts filesystem access, but System V shared memory remains accessible across indexing and search. Using the unchanged isolation launcher, grader cleanup, and index-locking functions, I confirmed that two fresh search processes running as UID 10001 could read shared-memory data left by an exited index process, while index_bytes reported only a one-byte marker file. This allows retrieval state to survive outside the measured index. Please block unnecessary IPC syscalls or isolate/reset IPC state between phases, and add a regression proving that search can reconstruct its state solely from the persisted index.
  2. Extended attributes are excluded from disk accounting. tests/benchmark.py:42 counts only regular-file st_size. I confirmed that a search process could read 4,096 bytes stored in an index file’s extended attribute after the grader made the index read-only, while the measured index remained one byte. Please reject extended attributes on index files and directories, or include their payload sizes in accounting, with a regression covering this case.

The instruction and model-observation runtime hashes match the reviewed revision. The reported near-threshold model failure is consistent with the exact quality gate; it does not justify lowering the threshold. Please revalidate the affected layers after the fixes and make the available model source, command logs, and trajectory evidence accessible for maintainer review outside the public task package.

Official asset publication, redistribution-rights review, and a passing merge-ready check remain separate merge prerequisites.

@hanhainebula
hanhainebula merged commit 205b3b5 into VectorSpaceLab:main Oct 14, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants