Draft the V5 diagnostic-prequalification harness - #3601
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## Gbwo6xhidw7eogojb7aokfi2gthtzr6wq #3601 +/- ##
==================================================================
Coverage 91.85% 91.85%
==================================================================
Files 20 20
Lines 6093 6093
==================================================================
Hits 5597 5597
Misses 496 496 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 091e9d6f5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def verify_draft() -> None: | ||
| from prepare import verify_draft as verify_preparation | ||
|
|
||
| verify_preparation() |
There was a problem hiding this comment.
On a clean writable checkout, a normal protocol.py verify-draft invocation imports prepare, which creates __pycache__/prepare.*.pyc inside the run tree; the same invocation then rejects that artifact at the scan near the end of verify_draft. This makes the documented draft verification fail unless callers happen to set PYTHONDONTWRITEBYTECODE, so load the helper without bytecode output or perform the artifact check before importing it.
Useful? React with 👍 / 👎.
| validate_direct_score( | ||
| read_json(args.score), | ||
| read_json(args.atoms), | ||
| read_json(args.rules), | ||
| args.scorer, |
There was a problem hiding this comment.
Every invocation of validate-score reaches validate_direct_score without an input packet, so it always raises ProtocolError("direct score validation requires the exact launch-bound input packet"); the neighboring consistency, adjudication, and merge dispatches are likewise still wired to older signatures and raise TypeError because their required score packets, two consistency reviews, and evidence digest are absent. The parser also exposes no options for these inputs, leaving the entire scoring pipeline unusable through protocol.py.
Useful? React with 👍 / 👎.
| elif args.command == "lease": | ||
| print(pretty_json(acquire_lease(args.state_root, args.slot, args.agent, args.envelope_spec)), end="") |
There was a problem hiding this comment.
Any protocol.py lease invocation immediately fails because acquire_lease requires (state_root, launch_path, agent_id, spec_path, attempt_root), while this dispatch passes only four arguments and treats the slot ID as the launch path. The parser has neither a launch-record option nor an attempt-root option, so callers cannot acquire the prerequisite lease and consequently cannot use the sealing workflow.
Useful? React with 👍 / 👎.
| unreadable = failed_output / "report.md" | ||
| unreadable.write_text("unreadable synthetic report\n", encoding="utf-8") | ||
| os.chmod(unreadable, 0) | ||
| try: | ||
| seal_attempt( |
There was a problem hiding this comment.
When the self-test runs as root or another process able to bypass discretionary read permissions, changing the report to mode 000 does not make it unreadable, so seal_attempt succeeds and the test raises synthetic failed seal unexpectedly succeeded. This occurs in privileged CI/container environments and prevents the advertised synthetic self-test from passing; trigger a deterministic capture failure instead of relying on file permission denial.
Useful? React with 👍 / 👎.
| "freeze/controls.json", | ||
| "freeze/controls-completeness.md", | ||
| "freeze/validate_controls.py", | ||
| "freeze/validate_oracle_materials.py", |
There was a problem hiding this comment.
The static verifier requires and runs the control and oracle validators but omits freeze/validate_fixture_manifests.py and every freeze/fixtures/*.json file. After working around the bytecode issue, protocol.py verify-draft still succeeds when both that validator and a fixture manifest are deleted, so missing or malformed hidden fixture definitions can pass the advertised harness validation and reach later integration unnoticed.
Useful? React with 👍 / 👎.
| chunk = os.read(fd, 1024 * 1024) | ||
| if not chunk: | ||
| break | ||
| chunks.append(chunk) | ||
| after = os.fstat(fd) |
There was a problem hiding this comment.
If an agent writes an unexpectedly large or sparse output file, scan_output reads the entire file into an in-memory chunk list and joins it before capture_envelope checks either the per-file or total-output limit. A malformed attempt can therefore exhaust coordinator memory and abort sealing instead of producing the intended oversize violation; enforce the bound while streaming or reject from the stable file size before allocating its contents.
Useful? React with 👍 / 👎.
| - Scoring: two blind mode-level direct-decision scorers each see all A–O in an | ||
| independent order (16 scorer agents total), one condition-blind consistency | ||
| review per mode, and at most one adjudicator per mode. The deterministic |
There was a problem hiding this comment.
The plan and scoring policy specify one consistency review per mode, but the generator schedules both c1 and c2, the schemas describe two independent reviews, and merge_final_scores requires both. An integration following the stated design would allocate eight consistency agents and then be unable to build or merge the required packets because all sixteen reviews are mandatory in the executable protocol; choose one topology and update the plan, policy, schedules, and validators consistently.
Useful? React with 👍 / 👎.
| lease = { | ||
| "schema_version": 1, | ||
| "status": "STARTED", | ||
| "slot_id": slot_id, | ||
| "attempt_id": f"{slot_id}-{secrets.token_hex(12)}", |
There was a problem hiding this comment.
For any otherwise valid slot ID longer than 103 characters, appending the hyphen and 24-character token produces an attempt_id longer than the 128-character limit enforced by require_safe_id. acquire_lease writes and returns that lease without validating the generated ID, but every later seal_attempt or verify_state call rejects it, permanently stranding a slot that the launch-record schema accepted.
Useful? React with 👍 / 👎.
| flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0) | ||
| fd = os.open(path, flags, 0o600) | ||
| try: | ||
| view = memoryview(data) | ||
| while view: | ||
| written = os.write(fd, view) |
There was a problem hiding this comment.
If the process crashes or a write/fsync fails after the final-path O_EXCL open, exclusive_write leaves a zero-length or partial ledger at its authoritative name. This helper publishes leases, terminal claims, failures, and canonical pointers, so subsequent operations treat the path as already claimed while verify_state cannot parse it; write and fsync a private temporary file first, then use an atomic no-replace publication step for the terminal name.
Useful? React with 👍 / 👎.
| def canonical_json_bytes(value: Any) -> bytes: | ||
| return ( | ||
| json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) | ||
| + "\n" | ||
| ).encode("utf-8") |
There was a problem hiding this comment.
When coordinator metadata contains Python's accepted NaN or infinity literals, json.dumps emits the same non-standard tokens because allow_nan defaults to true. The resulting content-addressed envelope is accepted by this module's equally permissive parser but is not valid JSON under the supplied schemas and fails strict downstream parsers, so canonical serialization must reject non-finite values.
Useful? React with 👍 / 👎.
Add an explicitly DRAFT/UNSEALED eight-mode, three-condition, five-replicate-per-cell diagnostic design. After independent review, correct and validate the 115-atom, 35-control oracle, including F's unavailable-root and fan-out separation and Q's invalid-str invariant escape and later-UB semantics. Bind exact fixture surfaces, frozen skill packages, V4 lineage, authority propositions and quotations, deterministic schedules, schemas, projection contracts, and strict-JSON semantic validators. Preserve synthetic self-tests for schedule generation, atom and gate composition, attempt lifecycles, projection, scoring, consistency, and aggregation data structures. Keep the design conspicuously non-executable as release evidence: blocking integration hooks and a static-integrity failure prevent promotion, and no reports, scores, adjudications, condition maps, seeds, lock, or result are recorded. gherrit-pr-id: Ghh32fbkyuqkrndyfjzwwarkb4dmfosug
091e9d6 to
953089a
Compare
ddfd561 to
3253c71
Compare
2 participants
Add an explicitly unsealed V5 evaluation design that can be reviewed before any semantic agent sees a fixture. The draft defines eight modes, three conditions, five replicates per cell, 109 theorem atoms, closed defect rules, 31 controls, fixture manifests, authority material, and exact schemas for the planned collection and scoring pipeline.
Add seven new fixture trees, reuse the V4 predicate fixture by declaration, and preserve the current V5 runtime skill as a content-addressed frozen package. The package bytes are intentionally exact, including existing Markdown hard line breaks.
Add deterministic schedule generation, atom and gate composition, scorer and consistency data structures, attempt leasing/finalization primitives, word counting, projection contracts, and synthetic self-tests. Keep the run conspicuously DRAFT/UNSEALED: twenty-five blocking integration hooks and a constant static-integrity failure prevent the scaffolding from producing a favorable diagnostic result or being mistaken for release evidence.
No reports, scores, adjudications, condition maps, seeds, lock, or results are included. This commit preserves the pre-integration design as a reviewable baseline; the next phase must repair and implement the promotion/runtime path before any evaluation launches.