Skip to content

Amend RFC 0020 with verifiable bulk records - #237

Merged
kikashy merged 7 commits into
mainfrom
rfc0020-bulk-228
Oct 22, 2026
Merged

kikashy merged 7 commits into
mainfrom
rfc0020-bulk-228

Conversation

@kikashy

@kikashy kikashy commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Closes #228.

What it adds

  • contributingRule on workflow output envelopes, outside Core's unchanged disposition
  • one closed summary and one reproducible seeded sample per configured stage
  • verifier recomputation and refusal for summary or sample disagreement
  • drift as a non-verdict comparison of two verified bundles of the same workflow
  • hand-written fixture, mutation, refusal, and resume cases in the RFC 0020 recording plan
  • workflow text-contract coverage and an unreleased changelog entry

Checks

== check summary (dev.json) ==
handover: boundary 2 paths, shared 3 paths; see AGENTS.md "Before you hand over"
PASS local-paths 0s
PASS agent-docs 0s
sync-agent-docs: CLAUDE.md matches AGENTS.md
PASS diff-check 0s
PASS repository-tests 105s
Ran 211 tests in 104.055s
OK
PASS policy-prototype 1s
Ran 38 tests in 1.290s
OK
PASS policy-contract 3s
Ran 13 tests in 3.453s
OK
PASS policy-official-draft 3s
Ran 16 tests in 3.353s
OK
== 7 passed, 0 failed, 0 skipped, 0 not run ==

Not in this PR

  • the Runtime evaluation record (judgment-pack-runtime#305)
  • the Runner executor (judgment-pack-runner#170)
  • the Runtime and Runner benchmark work (judgment-pack-runtime#306 and judgment-pack-runner#172)
  • a recorded workflow fixture, workflow implementation, or conformance claim

A Codex session wrote the change.

Define contributing-rule output metadata, closed stage summaries, and
deterministic seeded samples in the workflow companion. Require the verifier
to recompute summaries and samples and define drift as a comparison of two
verified bundles so bulk projections remain tied to retained records.

Extend the planned two-stage fixture, refusal cases, recovery coverage, and
text-contract tests for the new closed forms.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: kikashy <kikashy@hotmail.com>
@kikashy

kikashy commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Review round 1 (OpenAI gpt-5.6-sol, effort medium, 2026-10-22), reviewed SHA 0f363df

Clean-room clone at the head, probes under .review/; CI green before the round. The report verbatim:

Findings

  • HIGH — spec/workflow.md:591: The drift result defines only its top-level members; the element forms for stages and records are unspecified, so implementations can emit incompatible field names and structures while satisfying “carries those counter deltas.” Define exact closed forms for stage-delta and changed/added/removed record rows, including nested byRule deltas and null sides.

  • HIGH — spec/workflow.md:623: The decision requires schemas and a §11 fixture carrying every new member, but HEAD contains no workflow schema or recorded fixture and explicitly says none is recorded; consequently the required disposition digest and summary recomputation cannot be verified. Add closed workflow schemas and a hand-authored fixture containing all new forms, summaries, samples, refusals, and independently fixed digests.

Verified

  • Read the complete origin/main...HEAD diff and companion/RFC correspondence.
  • Checked contributing-rule placement, summary counters, canonical disposition byte rules, sample seed/size/ordering/algorithm, verifier refusals, recovery state, and drift semantics.
  • Ran python tests/test_workflow.py under ulimit -v 4000000 and timeout: 6 passed. The first package-style invocation loaded zero tests; the corrected direct invocation passed.
  • Ran .review/probe_bulk_contract.py under the same limits: found zero workflow fixture records and summaries; fixture digest comparison was unavailable. Confirmed the empty-disposition digest as sha256:4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945.
  • Ran git diff --check: clean.
  • Read the changelog and RFC index; both accurately state that only a fixture plan exists and no fixture is recorded.
  • No tracked files changed; only .review/ contains the probe.

merge: after fixes — 0f363df

Disposition (the coordinator's)

# Finding Severity Disposition
1 The drift result's element forms for stages and records are unspecified HIGH Fix in this pull request by the same builder: closed forms for the stage delta and the changed, added and removed record rows, nested byRule deltas and null sides included
2 No workflow schema and no recorded §11 fixture carry the new members, so the digest and the recomputation cannot be verified HIGH Fix in this pull request: a closed workflow schema and a hand-authored two-stage fixture with the summaries, the sample, the refusals and an independently computed digest; the gap predates this change, but the decision requires every new member carried, so it lands here

The same reviewer re-reads the fixed head. Same vendor as the builder, not a cross-vendor claim.

@github-actions

Copy link
Copy Markdown
Contributor

Carry needed from base commit a90dfc8.

kikashy and others added 4 commits October 22, 2026 14:23
Define every stage and record row in the drift result, including signed rule deltas and null-side semantics. Add the closed workflow schema and hand-written two-bundle fixture with summary, sample, verifier, refusal, and drift coverage so readers can recompute the recorded claims.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: kikashy <kikashy@hotmail.com>
…he fix (#228)

Signed-off-by: kikashy <kikashy@hotmail.com>
… entry (#236)

Signed-off-by: kikashy <kikashy@hotmail.com>
@kikashy kikashy closed this Oct 22, 2026
@kikashy kikashy reopened this Oct 22, 2026
@kikashy

kikashy commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Re-read after the fix (OpenAI gpt-5.6-sol, effort medium, 2026-10-23), reviewed SHA 6c141a1

The same reviewer, clean-room clone at the head (the fix plus the carries of #236 and #238), probes under .review/; CI green. The report verbatim:

[MEDIUM] The five sample-refusal vectors are not single-fault cases. Each records selected: [] (for example, conformance/workflow/vectors/sample-seed-empty.json:12), although the offered set contains one outcome record. After repairing only the bad seed or oversized size, the specified algorithm selects digest sha256:7171…7171, leaving a separate sample-mismatch. Update selected in all five vectors to the recomputed selection.

Verified

  • Drift stage/record forms are closed, typed, bounded, and define nested byRule deltas and null sides.
  • The drift fixture validates and independently recomputes exactly from both bundles.
  • All schema object forms use additionalProperties: false; the requested decision members are represented.
  • All four summaries, including canonical disposition digests and counters, independently recompute exactly.
  • The recorded sample independently redraws exactly.
  • Digest and sampling text is deterministic enough for independent implementations.
  • Focused workflow tests: 16 passed.

merge: after fixes
reviewed SHA: 6c141a139ca5c632edf9aedfa0be4b2d4fede1e2

Disposition (the coordinator's)

# Finding Severity Disposition
1 The five sample-refusal vectors are not single-fault: selected: [] leaves a second mismatch once the named defect is repaired MEDIUM Fixed in this pull request by the same builder: selected set to the digest the specified algorithm draws, so each vector fails for its stated reason only; no further round

Both round-1 HIGHs are confirmed fixed (closed drift forms; schema and recorded fixture recompute exactly). The merge follows CI on the fix. Same vendor as the builder, not a cross-vendor claim.

kikashy and others added 2 commits October 22, 2026 14:41
Record the deterministic sample member in all five sample-refusal vectors. Verify that repairing only the named seed or size defect produces a schema-valid selection whose recorded members match a fresh deterministic redraw.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: kikashy <kikashy@hotmail.com>
@kikashy kikashy closed this Oct 22, 2026
@kikashy kikashy reopened this Oct 22, 2026
@kikashy

kikashy commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

The re-read's MEDIUM is fixed at b55f8e4 (the five refusal vectors single-fault; the test proves repairing the named defect alone passes); CI green on the carried head 2889dd7; no further round. Merged on the standing rule.

@kikashy
kikashy merged commit cc2b6e7 into main Oct 22, 2026
5 checks passed
@kikashy
kikashy deleted the rfc0020-bulk-228 branch October 22, 2026 18:54
@github-actions

Copy link
Copy Markdown
Contributor

🎉 Thank you for your contribution to Judgment Pack!

If you’d like another task, choose a path that interests you:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC 0020 amendment: the contributing rule on outputs, a stage summary record, a seeded sample selection, and the verifier's recomputation

1 participant