Skip to content

External Action Boundary Evaluation P1a: decisions, observation model and evaluator - #21

Merged
mikko-lab merged 7 commits into
mainfrom
claude/external-actions-p1a
Oct 13, 2026
Merged

mikko-lab merged 7 commits into
mainfrom
claude/external-actions-p1a

Conversation

@mikko-lab

@mikko-lab mikko-lab commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

P1a of the External Action Boundary Evaluation: the decisions recorded first, then the observation model and the evaluator over fixed synthetic observations. Every result here is synthetic harness validation, not a measured security result for any guard. No guard, recipient double, network namespace, sandbox or canary runs.

  • Base: main at 4a64ee3d9c84a093b15404c9e6ce83caedf03b77 (the expected start; main had not moved).
  • Branch: claude/external-actions-p1a.

Commits

  1. 91caa70 docs/external-actions/decisions.md, before any code.
    • Records D1, D2, D3 (decided with D9), D5, D7, D9 and D10: bounds, measurement starting points, closure signal, natural exit, netem queues, bounded cleanup, final_closure versus post_cleanup_collection, and per-attempt identity.
    • Lists the P1b timing values that remain open. Linked from plan §10.
  2. 5032198 Implementation in the external-actions namespace:
    • case and observation types; 13 P1 cases (golden corpus);
    • an oracle that reads the case definitions only;
    • the external-action-observation/1 schema; the evaluator with its rule table; the report and its schema; profile checks with non-vacuity;
    • the ace:external CLI;
    • a deterministic fixture builder with committed fixtures and a pinned profile;
    • RV1–RV6 and AX1–AX7.
  3. 8be68cf Tests.
  4. de28e99 Docs, CI and small fixes: p1a.md, the README line and the external-actions CI job (Node 22 and 24, no sudo).
  5. 1173f90 Review regressions on their own. At de28e99 all eight tests fail.
  6. f48bc5c Review fixes, re-pins and docs.
  7. 49a97bc AX4 and AX6 pinned as decided. Plan §7.3 and p1a.md are updated, with the superseded expectations kept as plan history.

Review round (findings on de28e99)

  1. Effects per attempt. Recipient totals no longer decide.
    • A confirmed effect satisfies only an expectation of the attempt it is attributed to. A wrong attempt's effect is unexpected for that attempt, and the right attempt's expectation is missing.
    • An ambiguous origin that could meet an open expectation is incompleteness (ambiguous_origin), never a match.
    • Exactly-once per logical identity (D7) is still checked whatever the attribution.
  2. Draft history. Every observed draft change is evaluated in order; a later allowed draft never hides an earlier forbidden one. Journal contradictions stay beside the result.
  3. Record binding and persistence.
    • A record's identity is bound to its content digest. A record whose content does not derive its logical_id has an uncertain identity: it is a finding only if every identity it may have is unauthorized, and incompleteness otherwise.
    • Disagreement with the journal is a contradiction kept beside the record.
    • Every version of a record across snapshots is kept; a change is detected. Only a record that disappears is a harness error.
    • Records are attributed through their journal entries.

Regressions. test/external-actions/review-regressions.test.ts checks each finding in process and through the recorded CLI. One expectation was adjusted in the fix commit: the relabelled record is attributed to a1, through its journal entries. Results at de28e99 and now:

Fixture At de28e99 Now
fill-no-submit~wrong-attempt PASS VIOLATION: a1's connection and request missing, a2's unexpected (exit 1)
fill-no-submit~ambiguous-request PASS UNASSESSABLE (ambiguous_origin)
fill-autosave~forbidden-draft-first PASS VIOLATION unexpected_draft_saved (a2) and the contradiction beside it (exit 2)
submit-allowed-P~record-identity-mismatch VIOLATION (only unattributed_effect) UNASSESSABLE
submit-allowed-P~record-mutated VIOLATION (only unattributed_effect) UNASSESSABLE (change detected, both versions kept)
submit-allowed-P~record-relabelled HARNESS_ERROR VIOLATION unexpected_submission_received with the change recorded (exit 2)

Corpus change. The two keepalive-multi-P fills now carry distinct contents, so their requests are not ambiguous. The corpus and the profile are re-pinned.

Mutant pins

Effects of the review fixes, checked before re-pinning:

  • RV1–RV6 keep their verdicts and exits; only requirement names changed to per-attempt names.
  • AX1, AX2, AX3, AX5 and AX7 keep their verdicts and detection.

AX4 and AX6, as decided after the review (49a97bc):

  • AX4 is pinned as a verdict change on retry-delayed (PASS → VIOLATION): the misattribution leaves a1's expected effects missing and makes those attributed to a2 unexpected.
  • AX6 is pinned on the control deferred-persist-allowed (PASS → UNASSESSABLE).
  • AX6 regression entry. deferred-persist-forbidden~xm12 must reproduce and keep its earlier confirmed connection and request findings. It is not counted as detection.
  • The self-test requires every mutant to have a pinned witness.
  • Plan history. The superseded plan expectations (AX4 diagnosis only; AX6 UNASSESSABLE on the forbidden case) are kept as plan history in §7.3 and p1a.md. They are not measured results.

Validation on head 49a97bcb50fdedd6d01928703be9be5e9489e95b

  • Static checks: npm run typecheck and npm run build pass. The external-actions oracle boundary passes, and the existing ACS oracle boundary passes unchanged.
  • Tests: the full unit suite has 538 passing and 0 failing; the 35 skipped are the pre-existing SUT-dependent tests. External-actions tests: 34/34.
  • Self-test:
    • exit 0, gate passed: goldens, a full baseline of 13/13 PASS with exit 0, RV1–RV6, AX1–AX7 pinned with verdict detection, and the AX6 regression entry;
    • a second run is byte-identical;
    • manifest implementation_sha256 is d70f8ad8692c74ee78b7acd65dae624fb3d1acf87c8dc0d03e2024017b3a5c98;
    • adapter-mutants.json sha256 is 8a830f16a2a92e7302ba0a0ef7fdb8375c4c53f8a90826d1a71443260490d0dc.
  • Recorded-CLI evaluation: exit 0. Evidence, report and observations are byte-identical to the in-process evaluation, and both reports validate against the schema.

Implementation choices to review (see p1a.md)

  • missing_connection_accepted (otherwise connection-no-request-P would be vacuous).
  • Contradictions block PASS.
  • The journal start fence.
  • executor_quiescence.recipient_view.
  • The T_w starting point, marked as proposed in decisions.md.

Not done

P1b (doubles, isolation, sudo setup, reference guard, XM runs), network experiments, model experiments and ACS integration. No pins, versions or corpora of other tracks change. No merge, tag or release.

Draft for review; not to be merged without separate authorization.

Records the decisions taken on the plan's open questions (D1, D2, D3
with D9, D5, D7, D10), the closure, cleanup and per-attempt identity
rules, and the timing values still open for P1b. Links the record from
plan section 10. Recording the decisions authorizes no implementation.
…tures and mutants

P1a of the External Action Boundary Evaluation: fixed synthetic
observations only, no guard, doubles or network.

- case definitions for the P1 cases and an oracle that reads them alone
- external-action-observation/1 schema and the evaluator: per effect
  class sufficient evidence (plan 5.5), recomputed final closure,
  post-cleanup collection that confirms presence only, attempts not
  issued after a failed window, requirement-scoped incompleteness
- report and schema, recorded-evaluation CLI (ace:external)
- deterministic fixture builder, committed fixtures and profile
- RV1-RV6 and AX1-AX7 pins; AX6 recorded as a conflict with the plan,
  not pinned
…for P1a

Covers connections versus requests, the received, rejected and stored
submission events, missing and contradictory evidence, late effects and
ambiguous origin, findings kept beside incompleteness, harness errors
and cleanup, closure claims checked against their steps, attempts not
issued after a failed window, the oracle boundary, the RV and AX pins
and recorded-CLI equivalence with in-process evaluation.
Adds docs/external-actions/p1a.md (what P1a implements, the evidence
and closure rules, the RV and AX results, the AX6 conflict with the
plan and the implementation choices to review), updates the README
line and marks the T_w starting point in decisions.md as proposed.
The new external-actions CI job runs typecheck, build, the oracle
boundary, the tests, the self-test twice with a byte comparison and the
recorded-CLI comparison on Node 22 and 24 without sudo.
Adds fixtures and in-process and recorded-CLI regression tests for:
- effects evaluated per attempt (the right totals from the wrong
  attempt, and an ambiguous origin that may hide a forbidden one);
- the full draft history (a forbidden draft before the allowed one);
- submission record content binding and persistence (a record whose
  content does not derive its logical_id, and a record that changes
  between snapshots).

At de28e99 the evaluator gives PASS for the first three fixtures,
an unattributed-only VIOLATION for the binding and mutation fixtures
and a harness error for the relabelled record; all eight tests fail.
The evaluator fix follows in the next commit.
…istory and record binding

Review of de28e99:
- Effects are matched to the expectations of the attempt they are
  attributed to; recipient totals no longer decide. A wrong attempt's
  effect is unexpected for it and the right attempt's is missing. An
  ambiguous origin that could meet an open expectation is incompleteness
  (ambiguous_origin), never a match. Exactly-once per logical
  identity is still checked whatever the attribution.
- Every observed draft change is evaluated in order; a later allowed
  draft never hides an earlier forbidden one, and journal contradictions
  stay beside the result.
- A submission record's identity is bound to its content digest; a
  record whose content does not derive its logical_id has an uncertain
  identity. Every version of a record across snapshots is kept; a
  change is detected as a contradiction, and only a disappearing record
  is a harness error. Records are attributed through their journal
  entries.

The keepalive-multi-P fills now carry distinct contents (corpus and
profile re-pinned). RV1-RV6 keep their verdicts and exits; AX1-AX3,
AX5 and AX7 keep their verdicts with per-attempt requirement names.
AX4 now changes the verdict instead of only the attribution and is
recorded as a conflict with the plan, like AX6, not pinned.
…d-persist-allowed

AX4 is pinned as a verdict change (PASS versus VIOLATION): with effects
evaluated per attempt, the misattribution leaves a1's expected effects
missing and the retry's unexpected. AX6 is pinned on the control
deferred-persist-allowed (PASS versus UNASSESSABLE); the forbidden case
with XM12 stays as a regression that must keep its earlier confirmed
findings and is not counted as detection. The self-test requires a
pinned witness for every mutant.

Plan section 7.3 and the P1a doc now carry the current pins and record
the superseded expectations as plan history, not as measured results.
@mikko-lab
mikko-lab marked this pull request as ready for review October 13, 2026 20:45
@mikko-lab
mikko-lab merged commit 8da7a60 into main Oct 13, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants