docs: stop the source disclosing what the docs withhold - #33
Merged
Merged
Conversation
The committed Go source disclosed more than the published doc set withholds. Comments only; no behaviour change. Removed: - Measured figures — sample counts, precision/recall, per-population hit counts and lift, before/after tallies, and the measured cost of rejected alternatives. - Present-tense defeat descriptions — "bypassable by construction" claims and the input shapes that would smuggle a payload past a live check. - Dead pointers — ADR-NNNN citations and paths under docs/dogfood, none of which resolve in a clone. - Internal artefact names — measurement scripts, result files, programme/task numbering, and corpus sample identifiers. Kept, rewritten where needed: what the code does, that each narrow predicate is deliberate, and what a contributor must not change without the maintainer's sign-off. Historical rationale for holes that are closed and pinned by an adjacent regression test stays, since the assertion encodes it anyway. Verified comment-only mechanically: the go/scanner token stream (comments excluded) is identical to main for all 88 tracked .go files, and every changed diff line begins with //. Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
Fix round on the previous commit's under-scrub.
- injection_test.go kept, verbatim, the covert-channel construction
recipe that was removed from injection.go, plus four corpus specimen
paths and a task label. Removed; the cap is now stated as
load-bearing without describing what an uncapped carve-out permits.
- access_control_test.go still carried the "Task 6" label the previous
commit's report claimed to have removed, and a scrub-introduced
factual error ("pin every spelling in homePrefixes" — the four cases
cover the variable spellings only, not `~/`).
- Dead internal pointers repo-wide, same class as the ADR citations:
engine review F-08/F-09/F-10, Plan Task N, SP-1 Task N, Story N.N,
task-13, "the Task 3 brief", "A4 is NOT shipped", and the bare SP-1
programme label in four prose comments.
- scope_real_repo_test.go: a before/after tally and its programme label.
adversarial_test.go's `why:` field is read only as a t.Errorf/t.Fatalf
format argument, never compared, so editing those strings changes no
behaviour. Fifteen are changed here: the two covert-channel strings
(the per-pair encoding recipe and the 4N-joiner arithmetic), eight
carrying measured figures, and five dead dogfood/ADR citations. Every
one is enumerated in the branch report.
Verified: the go/scanner token stream is identical to main for all 88
tracked .go files except exactly those 15 STRING literals; no token of
any other kind changed anywhere.
Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
Revert the `why:` string-literal edits from the previous commit. The adversarial suite is published as a deliberate choice: its tables fail when a recorded case changes state, and that standing signal is worth more than the concealment dropping it would buy. Editing the prose changed nothing an attacker could reach, and spent a non-comment change to do it. Recorded as settled behaviour in AGENTS.md so the next reader does not "finish the job" and delete the gate. CHANGELOG.md carried the largest remaining disclosure: measurement tables, corpus pool composition, named samples, ADR citations and five dead `<workspace>/docs/product/research/...` paths. Removed. Every entry, version, date and grade-changing warning is kept intact, and where a figure is gone the sentence now says the change was validated against an internal corpus rather than resting on nothing. Four over-scrub repairs to the previous commit, where a removal left a comment that no longer carried its point: a circular "must leave them exactly as they were", "whole classes of local execution" in place of the reverse shells the deny-list argument rests on, an absolute "does not fire ... at all" stronger than the evidence, and a "recorded cost" that recorded nothing. The Go changes remain provably comment-only: the go/scanner token stream is byte-identical to main for all 88 tracked .go files. AGENTS.md and CHANGELOG.md are prose, outside that proof, verified by reading. Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
Residue from the previous commit's pass: - "recall on the headline pool moves to 1.0" and "precision up by one sample" were still figures; "unchanged to four decimal places" stated a measurement precision. All three now say the direction of the change without the number. - Six SP-N programme labels in prose. The only one left is inside the `## v0.2.0 — 2026-05-19 (SP-1: Multi-Axis Engine)` heading, which is a dated record's title and stays. Every entry, version, date and grade-changing warning is intact: 13 version headings and 78 entry bullets, both counts identical to main. Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
The adversarial suite is public by decision, and a gap's `why` cannot be scrubbed without destroying its purpose — the fixture beside it already carries the payload. Measurements are the exception: `testdata/adversarial/` shows that a case is uncaught, it does not say how broad or how stable the suppression around it is. That is the class this branch removed from every comment in the repo, and it was still being reprinted in strings a few lines away. Eight strings, figures out and what the case pins kept. Gap descriptions, deny-list observations and prefix observations are untouched. After this no `why:` string in the file carries a measurement of any kind — swept all 85 for counts, ratios, tallies and decimals: zero. Behaviour-neutral by construction: `why` is read only as a t.Errorf/t.Fatalf format argument, never compared. The go/scanner token stream differs from main by exactly these eight STRING tokens across all 88 tracked .go files, with token counts unchanged everywhere; every other change on this branch remains comment-only. Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
The scrub repeatedly turned a bounded measurement into an unconditional one: "0 malicious on corpus X" came back as "absent from the hostile side" or "does not appear in hostile ones". Zero-in-a-corpus is evidence about a set; the rewrite claimed something about all inputs, which nothing supports and which a rule author might build on. Worse in a changelog, where it misleads someone deciding whether to upgrade. Scope restored, counts not. Eleven sites: two in access_control.go, one in injection.go, two in exfiltration.go, two in exfiltration_test.go, one in injection_test.go, and four adversarial `why:` strings. The injection.go one was unflagged — main said "the corpus's real zero-width smuggling ... never has", and the scrub had dropped "the corpus's". Three claims restored to what the original said: the base64 threshold discarded no genuine payload *in the validation set*, and buys that by not catching every corpus path token; the skill.yaml gap shape occurred in *no* benchmark sample, which is what makes the byte-identical re-run follow and what the next sentence already assumed; behaviour-class recall is unchanged *to four decimal places*, which is not equality. Two disclosures this branch chased out of the code and left in the changelog: the one-bit-per-emoji-pair encoding, and the reverse-shell payload with the grade it earned. Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The doc restructuring in #32 published a careful general-terms doc set here while keeping design records, gap lists and measured results out of the repository. Reviewing that work turned up the obvious problem with it: the committed source disclosed more than the documentation withheld. Withholding a document while the code prints the same thing is theatre.
This is the follow-up. Comments and prose only.
The rule applied
Keep the constraint; drop the advantage.
These comments exist so nobody "simplifies" a deliberately narrow predicate, and that reasoning had to survive — losing it is worse than the disclosure, because the next contributor widens the predicate and reopens a hole. What went is anything handing a reader a measurement of how much room remains, the shape of an input that defeats a check, or a pointer to a record that does not exist in a clone.
Removed across 20 Go files and
CHANGELOG.md: measured precision, recall and F1 tables; corpus pool composition and per-sample counts; lift values and before/after tallies; two covert-channel constructions, one of which appeared in three separate places; a worked reverse-shell payload with the grade it received; named corpus specimens; internal decision-record and review-item citations; and paths into a directory tree that is not part of this repository.What was deliberately kept
cmd/skill-detector/adversarial_test.go, its tables and thetestdata/adversarial/fixtures describe cases the engine does not currently catch, and the tables fail when one of those cases changes state. That standing gate is worth more than the concealment giving it up would buy, andAGENTS.mdnow records the decision so the next person doing this work does not delete it.CHANGELOG.mdentry, version, date and grade-changing warning.Two failure modes this had to avoid, and one it hit
Scrubbing can go wrong in both directions, and only one of them is obvious.
Over-scrubbing — a comment that loses the reason a predicate is narrow. Reviewed file by file against "could someone reading only the new comment safely modify this code?"; four comments that had lost their point were repaired.
Strengthening a claim — this one it hit, in eleven places. Zero-in-a-corpus was repeatedly rendered as a universal: "0 malicious hits on the bench corpus" became "does not appear in hostile ones". A bounded measurement became a claim about all inputs that nothing supports and that a rule author might rely on. Worse, in
CHANGELOG.mda measured bound — "discards 0 of 20000 genuine payloads" — became "discards no genuine payload", an unconditional guarantee a case-stability threshold cannot make, with the cost half dropped so the entry read as free. Elsewhere "no sample has the gap shape" became "vanishingly rare" while the untouched next sentence still said "zero prevalence", so the entry argued with itself.All eleven were re-bounded. The scope was restored; the counts were not.
That class is the reason a token scanner is not a review: it sees the absence of a word, never the presence of a falsehood.
Verification
go/scannertoken dump across all 88 tracked.gofiles shows no non-comment token changed anywhere, and no token count moved, with exactly eight enumeratedSTRINGexceptions inadversarial_test.gowhere measured figures were removed fromwhy:metadata. That field is read only as at.Errorfformat argument, so the change is behaviour-neutral by construction.go build,go vet,go test -count=1 ./...— 12 of 12 packages pass.CHANGELOG.mdstructure checked againstmain: 13 version headings identical, 67 entry bullets against 67, all grade-changing warnings present by name.https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH