Skip to content

docs: stop the source disclosing what the docs withhold - #33

Merged
velzepooz merged 6 commits into
mainfrom
docs/source-comment-hygiene
Sep 7, 2026
Merged

velzepooz merged 6 commits into
mainfrom
docs/source-comment-hygiene

Conversation

@velzepooz

Copy link
Copy Markdown
Collaborator

The doc restructuring in #32 published a careful general-terms doc set here while keeping design records, gap lists and measured results out of the repository. Reviewing that work turned up the obvious problem with it: the committed source disclosed more than the documentation withheld. Withholding a document while the code prints the same thing is theatre.

This is the follow-up. Comments and prose only.

The rule applied

Keep the constraint; drop the advantage.

These comments exist so nobody "simplifies" a deliberately narrow predicate, and that reasoning had to survive — losing it is worse than the disclosure, because the next contributor widens the predicate and reopens a hole. What went is anything handing a reader a measurement of how much room remains, the shape of an input that defeats a check, or a pointer to a record that does not exist in a clone.

Removed across 20 Go files and CHANGELOG.md: measured precision, recall and F1 tables; corpus pool composition and per-sample counts; lift values and before/after tallies; two covert-channel constructions, one of which appeared in three separate places; a worked reverse-shell payload with the grade it received; named corpus specimens; internal decision-record and review-item citations; and paths into a directory tree that is not part of this repository.

What was deliberately kept

  • The adversarial suite is published on purpose. cmd/skill-detector/adversarial_test.go, its tables and the testdata/adversarial/ fixtures describe cases the engine does not currently catch, and the tables fail when one of those cases changes state. That standing gate is worth more than the concealment giving it up would buy, and AGENTS.md now records the decision so the next person doing this work does not delete it.
  • Comments describing bypasses found and closed, sitting above the tests that keep them closed.
  • Test names and fixtures for attacks the scanner detects.
  • Every CHANGELOG.md entry, version, date and grade-changing warning.

Two failure modes this had to avoid, and one it hit

Scrubbing can go wrong in both directions, and only one of them is obvious.

Over-scrubbing — a comment that loses the reason a predicate is narrow. Reviewed file by file against "could someone reading only the new comment safely modify this code?"; four comments that had lost their point were repaired.

Strengthening a claim — this one it hit, in eleven places. Zero-in-a-corpus was repeatedly rendered as a universal: "0 malicious hits on the bench corpus" became "does not appear in hostile ones". A bounded measurement became a claim about all inputs that nothing supports and that a rule author might rely on. Worse, in CHANGELOG.md a measured bound — "discards 0 of 20000 genuine payloads" — became "discards no genuine payload", an unconditional guarantee a case-stability threshold cannot make, with the cost half dropped so the entry read as free. Elsewhere "no sample has the gap shape" became "vanishingly rare" while the untouched next sentence still said "zero prevalence", so the entry argued with itself.

All eleven were re-bounded. The scope was restored; the counts were not.

That class is the reason a token scanner is not a review: it sees the absence of a word, never the presence of a falsehood.

Verification

  • Comment-only, proven mechanically. A go/scanner token dump across all 88 tracked .go files shows no non-comment token changed anywhere, and no token count moved, with exactly eight enumerated STRING exceptions in adversarial_test.go where measured figures were removed from why: metadata. That field is read only as a t.Errorf format argument, so the change is behaviour-neutral by construction.
  • go build, go vet, go test -count=1 ./... — 12 of 12 packages pass.
  • CHANGELOG.md structure checked against main: 13 version headings identical, 67 entry bullets against 67, all grade-changing warnings present by name.

https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH

The committed Go source disclosed more than the published doc set
withholds. Comments only; no behaviour change.

Removed:
- Measured figures — sample counts, precision/recall, per-population
  hit counts and lift, before/after tallies, and the measured cost of
  rejected alternatives.
- Present-tense defeat descriptions — "bypassable by construction"
  claims and the input shapes that would smuggle a payload past a
  live check.
- Dead pointers — ADR-NNNN citations and paths under docs/dogfood,
  none of which resolve in a clone.
- Internal artefact names — measurement scripts, result files,
  programme/task numbering, and corpus sample identifiers.

Kept, rewritten where needed: what the code does, that each narrow
predicate is deliberate, and what a contributor must not change
without the maintainer's sign-off. Historical rationale for holes
that are closed and pinned by an adjacent regression test stays, since
the assertion encodes it anyway.

Verified comment-only mechanically: the go/scanner token stream
(comments excluded) is identical to main for all 88 tracked .go
files, and every changed diff line begins with //.

Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
Fix round on the previous commit's under-scrub.

- injection_test.go kept, verbatim, the covert-channel construction
  recipe that was removed from injection.go, plus four corpus specimen
  paths and a task label. Removed; the cap is now stated as
  load-bearing without describing what an uncapped carve-out permits.
- access_control_test.go still carried the "Task 6" label the previous
  commit's report claimed to have removed, and a scrub-introduced
  factual error ("pin every spelling in homePrefixes" — the four cases
  cover the variable spellings only, not `~/`).
- Dead internal pointers repo-wide, same class as the ADR citations:
  engine review F-08/F-09/F-10, Plan Task N, SP-1 Task N, Story N.N,
  task-13, "the Task 3 brief", "A4 is NOT shipped", and the bare SP-1
  programme label in four prose comments.
- scope_real_repo_test.go: a before/after tally and its programme label.

adversarial_test.go's `why:` field is read only as a t.Errorf/t.Fatalf
format argument, never compared, so editing those strings changes no
behaviour. Fifteen are changed here: the two covert-channel strings
(the per-pair encoding recipe and the 4N-joiner arithmetic), eight
carrying measured figures, and five dead dogfood/ADR citations. Every
one is enumerated in the branch report.

Verified: the go/scanner token stream is identical to main for all 88
tracked .go files except exactly those 15 STRING literals; no token of
any other kind changed anywhere.

Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
Revert the `why:` string-literal edits from the previous commit. The
adversarial suite is published as a deliberate choice: its tables fail
when a recorded case changes state, and that standing signal is worth
more than the concealment dropping it would buy. Editing the prose
changed nothing an attacker could reach, and spent a non-comment change
to do it. Recorded as settled behaviour in AGENTS.md so the next reader
does not "finish the job" and delete the gate.

CHANGELOG.md carried the largest remaining disclosure: measurement
tables, corpus pool composition, named samples, ADR citations and five
dead `<workspace>/docs/product/research/...` paths. Removed. Every
entry, version, date and grade-changing warning is kept intact, and
where a figure is gone the sentence now says the change was validated
against an internal corpus rather than resting on nothing.

Four over-scrub repairs to the previous commit, where a removal left a
comment that no longer carried its point: a circular "must leave them
exactly as they were", "whole classes of local execution" in place of
the reverse shells the deny-list argument rests on, an absolute
"does not fire ... at all" stronger than the evidence, and a "recorded
cost" that recorded nothing.

The Go changes remain provably comment-only: the go/scanner token
stream is byte-identical to main for all 88 tracked .go files.
AGENTS.md and CHANGELOG.md are prose, outside that proof, verified by
reading.

Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
Residue from the previous commit's pass:

- "recall on the headline pool moves to 1.0" and "precision up by one
  sample" were still figures; "unchanged to four decimal places" stated
  a measurement precision. All three now say the direction of the
  change without the number.
- Six SP-N programme labels in prose. The only one left is inside the
  `## v0.2.0 — 2026-05-19 (SP-1: Multi-Axis Engine)` heading, which is
  a dated record's title and stays.

Every entry, version, date and grade-changing warning is intact: 13
version headings and 78 entry bullets, both counts identical to main.

Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
The adversarial suite is public by decision, and a gap's `why` cannot be
scrubbed without destroying its purpose — the fixture beside it already
carries the payload. Measurements are the exception: `testdata/adversarial/`
shows that a case is uncaught, it does not say how broad or how stable the
suppression around it is. That is the class this branch removed from every
comment in the repo, and it was still being reprinted in strings a few lines
away.

Eight strings, figures out and what the case pins kept. Gap descriptions,
deny-list observations and prefix observations are untouched. After this no
`why:` string in the file carries a measurement of any kind — swept all 85
for counts, ratios, tallies and decimals: zero.

Behaviour-neutral by construction: `why` is read only as a t.Errorf/t.Fatalf
format argument, never compared. The go/scanner token stream differs from
main by exactly these eight STRING tokens across all 88 tracked .go files,
with token counts unchanged everywhere; every other change on this branch
remains comment-only.

Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
The scrub repeatedly turned a bounded measurement into an unconditional
one: "0 malicious on corpus X" came back as "absent from the hostile
side" or "does not appear in hostile ones". Zero-in-a-corpus is evidence
about a set; the rewrite claimed something about all inputs, which
nothing supports and which a rule author might build on. Worse in a
changelog, where it misleads someone deciding whether to upgrade.

Scope restored, counts not. Eleven sites: two in access_control.go, one
in injection.go, two in exfiltration.go, two in exfiltration_test.go,
one in injection_test.go, and four adversarial `why:` strings. The
injection.go one was unflagged — main said "the corpus's real zero-width
smuggling ... never has", and the scrub had dropped "the corpus's".

Three claims restored to what the original said: the base64 threshold
discarded no genuine payload *in the validation set*, and buys that by
not catching every corpus path token; the skill.yaml gap shape occurred
in *no* benchmark sample, which is what makes the byte-identical re-run
follow and what the next sentence already assumed; behaviour-class
recall is unchanged *to four decimal places*, which is not equality.

Two disclosures this branch chased out of the code and left in the
changelog: the one-bit-per-emoji-pair encoding, and the reverse-shell
payload with the grade it earned.

Claude-Session: https://claude.ai/code/session_01XAY7LzfeuEZBgSLuzpWUAH
@velzepooz
velzepooz merged commit 6399f31 into main Sep 7, 2026
4 checks passed
@velzepooz
velzepooz deleted the docs/source-comment-hygiene branch September 7, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant