Skip to content

P68 (#2300) lexer observation totality + verify fixes: allowlists pin today's defect, scanner walrus arm observed - #2341

Merged
antonsynd merged 27 commits into
mainlinefrom
dev
Oct 22, 2026
Merged

antonsynd merged 27 commits into
mainlinefrom
dev

Conversation

@antonsynd

Copy link
Copy Markdown
Owner

P68 (#2300): lexer observation totality, plus the verify-round fixes. These are tests, build_tools and docs. The one production edit is behaviour-neutral: the TokenizeAll declaration move (e621d9e); the token dump is identical over 4,053 fixtures.

  • Implementation (20 commits, 220259b..0d58024) adds:
    • a CPython numeric-literal twin (C1);
    • a corpus position/line-ending/EOF twin and an EOF construct roster (C2/C3/C6);
    • a literal-loss scanner injection differential (C5);
    • an f/t/df-string arm matrix (C4);
    • a spec-anchored keyword roster (C7);
    • the Stryker triage script and ignore list;
    • a fix to the spec's stale to keyword row.
  • Verify fixes (6 commits, 5bedd44..96d7a27):
    • allowlist rows pin today's verdict, or the token's Value plus rules, so they no longer absorb a second defect;
    • an @walrus twin for the depth-0 := scanner arm;
    • a bulk-only fixed stride for the injection differential;
    • the RunBatch broken-pipe fix;
    • three pytest guards;
    • audit-doc wording.
      Every guard is mutation-tested; the commit bodies record the results.

Gate 81093/0/7 @ be4664e (measured). 96d7a27 is comment-only. build_tools pytest 1232 passed; format and allowlist issue-state checks clean.

#2300 stays open: the Stryker re-run is owner-slotted, and the 01/0_1 allowlist rows cite it. Residue: #2340 (new), cells on #2338 and #2300.

🤖 Generated with Claude Code

antonsynd and others added 27 commits October 21, 2026 08:52
…nds — a nested conditional, lambda, walrus, try or maybe there is parenthesized (refs #2169)

Ruling R-HP (chore, Pretty only). The parser reads a conditional's then-value
AND its test with ParseNullCoalesce (CPython: or_test); only the else-value is
a full `test`. The unparser wrote the test through a bare Visit (no precedence
context), and the then-value at PrecConditional with parentIsRightAssoc false,
which leaves an equal-precedence conditional or lambda bare. Both now go
through VisitExprInContext at PrecNullCoalesce. Parsed source is unchanged (it
carries a Parenthesized node); only synthesized ASTs (generators, normalized
trees) reach it.

Matrix (UnparserParenthesizationMatrixTests.ConditionalCells, 42 cells):
operand {conditional, lambda, walrus, try, maybe, ??, or} x position
{then, test, else} x source {synthesized (normalized, exact minimal text),
parsed (keeps Parenthesized; re-parse to an equal tree)}. Necessity verdicts
for conditional/lambda/walrus/or checked against python3 ast.parse/unparse.
At base 27bf064: 7 synthesized cells red (test: conditional, lambda, walrus,
try, maybe; then: conditional, lambda — the then-position is a sibling cell
of the same contract). The matrix lambda takes a parameter: the parameterless
`lambda :` spacing is a separate P22d cell this ruling leaves alone.

Drained with the fix: the KnownGeneratorPins #2169 pin (roster now empty) and
ParsePins_StillFire; NarrowedProgram_AnalysesClean back to fuel 2 (it was held
at fuel 1 for this cell); gap-discovery-contracts row updated.

Mutations (in tree, restored by copy, cmp-verified):
- M1 test position back to bare Visit -> 8 red: 5 synthesized test cells,
  TypedGeneratorRosterTests.YGenerated_RawCorpusContainsTheConstruct,
  TypedAnalyzePropertyTests.TypedProgram_AnalyzesClean and
  FilteredTypedProgram_AnalyzesClean (no pin hides them now); restored -> green.
- M2 then-value back to PrecConditional -> 2 red (conditional/lambda in then,
  synthesized); restored -> green.

Gate: 80453/0/4 @ 27bf064+this diff (measured; 80411 + 43 new - 1 deleted).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…l to" sentences — removed in 0.8.0 (type_casting.md); the roster test anchors to the corrected tables (refs #2300)

keywords.md listed `to` as a hard keyword ("Type coercion operator"), but the
lexer and parser have no such spelling (grep '"to"' over Lexer/ and Parser/ = 0
@ 220259b) and type_casting.md records the operator's removal in 0.8.0 (#1127).
Four sentences still called it a live or "transitional" cast:

- conversion_operators.md: explicit conversions now name `as!` / `as?`.
- operator_precedence.md: level 12 is `as!`, `as?` only.
- overload_resolution.md: a narrowing conversion needs an explicit `as!` cast.
- try_expressions.md: `try` binds lower than `as!`/`as?`.

No fenced block changed. Execution evidence @ 220259b (built sharpyc):
`take(big)` with `big: int64` into an `int` parameter → SPY0220;
`take(big as! int)` → prints 42; `to = 3; print(to)` → prints 3 (an ordinary
identifier); `r = try take(big as! int) + 1` → prints Ok(43) (try captures the
whole arithmetic expression). After the edit, the only `to` mentions in
docs/language_specification/ are type_casting.md's removal note.

Plan P68 Phase 4 Task 3 (Design Decision 7, docs chore). The C7 keyword
roster test anchors to the corrected tables (49 hard + 2 rejected + 2 future).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…ta — one reason per entry, C1–C7 class assignment by method (refs #2300)

P68 Phase 0 Task 1 (plan-d31acd, Design Decision 9).

- build_tools/stryker/triage.py: reads a mutation-testing-elements report
  (1-based line / UTF-16 column, per-file `source`, disk fallback) and
  buckets every mutant by (file, mutator, enclosing invocation, status).
  A small C# tokenizer handles strings, verbatim, interpolated and raw
  strings, char literals and comments. Mutator display names map to
  Stryker config identifiers. The names were read from dotnet-stryker
  5.0.0's Stryker.Core.dll, and the composed "{Kind} to {Kind} mutation"
  forms are matched by shape. An unknown name is counted as unknown:<name>.
  It prints score and scoreIgnoringListed by Stryker's formula,
  (killed+timeout)/(killed+timeout+survived+noCoverage), plus every
  status count, per-entry match counts, buckets, and classes C1–C7 and
  unclassified over the mutants still undetected after the list.
  --classes lists each class's mutants. --universe DIR widens the
  within-name check for partial-scope reports such as the Phase 0 probe.
  The script exits 2 on an unknown status, an out-of-range location, or
  a malformed entry. It also exits 2 when a top-level `within`
  alternative names no invoked method, so it never reports a silent zero.
- `within` matches the CALL a mutant sits in, never the declaring method.
  `fieldOf: "operator Value"` matches when the mutant is the value
  argument of CreateToken, written as a plain punctuation-only literal
  with a non-FString TokenType.
- build_tools/stryker/lexer-ignore.json has the two R-GB entries, each
  with a reason. It departs from Design Decision 9 in two ways:
  - LogDebug is dropped. It is invoked nowhere in
    src/Sharpy.Compiler/Lexer/ @ 220259b, and the per-alternative check
    refuses it.
  - AddError is added. Two dedented-string message literals in
    Lexer.Literals.cs reach the diagnostic bag directly through it.
  Positive control over every real string literal in Lexer/: the list
  covers 81 message literals and 30 operator/delimiter Values. "", "\n",
  the FString* "{"/"}" and the keyword table stay scored.
- build_tools/tests/test_stryker_triage.py runs a hand-built 20-mutant
  report over five Lexer-shaped sources: every Stryker status, both
  entries, both fieldOf negatives, each class rule and unclassified.
  It checks the score arithmetic (17.65 / 15.38), loud failures, CLI
  exit codes, mutator-name mapping, and the committed list against the
  real Lexer sources.

pytest: python3 -m pytest build_tools/tests/test_stryker_triage.py →
51 passed (python 3.12.13).

Guard, data direction (asserted in the pytest): with within ReportError…,
the "Consecutive underscores…" String mutant inside ReadNumber's
ReportError call is ignoredByList. With within=ReadNumber it is
survived, class C1.

Script mutations (each restored by cp from a scratchpad copy):
- within never matches → red (9 failed), restored → green (51 passed)
- absent-name check disabled → red (3 failed), restored → green (51 passed)
- fieldOf predicate skipped → red (6 failed), restored → green (51 passed)
- within also matches the declaring member → red (1 failed), restored → green (51 passed)
- score drops timeout from detected → red (2 failed), restored → green (51 passed)
- C2 column rule removed → red (2 failed), restored → green (51 passed)
- char literals tokenized as code → red (37 failed), restored → green (51 passed)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…ion, open bracket, header colon) as the oracle for the literal-loss scanner (refs #2300)

P68 Phase 3 Task 1 (#2300 row C5). CleanLineFacts reads every fact off the
clean lexer's own tokens and LiteralSpans.Of, never off a second hand model
of the text: OpenerCrossing(k, at), Continued(k) (a real bracket open at the
line's end, or a backslash continuation; the token reading is cross-checked
against the text and throws when inconsistent), BracketOpenerLine(k) (the
outermost real bracket), ColonOnLogicalLine(k) and TokenBefore(k, abortAt).

Domain restriction (verifier, Dimension 4): Continued, BracketOpenerLine and
PossibleHeader THROW on a line whose end is inside a literal span, so the
injection differential cannot assert them there by accident; there only
OpenerCrossing is a clean fact.

The f-string exclusion (brackets and colons inside FStringStart..FStringEnd
do not count) mirrors the lexer's own `_fstringStack.Count == 0` condition.
It cannot be falsified in-domain: a format-spec colon emits no Colon token,
and field brackets balance inside the field. It is defensive, and no guard
is claimed for it.

Oracle hand rows (24) plus an offset-fact test. The existing class gains
`partial`; its Corpus_* and hand rows are unchanged.

Mutations (worktree @ 76b7acc, filter LiteralLossScannerDifferential):
- domain restriction dropped (RequireDomain never throws on EndInsideLiteral)
  → red (3 failed), restored → green (46 passed)
- outermost opener read as the innermost (brackets[0] → brackets[^1])
  → red (1 failed), restored → green (46 passed)
- ColonOnLogicalLine ignores the logical line (a colon on an earlier
  logical line counts) → red (2 failed), restored → green (46 passed)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…icts on dropped text agree with the clean lexer's facts over the corpus × four error constructs; allowlist ratchet (refs #2300)

P68 Phase 3 Task 2 (#2300 row C5). Injected_ScannerAgreesWithTheCleanLexersFacts
runs over every clean document under FixtureRoots.All and its clean twins.
For each line it injects $ (SPY0015 at the first token start outside a
literal), \q (SPY0004 after the opening quote of each top-level non-raw
short string), !q (SPY0030 before the } of each plain outermost field of a
single-line f-/t-string) and a tab indentation (SPY0012 on a line that
starts its logical line). DroppedOpener must equal OpenerCrossing. In-domain,
the resume's continued and possible-header facts and BracketLeftOpenAtLine
must equal the clean facts. A construct that does not abort exactly once on
its line is a misfire and fails unconditionally.

Each injected document is the clean PREFIX through line k. Recovery resets
the indent stack and the bracket depth, so on a whole document the lines
below cascade into more recoveries whose flags pile into the cumulative
LiteralLoss. With the prefix, every recorded fact belongs to the injection
(one recovery, asserted).

Twins, the corpus-derived content axis, each built clean and asserted clean:
- @backtick: identifiers respelled as names holding :("""'#. No fixture has
  such a name.
- @escaped-quote: an escaped own quote after each short string's opening
  quote. No fixture's short string holds an odd count of escaped own quotes.
- @field-split: a line break before the } of plain single-quoted f-string
  fields (PEP 701, as in fstrings/multiline_hole_2022).

Measured @ 813df74 (stride 1, no sampling needed, 9.9 s wall):
SCANNER-INJECT documents 10296 sites 125268 constructs 4 cells 272201 disagreements 0 allowlisted 0 stride 1
(clean 4026 of 4053; dollar 125268, escape 20001, conversion 1852,
tab 125080; opener-crossing cells, DroppedOpener only, 1192; close lines
excluded 572; misfires 0; twins rejected 0.)

The PossibleHeader oracle reads the plan's `\q` row as the general rule:
!Continued and a token read before the abort and a depth-0 colon. The given-up
text starts at the aborted literal's prefix, so DroppedLineContinues reads that
literal closed and the end stays readable; L11 does not apply. The lexer agrees
on all 20001 escape cells.

Ratchet literal-loss-scanner-allowlist.txt: 0 rows. A row without an issue
cite, outside the generated universe, or whose cell agrees fails.

Mutations (worktree @ 76b7acc, filter LiteralLossScannerDifferential):
- DroppedLineContinues escape arm (LiteralLoss.cs:148) always false
  → red (1 failed; killed only by @escaped-quote), restored → green (46 passed)
- DroppedLineContinues SkipBacktickName arm (:163) skipped
  → red (1 failed; @backtick), restored → green (46 passed)
- ClosesOnItsLine (:462) inverted
  → red (1 failed; @field-split, nested_prefixed_literal_in_hole_2010.spy:8),
  restored → green (46 passed)
- WalkFrom text escape arm (:370) skipped
  → red (1 failed; @field-split, f_string_both_quotes.spy:3),
  restored → green (46 passed)
- WalkFrom SkipBacktickName arm (:338) skipped
  → red (2 failed: the existing hand row and the injection), restored → green (46 passed)
- NOT killed: WalkFrom `}}` text arm (:401) dropped → green (46 passed). The
  mutant is EQUIVALENT: a `}` in a literal's text does nothing either way.
- NOT killed: WalkFrom `{{` text arm (:391) dropped → green (46 passed). On
  balanced clean text a `{{` read as a field plus a brace pairs out with `}}`;
  no corpus or twin cell separates them. Aperture reported, not claimed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
… literal-loss and continuation facts against construction-known truth (refs #2300)

P68 Phase 3 Task 3 (#2300 row C5). Shapes_LexerFactsMatchTheirConstructionKnownTruth
asserts, per document, the verdicts the two shape sweeps in LiteralStateTests
leave unasserted:
- Budget stops (all four BudgetStopLines positions): LiteralLoss is EXACTLY
  UnreadRemainder when Q holds a literal spanning lines, EXACTLY None
  otherwise.
- dropped-SPY00NN: wherever the shape's code is the first error, on a line
  where one of Q's literals spanning lines opens, DroppedOpener is set.
- Every continuation shape gets the full continued / possible-header /
  bracket-at-injected truth table (ContinuationTruth), including both dropped
  headers of -nested and the dropped continuation line of -cont. The header
  fact had no corpus assertion before.

Loss shapes run over Q and its wide twin, continuation shapes over Q and its
lone-CR twin. Keys are file[@TwiN]:line:shape:NAME in the same ratchet file.

Deviation from the plan: its "LiteralStateUnknown == (Kind != null)" for
every loss shape is false for the closeline-* direction controls and
repair-comment. Their truth is R-FR/R-FV, owned by LiteralStateTests, so they
are not judged here. The parents are the whole clean corpus (stride 1)
rather than 300, and the run is within budget.

Measured @ 813df74 (11.5 s wall):
SCANNER-SHAPES parents 4026 of clean 4026 stride 1 documents 150240 cells 149331 disagreements 0 allowlisted 0
(judged per shape: budget* 8052 each, dropped-SPY0004/11/12/13/15 28 each,
dropped-SPY0014 9, continuation shapes 7982 each, -sibling 5226.)

Mutations (worktree @ 76b7acc, filter ...Shapes):
- RecoverFromError's possibleHeader forced false (Lexer.cs:551)
  → red (1 failed; recovery-header-* PossibleHeader lexer=False truth=True),
  restored → green (1 passed)
- the budget stop's UnreadRemainder never set (Lexer.cs:441)
  → red (1 failed; budget* LiteralLoss lexer=None truth=UnreadRemainder),
  restored → green (1 passed)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…f-string text kills the escaped-brace arm the corpus balanced out (refs #2300)

Follow-up to e316dda. WalkFrom's `{{` escaped-brace arm
(Lexer.LiteralLoss.cs:391) survived the injection differential because every
fixture's `{{` is balanced by a later `}}`. A scanner that reads `{{` as a
field opener plus a brace closes both again at `}}`.

The new clean twin @unbalanced-brace inserts `{{#` right after the opening
quote(s) of every single-line, top-level f-/t-string. It is an escaped brace
with no `}}` to pair it, then a `#` that is text. A scanner that misreads the
`{{` sees the `#` as a comment inside a field and the line as ending inside
one; the clean lexer says the literal closed.

A lone `{{` before plain text pairs out too. The program
`x = f"a{{b"` stays green with the arm dropped: in a field the enclosing quote
closes the literal when no literal opened there closes on its line (the
`f"{x" + 'z'` rule). The `#` is what makes the misread observable.

Measured @ ffdd7f7 (stride 1):
SCANNER-INJECT documents 10552 sites 130409 constructs 4 cells 283851 disagreements 0 allowlisted 0 stride 1
(twins as-is 4026, backtick 4026, escaped-quote 2022, unbalanced-brace 256,
field-split 222, rejected 0; misfires 0; wall 38.4 s under concurrent peer
load, 9.9 s at e316dda.)

Mutation (fresh worktree @ ffdd7f7, filter LiteralLossScannerDifferential,
plus a worktree-only probe that was never committed):
- `{{` skip at Lexer.LiteralLoss.cs:391–392 removed → red (2 failed:
  Injected_ScannerAgreesWithTheCleanLexersFacts with 1934 disagreements, all
  in @unbalanced-brace cells; e.g. Formatting/functions_with_comments.spy:5
  `msg = f"{{#Hello, {name}"` DroppedOpener lexer=True oracle=False. Also
  the probe's `x = f"{{#a"` row; the probe's `x = f"a{{b"` row stayed green),
  restored → green (48 passed).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…@unbalanced-brace twin (refs #2300)

Follow-up to e7bb3cf. That commit added the @unbalanced-brace twin to the
generated universe; the allowlist header's suffix list now names it too, so
the documented key format matches the keys the test generates.

This edits only a comment line, and the file still has 0 rows. No test reads
the header, so no mutation pair applies.
`scan_allowlist` returns [].

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…required-oracle mode so CI cannot soft-skip (refs #2300)

Sharpy.TestInfrastructure.Oracles.PythonInterpreter is the python3 locator for the P68 numeric
twin (#2300 row C1), so the twin does not add a fourth private copy beside
CPythonDifferentialParseTests.PythonOracle, FormatSpecDifferentialTests.PyOracle and
DifferentialExecutionTests (retrofitting those three soft-skippers is a chore note on #2300,
not this seam).

- TryLocate(minMinor = 12) / TryLocate(candidates, minMinor): python3 then python, a
  "--version" probe with a 10 s timeout, a missing executable is null.
- Required => SHARPY_REQUIRE_PYTHON_ORACLE == "1"; Locate(candidates, required, minMinor) throws
  PythonOracleMissingException when nothing answers and required is set. It is a PUBLIC
  overload taking the flag explicitly, so no InternalsVisibleTo was added, and no test sets the
  process-global variable inside the shared testhost.
- RunBatch(script, jsonlLines): one process, JSONL on stdin, a 120 s timeout, PYTHONUTF8; a
  non-zero exit or a timeout throws with stderr in the message.
- FindRepoFile(segments) walks up to sharpy.sln, as the existing locators do.

PythonInterpreterTests (namespace Sharpy.Compiler.Tests.Lexer, inside the Stryker filter):
required + bogus candidate list throws; not-required + the same list returns null (the
control); the located interpreter is >= 3.12 and the same executable fails minMinor+1000;
RunBatch round-trips stdin and reports exit code + stderr; FindRepoFile hits sharpy.sln and
misses an absent file.

mutation (worktree, filter NumericLiteralTwin|PythonInterpreterTests, 91 tests):
- Locate's required arm ignored ("located is null && false") → red (1 failed:
  Locate_RequiredAndMissing_Fails), restored → green (91 passed).
- TryLocate's version gate forced true → red (1 failed:
  TryLocate_FindsAPinnedInterpreter_AndRefusesAnOlderMinimum), restored → green (91 passed).

Lexer filter (Sharpy.Compiler.Tests.Lexer.|Properties.Lexer.): 2214 passed / 0 failed / 3 skipped @ 6e96220 (measured, with peers' untracked P68 files present; no failure outside these classes).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…onstant altitude) with its pytest (refs #2300)

build_tools/differential_lex/number_oracle.py is the CPython half of the P68 numeric-literal
twin (#2300 row C1). It answers one question per spelling: is `ast.parse(s, mode="eval")`'s
body ONE int/float Constant (ok, kind, repr value) or is the spelling refused (SyntaxError
message, "not a constant: …", or "imaginary" for a complex so the twin's exclusion of `1j` is
checkable). Batch protocol mirrors differential_parse/compare_ast.py: JSONL
{"id","source"} on stdin (or --batch <file>), one verdict line per request, import-only
without input. SyntaxWarnings are ignored (CPython runs such a program; a filter must never
promote one into a refusal).

build_tools/tests/test_number_oracle.py pins the plan's measured cells on CPython 3.12.13
(.5e-3 → float 0.0005; 01 → refused "leading zeros…"; .5e → refused; 1. / 1.e5 → float;
0x_FF → int 255; 0B101 / 0O755 → int; 1e500 → inf; 1j → refused "imaginary"; _1, 0xFFe+5,
-1, True → refused, not imaginary) plus the batch protocol with out-of-order ids and a
blank line, and --batch.

pytest: 27 passed @ 813df74 (measured, python 3.12.13).
mutation: the `complex` → "imaginary" arm dropped → red (2 failed: the 1j row and the batch
row), restored → green (27 passed).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…or × exponent × following character, suffix/trailing-dot/imaginary rows pinned as Sharpy's own — M1/M2 of the pilot now red (refs #2300)

NumericLiteralTwinTests (Sharpy.Compiler.Tests.Lexer, inside the Stryker filter) is the
generator for #2300 row C1.

Oracle altitude: the PROGRAM, not the token stream. This departs from the "python3 tokenize
twin" wording of #2300's C1 row and charter §3.6c, for this reason: CPython's tokenize splits
`.5e` into NUMBER `.5` + NAME `e` and refuses only at parse time, while Sharpy refuses in
the lexer (SPY0007). Comparing token boundaries would manufacture divergences the language
does not have. Each side therefore answers one question per spelling: is it ONE numeric
constant with value V, or refused? CPython answers through build_tools/differential_lex/
number_oracle.py, where ast.parse(mode="eval") must give an int/float Constant. Sharpy answers
through TokenizeAll(), which must give exactly [Integer|Float, Eof] with no error. Values are
compared, not only verdicts: BigInteger for ints, IEEE bits for floats. A Float Value that is
not a number (M2's "0.5-3") is unparseable and never equal.

Matrix: 7 forms (decimal int incl. 00/01, decimal float, trailing-dot float, leading-dot
float, hex/binary/octal × both prefix cases) × 13 separator placements (none, interior,
consecutive, interior of fraction, interior of exponent, leading, after prefix, trailing,
before/after '.', before/after e, after sign) × 22 exponents (none, e/E × {,+,-} × {1,3,10},
three digits-missing shapes) × following character. A separating follower (EOF, newline,
space, ')') is a Sharpy self-consistency axis: spelling+follower must lex to the bare verdict
plus the follower's token. A gluing follower ('_', '.', g, x, z) makes a new spelling that
CPython judges.

Sharpy-only syntax is excluded but still checked. The suffix roster and j are excluded by
construction, and a test asserts no roster letter follows a digit. The trailing-dot floats
(`12.`, `1.e3`) go through the literal SharpyOnly (predicate, float_literals.md sentence)
table. Sharpy must refuse those cells AND CPython must accept them, else the exclusion is
over-broad. SharpyOnlyRows_LexAsTheSpecSays pins 83 rows: 14 suffixes on 42; f/d/m on 3.14;
integer suffixes on 0xFF/0b101/0o17; f/d/m on binary/octal and m on hex → SPY0023; 0xFFf/0xFFD
as hex digits; lu/x/j × 5 bases → SPY0023; 1. / 12. / 1_000. / 1.e5 → Integer, Dot[, Identifier];
1j / 1.5j → SPY0023.

Allowlist numeric-literal-twin-allowlist.txt (rows `spelling # #NNNN reason`): a row with no
cite fails, a row naming a spelling the twin does not compare fails, a stale (now agreeing) row
fails. Positive control: three hand-broken rows give three violations. Rows: `01` and `0_1`,
the leading-zero decimal cell (DEVIATION-by-default D5 a; already a ledger line). Issue-state
gate: with the header paragraph naming docs/deviations.yaml and no blank line,
scan_allowlist marked both rows EXEMPT. With a blank line and the header reworded, both rows
are checked (exempt=False).

Python absent: a loud skip, or a failure under SHARPY_REQUIRE_PYTHON_ORACLE=1.

Measured @ 813df74 (python 3.12.13):
  NUMERIC-TWIN spellings 49432 agree 1477 refusedBoth 47953 allowlisted 2
  NUMERIC-TWIN python Python 3.12.13 (python3) sharpyOnly 144 followCells 148728 elapsed 1.3s

mutation (Lexer.Literals.cs in a private worktree, filter NumericLiteralTwin|PythonInterpreterTests, 91 tests):
- M1 :736-737 leading-dot fraction consecutive-underscore refusal removed → red (1 failed,
  57 twin failures, first ".1__2": CPython refused, Sharpy Float '0.12'), restored → green (91 passed).
- M2 :753 exponent letter not appended → red (1 failed, 150 twin failures incl. ".5e-3":
  CPython 0.0005, Sharpy Float '0.5-3' unparseable), restored → green (91 passed).
- ReadHexNumber :979-980 trailing-underscore check removed → red (1 failed, 296 twin failures,
  first "0xF_": CPython refused, Sharpy Integer '0xF'), restored → green (91 passed).
- validSuffixes :923 "m" deleted → red (2 failed: SharpyOnlyRows 42m, 3.14m), restored →
  green (91 passed).

Findings-ledger lines added (outside the contract, not pinned): an integer suffix on a decimal
float (`3.14L` runs, typed float64, suffix dropped); a suffix on a leading-dot float (`.5f` →
SPY0103 while `0.5f` runs).

Lexer filter (Sharpy.Compiler.Tests.Lexer.|Properties.Lexer.): 2214 passed / 0 failed / 3 skipped @ 6e96220 (measured, with peers' untracked P68 files present; no failure outside these classes).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…s missing (refs #2300)

The compiler: job in dotnet10.yml had no job-level env block. This adds one with
SHARPY_REQUIRE_PYTHON_ORACLE: "1", at job level so all 7 shards inherit it. With it set,
NumericLiteralTwinTests.SharedGrammar_AgreesWithCPython FAILS when no Python 3.12+ interpreter
is found, so a runner that lost the existing "Setup Python" step can no longer pass the twin
vacuously. Only the new twin reads the variable; the three older differential twins keep
soft-skipping (a chore note on #2300).

YAML checked with yaml.safe_load: jobs.compiler.env == {'SHARPY_REQUIRE_PYTHON_ORACLE': '1'},
matrix unchanged. The failing arm is covered by PythonInterpreterTests.Locate_RequiredAndMissing_Fails
(mutation-tested in the infra commit).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…, escape, dedent and refusal arm × prefix × quoting, values hand-verified against python3 (refs #2300, refs #2337)

FStringArmMatrixTests: 55 arms × prefix {f, t, df} × quoting {single,
triple} through a literal applicability table (a gap carries its reason).
Each cell asserts the whole token (Type, Value) sequence, or the code AND
line of the FIRST diagnostic. Every f value was probed with python3 3.12.13
(value + tokenize FSTRING_MIDDLE) before it was written; t/df outcomes
come from executed spec examples. A df-triple cell wraps the body in the
spec's dedent shape, so every such cell also exercises the swallowed first
and pre-close newlines. The nested-literal arms are pinned set-equal to
Lexer.StringLiteralPrefixes as a positive control.

FSTRING-ARMS arms 55 cells 309 @ 76b7acc + this file (measured).
Lexer filter (Lexer. | Properties.Lexer.) 2214/0/3 @ d15d719 + working
tree (measured). The 3 skips are the #2337 df cells below.

First diagnostic, not the whole list (lead-approved): after a line-break
refusal the lexer resumes on the next line and reads the leftover text as
code. `f"a⏎b"` gives SPY0002 then SPY0026 (`b"`), and a df-triple refusal
gives its code then SPY0001 (`    """`). CPython reports only the first.
The arm asserts the first code on the body's line, so a recovery code
cannot stand in. Both cascade shapes are findings-ledger lines (message ·
program · "lexer recovery cascade after a line-break refusal").

PEP 822 refusal arms (#2337): dstrings.md gives SPY0029 for a shallow
content line and for a non-whitespace close line, and drops the first and
pre-close newlines for a zero-indent close. The d"""…""" controls are
live and green. The df twins assert the spec's verdict and are
Skip("#2337 …"), so they un-skip when #2337 closes.

Not pinned (findings ledger, lead classifies): `{{` inside a spec (lexer:
literal '{', CPython 3.12 refuses) and a line break in a single-quoted
spec (lexer keeps \n; 3.12 drops it, 3.13 refuses). The `\"` spec token
IS pinned because it matches CPython's FSTRING_MIDDLE; its SPY0609
downstream is a ledger line.

Mutations (worktree @ 76b7acc, FullyQualifiedName~FStringArmMatrix):
- `==` guard removed from the '=' self-doc arm (Lexer.FStrings.cs:685)
  → red (6 failed: the `{a==b}` cell × 6), restored → green
- `\"` spec-text arm removed (:1014) → red (3 failed: the single-quoted
  `{x:\">4}` cells; triple stays green by design), restored → green
- no-`}}`-escape-in-spec rule inverted (:985 close skips a `}}`, spec
  text reads `}}` as '}', the #1884 bug) → red (6 failed: `{x:{y:{z}}}`
  × 6), restored → green
- IsPreCloseNewline inverted (:359) → red (42 failed: df-triple cells
  only), restored → green
Restore = cp from backups; cmp identical; restored run 477/0/3
(both classes), green again with the line condition 477/0/3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…rd, rejected, future) + the contextual self in KeywordNames — union/delegate/super/Self/self now observed inside the Lexer namespaces (refs #2300)

KeywordRosterTests: a literal (Keyword, TokenType, SpecTable) table spelled
from keywords.md. It has Hard Keywords 49 (after the `to` row was dropped
in 76b7acc), Reserved Rejected 2 (global, nonlocal) and Future 2 (defer,
do). The contextual `self` is a fourth kind of row: it is in KeywordNames
but lexes as an Identifier. Its consumers are Parser.IsReservedWordToken
(Parser.cs:410) and the LSP rename handler (RenameHandler.cs:259).
Theories: alone (`kw\n`), between identifiers (`a kw b\n`), at EOF
without a newline (`a kw`), each asserting the full token sequence.
KeywordNames set-equals the table's keys. The whole-suite control is
Integration.ReservedWordAsNameMatrixTests.Roster_IsTheLexerKeywordTable,
which sits outside the Stryker filter. P69 (`out`) and P70 (`property`)
edit this table, one commit each, and never widen it.

KEYWORD-ROSTER rows 54 @ 76b7acc + this file (measured).
Lexer filter 2214/0/3 @ d15d719 + working tree (measured).

Mutations (worktree @ 76b7acc, FullyQualifiedName~KeywordRoster):
- `{ "union", TokenType.Union }` deleted from Keywords → red (4 failed:
  union × 3 sites + KeywordNames_IsTheSpecRoster), restored → green
- "self" removed from _keywordNames → red (1 failed:
  KeywordNames_IsTheSpecRoster), restored → green
Restore = cp from backups; cmp identical; restored run of both classes 477/0/3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…umn, position, length) maps back to its source slice; P67 extends it with node ends (refs #2300, #2295)

TokenPositionReconstruction (Sharpy.TestInfrastructure/Lexing) is the token half of the C2
position instrument shared with P67 (#2295 adds node ends). Violations(source, tokens) yields
one line per broken rule:
- R1: LineStarts[Line-1] + Column - 1 == Position. The position must also lie on that line, so
  a column run past the line's end cannot sum to the next line's offset. Indent/Dedent may sit
  at column 1 with spaces-only before the position, which is how Lexer.Indentation.cs stamps
  them. R1 is vacuous at column 1: a dropped column advance is seen on the FOLLOWING token on
  the same line.
- R2: span within the source.
- R3: verbatim kinds have slice == Value. Integer/Float are read through the lexer's
  normalization: underscores stripped, 0X/0B/0O folded, leading '.' as '0.'.
- R4: String/RawString/ByteString are prefix + quote run ... quote run. A backtick name is
  `Value`. FStringConversion spans '!' + Value. FStringSelfDoc is '=' + trivia.
- R5: a Newline sits on a line break and the next line starts right after it.
- R6: Eof at the source end.
- R7: tiling. Gaps hold trivia only. The first FStringFormatSpec's gap may end in the field's
  ':'. A gap holding a recovery resume (or the end after an abort) is exempt as dropped text,
  but only for its content, never its order.
Recorded limit: FStringText/FStringFormatSpec carry no SourceLength, so only their (line,
column) <-> position identity and order are asserted.

TokenPositionReconstructionTests: a clean lex reconstructs under LF/CRLF/CR/no final break.
One hand-broken token per rule yields that rule's line.

mutation: helper R1 made vacuous (`if (expected == token.Position) yield break;` -> always
yield break) -> red (3 failed: R1_AColumnOneShortAfterAConstruct…, R1_ALineOffByOne,
R1_AnIndentWithTextBeforeItsPosition), restored -> green (45 passed with the Phase 2 classes;
worktree @ 76b7acc + these files).
Lexer filter (Lexer. | Properties.Lexer.): 2214 passed / 0 failed / 3 skipped @ 22234d0 +
the P68 position files (measured).

refs #2300, #2295

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…cs/facts are invariant under LF/CRLF/CR and a present/absent/doubled trailing newline, over every fixture (refs #2300)

LexerCorpusTwinTests lexes every .spy under FixtureRoots.All (build copies excluded via
CrashBundleWriter.IsNonSourceSegment) in 9 twins: line ending {as-is, Crlf, Cr} x EOF {as-is,
trailing break stripped, one extra break}. Each twin must not throw, must reconstruct
(TokenPositionReconstruction), and must agree with the as-is file on several things: the
(Type, Value) sequence, the lexer diagnostics' (Code, Line, Column) multiset, LiteralLoss,
RecoveryResumes, the continued/possible-header resumes (as LINE numbers) and
BracketLeftOpenAtLine.

The line-ending twins are compared exactly. The EOF twins use three stated end-only
normalizations:
- a final Newline before only Dedent/Eof is dropped;
- a diagnostic in the trailing breaks or at the source end reads END;
- a resume there is dropped.
A source whose text ends in a backslash compares only its line-ending twins, because its
trailing break is the backslash's operand: a continuation with one, SPY0016 without
(errors/backslash_eof.spy; the EOF roster's rule). Its EOF twins must still not throw and must
reconstruct.
SHARPY_DUMP_TOKENS=<dir> writes each file's as-is token dump plus a SHA-256 per twin, as the
direction record for P68 DD8. No GapDiscovery trait (inside the Stryker filter).

CORPUS-TWIN files 4053 clean 4026 tokens 3564459 twins 9 backslashEnds 1 violations 0
allowlisted 18 elapsed 3.5s @ 92f4cc4 + these files (measured). Floors pinned at files >=
3,000 and clean >= 2,500.

Ratchet: lexer-position-allowlist.txt, rows `corpus <file> :: <TokenType>@l<l>:C<c> # #NNNN
reason` (also `roster …`). A row suppresses violations about that ONE token only. An uncited
row fails, a stale row fails, and disagreements (TOKENS/DIAGNOSTICS/FACTS/THROW) are never
allowlistable. allowlist_issue_state.scan_allowlist: cites [2338], exempt=False. One row at
base, citing #2338 (lexer position class, filed by the lead): strings/d_string_fstring.spy's
df""" FStringEnd is stamped at the dropped pre-close newline (L5:C24, slice "\n  ") instead of
the closing quotes on line 6. It shows in all 9 twins as 18 suppressed lines (R3 + R7).

mutations (worktree @ 76b7acc + the P68 position files, filter = the four position classes,
45 tests; control 45 passed before and after):
- C2 `>>`: `_column += 2` -> `+= 1` (Lexer.cs ReadOperatorOrDelimiter) -> red (7 failed;
  corpus 117 violations, e.g. R1 Integer '2' L7:C16 in expressions/bitwise_operators.spy,
  the token AFTER `>>` — R1 is vacuous at column 1, so the red comes from the following
  token on the same line). The #2338 rows were in place (allowlisted 18/6), so the row is
  not the subject. Restored -> green (45 passed).
- C2 ReadLiteralName `sourceLength:` argument removed -> red (7 failed; corpus 7290
  violations, R4 slice '`na m' is not '`na me`' + R7 gap 'e` '); restored -> green.
- C3 ConsumeLineBreak `\r\n` two-step reduced to one char -> red (3 failed; corpus 2407
  violations, e.g. R1 Newline L90:C76 in Formatting/blank_line_placements.spy [crlf/as-is]);
  restored -> green.
Lexer filter: 2214 passed / 0 failed / 3 skipped @ 22234d0 + the P68 position files
(measured).

refs #2300, refs #2338

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…iteral prefix, number form and open construct lexes the same at end of file, no exception escapes (refs #2300)

EofConstructRosterTests is a literal roster of 142 constructs: every operator/delimiter
spelling ReadOperatorOrDelimiter reads, the lone backslash, string prefix {"", f, t, r, b, d,
dr, df} x quote {' " ''' """} closed and unterminated, a closed multi-line d""", the 11 number
forms, a comment, a backtick name closed/unterminated, a keyword, an identifier, an open
block, and the f-string open field/spec/conversion. Escape-at-EOF and short hex escape at EOF
are included too. Each construct is placed after `x = ` (statements stand alone), at the end
and, where a token can follow on its line, also followed by ` b`, because R1 is vacuous at
column 1. Seven endings: none, LF, CRLF, CR, LF LF, trailing spaces, trailing comment.

Each cell must:
- let no exception escape TokenizeAll;
- reconstruct;
- match the LF cell's (Type, Value) sequence (final Newline before Dedent/Eof dropped) and its
  diagnostic-code multiset.
Constructs whose text ends in a backslash (the lone `\`, `"abc\`, `b"\x4`) read the ending's
first char as their operand, so only their LF/CRLF/CR cells are held equal (the C3
invariance). Their other endings must still not throw and must reconstruct.

Totality: every TokenType in Plus..Backslash is the first token of some construct lexed alone.
The enum sweep is the positive control for the literal list, and the list stays the anchor.

EOF-ROSTER constructs 142 endings 7 cells 1708 violations 0 allowlisted 6 @ 92f4cc4 + these
files (measured).

Allowlist rows added (cite #2338, the lexer position class the lead filed): a line break read
as an escape's operand is consumed without advancing the line. `x = "abc\` + LF and
`x = b"\x4` + LF end with Eof on line 1 at the offset where line 2 starts; with LF LF, on line
2 where line 3 starts. Four rows, keyed by construct + Eof@L:C. scan_allowlist: cites [2338],
exempt=False.

mutations (worktree @ 76b7acc + the P68 position files; control 45 passed before and after):
- C6: ReadDedentedRawString's single-line `while (_position < _source.Length)` -> `<=`
  (Lexer.Literals.cs; the arm the pilot's `dr"abc` at EOF mutant hit) -> red (2 failed;
  12 THROW IndexOutOfRangeException lines, e.g. `x = dr"abc` with ending none/trailing
  spaces/trailing comment; Roster_HasAConstructFor… throws too); restored -> green (45 passed).
- C2 `>>` column advance dropped -> roster 21 violations, e.g. R1 on the token after `>>`,
  with the #2338 rows in place (allowlisted 6) -> red; restored -> green.
- C2 ReadLiteralName sourceLength removed -> roster 28 violations (R4 '`a ' is not '`a b`')
  -> red.
- C3 ConsumeLineBreak CRLF one-step -> roster 17 violations -> red.
Lexer filter: 2214 passed / 0 failed / 3 skipped @ 22234d0 + the P68 position files
(measured).

refs #2300, refs #2338

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
)

This adds four rows covering the six new harnesses: the numeric twin (C1); the corpus twin with
reconstruction plus the EOF roster (C2/C3/C6, which share one ratchet); the literal-loss injection and shape
differential (C5); and the f-string arm matrix plus the keyword roster (C4/C7). Each row names its contract,
its harness, its ratchet file and its umbrella (#2300), along with the class issues its cells were filed to:
#2337, #2338, #2339 and #2044.

No test reads this document. A grep of src for gap-discovery-contracts.md finds only message strings, so
there is no guard to mutation-test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…r can score the method (CS0165 safe-mode hole); token dump identical over 4,053 fixtures (refs #2300)

P68 Design Decision 8: `Token token;` was declared before the try. It was assigned only in
the try and read after it, so Stryker's statement-removal mutant of the assignment is CS0165
and safe mode then drops every mutation in TokenizeAll (unscored in the pilot). Now
`var token = NextToken(); tokens.Add(token); if (token.Type == TokenType.Eof) break;` all sit
inside the try. Both catch exits already leave the iteration: `break` at the budget/EOF stop
and `continue` after RecoverFromError. Nothing after the try reads `token`, and
tokens.Add cannot throw LexerAbortException. No behaviour change. Whether Stryker now scores
the method is a hypothesis until the Phase 5 re-run.

Direction record:
- LexerCorpusTwinTests with SHARPY_DUMP_TOKENS before the edit (HEAD 7792537) and after it
  (working tree on 0b35287; no Lexer/ change between them except this one).
- Each dump holds 4,053 files: each file's as-is token dump (Type|Value|L:C@Position+Length,
  every diagnostic, the lexer facts) plus a SHA-256 for each of its 9 LF/CRLF/CR x EOF twins.
  The dumps include 27 diagnostic lines and 26 files with recovery resumes, so the catch path
  is covered.
- `diff -r dump-before dump-after`: empty (exit 0, 0 bytes) (measured).
- CORPUS-TWIN files 4053 clean 4026 tokens 3564459 twins 9, violations 0, both sides.
Lexer filter (Lexer. | Properties.Lexer.): 2214 passed / 0 failed / 3 skipped @ 0b35287 +
this edit (measured).

refs #2300

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
… record the P68 ignore-list policy (refs #2300)

The probe config (lexer-probe.json) mutates Token.cs, LiteralSpans.cs and
Lexer.Indentation.cs, with ignore-methods set to ReportError, LogInfo,
AddWarning and AddError. Neither of the first two files calls ReportError, so
the probe was widened to Indentation, as the plan allowed. It ran through
run.sh in a detached worktree at e621d9e.

Measured:
- ignore-methods skips argument mutations. All 4 ReportError message-string
  mutants in Lexer.Indentation.cs (lines 47, 51, 55, 72) are Ignored with
  "Removed by method filter".
- Score 82.16 % @ e621d9e over the three files: 144 killed, 31 timeout,
  38 survived, 0 without coverage.
- Wall clock 5,835 s. Setup took 24.5 min, 17 of it coverage capture over
  2,217 tests. Testing ran at about 20.5 s per mutant at concurrency 4,
  against 8.8 s for the pilot. Maximum single-process RSS was 5.55 GB.

The audit doc gains an "Ignore list (P68)" section. It records the semantics,
the cost, and the derived estimate of about 16.7 h for a full re-run at the
probe's rate. That estimate sizes the owner slot for Phase 5 Task 3. It also
records the decision: lexer.json carries ignore-methods, and the data file
stays authoritative for what the config cannot express.

The audit doc also records the per-class mutation-control table from P5T2:
each of C1-C7 goes red on its recorded mutant, and the restored tree is
green at 2214/0/3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…ed @ e621d9e (refs #2300)

The probe's log (Stryker compiles every mutant of Sharpy.Compiler before the
mutate filter: 73,308 created) has 85 "Safe Mode!" lines, and none is in
Lexer/. At 258a7b6, safe mode removed every mutation in TokenizeAll. This
measures Design Decision 8's caveat, which was a hypothesis until a Stryker
compile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHwfTQy7v2FAujH3NSLQg
…n allowlisted numeric spelling or position token fails (refs #2300, #2338)

P68 verify round (agent3 refuted, M1 and M9). Both ratchets keyed a row on the
subject alone, so a row admitted ANY disagreement on that subject:
- numeric twin: `01 # #2300` passed with `01` lexing as Integer '017';
- position ratchet: `FStringEnd@L5:C24` absorbed a wrong Value on that token,
  and by reading any further rule (R1..R7) on it.

Rows now pin what they admit:
- numeric-literal-twin-allowlist.txt: `<spelling> => <int:n | float:bits |
  refused> # #NNNN reason`. A disagreement whose Sharpy verdict differs from the
  pin fails (PIN). A row with no pin is malformed. The positive control gains
  a no-pin row (4 broken rows → 4 violations).
- lexer-position-allowlist.txt: `<scope> :: <Type> '<Value>'@l<l>:C<c>
  <R-rules> # #NNNN`. A violation is absorbed only when its token's Value and
  its rule both match. Each named rule that suppresses nothing is stale. The
  unit test's positive control adds the same token with another Value, the
  same token under an unnamed rule, a row with no rule, and a row naming a
  rule that never fires.
- The numeric matrix gains the digit string `1000` (values of four or more
  digits; the oracle's int formatting was otherwise unobserved, agent3 P1).

Mutations (private worktree @ 0d58024 plus this change; copies restored
with shutil.copy + touch):
- `01 => int:1` pin edited to int:2 → red (SharedGrammar 1 failed), restored → green (1 passed)
- Lexer.FStrings.cs:834 FStringEnd Value '""' only when mispositioned (the #2338
  token, agent3's M9) → red (corpus/roster 1 failed of 11), restored → green (11 passed)
- position row Value pin '"""' → '}' → red (1 failed of 11), restored → green
- position row rules R3,R7 → R3 → red (1 failed of 11), restored → green

Lexer filter 2214/0/3 (measured, working tree on 0d58024).
NUMERIC-TWIN spellings 50422 agree 1521 refusedBoth 48899 allowlisted 2 (python 3.12.13, 1.4 s).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… 60 s budget, plus an @walrus twin for the depth-0 := arm (refs #2300)

P68 verify round:
- Budget (agent2): the whole-solution gate measured
  Injected_ScannerAgreesWithTheCleanLexersFacts at 96 s @ 2c15c47, stride 1,
  against Design Decision 10's 60 s per class. Fixed stride 4 now, over BULK
  cells only: the dollar and tab constructs on the as-is document and its
  @backtick twin. Each construct keeps its own counter so the sample is even.
  Everything else is lexed always: escape/conversion, and the @escaped-quote,
  @unbalanced-brace, @field-split and @walrus twins, the cells that alone kill
  the scanner-arm mutants.
- Aperture (agent3 M6): with the `:=` exclusion removed from
  DroppedLineContinues' depth-0 colon arm (Lexer.LiteralLoss.cs:178), the
  Lexer namespaces stayed green. Every fixture's walrus is inside brackets or
  on a header line. The @walrus twin appends `; w__ := 0` after the last code
  token of each comment-free line that ends its logical line, in documents
  that spell a walrus (54). An escape/conversion abort there has read tokens,
  and its given-up text ends in a depth-0 `:=` that is not a header colon.
  A first spelling, `w__ := 0; ` at the line START, did NOT kill M6: the `$`
  construct aborts before any token is read, so the possible-header fact is
  false either way. Measured and replaced.

SCANNER-INJECT documents 10606 sites 86333 constructs 4 cells 146471 disagreements 0 allowlisted 0 stride 4 (bulk only; sampled out 138481)
(twins as-is 4026, backtick 4026, escaped-quote 2022, field-split 222, unbalanced-brace 256, walrus 54; dollar 61611, escape 20818, conversion 2516, tab 61526; wall 6.3 s quiet)

Mutations at stride 4 (private worktree @ 0d58024 plus this change;
restored with shutil.copy + touch). Filter: LiteralLossScannerDifferentialTests.
- :178 `:=` exclusion removed → red (1 failed, 52 @walrus cells), restored → green (46 passed)
- :148 DroppedLineContinues escape arm off → red (1 failed), restored → green (46)
- :163 DroppedLineContinues SkipBacktickName off → red (1 failed), restored → green (46)
- :338 WalkFrom SkipBacktickName off → red (2 failed), restored → green (46)
- :370 WalkFrom text escape arm off → red (1 failed), restored → green (46)
- :391 WalkFrom `{{` skip off → red (1 failed), restored → green (46)
- ClosesOnItsLine closing-quote arm returns false → red (1 failed), restored → green (46)
All six scanner mutants recorded at stride 1 in e316dda/e7bb3cfa6 stay red at stride 4.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ops reading stdin — never a bare broken-pipe IOException (refs #2300)

P68 verify round (agent3 W3). A script that exits before reading the whole
batch (a crash, an import error) broke the stdin pipe. WriteLine then threw
IOException with no stderr, and under load the
Assert.Throws<InvalidOperationException> in PythonInterpreterTests could flake.
The write loop now stops on IOException, and the exit-code check reports the
script's exit code and stderr. An exit 0 before the batch was read is its own
InvalidOperationException.

Guard: fail.py (exit 3, no stdin read) is sent 2000 lines × 1000 chars,
larger than any pipe buffer.
- try/catch reverted → red (PythonInterpreterTests 1 failed of 5: IOException), restored → green (5 passed)
  (private worktree @ 0d58024 plus this change; restored with shutil.copy + touch)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ist never touches unscored mutants; the oracle prints plain integer digits (refs #2300)

P68 verify round (agent3 T1, T2, P1): three mutants survived the pytest.
- T1: `ig.regex.fullmatch(inv.name)` → `search`. A LogInfo entry then ignored
  a string inside LogInfoVerbose(...). New test over a one-file report.
- T2: the `status in SCORED` gate removed. The list then reclassified a
  CompileError mutant inside ReportError. New test: it stays compileError, and
  the scored twin is listed.
- P1: number_oracle printing ints as `f"{value:_}"`. New rows: 1_000,
  0xFFFF, and a value past Int64.

Mutations (shared tree, file copies restored with cp + touch):
- T1 → red (1 failed of 53), restored → green
- T2 → red (1 failed of 53), restored → green
- P1 → red (3 failed of 30), restored → green
Both files: 83 passed (measured).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…json has no ignore-methods key yet (P5T3 adds it) (refs #2300)

P68 verify round (agent1). The "Ignore list (P68)" Decision line said
lexer.json carries ignore-methods; the file has none @ 0d58024. The plan
(P5T3) adds it before the re-run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…bound the in-gate wall clock — measured 96 s → 102 s, load-dominated (refs #2300)

Correction to bd23358, whose title said "inside the 60 s budget". The
whole-solution gate @ be4664e measured Injected_ScannerAgreesWithTheCleanLexersFacts
at 1 m 42 s with stride 4, against 1 m 36 s @ 2c15c47 with stride 1. In the
same pair of runs, the unchanged Shapes_ test went 28 s → 1 m 23 s, so in-gate
per-test durations track machine load, not cell count. Quiet, in the Lexer
filter alone, the test takes 6.3 s at stride 4. Comment-only change; the
stride, the cells and the mutation record of bd23358 stand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@antonsynd
antonsynd merged commit e7e0a0e into mainline Oct 22, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant