Skip to content

tests: lexer observation totality — the Stryker pilot's seven survivor classes C1–C7 (numeric literals, token positions, line endings, f-string arms, LiteralLoss scanner, end of file, keyword roster) #2300

Open
Open
tests: lexer observation totality — the Stryker pilot's seven survivor classes C1–C7 (numeric literals, token positions, line endings, f-string arms, LiteralLoss scanner, end of file, keyword roster)#2300
Labels
class-trackerThe one issue per class: carries the contract and the cell tablelane:toolingDocs, formatter, LSP, build tools, tests, CIreach:probeNeeds an escaped spelling, a wrong-annotation probe or a python3 message difftesting

Description

@antonsynd
antonsynd
opened on Oct 12, 2026

Class

Contract: every lexer arm — each numeric-literal form, each token's position, each line-ending, each f/t-string and dedented-literal arm, the Lexer.LiteralLoss.cs scanner's arms, end of file with and without a trailing newline, and every keyword in the spec's roster — is observed by a test that fails when the arm is removed. The Stryker pilot (#2289, R-GB) measured Lexer/ at 73.27 % @ 258a7b6 (measured): 2,146 of 2,929 mutants detected, 783 survived; a seeded sample of 30 read as 23 test gaps / 3 plausible / 3 equivalent / 1 out of reach. Two leading-dot-float mutants (M1 drops the .1__2 refusal, M2 drops the exponent letter so .5e-3 dies with SPY0909) pass the WHOLE solution 79207/0/4 — no test anywhere lexes a leading-dot float. Report: docs/audits/stryker-lexer-pilot-2026-10-11.md (7f0502d).

This is a tests-only class: the compiler's behaviour on every cell below is believed correct (the C3 probes print 3 7 on crlf/cr @ 258a7b6); the defect is the suite's aperture. Lane tooling, reach probe.

Ruling R-GR (owner, 2026-10-12)

(a) ONE class tracker with seven rows, drained by one tests-only group (P68, directly after P63) — C2 shares P67's position instrument, whichever lands first builds it. Rejected: (b) seven issues, one per class (seven one-row chores; the open count rises for tests-only work); (c) a cell table on #2289 and a chore drain with no new issue (#2289 is the pilot's tracker and closes when Semantic/Validation/ is done — the work would lose its home).

Cells (survivor classes of the pilot, counts by mutated method @ 258a7b6)

class survivors contract a guard states representative cells harness lane reach found by
C1 numeric literals 152 (ReadNumber, hex/octal/binary, prefixed-integer suffix) every numeric-literal form × {underscore placement, exponent sign, suffix, end of file} lexes to CPython's verdict and value (Sharpy suffixes on their own rows) leading-dot floats .5e3 .5e-3 .1__2 .12_ .5e .5e1_; 0xFFu + any following character; trailing-dot 1. generator FIRST: a python3 tokenize twin over form × prefix × suffix × separator with an allowlist ratchet; Sharpy-only suffix rows are the hand axis tooling probe #2289 pilot
C2 token positions 146 column + span/SourceLength mutants every token's (line, column, position, length) maps back to its source slice a >> b; a token after b"…", dr"…", """…""" the position-reconstruction instrument P67 (#2295) builds over the corpus — tokens AND node ends; shared, built by whichever lands first tooling probe #2289 pilot
C3 line-ending axis NextToken, ConsumeLineBreak, IsPreCloseNewline, hole trivia {LF, CRLF, CR} at every line-break site gives the same tokens backslash continuation over CRLF; newline inside brackets on CR; CRLF in a triple-quoted literal a line-ending twin axis on the existing lexer theories (every theory runs ×3) tooling probe #2289 pilot
C4 f/t-string and dedented-literal arms 266 every f-string arm (spec escape, {{/}}, == in a field, nested braces, a comment in hole trivia, dedented triple prescan) is exercised f"{a==b}" (the mutant reads a self-documenting =); f"{x:\">4}"; a df"""…""" hole holding a triple-quoted string or a tab hand cells on the f-string theories (closed axis) tooling probe #2289 pilot
C5 Lexer.LiteralLoss.cs scanner 60 the hand-model scanner agrees with the lexer on every escape and brace arm a backslash escape and nested { } in a field on a dropped line; spec \" the differential the scanner never got: scanner verdict vs lexer verdict over the P22 corpus tooling probe #2289 pilot
C6 end of file 89 every literal and operator form lexes the same at end of file with and without a trailing newline dr"abc at EOF (mutant: IndexOutOfRange instead of the unterminated-string diagnostic) an EOF twin axis beside C3's (with / without trailing newline) tooling probe #2289 pilot
C7 keyword roster 6 every keyword in the spec's list lexes to its token type, anchored to literals, never to Keywords.Keys union, delegate, super, Self, self a roster test anchored to the spec's literal list (a totality count from the same enum is vacuous) tooling probe #2289 pilot

Acceptance

  • Each row's harness lands with its mutation record (one arm removed → red; restored → green) in the commit body; the C1 and C5 generators ratchet on an allowlist that cites this issue and drains to 0.
  • Re-run the pilot on Lexer/ (build_tools/stryker/run.sh, --since form if measured) and record score @ sha (measured) against the 73.27 % baseline; the message-text / logger-string mutants get the ignore list R-GB asked for, with a reason per entry.
  • No production change: a cell that turns out to be a lexer defect is filed by class (contract §10), never fixed under this issue.

Labels: class-tracker, lane:tooling, reach:probe, testing.

Charter: .claude/plans/remediation-round-2026-09-20-batching.md §3.6c — P68 lexer observation totality (directly after P63). Siblings: #2289 (the pilot, stays open for Semantic/Validation/), #2295 (P67 builds the shared position instrument).

Activity

  1. antonsynd
    added
    testing
    lane:toolingDocs, formatter, LSP, build tools, tests, CI
    reach:probeNeeds an escaped spelling, a wrong-annotation probe or a python3 message diff
    class-trackerThe one issue per class: carries the contract and the cell table
    on Oct 12, 2026

  2. antonsynd commented on Oct 21, 2026

    @antonsynd
    antonsynd
    on Oct 21, 2026
    OwnerAuthor

    P68 record post: lexer observation totality (220259b..0d58024, 20 commits)

    State: this issue stays open. Two acceptance items are not done:

    Acceptance bullets → evidence

    # Bullet Evidence
    1 Each row's harness landed with its mutation record in the commit body C1 2eeb1b2e2 (+ 92f4cc430, 40d771acd, d15d7194a) · C2/C3 a3838d719, 93c6c92ca · C6 7792537c0 · C5 5baaeb0b7, e316dda15, ffdd7f7d3, e7bb3cfa6 · C4 78ddc4bc1 · C7 22234d00e (+ spec 76b7acca7). The roster rows are in 0b3528716.
    1b One recorded mutant per class, re-applied before the re-run Private worktree @ e621d9e, Lexer filter (2,217 tests): C1 red 1 · C2 red 7 · C3 red 9 · C4 red 46 · C5 red 2 · C6 red 2 · C7 red 1 · all restored green 2214/0/3. The table is in docs/audits/stryker-lexer-pilot-2026-10-11.md. The lead also re-ran the hex :979 (red 1/91), IsPreCloseNewline (red 42/317) and {{ (red 2/48) mutants per wave.
    2 The re-run's score @ sha against 73.27 %, ignore list applied, survivors by class Pending: the owner's weekend slot, concurrency 4 (see "Re-run cost" below). The probe below stands in for the instrument check only, not for the score.
    3 No production change beyond Design Decision 8 git diff --stat 220259b51..HEAD -- src/Sharpy.Compiler = Lexer/Lexer.cs only, 8+/7− (e621d9e94). The token dump over 4,053 fixtures is identical.
    — Whole-solution gate 81093 passed / 0 failed / 7 skipped @ 2c15c47 (measured), against a baseline of 80453/0/4 @ 220259b. All of the change is in Sharpy.Compiler.Tests: +643 tests, of which +640 pass and +3 are skipped (the #2337 df cells). The other four projects are identical. 0d58024dd is docs-only.
    — build_tools pytest · check_spec_blocks.sh · allowlist issue-state · format pytest 1232 passed / 1 skipped · spec blocks 1206, 0 failing · issue-state exit 0 · dotnet format whitespace --verify-no-changes exit 0
    — Refuting verifier @ 0d58024 Claims 1–3, 5 and 6 not refuted. It ran three mutants spelled differently from the lead's: C7 (delegate removed) red 4, C3 (lone CR swallows the next char) red 6, C6 (ReadByteString bound <=) red 15; restored green 2214/0/3. Bogus, uncited and stale allowlist rows fail in all three ratchets. Claim 4 was partly refuted (see the chore note).
    — regen no diff: no Semantic/Lowering/CodeGen change, so no generated artifact is reached

    Stryker instrument (Phase 0, measured @ e621d9e)

    • ignore-methods skips argument mutations. All 4 ReportError("…") message-string mutants in Lexer.Indentation.cs are Ignored with "Removed by method filter". lexer.json will carry ignore-methods. lexer-ignore.json plus triage.py stay authoritative for what the config cannot express.
    • Probe score: 82.16 % on Token.cs + LiteralSpans.cs + Lexer.Indentation.cs (213 tested: 144 killed, 31 timeout, 38 survived).
    • The TokenizeAll CS0165 safe-mode hole is closed. There are 85 safe-mode lines in a compile of all 73,308 mutants, and none is in Lexer/.
    • Re-run cost (derived, not measured). The probe took about 20.5 s of wall clock per mutant at concurrency 4. The pilot took 8.8 s.
      • The P68 harnesses cover nearly every Lexer line, so under perTest each mutant now runs the corpus twin, the injection differential and the CPython twin.
      • At the probe's rate, the pilot's 2,929 mutants would take about 16.7 h, not 7 h. The weekend slot is sized from this. Concurrency stays at 4: higher concurrency risks swapping on this 36 GB machine, and contention timeouts would inflate the score against the 73.27 % baseline.

    Cells (allowlist rows and ledger)

    Cell Program Observed Expected Lane · reach Disposition
    01 leading-zero decimal (also 0_1) print(01) runs, prints 1 CPython: SyntaxError "leading zeros in decimal integer literals are not permitted"; the spec is silent missing-refusal · program DEVIATION-by-default (python-fidelity-scope D5 a). Two rows in numeric-literal-twin-allowlist.txt cite #2300. Rule at /batch-issues: a deviations.yaml row (the spelling moves to the twin's exclusion table) or promotion to a lexer-defect class issue.
    keywords.md to row keywords.md:46 listed as a hard keyword removed in 0.8.0 (type_casting.md); neither the lexer nor the parser knows it message · probe Fixed in 76b7acca7 (the row and four sentences); the C7 roster anchors to the corrected tables.
    Recovery cascade after a single-quoted line break f"a⏎b" (also t", df", CRLF, CR, f"a {x}⏎b") SPY0002 @1, then SPY0026: recovery reads the leftover b" as an unterminated byte string one diagnostic (CPython: "unterminated f-string literal") message · program Comment here; message lane.
    Recovery cascade after a df-triple refusal df"""⏎ a}b⏎ """ (also {x!q}, {x!r x}, {x)}, {x]}) first code (SPY0021/0030/0020) @2, then SPY0001 on the leftover """ one diagnostic message · program Comment here; message lane.
    C2 instrument limit f-string interiors FStringText/FStringFormatSpec/FStringConversion carry no SourceLength (Value is decoded text; !r spans 2 but GetSpan() covers 1). A field's spec-introducing : lies in no token's span. each token's span equals its source slice tooling · probe TokenPositionReconstruction reads those spans as next.Position − Position. The limit is recorded; a span carrier for interior tokens is a candidate for P67 (#2295).

    Filed from this round's ledger (one per class):

    The scanner allowlist (literal-loss-scanner-allowlist.txt) is empty: 283,851 injection cells and 149,331 shape cells, 0 disagreements. The }} arm (Lexer.LiteralLoss.cs:401) is an accepted equivalent mutant.

    Oracle altitude (Design Decision 1)

    The C1 twin asks ast.parse(s, mode="eval") for a numeric Constant (value or refused). It does not compare token streams with tokenize, because CPython refuses .5e at parse time while Sharpy refuses in the lexer. Comparing token boundaries would invent divergences the language does not have.

    • Matrix: 49,432 spellings, with 2 allowlisted (01, 0_1).
    • Hand rows: the Sharpy-only syntax (the suffix roster, j, trailing dot) is pinned as hand rows citing the spec, not left to the twin.

    Hand-offs and chores

    • P67 (parser: a node whose last token spans several lines ends on that token's START line — LineEnd/ColumnEnd ignore the lexer's span (multi-line strings) #2295): Sharpy.TestInfrastructure/Lexing/TokenPositionReconstruction is the token half of the position authority. P67 extends it with node ends. The f-string interior span limit above is the open input.

    • Chore: the three existing python twins still soft-skip when python 3.12 is absent:

      • CPythonDifferentialParseTests.PythonOracle.TryLocate
      • FormatSpecDifferentialTests.PyOracle
      • DifferentialExecutionTests

      Only the new numeric twin honours SHARPY_REQUIRE_PYTHON_ORACLE=1, which CI sets for every Compiler shard (d15d7194a). The retrofit is moving each of them onto Sharpy.TestInfrastructure.Oracles.PythonInterpreter.TryLocate(minMinor: 12) and the required mode. It is seam-free; it goes in the chores band.

    • The same chore covers the "skip" itself (found by the refuting verifier @ 0d58024). With no Python 3.12 and the env var unset, every python twin is counted Passed, not Skipped:

      • the new numeric twin (NumericLiteralTwinTests.cs:464), PythonInterpreterTests, and the three existing twins all write SKIP: to ITestOutputHelper and return. That output is visible only at detailed verbosity.
      • xUnit 2.9.3 has no dynamic skip, and the repo references no Skippable package. A dev-machine gate count therefore hides an oracle that did not run.
      • CI is unaffected: SHARPY_REQUIRE_PYTHON_ORACLE=1 is set at the compiler: job level and makes the numeric twin fail (verified with PATH reduced to python 3.9: 1 failed, PythonOracleMissingException).
      • The cure: a real dynamic skip (Xunit.SkippableFact, or Assert.Skip after an xUnit v3 move), applied to all five sites in one place.

    Close criterion: no allowlist row cites #2300, and the re-run's score is recorded.

  3. antonsynd commented on Oct 22, 2026

    @antonsynd
    antonsynd
    on Oct 22, 2026
    OwnerAuthor

    P68 /verify-implementation record (220259b..96d7a27)

    Gate: 81093 passed / 0 failed / 7 skipped @ be4664e (measured). The same count as @ 2c15c47. 96d7a27 changes one code comment. Format clean. build_tools pytest 1232 passed. check_allowlist_issue_state.sh clean. Spy, spy-test and spec-block staleness gates are clean @ 0d58024.

    Four refuting auditors ran: completeness, regression, guard mutation and sibling cells. No regression. The production edit (TokenizeAll) is behaviour-neutral by reading, and by eight fixtures run.

    Fixed in this round (each mutation-tested, recorded in the commit body)

    Commit Finding Guard
    5bedd44 Allowlist rows admitted ANY disagreement on their subject: 01 lexing as '017' passed, and a wrong Value on the #2338 FStringEnd was absorbed Rows pin the verdict (01 => int:1) or the token's Value and rules (FStringEnd '"""'@L5:C24 R3,R7). FStringEnd Value mutant red, pin edits red
    bd23358 DroppedLineContinues' depth-0 := exclusion (LiteralLoss.cs:178) went unobserved in the Lexer namespaces New @walrus clean twin, mutant red. A first spelling (walrus at line start) did not kill it and was measured and replaced
    bd23358 + 96d7a27 Injection test over the 60 s budget in the gate (96 s) Fixed stride 4 over BULK cells only: 146,471 of 284,952 cells. All seven scanner mutants are still red at stride 4. The in-gate wall clock did not drop (102 s). Gate durations are load-dominated: the untouched Shapes test went 28 s → 83 s. Quiet: 6.3 s.
    2f4ee45 RunBatch broken pipe produced a bare IOException with no stderr Large-batch early-exit test, reverted → red
    912f22f triage within fullmatch→search, unscored-status gating, and the oracle's int formatting survived pytest Three new pytest guards, each red under its mutant
    be4664e The audit doc's Decision line said lexer.json carries ignore-methods; it has none Wording: P5T3 adds it

    Cells for this tracker (lane · reach)

    Filed elsewhere

    Still open on this tracker

    P5T3, the Stryker re-run, is owner-slotted for the weekend. The 01/0_1 rows still cite #2300 (Design Decision 4). #2300 must not close while they do: the issue-state gate would turn red.

  4. antonsynd
    added 9 commits that reference this issue on Oct 22, 2026

    docs(spec): drop the stale `to` keyword row and the four "transitiona…

    76b7acc

    test(stryker): survivor triage script with the R-GB ignore list as da…

    813df74

    test(lexer): the clean lexer's line facts (opener crossing, continuat…

    5baaeb0

    test(lexer): injection differential — the literal-loss scanner's verd…

    e316dda

    test(lexer): the P22 shape builders over the corpus judge the lexer's…

    ffdd7f7

    test(lexer): an @unbalanced-brace clean twin — a lone {{ then a # in …

    e7bb3cf

    docs(lexer): the scanner allowlist's documented key format names the …

    6e96220

    test(infra): one python-oracle locator for the differential twins; a …

    92f4cc4

    test(build-tools): CPython numeric-literal oracle (ast.parse at the c…

    40d771a

  5. 7 remaining items

  6. antonsynd
    added 11 commits that reference this issue on Oct 22, 2026

    docs(gap-discovery): roster rows for the P68 lexer harnesses (refs #2300

    0b35287

    refactor(lexer): TokenizeAll reads the token inside its try so Stryke…

    e621d9e

    docs(stryker): measure what ignore-methods skips on a two-file probe;…

    2c15c47

    docs(stryker): the TokenizeAll CS0165 safe-mode hole is measured clos…

    0d58024

    test(lexer): allowlist rows pin today's defect — a second change on a…

    5bedd44

    test(lexer): injection differential at a fixed bulk stride inside the…

    bd23358

    test(infra): RunBatch reports the script's exit and stderr when it st…

    2f4ee45

    test(build-tools): triage's `within` is a full callee match and the l…

    912f22f

    docs(stryker): the ignore-methods decision is for the re-run — lexer.…

    be4664e

    docs(lexer): the injection stride halves the test's CPU; it does not …

    96d7a27

    Merge pull request #2341 from antonsynd/dev

    e7e0a0e

  7. antonsynd commented on Oct 22, 2026

    @antonsynd
    antonsynd
    on Oct 22, 2026
    OwnerAuthor

    Batching 2026-10-21 @ e7e0a0e — charter .claude/plans/remediation-round-2026-09-20-batching.md §3.6c (P68 IMPLEMENTED + VERIFIED, gate 81093/0/7 @ be4664e measured; moved to §2) and §3.0 (two ledger rows ride the chores band). #2300 stays OPEN for the P5T3 Stryker re-run (owner: WEEKEND slot, concurrency 4, ~16.7 h derived; build_tools/stryker/lexer.json gains ignore-methods first) and the two 01/0_1 rows in numeric-literal-twin-allowlist.txt (lines 16–17) until R-HT below moves them.

    Ruling R-HT (owner, 2026-10-21; D1 a): RECORD 01 / 0_1 — a docs/deviations.yaml row (leading-zero-decimal-integer, syntax, python audience, severity none, example executed at HEAD) plus one integer_literals.md sentence "leading zeros are digits (C#'s rule)"; the two allowlist rows move into NumericLiteralTwinTests.SharpyOnlyRows_LexAsTheSpecSays citing the row; no compiler change. Measured @ e7e0a0e: print(01, 0_1, 007) runs printing 1 1 7; CPython 3.12.13 SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers (00 and 0 accepted by both); C# accepts 007; the spec has no grammar block and is silent (fidelity-scope D5: DEVIATION until promoted). Rejected: (b) promote — refuse like CPython with a 0o steer as a cell on the #2339 chore (the analysis's recommendation). Door: (a) accepts a spelling users may come to depend on; a → b later breaks programs — the one-way choice, taken knowingly; do not re-ask.

    Ruling R-HW (owner, 2026-10-21; D4 a): RECORD supplementary-plane identifier letters — a docs/deviations.yaml row (identifier-letters-bmp-only, syntax, python audience) plus one identifiers.md sentence beside line 19: identifier letters are letters of the Basic Multilingual Plane, C#'s identifier alphabet (Axiom 1); the message naming half a surrogate pair is written into the row's sharpy_behavior as a message-lane cell (printing the full code point is a one-line courtesy the #2339 chore may take, not a requirement). Measured @ e7e0a0e: 𝒳 = 1 (U+1D4B3) → SPY0015 "Unexpected character: '�'" at 2:5 naming the lone high surrogate \uD835; control été = 1 runs printing 1; CPython accepts 𝒳. Roslyn's SyntaxFacts.IsIdentifierStartCharacter(char) / IsIdentifierPartCharacter(char) take a UTF-16 unit (grep-only of the 5.9.0 API doc; a surrogate's category is Cs), so C# has no spelling for the identifier, not even a \u escape. Rejected: (b) promote — accept and mangle to a C#-legal name (a new mangling is an interop surface C# consumers can depend on; one-way; a spelling nobody writes). (a) is relaxable to (b) later.

    The other cells of the record and verify posts, re-homed (no ruling): the backtick-content cell (empty → Identifier `''` → `Debug.Assert` at `Parser/Ast/Expression.cs:506`, exit -6; a b / 1x `` → SPY0599) is posted on #2287 (P62, whose construct axis lists backtick identifiers); the two recovery-cascade message cells (f"a⏎b" → SPY0002 then SPY0026; the df-triple refusal then SPY0001) are posted on #2274 (the permanent lexer known-limit tracker, R-FQ); the C2 instrument limit (f-string interior tokens `FStringText`/`FStringFormatSpec`/`FStringConversion` carry no `SourceLength`, so `TokenPositionReconstruction` cannot judge them) is posted on #2295 (P67 extends the instrument); the three format-spec cells stay on #2044 (P22c). The lexer DEFECTS filed by the rounds are placed by R-HX: #2337 → P22c (pulled to row 6c‴ by R-HY), #2338 → P67, #2339 → chore; #2340 → chore (R-HU R-HV); the python-twin `SKIP:`-and-return note is chore #2342 (R-HZ).

  8. antonsynd
    added a commit that references this issue on Oct 22, 2026

    docs(deviations): leading-zero decimal integers and BMP-only identifi…

    9e6dcb2

  9. antonsynd commented on Oct 22, 2026

    @antonsynd
    antonsynd
    on Oct 22, 2026
    OwnerAuthor

    Landed (chores band 2026-10-22) on dev @ 4d22212, PR #2346 to mainline. Whole-solution gate 81669/0/7 @ 4d22212 (measured), against 81093/0/7 @ be4664e.

    Ledger rows, commit 9e6dcb2 (R-HT, R-HW):

    • leading-zero-decimal-integer, with an integer_literals.md sentence and an output-asserted spec block.
    • identifier-letters-bmp-only, with an identifiers.md sentence.

    The 01 / 0_1 rows are deleted from numeric-literal-twin-allowlist.txt. They now pin the SharpyOnly table, with the Sharpy verdict per row. The oracle ledger is regenerated: --check reports 104 deviations, up to date.

    Mutations:

    • The lexer refusing a leading-zero integer turned 6 rows red.
    • Flipping SharpyAccepts turned 1 row red.

    Stays open for P5T3, the Stryker re-run on Lexer/. It is scheduled for the weekend at concurrency 4.

  10. antonsynd
    added a commit that references this issue on Oct 22, 2026

    Merge pull request #2346 from antonsynd/dev

    37bb851
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    class-trackerThe one issue per class: carries the contract and the cell tablelane:toolingDocs, formatter, LSP, build tools, tests, CIreach:probeNeeds an escaped spelling, a wrong-annotation probe or a python3 message difftesting

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions