Repository navigation
chore(tests): python-twin tests count Passed when python 3.12 is absent — SkippableFact + one RequireOrSkip helper at the eight SKIP:-and-return sites (R-HZ) #2342
Description
Class
Contract (verification-contract §2, "a guard is falsifiable"): a test that cannot reach its oracle reports SKIPPED, never PASSED — a gate count that says passed for a twin that compared nothing is a vacuous instrument.
Read @ e7e0a0e (grep-only; no dotnet test was run at batching): eight python-twin sites write a SKIP: line to ITestOutputHelper and return when python 3.12 is absent, so xunit counts the test Passed:
| file | lines | oracle |
|---|---|---|
src/Sharpy.Compiler.Tests/Lexer/PythonInterpreterTests.cs |
49, 68 | python3 tokenize |
src/Sharpy.Compiler.Tests/Lexer/NumericLiteralTwinTests.cs |
479 | python3 tokenize (the only site that honours SHARPY_REQUIRE_PYTHON_ORACLE via PythonInterpreter.TryLocate(minMinor: 12)) |
src/Sharpy.Compiler.Tests/Properties/Differential/FormatSpecDifferentialTests.cs |
119, 270 | python3 format() |
src/Sharpy.Compiler.Tests/Properties/Differential/DifferentialExecutionTests.cs |
160 | python3 execution |
src/Sharpy.Compiler.Tests/Properties/Parser/CPythonDifferentialParseTests.cs |
67, 167 | python3 ast |
CI sets SHARPY_REQUIRE_PYTHON_ORACLE: "1" at .github/workflows/dotnet10.yml:361, so CI is honest for the one site that reads it; every other site is a silent pass on any machine (CI runners included) where the oracle is missing. xunit is 2.9.3 + xunit.runner.visualstudio 4.0.1 in every test csproj; no skippable-fact package is referenced, and xunit 2 has no dynamic skip of its own. Found by the P68 record round (#2300 issuecomment-6327310294, "chore from the record").
Ruling R-HZ (owner, 2026-10-21; /batch-issues call 27 D7 a)
(a) Add Xunit.SkippableFact and ONE helper PythonInterpreter.RequireOrSkip(minMinor: 12) used at all eight sites; a missing oracle shows in the gate's skipped count; the required mode (SHARPY_REQUIRE_PYTHON_ORACLE=1, CI) stays and makes the helper throw instead of skip. Rejected: (b) required mode as the default everywhere with no skip path (a machine without python 3.12 cannot run a green gate; an opt-out env var would reopen the hole); (c) leave it (CI covers one site; local counts keep lying — the vacuous-instrument class). (a) → (b) stacks later by flipping a default; two-way.
Cells
| axis values | program | observed @ sha | expected | lane | reach | found by |
|---|---|---|---|---|---|---|
oracle absent × SHARPY_REQUIRE_PYTHON_ORACLE unset × any of the 8 sites |
dotnet test --filter FullyQualifiedName~PythonInterpreterTests on a host without python3.12 |
Passed (grep-only @ e7e0a0e: the method writes SKIP: and returns) |
Skipped, with the reason | tooling | probe | P68 record round |
oracle absent × required mode × the 7 sites other than NumericLiteralTwinTests |
same, with SHARPY_REQUIRE_PYTHON_ORACLE=1 |
Passed (only the numeric twin throws) | Failed, naming the missing oracle | tooling | probe | P68 record round |
| oracle present (positive control) | same on this machine (python3 3.12.13) | Passed with comparisons made | Passed | tooling | probe | — |
Acceptance
- One helper (
RequireOrSkipbesideTryLocateinSharpy.TestInfrastructure.Oracles.PythonInterpreter) with the three verdicts {located → proceed; absent + not required →throw new SkipException(reason); absent + required → fail naming the oracle}; the eight sites call it and theSKIP:-and-return bodies are deleted (a grep forSKIP:in test sources returns 0 — the census guard). - The eight
[Fact]/[Theory]attributes become[SkippableFact]/[SkippableTheory];Xunit.SkippableFactadded toDirectory.Packages.props(or each csproj) at one pinned version. - Mutation record in the commit body: run one site with
PATHscrubbed of python (a child-process env, never an in-process override — P54's class) → the gate reports it underskipped, notpassed; withSHARPY_REQUIRE_PYTHON_ORACLE=1→failed; restore →passed. Both recorded@ sha (measured). - A positive control: the run with python present reports the same
passedcount as before the change (the twins still compare). - Seam-free with the current chores band (tests-infra only; no compiler code); rides P21e's gate.
Labels: class-tracker, lane:tooling, reach:probe.
Charter: .claude/plans/remediation-round-2026-09-20-batching.md §3.0 — chores band (R-HZ). Sibling: #2300 (the record that found it).
Activity
test(infra): a python twin that cannot reach its oracle reports Skipp…
antonsynd commentedon Oct 22, 2026
Landed (chores band 2026-10-22) on
dev@ 4d22212, PR #2346 to mainline. Whole-solution gate 81669/0/7 @ 4d22212 (measured), against 81093/0/7 @ be4664e.Commit fed8a7c (R-HZ a).
Xunit.SkippableFact1.5.85.PythonInterpreter.RequireOrSkip(minMinor: 12), with three verdicts: located, Skip, or fail underSHARPY_REQUIRE_PYTHON_ORACLE=1.SKIP:-and-return sites now call it under[SkippableFact]/[SkippableTheory].Census guard:
PythonOracleSkipCensusTests, a Roslyn scan. It enforces noSKIP:strings, a skippable attribute on every caller, a floor of 8 callers and more than 500 files scanned, with a positive control.Sibling seam fixed:
CiFilterCoverageConformanceTestsmatched tests by attribute name, so[SkippableFact]tests had dropped out of its universe. It now matches by type derivation.Record (child
dotnet testwith PATH scrubbed of python 3.12):Every acceptance bullet is met. Closing.
Merge pull request #2346 from antonsynd/dev
antonsynd commentedon Oct 22, 2026
Batching 2026-10-22 @ 4d22212 — charter
.claude/plans/remediation-round-2026-09-20-batching.md§3.0, chores — call-28 band: #2342 part 2 (order 0, one commit, no plan; rides P21e's gate). Ruling R-IH (owner, 2026-10-22): D8 a — REOPENED as the "no vacuous pass" tracker. Part 1 (R-HZ) landed @ fed8a7c.Contract (verification-contract §2): a test that cannot run its subject reports Skipped on a missing PRECONDITION and FAILS on a failed STEP — never Passed. Two sibling patterns survived part 1 because their strings differ from
SKIP:(read @ 4d22212, grep-only):RuntimeClosureResolverTests.cs:112,153("No e_sqlite3 native asset for the current RID; skipping native assertion" / "Numpy per-module DLL or MathNet.Numerics.dll not available; skipping." +return) are preconditions and should report Skipped;LineDirectiveRuntimeTests.cs:66,124("Skipping runtime test: C# compilation failed" +return) swallow the very defect the test exists to catch and should FAIL. Lane tooling, reach probe.R-IH (a): one
RequireOrSkip(bool condition, string reason)overload beside the python one; the two precondition sites use it under[SkippableFact]; the two swallowed-failure sites become hard failures carrying the Roslyn diagnostics;PythonOracleSkipCensusTestswidens to"skipping"/"Skipping"strings in test sources with a positive control; mutation = a childdotnet testwith the native asset absent counts Skipped (never an in-process override — P54's class), the census guard red on a re-insertedreturn. Rejected: (b) a new chore issue with the same body (contract §10: tooling cells never get a new issue; the commit-cite is satisfied by this reopen); (c) ledger only (leaves a test that passes when the compiler emits uncompilable C#). Compiler.Tests only; two-way.test(infra): a test whose subject cannot run reports Skipped on a mis…
Merge dev into mainline: call-28 band slice 2 (refs #2057 #2143 #2342 #…
antonsynd commentedon Oct 22, 2026
Landed: part 2 @ 2a87810 (R-IH). Contract §2: a missing PRECONDITION reports Skipped, and a failed STEP fails.
Helper:
TestPreconditions.RequireOrSkip(bool, reason)(TestInfrastructure).Sites. The census found more than the charter's grep:
RuntimeClosureResolverTestsreturns, the Numpy siteRuntimeClosureResolverTests[SkippableFact]LineDirectiveRuntimeTestsIncrementalCompilationPropertyTestsUnparserFixtureRoundTripTests|> _fixturesfunctions/pipe_with_partialwas a hidden pass)UnparserFixtureRoundTripTestsCensus:
PythonOracleSkipCensusTestsgains rule 3: noWriteLineof a "skipping" string, holes included. Its positive control covers both spellings and an allowed assertion message.Mutation records:
dotnet test: Skipped 1.test(compiler): LineDirectiveRuntimeTests compiles the generated prog…
Merge dev into mainline: the line-directive runtime tests reference t…
antonsynd commentedon Oct 22, 2026
Follow-up @ 03eee75 (mainline 0c90a73, CI green): part 2 exposed one hidden failure, now fixed.
On mainline b99221c, the Windows CI job's line-directive step went red: 2
LineDirectiveRuntimeTests, CS0400'Sharpy' could not be found.bin/Debug/netstandard2.{1,0}. Only a whole-solution build makes those. The Windows job buildssrc/Sharpy.Compiler.Testsalone.