Repository navigation
tests: two standing thresholds on random/wall-clock quantities flake CI — ILCompiles emitted-sample yield halved (P(<5) ≈ 4%/run), CallTimingCoverage fails an arm on one runner stall #2387
Description
Contract
A standing test that asserts a threshold on a random or wall-clock quantity has to keep its false-red rate per CI run negligible (well under 1 in 1000). That rate has to be justified against the quantity's current distribution. The distribution measured when the threshold was chosen doesn't count. When the distribution drifts, the fix is to restore the quantity, or to make the assertion robust to the noise source. Moving the threshold to agree with whatever the suite now produces is the #1432 failure mode.
Evidence
Three Dependabot PRs (#2375, #2378, #2380, all @ base 1ed48de, patch-version NuGet bumps) went red in CI on tests their diffs can't reach. All three were green on a full workflow re-run, and mainline was green @ 393e533.
| PR | run / job | test | observed | threshold |
|---|---|---|---|---|
| #2375 | 40279465395 / 121430055734 | ILCompilesPropertyTests.GeneratedCSharp_CompilesToValidIL |
IL emission: 2/2 (248 rejected) |
≥ 5 emitted of 250 |
| #2378 | 40280449879 / 121434861749 | CallTimingCoverageTests.EveryExecuteArm_AttributesWithinFivePercentOfItsWallClock_ToItsComponents |
arm project: 239.7 of 254.7 ms, ratio 0.9410 |
|1 − ratio| ≤ 0.05 |
| #2380 | 40280487430 / 121434777435 | ILCompilesPropertyTests (seed 2Y1t7NHjkKKc) |
return None.qux → CS0023 |
— not this class: a real defect, already folded into #2365 (#2139 cells) |
The ILCompiles floor also tripped locally on the CsCheck 5.0.0 branch (#2374): 4/4 (246 rejected). So it isn't specific to CsCheck 4.9.1.
Cell 1: ILCompiles emitted-sample yield has halved
The floor (MinimumEmittedSamples = 5, SampleCount = 250) was set in 8f188d8 (#1432) against a measured yield of 3.6 emitted per 50 (≈7.2%). That gives an expected 18 per 250, where P(<5) is about 2e-4. That reasoning was sound when it was made.
Measured now (10 consecutive local runs @ dev 21dfc83): emitted = 5, 10, 8, 14, 6, 11, 13, 7, 13, 8. That's a mean of 9.5 per 250 (≈3.8%), roughly half the design yield. Under a Poisson mean of 9.5, P(<5) ≈ 4% per run, which matches the 2 CI reds plus 1 local red seen across a handful of runs.
The top rejection reasons are front-end refusals of constructs the shared untyped generator (GenSharpy.Module) still draws:
rejected x46: SPY0287: super() must be followed by a method call (e.g., super().__init__())
rejected x45: SPY0708: an import is not allowed inside a function; import at module level
rejected x20: SPY0708: a property declaration is not allowed inside a function; …
rejected x16: SPY0200: Undefined identifier 'x'
rejected x8: SPY0242: Cannot raise a value of type 'bytes' — 'raise' requires an exception …
The floor's own doc comment says not to lower it. The cure is to restore yield. Possible routes: the WithEntryPoint-style shaping this test already applies, or not drawing function-scoped imports, properties and bare super() for this consumer. Raising SampleCount would also work. Any of these should bring P(<5) back under about 1e-3, and the result should be recorded as a measured distribution like the one above.
Cell 2: CallTimingCoverage sums over calls, so one stall fails the arm
Measured locally (10 runs @ 21dfc83): every arm has ratio 0.9986–0.9992, with about 0.2 ms unattributed per arm, which agrees with the 0.998 in the class doc. The CI red had 15 ms unattributed in one arm, about 75× the local gap. The other two arms were fine. This looks like a single GC pause or preemption on a loaded 2-core runner landing between two timed components. A dropped component (the thing this test exists to catch) would look different: P41 P1.2 produced 0.729 / 0.936 / 0.747, a deficit in every call.
The assertion sums component and total time over CallsPerArm = 3 calls of a 2-line program (about 85 ms per call), so one 15 ms stall costs the whole arm 6%. A fix should separate "a region stopped being recorded" (systematic: every call is short) from "the runner stalled once" (one call is short). Two options: assert on the per-call median or best ratio instead of the summed ratio, or require the deficit in every call. Either way, mutation-test it against the P41 P1.2 drop (red) and an injected one-call sleep between components (green).
Not in this issue
- The
None.quxCS0023 cell is semantic: an unknown member on a constructed Sharpy generic receiver (Box[int], Pair[int, str], IProducer[int]) is CS1061 behind SPY0908 — the SPY0203 not-found arm skips this receiver kind #2365. - The other
Minimum*floors in the suite (MinimumValidatorCount,MinimumSideTableFieldCount) count deterministic reflection results. They aren't random quantities. - Must-complete deadlines on loaded runners were tests: must-complete deadlines are too tight for a loaded CI runner — socket 1 s connects and LSP E2E 15 s waits time out on healthy code (mainline 926670989) #2278 (closed). That's the same family, a different quantity.
Activity
antonsynd commentedon Oct 28, 2026
Another occurrence, recorded during the P21g verify round. The whole-solution gate @ 8ff3417 (dev after merging origin/mainline, which brings in CsCheck 5.0.0) gave 82563/1/9 (measured). The only red was
GeneratedCSharp_CompilesToValidIL:IL emission: 4/4 emitted to valid IL (246 rejected by the front end), which is below the floor of 5.Yield per run, from six reruns of the single test on each sha (
--no-build):The pre-merge sha falls below the floor too, so this is the variance this issue already describes. It is not caused by P21g or by the dependabot merge. The top rejection reasons have not changed: SPY0708 import inside a function, SPY0287 bare super(), SPY0708 property inside a function, and SPY0200 undefined identifiers.
antonsynd commentedon Oct 28, 2026
Batching 2026-10-28 @ 8ff3417 — charter
.claude/plans/remediation-round-2026-09-20-batching.md§3.0, call-32 chores band — FIRST (it reddens every gate: the P21g gate @ 8ff3417 was 82563/1/9 on this flake, control 1/6 @ e62f8fa; rides P21h's gate;Sharpy.Compiler.Testsonly). Class tracker for the random/wall-clock-threshold contract.Contract (yours): a standing threshold on a random or wall-clock quantity keeps its false-red rate per run well under 1/1000, justified against the quantity's CURRENT distribution; a drifted distribution is cured by restoring the quantity or making the assertion robust, never by moving the floor (#1432).
Two commits: (1) restore the ILCompiles yield — shape the shared untyped draw for this consumer (SPY0287 bare
super(), SPY0708 function-scoped import/property, SPY0200 are the top rejections) and/or raiseSampleCount; record the new distribution@ shain the commit body and require P(<floor) < 1e-3 under it; (2)CallTimingCoveragereads the per-call median or requires the deficit in EVERY call; mutation-tested against the P41 P1.2 drop (red) and an injected one-call sleep between components (green). TheNone.quxCS0023 cell is #2365's (P13b). No ruling needed.Lane / reach: tooling · probe.
test(compiler): ILCompiles draws no shape the front end refuses uncon…
test(infra): the call-timing coverage verdict reads the median per-ca…
test(compiler): ILCompiles walks the AST through Node.GetChildNodes, …
Merge dev into mainline — call-32 band: ILCompiles draws no shape the…
antonsynd commentedon Oct 28, 2026
Fixed in the call-32 band. dev 9aae969, mainline a31360f. Gate 82587/0/9 @ 9aae969 (measured).
Cell 1 (ILCompiles yield): cfbaff3, followed by 9aae969. The draw now skips function-scoped imports and properties (SPY0708) and a bare
super()(SPY0287).MinimumEmittedSamples = 5andSampleCount = 250are unchanged.AstRewriter. Naming that namespace put ILCompiles inPropertyOracleConformanceTests' universe, so its floor was flagged as a weak oracle. That was the one red in the first gate (82586/1/9 @ 1fc85b6). The walk now usesNode.GetChildNodes.Cell 2 (CallTimingCoverage): 6f6f1dc. The verdict reads the median per-call ratio over 5 calls per arm. The report row prints
median_call_ratio=.