gpu: the Phase 6 gate — and three product gaps it found - #92
Open
dpsoft wants to merge 1 commit into
Open
Conversation
The plan writes the gate as an extension of
TestStubDrivesThePipelineToPprofWithoutAGPU, which needs CAP_BPF, CAP_PERFMON
and CAP_CHECKPOINT_RESTORE and SKIPS without them. Written only there, a
twelve-point gate cannot run on any machine without a GPU box - which is the
most expensive available instance of the failure this project has hit nineteen
times, a check that reads green when things are worst.
So the gate is in three places:
gpu/gate_test.go TestPhase6Gate - 1-9, 10b, no privilege
gpuprobe/gate_compose_test TestPhase6GateConsumerHalf - 10, 10a, 11, 12, no privilege
gpuprobe/gate_test.go TestStubDrivesPCSamplingToPprofWithoutAGPU - end to end
The two unprivileged entry points COMPOSE the assertions the tasks already
made rather than restating them: they call those tests as sub-tests named for
the gate assertion, so deleting or weakening any of them fails the gate by
assertion number, and the twelve are enumerated in one place where a missing
one is visible as a gap. Two of the twelve had no test behind them and are
written out: assertion 2 (a source line reached from a CPU stack, the Phase 6
exit condition, through the join and the projection together) and assertion 3's
aggregation clause (the resolvable and unresolvable populations must share a
stack, or a partially-built-with-lineinfo workload splits its kernel block in
two). Assertion 11 is also new - the no-cap_sys_admin claim existed only as
prose.
The end-to-end test is a SECOND privileged test rather than an edit to the
baseline: that one asserts require.Len(samples, len(snap.Executions)), true
precisely because it emits no PC samples, and several other equalities that PC
sampling would have forced weaker. It is unchanged - 803 insertions, 0
deletions.
Three product gaps the gate surfaced, reported rather than worked around, each
pinned by a passing test that fails the moment the gap is closed:
1. The stub's PC records cannot attribute to anything in either tier. Their
cubin_crc is two compile-time constants unrelated to the cubins the same
run delivers, their correlation is 0, and their kernel names cannot match
any function in the fixtures. The pipeline is right to leave all 64
pending, and the gate asserts that as an equality - but assertions 2, 3,
4, 7 and 9 cannot be driven from the producer, so the end-to-end test
supplies 44 PC records at Timeline.EmitPCSample on correlations a
wire-delivered stack-carrying launch occupies.
2. The cubin transport does not feed gpu.ModuleStore. Attach installs the
placeholder memCubinStore, Config has no field for a store, and
cmd/gpu-cuda-profile builds neither. On hardware today every cubin is
received, sealed, verified and stored - and never read, so every PC sample
in a real profile reads gpu_src_status="no-module" and the Phase 6 exit
condition is unsatisfiable by the shipping product. One hop, both ends
built and tested.
3. Tier A does not refuse to start where CUDA graphs have been observed.
Nothing consumes DropClassGraphExec, so there is no counter, no Snapshot
field, no joinhealth anomaly and no input by which PCSamplingRequest could
be told. Tier A starts happily in a graph-using process and produces
confident, exact-LOOKING attribution of N kernels to one call site.
Mutation-checked: an extra frame in projectionFrames fails assertions 1-3; a
nearest-line fallback in ModuleStore.Resolve fails 3 and 4; sizing the cubin
admission bucket at enrollUIDBurst fails 10.
Assertions 13-16 need the RTX 3090 and are stated as outstanding, with the
plan's full "cannot verify without hardware" list, in
.superpowers/sdd/task-13-gate-report.md.
No gpu/, shim/ or bpf/ behaviour changed. Test-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 13 of the GPU PC sampling plan, GPU-free half (assertions 1–12).
The gate is in three places, deliberately
The plan writes it as an extension of
TestStubDrivesThePipelineToPprofWithoutAGPU— which needsCAP_BPF/PERFMON/CHECKPOINT_RESTOREand skips without them. Written only there, a twelve-point gate cannot run on any machine without a GPU box.gpu/gate_test.goTestPhase6Gategpuprobe/gate_compose_test.goTestPhase6GateConsumerHalfgpuprobe/gate_test.goTestStubDrivesPCSamplingToPprofWithoutAGPUThe unprivileged halves compose the tasks' own tests as sub-tests named for the assertion number, so weakening any of them fails the gate. ~35 existing tests reused across Tasks 3, 4, 6, 8a, 8b, 9, 10, 11.
New, because nothing asserted them: assertion 2 (the conjunction of a real CPU stack and a resolved source line — three tests each covered a segment, none the join); assertion 3's aggregation clause; and assertion 11 (no test anywhere asserted no-
cap_sys_admin).The privileged test is a second test, not an edit — 803 insertions, 0 deletions. Nothing existing was weakened.
Three product gaps, reported rather than worked around
Each is pinned by a passing test that fails the moment the gap closes — the #44/#45 idiom — so the gate gets updated rather than left claiming an assertion it never made.
1. The stub's PC records cannot attribute to anything, in either tier
Their
cubin_crcis two compile-time constants unrelated to the cubins the same run delivers (measured:0x9d57accad01046eb), their correlation is 0, and their kernel names cannot match any fixture function. The pipeline correctly leaves all 64 pending — but assertions 2/3/4/7/9 cannot be driven from the producer, so the end-to-end test injects 44 records atTimeline.EmitPCSampleon correlations a wire-delivered stack-carrying launch occupies.2. The cubin transport never feeds — the exit condition is unsatisfiable
Attachinstalls the placeholdermemCubinStore;Confighas no field for a store;cmd/gpu-cuda-profilebuilds neither.On hardware today every cubin is received, sealed, verified, stored — and never read. Every PC sample reads
gpu_src_status="no-module", and Phase 6's exit condition (a flame graph reaching a real source line from a CPU stack) cannot be satisfied by the shipping product regardless of what the GPU says.This is the risk the plan named as biggest, arriving from an unexpected direction: the transport did not fail. Both ends are built and one hop is missing.
3. Tier A does not refuse to start where CUDA graphs were observed
Nothing consumes
DropClassGraphExec— no counter, noSnapshotfield, no joinhealth anomaly. Tier A runs happily in a graph-using process and produces exact-looking attribution of N kernels to one call site, which is the precise failure the refusal exists to prevent. Assertion 10b's first clause is therefore not asserted, and says so.Ran vs. compiled
Ran (unprivileged,
-count=1and-race -count=4): every one of the twelve has an executed assertion behind it.Compiled only: the privileged end-to-end run —
CapEff: 0here. Its constants were derived, not guessed: the stub was run standalone with the gate's exact environment, confirming 64 PC records / 8 stall names / 2 cubins / 4 drop classes / 8 window records, and the test's FNV CRC replica was checked against the producer's own output.Mutation-checked (applied, run, reverted): an extra frame in
projectionFrames→ assertions 1–3 fail; a nearest-line fallback inModuleStore.Resolve→ 3, 4 fail; the cubin bucket sized atenrollUIDBurst→ 10 fails.One honest deviation
Assertion 12 end-to-end asserts
Undecoded == 4, not 0:gpu_dropped_v1is not a kind this phase decodes — drop-class normalization is the task after Task 7 — and the stub emits one record per class.Assertions 13–16 and the plan's full cannot-verify list are outstanding, in the report.