From 3494793c3c87514f0b215568c5624ac3031a4451 Mon Sep 17 00:00:00 2001 From: diego Date: Tue, 25 Aug 2026 22:09:46 -0300 Subject: [PATCH] Task 13: the Phase 6 phase gate, GPU-free half (assertions 1-12) The plan writes the gate as an extension of TestStubDrivesThePipelineToPprofWithoutAGPU, which needs CAP_BPF, CAP_PERFMON and CAP_CHECKPOINT_RESTORE and SKIPS without them. Written only there, a twelve-point gate cannot run on any machine without a GPU box - which is the most expensive available instance of the failure this project has hit nineteen times, a check that reads green when things are worst. So the gate is in three places: gpu/gate_test.go TestPhase6Gate - 1-9, 10b, no privilege gpuprobe/gate_compose_test TestPhase6GateConsumerHalf - 10, 10a, 11, 12, no privilege gpuprobe/gate_test.go TestStubDrivesPCSamplingToPprofWithoutAGPU - end to end The two unprivileged entry points COMPOSE the assertions the tasks already made rather than restating them: they call those tests as sub-tests named for the gate assertion, so deleting or weakening any of them fails the gate by assertion number, and the twelve are enumerated in one place where a missing one is visible as a gap. Two of the twelve had no test behind them and are written out: assertion 2 (a source line reached from a CPU stack, the Phase 6 exit condition, through the join and the projection together) and assertion 3's aggregation clause (the resolvable and unresolvable populations must share a stack, or a partially-built-with-lineinfo workload splits its kernel block in two). Assertion 11 is also new - the no-cap_sys_admin claim existed only as prose. The end-to-end test is a SECOND privileged test rather than an edit to the baseline: that one asserts require.Len(samples, len(snap.Executions)), true precisely because it emits no PC samples, and several other equalities that PC sampling would have forced weaker. It is unchanged - 803 insertions, 0 deletions. Three product gaps the gate surfaced, reported rather than worked around, each pinned by a passing test that fails the moment the gap is closed: 1. The stub's PC records cannot attribute to anything in either tier. Their cubin_crc is two compile-time constants unrelated to the cubins the same run delivers, their correlation is 0, and their kernel names cannot match any function in the fixtures. The pipeline is right to leave all 64 pending, and the gate asserts that as an equality - but assertions 2, 3, 4, 7 and 9 cannot be driven from the producer, so the end-to-end test supplies 44 PC records at Timeline.EmitPCSample on correlations a wire-delivered stack-carrying launch occupies. 2. The cubin transport does not feed gpu.ModuleStore. Attach installs the placeholder memCubinStore, Config has no field for a store, and cmd/gpu-cuda-profile builds neither. On hardware today every cubin is received, sealed, verified and stored - and never read, so every PC sample in a real profile reads gpu_src_status="no-module" and the Phase 6 exit condition is unsatisfiable by the shipping product. One hop, both ends built and tested. 3. Tier A does not refuse to start where CUDA graphs have been observed. Nothing consumes DropClassGraphExec, so there is no counter, no Snapshot field, no joinhealth anomaly and no input by which PCSamplingRequest could be told. Tier A starts happily in a graph-using process and produces confident, exact-LOOKING attribution of N kernels to one call site. Mutation-checked: an extra frame in projectionFrames fails assertions 1-3; a nearest-line fallback in ModuleStore.Resolve fails 3 and 4; sizing the cubin admission bucket at enrollUIDBurst fails 10. Assertions 13-16 need the RTX 3090 and are stated as outstanding, with the plan's full "cannot verify without hardware" list, in .superpowers/sdd/task-13-gate-report.md. No gpu/, shim/ or bpf/ behaviour changed. Test-only. --- .superpowers/sdd/task-13-gate-report.md | 482 ++++++++++++++ gpu/gate_test.go | 429 +++++++++++++ gpuprobe/gate_compose_test.go | 276 ++++++++ gpuprobe/gate_test.go | 803 ++++++++++++++++++++++++ 4 files changed, 1990 insertions(+) create mode 100644 .superpowers/sdd/task-13-gate-report.md create mode 100644 gpu/gate_test.go create mode 100644 gpuprobe/gate_compose_test.go diff --git a/.superpowers/sdd/task-13-gate-report.md b/.superpowers/sdd/task-13-gate-report.md new file mode 100644 index 0000000..cd989bf --- /dev/null +++ b/.superpowers/sdd/task-13-gate-report.md @@ -0,0 +1,482 @@ +# Task 13 — The Phase 6 gate, GPU-free half (assertions 1–12) + +Branch `feat/phase6-gate`, one commit, test-only. No file under `gpu/`, `shim/` or +`bpf/` changed; `git diff --stat` on `gpuprobe/gate_test.go` is 803 insertions and +**zero deletions**, so no existing gate assertion was weakened. + +Verified on this machine: `CapEff: 0`, no GPU, no passwordless sudo. That shapes the +whole deliverable and is stated plainly below rather than glossed. + +## Where the gate lives, and why it is in three places + +| file | package | privilege | what | +| --- | --- | --- | --- | +| `gpu/gate_test.go` | `gpu` | none | `TestPhase6Gate` — assertions 1–9 and 10b | +| `gpuprobe/gate_compose_test.go` | `gpuprobe` | none | `TestPhase6GateConsumerHalf` — assertions 10, 10a, 11, 12 | +| `gpuprobe/gate_test.go` | `gpuprobe_test` | CAP_BPF + CAP_PERFMON + CAP_CHECKPOINT_RESTORE | `TestStubDrivesPCSamplingToPprofWithoutAGPU` — 1–5, 8, 9, 12 end to end | + +The plan writes the gate as an extension of `TestStubDrivesThePipelineToPprofWithoutAGPU`. +Written only there, the gate **skips on every machine without capabilities**, including +this one — and a twelve-point gate that cannot run is the most expensive instance of the +failure this project has hit nineteen times. So every assertion that is a statement about +the `gpu` or `gpuprobe` package is *also* asserted where it needs no privilege, and the +privileged test adds what only a real producer can add: a CPU stack walked by BPF through +two `-fomit-frame-pointer` frames, real cubins crossing a real socket, 500 real executions. + +The two unprivileged entry points **compose** rather than restate: they call the tests the +tasks already wrote, as sub-tests named for the gate assertion. Two assertions of one fact +drift apart; one assertion referenced from the gate does not. What composing buys is that +deleting or weakening any of them now fails *the gate*, by assertion number. + +`TestStubDrivesPCSamplingToPprofWithoutAGPU` is a **second** privileged test rather than an +edit to the baseline one. The baseline asserts `require.Len(samples, len(snap.Executions))`, +which is true precisely because that run emits no PC samples; turning PC sampling on inside +it would have meant weakening that and several others. + +--- + +## The twelve + +"Reused" means the gate calls an existing test. "New" means no test asserted the property. +"Ran" means it executed on this machine, unprivileged. "Compiled" means it type-checks and +is wired into the privileged gate, which cannot run here. + +### 1. Frames stop at the kernel + +**Asserts.** Frames are exhaustively ` → [gpu:launch]|[gpu:launch unsampled] → +[gpu:kernel:]`. No frame carries the PC, the stall reason, the source location or the +attribution quality. + +**Catches.** Promoting any per-sample detail to a frame, in any spelling. At PC-sampling +rates one frame per PC destroys aggregation and fragments the kernel's own block, which is +the §8 ruling the whole label design rests on. + +**Reused** — `TestProjectionAddsNoFrames`, `TestProjectionKeepsPCOutOfStackIdentity`. +**New end-to-end**: the privileged gate compares each sample's frames against the launch's +own `CPUStack` as a *whole slice*, so nothing may be inserted anywhere rather than merely +appended. Whole-slice comparison also avoids a false positive a substring scan would hit: +three of the stub's stall reasons are spelt `wait`, `barrier` and `membar`, and libc frame +names contain all three. + +**Ran** (unprivileged half) / **compiled** (end-to-end half). +**Mutation-checked**: appending `[gpu:src:mutant]` in `projectionFrames` fails assertions +1, 2 and 3 of `TestPhase6Gate`. + +### 2. A source line is reached from a CPU stack — the Phase 6 exit condition + +**Asserts.** One pprof sample carrying *both* a real CPU call path in its frames *and* +`gpu_src_status="resolved"` with `gpu_src_file`/`gpu_src_line` naming a real line of +`internal/cubin/testdata/single.cu`. The line number is checked against the `.cu` itself, +not against a constant, so a fixture rebuilt from edited source cannot leave it green. + +**Catches.** A projection that resolves source labels but loses the launch's frames, or the +reverse; a join that reaches the execution but drops `PCSamples` so nothing projects; a +store keyed on something the sample does not carry; a resolution naming a line the table +does not contain. + +**New.** Three tests each covered a segment and none the conjunction: +`TestTierBSampleReachesItsExecution` drives the join but its execution has no launch and +never projects; `TestProjectionEmitsAllFourSrcStatuses` reaches `resolved` from a hand-built +`ExecutionView` with no `Launch`; `TestProjectionAddsNoFrames` has the CPU stack but asserts +the *negative*. `TestGateASourceLineIsReachedFromACPUStack` drives a real `Timeline` so the +module join and the resolution both run for real. + +**Ran** (through the `gpu` package) / **compiled** (end to end). +**This is the assertion with the largest gap between what the gate proves and what the +product does — see "Findings" below.** + +### 3. No `-lineinfo`, no invention + +**Asserts.** Every no-lineinfo sample carries `gpu_src_status="no-lineinfo"` and no +`gpu_src_file`/`_line`/`_func`; the kernel frame is unchanged, so the resolvable and +unresolvable populations still aggregate at the same kernel. + +**Catches.** Synthesizing a nearest line or a function's first line. And, for the second +clause: any frame that varies with `gpu_src_status`, which would split one kernel's block +in two in a flame graph while every label assertion still passed. + +**Reused** for the labels — `TestProjectionEmitsAllFourSrcStatuses`. +**New** for the aggregation clause — `TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel`; +nothing asserted that the two populations share a stack. + +**Ran** / **compiled** (end to end, generalized there to all three unresolvable statuses). +**Mutation-checked**: making `Resolve` fall back to `Resolve(fn, 0)` when the PC is +uncovered fails assertions 3 and 4. + +### 4. All four `gpu_src_status` values reachable, each by the fixture that should produce it + +**Asserts.** `resolved` from the `-lineinfo` fixture, `no-lineinfo` from the no-lineinfo +fixture, `no-module` from a CRC no cubin was stored under, `unmapped` from a PC past the +function — in the store *and* in the labels; the enum is exhaustive and its zero value is +not a status. + +**Catches.** A fifth value; a silent default; the two "cannot resolve" statuses collapsing +into one, which is the difference between "recompile with `-lineinfo`" and "the compiler +emitted no line here". + +**Reused** — `TestModuleStoreAllFourStatusesAreReachable`, +`TestProjectionEmitsAllFourSrcStatuses`, `TestSrcStatusesIsExhaustiveAndStable`, +`TestSrcStatusZeroValueIsNotAStatus`. **Ran** / **compiled** (end to end, with exact +per-status counts). + +### 5. Reconciliation + +**Asserts.** Every PC record accepted by the sink lands in exactly one of attributed-exact, +attributed-kernel, still-pending (in either of the two pending stores) or evicted (from +either), and the counters sum to what was emitted. Plus the group-level identity: every +pending module group is joined or left pending for exactly one *counted* reason. + +**Catches.** A join that attaches samples without counting them (satisfies the sample +identity, reports a wrong per-tier split); a refusal path that returns without incrementing +any counter — the easiest and most invisible mistake available in that function. + +**Reused** — `TestConformancePCSampleReconciliationCoversPendingAndEvicted`, +`...CoversAttributedByKernel`, `TestTierBSamplesReconcile`, +`TestTierBJoinAccountsForEveryGroup`. **New end-to-end**: the identity over 64 wire records +plus 44 injected ones, with `PendingModuleSamples == 64` as an equality (see finding 1). +**Ran** / **compiled**. + +### 6. Tier B does not collapse + +**Asserts.** 10,000 samples across 50 distinct `(crc, functionIndex)` pairs attribute with +no loss to `EvictedPendingSamples`. + +**Catches.** Task 8a's pathology returning — a Tier B sample keyed on its (empty) +correlation value, collapsing a whole process onto `{backend, pid, ""}` and evicting +everything past `pendingSampleCap`. + +**Reused** — `TestTierBPCSamplesDoNotCollapseOntoOneKey`, +`TestTierBDistinctFunctionIndexSplitsGroups`. **Ran.** Not reachable end to end: the stub +emits 64 records, not 10,000, and none of them attributes at all. + +### 7. Ambiguity is marked in its own label + +**Asserts.** Two executions of one kernel in the horizon with one Tier B batch give +`gpu_pc_attrib="kernel-ambiguous"` **and leave `gpu_ambiguous` unset**. + +**Catches.** Reusing `ExecutionView.Ambiguous` for PC ambiguity, which would put +`gpu_join="exact" gpu_ambiguous="true"` on one sample — two unrelated facts on one boolean, +and `AmbiguousHeuristicMatchCount` no longer meaning what its name says. + +**Reused** — `TestTierBAmbiguityIsMarkedWithoutTouchingAmbiguous`, +`TestProjectionKernelAmbiguousNeverCoincidesWithGpuAmbiguous`. **Ran.** Not reachable end to +end: the stub's PC records cannot join through the module at all (finding 1). + +### 8. Tier A disclosure + +**Asserts.** Inside a window `"true"`, in a proven gap `"false"`; Tier A selected with no +window arriving gives `"unknown"` on every execution and `"false"` on none; the three +outcomes partition the executions exactly. + +**Catches.** The one answer that must never be reachable by accident — a profile reporting +"not perturbed" when it means "cannot tell". + +**Reused** — `TestSerializationMarksExecutionsOverlappingABurst`, +`TestSerializationIsFalseInAProvenGap`, `TestSerializationIsUnknownWhenNoWindowsArrived`, +`TestSerializationFalseIsOnlyEverReachedFromPositiveEvidence`, +`TestSerializedLabelIsUnconditionalAndHasThreeValues`. +**New end-to-end**: real windows from the producer bracket real executions, and the +"no windows" half is driven by re-emitting *this run's own* executions into a second Tier A +`Timeline` that never sees a window — same population, only the evidence differs. +**Ran** / **compiled**. + +### 9. Cardinality cap + +**Asserts.** Past the ceiling `gpu_pc` is suppressed and nothing else — `gpu_stall`, +`gpu_src_*` and `gpu_pc_attrib` survive, the sample keeps its full share of the execution's +duration — with `ProjectionPCLabelsSuppressed` **equal to the suppressions actually visible +in the output**, and the loss surfaced in `joinhealth`. + +**Catches.** A cap that drops the coarser, more actionable labels instead of the numerous +one; a suppression that is silent, since a profile that lost its PC labels looks identical +to one that never had any. + +**Reused** — `TestProjectionCapSuppressesGpuPCAndOnlyGpuPC`, +`TestProjectionCapIsSurfacedInJoinHealth`. **New end-to-end**: 40 distinct injected offsets +against a ceiling of 8, with the counter tied to the observed output rather than to a +hard-coded number. **Ran** / **compiled**. + +### 10. The cubin channel cannot touch enrolment + +**Asserts.** The isolation test runs *as part of the gate*, in **both** orders including +offers flooded *ahead of* an enrolment with the cubin listener's accept loop deliberately +wedged: `CubinsThrottled` non-zero, `UnwindEnrollThrottled` unchanged, the enrolment still +confirmed. Plus: the two admission buckets are separate objects with different numbers, the +two addresses are siblings and not one socket, and the enrolment handler performs **no read** +on the producer's connection — asserted behaviourally and structurally. + +**Catches.** Moving cubin traffic onto the enrolment listener, its goroutine, its accept +loop or its bucket. Any of those restores issue #49's ~38 % stack loss on a module-heavy +workload, silently, with only `UnwindEnrollThrottled` moving. + +**Reused** — `TestFloodingTheCubinChannelCannotStarveOrThrottleAnEnrolment`, +`TestTheCubinAdmissionBucketIsItsOwn`, +`TestTheCubinAddressIsASiblingOfTheRendezvousAndNotTheSameSocket`, +`TestAnEnrolmentCompletesWithNoReadOnThatConnection`. **Ran.** The privileged gate adds the +live form: real cubin traffic crossing while a real enrolment happens, with +`UnwindEnrollThrottled == 0` and `UnwindEnrollConfirmed == 1`. +**Mutation-checked**: sizing the cubin per-uid bucket at `enrollUIDBurst` fails +`assertion-10-the-buckets-are-separate-objects`. + +### 10a. Seals are enforced + +**Asserts.** A memfd missing each required seal *in turn* is rejected, counted in +`CubinsRejectedUnsealed`, and **never mapped**; a descriptor that is not a sealed memfd at +all (a pipe, an unsealed tmpfs file) is refused; the required set is spelt out. + +**Catches.** Dropping a seal — without `F_SEAL_SHRINK` a peer can `ftruncate` under our +`mmap` and SIGBUS the agent; without `F_SEAL_WRITE` the ELF mutates under the parser +mid-parse — and, worse, any fallback that maps it anyway. + +**Reused** — `TestEachRequiredSealMissingInTurnIsRejectedAndNeverMapped`, +`TestADescriptorThatIsNotASealedMemfdIsRefused`, `TestSealNamesAreSpeltOut`. **Ran.** + +### 10b. Out-of-scope conditions refuse rather than guess + +Three clauses, and they are not in the same state. + +- **Two device ids give `gpu_pc_attrib="kernel-multidevice"`** — **reused** + (`TestTierBMultiDeviceProcessIsMarked`, `TestMultiDeviceOutranksAmbiguity`), **ran**. +- **A window with `end_ns == 0` gives `"unknown"` on everything after it and `"false"` on + nothing** — **reused** + (`TestSerializationOpenWindowMakesEverythingFromItsStartUnknownAndNeverFalse`), **ran**; + the privileged gate also drives it over this run's own executions. +- **A graph execution makes Tier A refuse to start** — **NOT IMPLEMENTED BY THE PRODUCT.** + See finding 3. Pinned, not asserted, by + `TestGateGraphExecutionRefusalIsNotAssertableYet`, which **ran**. + +### 11. `getcap` on the gate binary shows no `cap_sys_admin` + +**Asserts.** From two independent directions, because either alone is escapable: the file +capabilities of the test binary itself (via `cap.GetFile`, the same +`security.capability` xattr `getcap` prints — no external tool needed, so it cannot degrade +to a skip on a container image without `getcap`), and this process's own Permitted set. A +separate test runs `getcap(8)` when present and requires it to agree. Vacuous under root, so +the process half is skipped there and said so, while the file half still runs. + +**Catches.** The two quiet one-liners that acquire the requirement: attaching with +`link.Uprobe` instead of `link.UprobeMulti` (the `perf_uprobe` PMU needs CAP_SYS_ADMIN, the +BPF link does not), and reading the producer's address space to follow +`gpu_module_load_v1.bytes_ptr` (needs CAP_SYS_PTRACE) — both of which work fine on a +developer box that runs tests as root and say nothing about why. + +**New.** No test asserted it anywhere; the standing Phase 1 assertion existed only as prose. +Composed alongside `TestEmbeddedProgramIsUprobeMulti`, which is *why* no CAP_SYS_ADMIN is +needed. **Ran.** + +### 12. `Stats.Undecoded` is zero for every kind this phase decodes + +**Asserts.** All five PC-sampling kinds (module, PC, stall map, sampling window, config) +have an `applyBatch` arm and are counted in `Records`, not `Undecoded`; a healthy Tier B run +leaves every new loss counter at zero; an unknown kind is still counted rather than dropped; +`kindMax` matches the embedded BPF object's `dropped` map; every cookie has a sized kind. + +**Catches.** A `KIND_*` added on one side of the wire and not the other — silent loss unless +counted — and, in the other direction, a decode arm removed, which would put a kind back +into the default arm while every other counter still read healthy. + +**Reused** — `TestTheFivePCSamplingKindsAreDecodedNotCountedUndecoded`, +`TestHealthyTierBRunLeavesEveryNewLossCounterAtZero`, +`TestUndecodedKindsAreCountedNotDropped`, `TestEmbeddedProgramIsUprobeMulti`, +`TestBPFSizesEveryKindCookieForInstalls`. **Ran.** + +**End to end this is `Undecoded == 4`, not zero, and that is correct.** `gpu_dropped_v1` is +*not* a kind this phase decodes: it is decoded into `batch.Drops` and carried, and +normalizing a drop class into an operator-visible number is the consumer task after Task 7 +(see `Stats.Undecoded`'s own comment). The stub emits exactly one record per drop class when +PC sampling is on, so the gate asserts the equality — a fifth undecoded record means a kind +arrived that nothing on this side knows about. **Compiled.** + +--- + +## Findings — three product gaps the gate surfaced + +All three are **reported, not silently worked around**, and each is pinned by a passing test +that **fails the moment the gap is closed**, so the gate is updated rather than left +claiming an assertion it never made. That is the shape issue #44 used and #45 inverted. + +### Finding 1 — the stub's PC records cannot attribute to anything, in either tier + +`shim/stub/stub.cc` emits real module loads carrying the real checked-in cubins, and real PC +records. They are unrelated: + +- the PC records' `cubin_crc` is one of two compile-time constants, + `kStubCubinCRC = {0xC0FFEE01, 0xC0FFEE02}`, while the modules the same run delivers are + keyed by a content hash of the fixture bytes — measured on this machine, + `0x9d57accad01046eb` for `single_lineinfo.cubin`. No cubin is ever stored under a + `0xC0FFEE0n` key; +- their `correlation` is 0 in every tier — correct, that is what CONTINUOUS collection + produces — so the exact-correlation path is unavailable to them; +- Tier B attribution runs `crc → module → function name → the execution's KernelName`, and + the stub's kernel names are `kernel_1111`/`kernel_2222` while the fixtures' only function + is the CUDA kernel they were compiled from (`addOne`). No name can match. + +So neither join path can fire, and the stub cannot drive assertions 2, 3, 4, 7 or 9. The +pipeline is not at fault — it correctly counts all 64 as pending, which the gate asserts as +an exact equality, and `TestTierBKernelNameMismatchStaysPending` already pins that samples +stay pending rather than being attached to a plausible neighbour. + +**Consequence for the gate.** `TestStubDrivesPCSamplingToPprofWithoutAGPU` injects 44 PC +samples of its own at `Timeline.EmitPCSample` — the same entry point the consumer calls — on +correlations that a wire-delivered, stack-carrying launch is known to occupy (replayed +exactly by `gpuabi.SampleSchedule`, which is pinned against the shim's own sampler). What is +injected is the *record*; everything downstream is product code — the join, the store's +four-valued resolution, the label set, the cardinality budget. What the injection skips is +the consumer's `KIND_PC` decode arm, and that is asserted separately and exactly by the 64 +wire records. + +**Fix** (a small change to `shim/stub/stub.cc`, out of scope on a test branch): record the +CRC each capture computed and use it on the PC records; name the kernels after the fixtures' +own functions. **Pinned by** `TestGateTheStubsPCRecordsCannotAttributeToAnything` +(unprivileged, ran). + +### Finding 2 — the cubin transport does not feed `gpu.ModuleStore`, so the product cannot resolve a source line at all + +Task 3 built the channel; Task 4 built the store. Nothing connects them: + +- `Attach` calls `newCubinListener(cfg, nil)`, and a nil sink becomes `memCubinStore` — a + bounded CRC→bytes map with no line table, no LRU and no `Resolve`. Its own comment says + "Task 4 replaces it"; +- `gpuprobe.Config` has no field by which a caller could supply one, and `gpu.ModuleStore` + does not satisfy `cubinSink` (`Put`/`HasCubin` versus `PutCubin`/`HasCubin`) even if it + had; +- `cmd/gpu-cuda-profile` builds neither: `gpu.NewTimeline(gpu.TimelineConfig{PCSampling: tier})` + with no `Modules`, and `gpu.ProjectExecutionsWith(snap, gpu.ProjectionConfig{})`. + +**On hardware today every cubin is received, sealed, verified, identity-checked, stored — +and then never read, and every PC sample in a real profile reads +`gpu_src_status="no-module"`.** Gate assertion 2, the Phase 6 exit condition, is therefore +not satisfiable by the product as shipped. This is one hop, with both ends built and tested; +it is a wiring task, not a design gap. It is also, per `ProjectionConfig.Modules`' own +comment, *designed* to be visible rather than silent — "no-module" on every sample points +straight at the missing store. + +**Consequence for the gate.** The privileged gate builds the `ModuleStore` itself and keys +it on the CRCs the **producer** declared (parsed from the producer's own report and +independently recomputed in Go over the checked-in fixture bytes), not on numbers the test +invented — so the identity the store keys on is the identity that went on the wire. It also +asserts that the decoded `gpu_module_load_v1` records name the same `(crc, size)` pairs. +That is the offline half of hardware assertion 13. + +**Pinned by** `TestGateTheCubinTransportDoesNotYetFeedTheModuleStore` (unprivileged, ran). + +### Finding 3 — Tier A does not refuse to start where CUDA graphs have been observed + +The plan's Task 10 and its out-of-scope section require: *"Tier A refuses to start in a +process where graph executions have been observed … The refusal is loud and counted, not a +silent downgrade to Tier B."* + +The wire signal exists — `internal/gpuabi.DropClassGraphExec` decodes and spells itself +`graph-exec`, and the stub emits one such record so the class is reachable from a test. The +refusal does not. **Nothing in `gpu/`, `gpuprobe/` or `cmd/` consumes that drop class**: the +consumer decodes `gpu_dropped_v1` into `batch.Drops` and stops (finding above), so there is +no counter, nothing on the `Snapshot`, no `joinhealth` anomaly, and no input by which +`PCSamplingRequest` could be told. Tier A starts happily in a graph-using process and +produces confident, exact-*looking* attribution of N kernels to one call site — which is the +plan's own finding 4, and the condition it says must be visible rather than merely out of +scope. Graph launches are the norm in inference serving. + +What *is* true today, and is asserted: the Tier A acknowledgement refusal an operator must +read before the tier can run names CUDA graphs, and so does the standing warning. + +**Pinned by** `TestGateGraphExecutionRefusalIsNotAssertableYet` (unprivileged, ran), which +uses reflection over `Snapshot`, `TimelineDropStats`, `PCJoinStats`, `TimelineConfig` and +`PCSamplingRequest` and fails the moment any of them grows a field naming graph executions. + +--- + +## What ran and what only compiled + +**Ran, unprivileged, on this machine** — `go test ./gpu/ ./gpuprobe/ ./internal/... -count=1` +and `-race -count=4`, both green: + +- assertions **1, 3, 4, 5, 6, 7, 8, 9, 10b** (two of three clauses) via `TestPhase6Gate`; +- assertion **2** via `TestGateASourceLineIsReachedFromACPUStack`, through a real `Timeline` + and a real `ModuleStore` over the checked-in `-lineinfo` fixture; +- assertions **10, 10a, 11, 12** via `TestPhase6GateConsumerHalf`; +- all three outstanding-gap pins. + +**Compiled only** — `TestStubDrivesPCSamplingToPprofWithoutAGPU`. `CapEff: 0` and no +passwordless sudo on this machine (`sudo -n true` → "a password is required"), so it skips +with the message that names the setcap line. It type-checks, vets and lints clean, and every +number in it is derived rather than guessed: + +- 58 sampled launches from `gpuabi.SampleSchedule(500, 8, DefaultSampleSeed)`, the same + constant the baseline gate uses; +- 64 PC records, 2 cubins, 8 stall reasons, 4 drop records, 4 bursts — all read out of + `shim/stub/stub.cc` and confirmed by **running the producer standalone** with the exact + environment the gate sets. That run printed `pc_sampling=serialized`, + `pc_samples=64 stall_reasons=8 cubins=2 functions=4 drop_classes=4`, and captured both + fixtures with the CRCs above; +- the FNV-1a CRC replica in the test was checked against the producer's own output for both + fixtures and matches exactly. + +What has *not* been executed is the attach, the walk, the symbolization and the assertions +that read `Stats` after a live run. Those need `cap_bpf,cap_perfmon,cap_checkpoint_restore`. + +**Mutation checks** (each applied, run, and reverted): + +| mutation | gate result | +| --- | --- | +| `projectionFrames` appends `[gpu:src:mutant]` | assertions 1, 2, 3 FAIL | +| `ModuleStore.Resolve` falls back to `Resolve(fn, 0)` for an uncovered PC | assertions 3, 4 FAIL | +| cubin per-uid admission bucket sized at `enrollUIDBurst` | assertion 10 FAIL | + +--- + +## Outstanding: assertions 13–16 (RTX 3090) + +None of these is attempted here; all four need the hardware. + +13. **`cuptiGetCubinCrc()` over the received copy equals the PC records' `cubinCrc`.** The + gate proves the *shape* of this offline — one number identifies one set of bytes and the + same number reaches both ends — against the stub's FNV-1a stand-in. CUPTI's polynomial + is unpublished and there is no CUDA toolkit on the agent path, so the real equality is + hardware-only. +14. **Tier B: a flame graph reaching a real line of `cuda_workload.cu` from a CPU stack, + through labels** — the Phase 6 exit condition on real hardware. Blocked additionally by + finding 2: the transport→store hop must be wired before this can pass on any machine. +15. **Tier A: `correlationId` non-zero on ≥ 99 % of PC records; `gpu_pc_attrib="exact"` on + the resulting samples; windows bracketing the executions that ran in them.** +16. **Overhead within the Task 12 thresholds, or the tier decision they dictate.** Task 12 + is active on `.worktrees/pc-overhead` and its numbers are not in the plan file yet. + +## The plan's own "cannot verify without hardware" list, restated + +Beyond assertions 13–16, and unchanged by this task: + +- whether `functionIndex` **is** the cubin's `.symtab` index — the finding-2 question of the + plan, and the trigger for `gpu_pc_sample_batch_v2`. Every test in this tree, including the + gate's, reads the index out of the fixture and asserts only that the store and the sample + use the same one consistently; +- whether `pcOffset` is function-relative in the sense the line table is; +- every rate and buffer-sizing number (the spike's 352 records for ~103 k samples drives all + of them); +- hardware-buffer overflow behaviour — `droppedSamples` / `hardwareBufferFull` under a + saturating workload; +- whether the collection mode can change between `Stop` and `Start` without a full + `Disable`/`Enable` — undocumented, and it decides whether a future tier switch is possible + at all; +- whether per-context enable holds when contexts are created lazily on other threads; +- whether the `MODULE_UNLOAD_STARTING` drain preserves PC uniqueness across a + load–unload–load cycle; +- whether a `cuptiFinalize` handler runs at all, and whether disabling per context inside it + errors. + +**Not verifiable at all, on hardware or otherwise:** MPS, and cross-process contention for +the per-device PC-sampling hardware. A process cannot observe either from inside itself. +Stated so a quiet profile is not read as a correct one. + +## Verification run + +``` +make -C shim && make -C shim test && make -C shim check-fpless \ + && make -C shim check-cubin-defer && make -C shim nvidia # all OK +go build ./... && go vet ./... # clean +go test ./gpu/ ./gpuprobe/ ./internal/... -count=1 # ok +go test ./gpu/ ./gpuprobe/ -race -count=4 # ok +~/go/bin/golangci-lint run --timeout=5m # 0 issues +``` + +`git diff --stat` for the one modified file: `gpuprobe/gate_test.go`, 803 insertions, +0 deletions. diff --git a/gpu/gate_test.go b/gpu/gate_test.go new file mode 100644 index 0000000..2df5138 --- /dev/null +++ b/gpu/gate_test.go @@ -0,0 +1,429 @@ +package gpu + +import ( + "os" + "reflect" + "strconv" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + pp "github.com/dpsoft/perf-agent/pprof" +) + +// TestPhase6Gate is the GPU-free half of the Phase 6 phase gate, assembled in +// the package whose behaviour it is about. +// +// # Why the gate lives in two places and why this half exists at all +// +// The plan's gate (Task 13, assertions 1-12) is written as an extension of +// gpuprobe's TestStubDrivesThePipelineToPprofWithoutAGPU, which needs CAP_BPF, +// CAP_PERFMON and CAP_CHECKPOINT_RESTORE to attach the uprobes and symbolize +// the producer's stacks. On a machine without those it SKIPS - and a gate that +// skips is a gate that cannot fail. Nineteen defects on this project have been +// checks that read green when things were worst; a twelve-point gate nobody can +// run is the most expensive available instance of that. +// +// So every assertion of the twelve that is a statement about the gpu package - +// which is nine of them - is also asserted HERE, where it needs no capability +// and runs on every `go test ./gpu/`. The privileged end-to-end gate in +// gpuprobe/gate_test.go adds what only a real producer can add: a real CPU +// stack walked by BPF, a real cubin crossing a real socket, real executions. +// +// # Why this composes rather than restates +// +// Every property below is already asserted by a task's own test. Restating it +// here would give the project two assertions of one fact, which is how two +// assertions of one fact drift apart. So this calls them. What that buys, over +// leaving them scattered: +// +// - deleting or weakening any one of them fails THE GATE, by name, with the +// assertion number attached, rather than removing a test nobody notices; +// - the twelve are enumerated in one place, so an assertion that has no test +// behind it is visible as a gap rather than as an absence; +// - the sub-test names carry the gate numbering, so a failure reads as +// "assertion 7 failed" instead of as a test name that has to be traced back +// to the plan. +// +// Two of the twelve are NOT compositions, because no test asserted them: +// assertion 2 (a source line reached from a CPU stack, through the join and the +// projection together) and assertion 3's aggregation clause. Both are written +// out below. +// +// Assertions 10, 10a, 11 and 12 are consumer-side and live in +// gpuprobe/gate_compose_test.go. Assertions 13-16 need an RTX 3090 and are +// stated as outstanding in .superpowers/sdd/task-13-gate-report.md. +func TestPhase6Gate(t *testing.T) { + // 1. Frames stop at the kernel. The §8 pin, and first for a reason: at PC + // sampling rates one frame per PC destroys aggregation and fragments the + // kernel's own block, so every other assertion here is about a label. + // Catches: promoting the PC offset, the stall reason, the source + // location or the attribution quality to a frame, in any spelling. + t.Run("assertion-01-frames-stop-at-the-kernel", TestProjectionAddsNoFrames) + t.Run("assertion-01-pc-is-not-stack-identity", TestProjectionKeepsPCOutOfStackIdentity) + + // 2. A source line is reached from a CPU stack. The Phase 6 exit + // condition, satisfied through labels rather than frames. New here - + // see the test's own comment for what no existing test covered. + t.Run("assertion-02-source-line-from-a-cpu-stack", TestGateASourceLineIsReachedFromACPUStack) + + // 3. No -lineinfo, no invention. + // Catches: synthesizing a nearest line, or a function's first line, for + // a module that carries no line table at all. + t.Run("assertion-03-no-lineinfo-no-invention", TestProjectionEmitsAllFourSrcStatuses) + t.Run("assertion-03-populations-aggregate-at-the-kernel", + TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel) + + // 4. All four gpu_src_status values reachable, each by the fixture that + // should produce it. Catches: a fifth value; a silent default; the two + // "cannot resolve" statuses collapsing into one. + t.Run("assertion-04-four-statuses-in-the-store", TestModuleStoreAllFourStatusesAreReachable) + t.Run("assertion-04-four-statuses-in-the-labels", TestProjectionEmitsAllFourSrcStatuses) + t.Run("assertion-04-status-enum-is-exhaustive", TestSrcStatusesIsExhaustiveAndStable) + t.Run("assertion-04-zero-value-is-not-a-status", TestSrcStatusZeroValueIsNotAStatus) + + // 5. Reconciliation: every emitted PC record lands in exactly one of + // attributed-exact, attributed-kernel, pending, evicted. + // Catches: a join that attaches samples without counting them; a + // refusal path that leaves a group pending without incrementing any + // counter - the easiest and most invisible mistake in that function. + t.Run("assertion-05-reconcile-pending-and-evicted", + TestConformancePCSampleReconciliationCoversPendingAndEvicted) + t.Run("assertion-05-reconcile-attributed-by-kernel", + TestConformancePCSampleReconciliationCoversAttributedByKernel) + t.Run("assertion-05-tier-b-samples-reconcile", TestTierBSamplesReconcile) + t.Run("assertion-05-join-accounts-for-every-group", TestTierBJoinAccountsForEveryGroup) + + // 6. Tier B does not collapse: 10,000 samples across 50 (crc, + // functionIndex) pairs attribute without loss to EvictedPendingSamples. + // Catches: the Task 8a pathology returning - a Tier B sample keyed on + // its (empty) correlation value, collapsing an entire process onto one + // pending entry and evicting the rest. + t.Run("assertion-06-tier-b-does-not-collapse", TestTierBPCSamplesDoNotCollapseOntoOneKey) + t.Run("assertion-06-distinct-function-indexes-split", TestTierBDistinctFunctionIndexSplitsGroups) + + // 7. Ambiguity is marked in its OWN label, and leaves gpu_ambiguous unset. + // Catches: reusing ExecutionView.Ambiguous for PC ambiguity, which would + // emit gpu_join="exact" gpu_ambiguous="true" on one sample - two + // unrelated facts on one boolean, and a counter that stops meaning what + // its name says. + t.Run("assertion-07-ambiguity-has-its-own-label", + TestTierBAmbiguityIsMarkedWithoutTouchingAmbiguous) + t.Run("assertion-07-never-coincides-with-gpu-ambiguous", + TestProjectionKernelAmbiguousNeverCoincidesWithGpuAmbiguous) + + // 8. Tier A disclosure: "true" inside a window, "false" outside, and + // "unknown" - never "false" - when Tier A ran and no window arrived. + // Catches: the one answer that must never be reachable by accident, a + // profile saying "not perturbed" when it means "cannot tell". + t.Run("assertion-08-true-inside-a-burst", TestSerializationMarksExecutionsOverlappingABurst) + t.Run("assertion-08-false-in-a-proven-gap", TestSerializationIsFalseInAProvenGap) + t.Run("assertion-08-unknown-when-no-window-arrived", TestSerializationIsUnknownWhenNoWindowsArrived) + t.Run("assertion-08-false-only-from-positive-evidence", + TestSerializationFalseIsOnlyEverReachedFromPositiveEvidence) + t.Run("assertion-08-label-is-unconditional-and-three-valued", + TestSerializedLabelIsUnconditionalAndHasThreeValues) + + // 9. Cardinality cap: past the ceiling gpu_pc is suppressed and NOTHING + // else, with an exact ProjectionPCLabelsSuppressed. + // Catches: a cap that also drops gpu_stall or gpu_src_* (the coarser, + // more actionable labels), and a suppression that is silent - a profile + // that lost its PC labels looks identical to one that never had any. + t.Run("assertion-09-cap-suppresses-gpu-pc-and-only-gpu-pc", + TestProjectionCapSuppressesGpuPCAndOnlyGpuPC) + t.Run("assertion-09-suppression-is-surfaced", TestProjectionCapIsSurfacedInJoinHealth) + + // 10b. Out-of-scope conditions refuse rather than guess. The graph clause + // is asserted at the level the product implements it - see the test. + t.Run("assertion-10b-two-devices-are-marked", TestTierBMultiDeviceProcessIsMarked) + t.Run("assertion-10b-multidevice-outranks-ambiguity", TestMultiDeviceOutranksAmbiguity) + t.Run("assertion-10b-open-window-is-unknown-never-false", + TestSerializationOpenWindowMakesEverythingFromItsStartUnknownAndNeverFalse) + t.Run("assertion-10b-tier-a-refusal-names-cuda-graphs", + TestSerializedIsRefusedWithoutAnExplicitAcknowledgement) + // The first clause of 10b - "a graph execution makes Tier A refuse to + // start" - is NOT implemented by the product. This pins that, and fails + // when it becomes implementable so the gate is updated rather than left + // claiming an assertion it never made. + t.Run("assertion-10b-graph-refusal-is-outstanding", + TestGateGraphExecutionRefusalIsNotAssertableYet) +} + +// gateCubinCRCs are the CRCs this file stores its fixtures under. The values +// are arbitrary - what matters is that the sample and the store agree, exactly +// as cubin_crc makes them agree on the wire. +const ( + gateCRCLineInfo uint64 = 0x6A7E0001 + gateCRCNoLineInfo uint64 = 0x6A7E0002 +) + +// fixtureSourceLines reads the .cu the cubin fixtures were built from, so an +// assertion about a source LINE can be checked against the source rather than +// against a number someone copied out of a run. +func fixtureSourceLines(t *testing.T, name string) []string { + t.Helper() + b, err := os.ReadFile(fixturePath(name)) + require.NoError(t, err, "fixture source %s", name) + return strings.Split(strings.TrimRight(string(b), "\n"), "\n") +} + +// TestGateASourceLineIsReachedFromACPUStack is gate assertion 2, and the Phase +// 6 exit condition: an instruction sampled inside a GPU kernel is reported at a +// named line of the CUDA source that produced it, on a sample whose frames are +// the CPU call path that launched the kernel. +// +// # Why this is not covered by anything that already exists +// +// Three tests each cover a segment and none covers the join of them: +// +// - TestTierBSampleReachesItsExecution drives a Tier B sample to its +// execution through the module, but the execution has no launch and +// therefore no CPU stack, and it never projects to a pprof sample; +// - TestProjectionEmitsAllFourSrcStatuses reaches "resolved" with real file +// and line, but from a hand-built ExecutionView with no Launch at all; +// - TestProjectionAddsNoFrames has a Launch with a CPU stack, but asserts the +// NEGATIVE - that nothing about the PC reached the frames. +// +// The exit condition is the conjunction: one pprof sample carrying BOTH. It is +// asserted here through a real Timeline, so the module join and the source +// resolution run for real rather than being assumed by a literal ExecutionView. +// +// The line is checked against internal/cubin/testdata/single.cu itself - not +// against a number copied out of a previous run - so "a real line of the +// fixture's source" is a fact about the fixture rather than a constant that +// could go stale with it. +// +// Mutations this catches: a projection that resolves source labels but loses +// the launch's frames (or vice versa); a join that reaches the execution but +// drops PCSamples so nothing projects; a store keyed on something the sample +// does not carry; a resolution that reports a line the cubin's table does not +// contain. +func TestGateASourceLineIsReachedFromACPUStack(t *testing.T) { + const pid = 4242 + b := fixture(t, "single_lineinfo.cubin") + store := NewModuleStore(ModuleStoreConfig{}) + require.NoError(t, store.Put(gateCRCLineInfo, b)) + fnIndex := symIndexOf(t, b, "addOne") + + tl := NewTimeline(TimelineConfig{Modules: store}) + + // A real CPU call path. The privileged half of the gate replaces these + // names with frames the DWARF walker produced from a live process; here + // what matters is that they are the LAUNCH's frames and that they survive + // to the projected sample unchanged. + corr := CorrelationID{Backend: BackendCUPTI, PID: pid, Value: "17"} + require.NoError(t, tl.EmitLaunch(GPUKernelLaunch{ + Correlation: corr, + KernelName: "addOne", + TimeNs: 10, + Launch: LaunchContext{ + PID: pid, + TimeNs: 10, + CPUStack: pp.FramesFromNames([]string{"main", "run_training_step", "cudaLaunchKernel"}), + SamplePeriod: 8, + }, + })) + + // Tier B: no correlation value on the sample. It reaches the execution + // through cubin_crc -> module -> function name, and nothing else. + require.NoError(t, tl.EmitPCSample(GPUPCSample{ + Correlation: CorrelationID{Backend: BackendCUPTI, PID: pid}, + Module: ModuleRef{Backend: BackendCUPTI, CRC: gateCRCLineInfo}, + FunctionIndex: fnIndex, + TimeNs: 20, + PCOffset: 0x10, + StallReason: "long_scoreboard", + Count: 1, + })) + require.NoError(t, tl.EmitExec(GPUKernelExec{ + Correlation: corr, + KernelName: "addOne", + StartNs: 30, + EndNs: 130, + })) + + snap := tl.Snapshot() + require.Len(t, snap.Executions, 1) + view := snap.Executions[0] + require.Len(t, view.PCSamples, 1, + "the sample never reached its execution, so nothing downstream can be the exit condition") + require.NotNil(t, view.Launch, "the execution lost the launch that carries the CPU stack") + + samples, _ := ProjectExecutionsWith(snap, ProjectionConfig{Modules: store}) + require.Len(t, samples, 1) + s := samples[0] + + // --- the CPU stack half. + names := frameNames(s.Stack) + assert.Equal(t, + []string{"main", "run_training_step", "cudaLaunchKernel", FrameLaunch, "[gpu:kernel:addOne]"}, + names, + "the sample's frames must be the launching CPU call path, the boundary marker and the kernel - nothing more and nothing less") + + // --- the source-line half. + require.Equal(t, "resolved", s.Labels["gpu_src_status"], + "the -lineinfo fixture's line table covers this pcOffset; anything else means the store never saw the module or never parsed it") + assert.Equal(t, "single.cu", s.Labels["gpu_src_file"], + "the basename, never the build-host path") + assert.Equal(t, "addOne", s.Labels["gpu_src_func"]) + require.Contains(t, s.Labels, "gpu_src_line") + + // The line names a real line of the fixture's own source. Read from the + // .cu rather than pinned as a constant: a fixture rebuilt from edited + // source would otherwise keep this assertion green while the label pointed + // somewhere else. + line, err := strconv.Atoi(s.Labels["gpu_src_line"]) + require.NoError(t, err) + src := fixtureSourceLines(t, "single.cu") + require.Positive(t, line) + require.LessOrEqual(t, line, len(src), + "gpu_src_line=%d is past the end of single.cu (%d lines): the label does not name a line of the source it claims", + line, len(src)) + body := strings.TrimSpace(src[line-1]) + assert.NotEmpty(t, body, "gpu_src_line names a blank line of single.cu") + t.Logf("assertion 2: %s -> %s:%s %q (stall=%s, pc=%s, attrib=%s)", + strings.Join(names, " -> "), s.Labels["gpu_src_file"], s.Labels["gpu_src_line"], + body, s.Labels["gpu_stall"], s.Labels["gpu_pc"], s.Labels["gpu_pc_attrib"]) + + // And the attribution quality is stated, not implied: one execution of this + // kernel was in the horizon, so the module join is not an inference about + // WHICH invocation. + assert.Equal(t, string(PCAttribKernel), s.Labels["gpu_pc_attrib"]) + assert.NotContains(t, s.Labels, "gpu_ambiguous", + "gpu_ambiguous means a heuristic LAUNCH join and nothing else; this join was exact") +} + +// TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel is the second half of +// gate assertion 3: "the kernel frame is unchanged, so the two populations +// still aggregate together at the kernel level". +// +// TestProjectionEmitsAllFourSrcStatuses already asserts that a no-lineinfo +// sample carries the explicit status and no location. What it does not assert - +// and what the gate's own wording asks for - is that the resolvable and +// unresolvable populations still SHARE A STACK. That is the property that +// decides whether a flame graph of a partially-built-with-lineinfo workload +// shows one kernel block or two, and it is not implied by the labels: a +// projection that appended the source location, or the status, or a +// "[gpu:src:unknown]" placeholder to the frames would satisfy every label +// assertion and split the block in half. +// +// Mutation this catches: any frame that varies with gpu_src_status. +func TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel(t *testing.T) { + withInfo := fixture(t, "single_lineinfo.cubin") + noInfo := fixture(t, "single_nolineinfo.cubin") + st := NewModuleStore(ModuleStoreConfig{Capacity: 8}) + require.NoError(t, st.Put(gateCRCLineInfo, withInfo)) + require.NoError(t, st.Put(gateCRCNoLineInfo, noInfo)) + + // One execution of one kernel, sampled twice: once in a module built with + // -lineinfo and once in a module built without. On real hardware this is a + // process that links one library built each way. + view := pcView(PCAttribKernel, + pcSampleAt(gateCRCLineInfo, symIndexOf(t, withInfo, "addOne"), 0x10), + pcSampleAt(gateCRCNoLineInfo, symIndexOf(t, noInfo, "addOne"), 0x10), + ) + view.Launch = &GPUKernelLaunch{Launch: LaunchContext{ + CPUStack: pp.FramesFromNames([]string{"main"}), + }} + + samples, _ := ProjectExecutionsWith(Snapshot{Executions: []ExecutionView{view}}, + ProjectionConfig{Modules: st}) + require.Len(t, samples, 2) + + assert.Equal(t, "resolved", samples[0].Labels["gpu_src_status"]) + assert.Equal(t, "no-lineinfo", samples[1].Labels["gpu_src_status"]) + for _, forbidden := range []string{"gpu_src_file", "gpu_src_line", "gpu_src_func"} { + assert.NotContains(t, samples[1].Labels, forbidden, + "no-lineinfo must invent nothing: %s rode on a sample with no line table behind it", forbidden) + } + + assert.Equal(t, frameNames(samples[0].Stack), frameNames(samples[1].Stack), + "the resolvable and unresolvable populations must aggregate at the same kernel; a frame that varies with gpu_src_status splits the kernel's own block in two") + assert.Equal(t, []string{"main", FrameLaunch, "[gpu:kernel:addOne]"}, frameNames(samples[0].Stack)) +} + +// TestGateGraphExecutionRefusalIsNotAssertableYet is gate assertion 10b's +// FIRST clause - "a graph execution makes Tier A refuse to start" - pinned as +// an outstanding gap rather than asserted, because the product does not +// implement it. +// +// # What exists and what does not +// +// The wire signal exists: the plan's Task 6 gave the adapter a +// classGraphExec drop class, `internal/gpuabi.DropClassGraphExec` decodes it +// and spells it "graph-exec", and the stub emits one such record so the class +// is reachable from a test. The operator warning names CUDA graphs, and the +// Tier A acknowledgement refusal names them too. +// +// What does not exist is the REFUSAL the plan specifies: +// +// Tier A refuses to start in a process where graph executions have been +// observed, because Tier A's whole claim is exact launch attribution and a +// graph makes that claim false. The refusal is loud and counted, not a +// silent downgrade to Tier B. +// +// Nothing in gpu/, gpuprobe/ or cmd/ consumes DropClassGraphExec. The +// consumer decodes gpu_dropped_v1 into batch.Drops and stops there (see +// Stats.Undecoded's own comment, which says normalizing a drop class is the +// task after Task 7). So there is no counter for graph executions, nothing on +// the Snapshot that names them, no joinhealth anomaly, and no input by which +// PCSamplingRequest could be told about them. Tier A therefore starts happily +// in a graph-using process and produces confident, exact-LOOKING attribution +// of N kernels to one call site - which is finding 4 of the plan, and the +// condition it says must be visible rather than merely out of scope. +// +// # Why this is a passing test rather than a failing one +// +// It is the shape issue #44 used and #45 inverted: an assertion that pins the +// CURRENT state, with the note that fixing the defect must fail it. Writing +// the real assertion now would leave the gate red on a branch that is not +// allowed to change product behaviour; leaving nothing at all would let the +// gate ship claiming twelve assertions while one of them was never written. +// +// When Task 10's refusal lands, this test fails - by name, in the gate's own +// file - and the person landing it replaces it with the real assertion: +// a stub reporting a graph execution makes Tier A refuse to start, loudly and +// counted. +func TestGateGraphExecutionRefusalIsNotAssertableYet(t *testing.T) { + mentionsGraph := func(v any) []string { + var out []string + typ := reflect.TypeOf(v) + for i := range typ.NumField() { + if name := typ.Field(i).Name; strings.Contains(strings.ToLower(name), "graph") { + out = append(out, name) + } + } + return out + } + for _, tc := range []struct { + name string + v any + }{ + {"Snapshot", Snapshot{}}, + {"TimelineDropStats", TimelineDropStats{}}, + {"PCJoinStats", PCJoinStats{}}, + {"TimelineConfig", TimelineConfig{}}, + {"PCSamplingRequest", PCSamplingRequest{}}, + } { + assert.Empty(t, mentionsGraph(tc.v), + "%s now names graph executions (%v). Gate assertion 10b's first clause has become "+ + "assertable: replace this test with the real one - a stub reporting a graph "+ + "execution makes Tier A refuse to start, loudly and counted.", + tc.name, mentionsGraph(tc.v)) + } + + // The half that IS true today, so this test is not purely negative: the + // operator is told, in the refusal they must read before Tier A can run at + // all, that the tier is unavailable where graphs are in use. + _, err := PCSamplingRequest{Flag: "serialized"}.Select() + require.Error(t, err) + assert.Contains(t, err.Error(), "CUDA graphs", + "the only place the graph limitation is stated to an operator is the Tier A acknowledgement refusal; if that text loses it, nothing anywhere names the condition") + warning := strings.Join(PCSamplingStandingWarning(PCSamplingSerialized), "\n") + assert.Contains(t, warning, "CUDA GRAPHS", + "the standing warning must keep naming the third perturbation") + t.Log("gate assertion 10b, first clause: OUTSTANDING - see this test's doc comment and " + + ".superpowers/sdd/task-13-gate-report.md") +} diff --git a/gpuprobe/gate_compose_test.go b/gpuprobe/gate_compose_test.go new file mode 100644 index 0000000..45af8e7 --- /dev/null +++ b/gpuprobe/gate_compose_test.go @@ -0,0 +1,276 @@ +package gpuprobe + +import ( + "os" + "os/exec" + "reflect" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "kernel.org/pub/linux/libs/security/libcap/cap" +) + +// TestPhase6GateConsumerHalf is the consumer-side quarter of the Phase 6 phase +// gate: assertions 10, 10a, 11 and 12 of Task 13. +// +// It is a SECOND gate entry point, in package gpuprobe rather than +// gpuprobe_test, and both halves of that are load-bearing: +// +// - second, because TestStubDrivesThePipelineToPprofWithoutAGPU and the +// end-to-end PC-sampling gate beside it need CAP_BPF, CAP_PERFMON and +// CAP_CHECKPOINT_RESTORE and SKIP without them. Assertions 10, 10a, 11 and +// 12 need no capability at all - the cubin channel is an AF_UNIX socket and +// a memfd, the decode path is pure Go - so putting them behind the +// privileged skip would be putting four assertions behind a gate nobody +// without a GPU box can run; +// - in-package, because the tests these compose are in-package. Calling them +// is the point: the plan asks the gate to COMPOSE the assertions the tasks +// already made rather than to write second copies of them, and a second +// copy of an assertion is two assertions of one fact that drift apart. +// +// See gpu/gate_test.go for assertions 1-9 and 10b, and +// gpuprobe/gate_test.go for the privileged end-to-end run. +func TestPhase6GateConsumerHalf(t *testing.T) { + // 10. The cubin channel cannot touch enrolment, in BOTH orders - + // including offers flooded AHEAD of an enrolment, which is the + // direction a shared socket fails and the easy direction is not. + // CubinsThrottled non-zero while UnwindEnrollThrottled is unchanged, + // and the enrolment still succeeds. + // + // Catches: moving cubin offers onto the enrolment listener, its + // goroutine, its Accept loop or its admission bucket. Any of those + // restores issue #49's ~38% stack loss on a module-heavy workload, + // silently, with only UnwindEnrollThrottled moving. + t.Run("assertion-10-cubins-cannot-starve-or-throttle-enrolment", + TestFloodingTheCubinChannelCannotStarveOrThrottleAnEnrolment) + t.Run("assertion-10-the-buckets-are-separate-objects", TestTheCubinAdmissionBucketIsItsOwn) + t.Run("assertion-10-they-are-not-the-same-socket", + TestTheCubinAddressIsASiblingOfTheRendezvousAndNotTheSameSocket) + // The regression that keeps them apart forever. The enrolment handler must + // never read from its connection: a read would block until the producer's + // 2s budget expired, turning every rendezvous into a 2s stall ending in + // kEnrollError. Discriminating an offer from an enrolment on one socket + // REQUIRES such a read, so this is the assertion that makes the shared + // socket unbuildable rather than merely unbuilt. + t.Run("assertion-10-enrolment-performs-no-read", TestAnEnrolmentCompletesWithNoReadOnThatConnection) + + // 10a. Seals are enforced: a memfd missing ANY required seal is rejected, + // counted in CubinsRejectedUnsealed, and never mapped. + // + // Catches: dropping a seal from the required set, or - worse - a + // fallback that maps it anyway. Without F_SEAL_SHRINK a peer can + // ftruncate under our mmap and SIGBUS the agent; without F_SEAL_WRITE + // the ELF mutates under the parser mid-parse. Falling back is how a + // defended path becomes an undefended one. + t.Run("assertion-10a-each-missing-seal-is-rejected-and-never-mapped", + TestEachRequiredSealMissingInTurnIsRejectedAndNeverMapped) + t.Run("assertion-10a-a-plain-fd-is-not-a-sealed-memfd", TestADescriptorThatIsNotASealedMemfdIsRefused) + t.Run("assertion-10a-the-required-seals-are-spelt-out", TestSealNamesAreSpeltOut) + + // 11. The capability set does not grow: no cap_sys_admin, anywhere. + t.Run("assertion-11-no-cap-sys-admin", TestPhase6GateBinaryDoesNotAskForCapSysAdmin) + + // 12. Stats.Undecoded is zero for every kind this phase decodes. + // + // Catches: a KIND_* added on one side of the wire and not the other, + // which is silent loss unless counted - and, in the other direction, a + // decode arm removed, which would put a kind back into the default arm + // while every other counter still read healthy. + t.Run("assertion-12-the-five-pc-kinds-are-decoded", + TestTheFivePCSamplingKindsAreDecodedNotCountedUndecoded) + t.Run("assertion-12-a-healthy-tier-b-run-loses-nothing", + TestHealthyTierBRunLeavesEveryNewLossCounterAtZero) + t.Run("assertion-12-unknown-kinds-are-still-counted", TestUndecodedKindsAreCountedNotDropped) + t.Run("assertion-12-kindmax-matches-the-bpf-object", TestEmbeddedProgramIsUprobeMulti) + t.Run("assertion-12-every-cookie-has-a-sized-kind", TestBPFSizesEveryKindCookieForInstalls) + + // Not one of the twelve: the wiring gap that stops assertion 2 from being + // reachable by the shipping product. Pinned here so it fails the moment it + // is closed, rather than being remembered only in a report. + t.Run("outstanding-cubin-transport-does-not-feed-the-module-store", + TestGateTheCubinTransportDoesNotYetFeedTheModuleStore) +} + +// TestPhase6GateBinaryDoesNotAskForCapSysAdmin is gate assertion 11, the +// standing Phase 1 assertion: the capability set this pipeline runs under is +// cap_bpf,cap_perfmon,cap_checkpoint_restore and it does not grow. +// +// # Why this is worth a test rather than a README line +// +// CAP_SYS_ADMIN is the difference between an agent that can run as one pod +// among many and an agent that is effectively root on the node. Nothing about +// losing it is loud: the two ways to acquire the requirement are both quiet +// one-liners a reviewer would wave through - +// +// - attaching with link.Uprobe instead of link.UprobeMulti, which routes +// through the perf_uprobe PMU (measured: needs CAP_SYS_ADMIN, while the BPF +// link does not - see Attach's doc comment and +// TestEmbeddedProgramIsUprobeMulti, composed above); +// - reading the producer's address space to follow gpu_module_load_v1's +// bytes_ptr, which needs /proc//mem or process_vm_readv and therefore +// CAP_SYS_PTRACE, and which the whole cubin channel exists to avoid. +// +// Both would be found on a developer box that runs the tests as root, where +// everything works and nothing says why. So this asserts the negative from two +// independent directions, because either one alone is escapable: +// +// 1. the FILE capabilities of the test binary itself, read with getcap(8) - +// which is the plan's own wording, and which is the thing a CI job or a +// developer following the plan's setcap line actually installs; +// 2. this process's own Permitted set, which is where a capability acquired +// any other way (a setuid wrapper, an inherited ambient set) would show up. +// +// Running as root makes (2) vacuous - root has the full set - so it is skipped +// there rather than asserted falsely, and (1) still runs, because a file +// capability set naming cap_sys_admin is wrong whoever executes it. +func TestPhase6GateBinaryDoesNotAskForCapSysAdmin(t *testing.T) { + self, err := os.Executable() + require.NoError(t, err) + + // --- 1. the file capabilities of this very binary. + // + // cap.GetFile rather than shelling out to getcap(8): it reads the same + // security.capability xattr getcap prints, needs no external tool, and + // cannot be defeated by getcap being absent from a container image - which + // would otherwise turn this assertion into a skip on exactly the machines + // that most need it. + set, err := cap.GetFile(self) + switch { + case err != nil: + // No file capabilities at all. That is the ordinary state for an + // unprivileged `go test` run and for a sudo run, and it is the + // strongest possible form of "no cap_sys_admin". + t.Logf("no file capabilities on %s (%v): nothing to grant cap_sys_admin", self, err) + default: + for _, flag := range []cap.Flag{cap.Effective, cap.Permitted, cap.Inheritable} { + have, gerr := set.GetFlag(flag, cap.SYS_ADMIN) + require.NoError(t, gerr) + assert.False(t, have, + "the gate binary's file capabilities name cap_sys_admin in the %v set (%s): "+ + "this pipeline runs on cap_bpf,cap_perfmon,cap_checkpoint_restore and the "+ + "capability set does not grow. The two ways to acquire this requirement are "+ + "attaching through the perf_uprobe PMU instead of a uprobe_multi BPF link, "+ + "and reading the producer's address space to follow bytes_ptr", + flag, set) + } + t.Logf("file capabilities on %s: %s", self, set) + } + + // --- 2. this process's own set. + if os.Geteuid() == 0 { + t.Log("running as root: the process capability half of this assertion is vacuous and is skipped; " + + "the file-capability half above still ran") + return + } + proc := cap.GetProc() + require.NotNil(t, proc) + for _, flag := range []cap.Flag{cap.Effective, cap.Permitted} { + have, gerr := proc.GetFlag(flag, cap.SYS_ADMIN) + require.NoError(t, gerr) + assert.False(t, have, + "this process holds cap_sys_admin in the %v set (%s); the gate is supposed to prove "+ + "the pipeline works WITHOUT it, and a run that holds it cannot", + flag, proc) + } + t.Logf("process capabilities: %s", proc) +} + +// TestGetcapAgreesWithTheLibraryReading cross-checks the assertion above +// against the tool the plan's wording actually names. +// +// Assertion 11 is written as "getcap on the gate binary shows no +// cap_sys_admin". The test above reads the same xattr through libcap rather +// than through getcap(8), which is stricter in one direction (it cannot be +// skipped by the tool being missing) and weaker in another: if libcap's +// reading and getcap's ever disagreed, the assertion would be about something +// other than what an operator sees. So when getcap IS available, run it and +// require the two to agree. +// +// Skipped, not failed, when getcap is absent: the assertion it backs has +// already run through libcap, and a missing binutils-ish tool is not a defect +// in this pipeline. +func TestGetcapAgreesWithTheLibraryReading(t *testing.T) { + path, err := exec.LookPath("getcap") + if err != nil { + t.Skip("getcap(8) unavailable; TestPhase6GateBinaryDoesNotAskForCapSysAdmin " + + "reads the same xattr through libcap and has already run") + } + self, err := os.Executable() + require.NoError(t, err) + + out, err := exec.Command(path, self).CombinedOutput() + require.NoError(t, err, "getcap %s: %s", self, out) + text := string(out) + t.Logf("getcap %s -> %q", self, strings.TrimSpace(text)) + assert.NotContains(t, strings.ToLower(text), "cap_sys_admin", + "getcap names cap_sys_admin on the gate binary") +} + +// TestGateTheCubinTransportDoesNotYetFeedTheModuleStore pins the gap that +// stops the Phase 6 exit condition from being reachable by the shipping +// product, so that it is a named, self-invalidating fact rather than prose in +// a report nobody re-reads. +// +// # The gap +// +// Task 3 built the cubin channel and Task 4 built gpu.ModuleStore - the store +// that turns (cubin_crc, functionIndex, pcOffset) into a source line and is +// "the single place gpu_src_status is decided". Nothing connects them: +// +// - Attach calls newCubinListener(cfg, nil), and a nil sink becomes +// memCubinStore - a bounded map of CRC to bytes with no line table, no LRU +// and no Resolve. Its own comment says "Task 4 replaces it"; +// - gpuprobe.Config has no field by which a caller could supply one, and +// gpu.ModuleStore does not implement cubinSink (Put/HasCubin versus +// PutCubin/HasCubin) even if it had; +// - cmd/gpu-cuda-profile builds neither: it calls ProjectExecutionsWith with +// a zero ProjectionConfig and NewTimeline without Modules. +// +// So on hardware today, every cubin is received, sealed, verified, size- and +// identity-checked, stored - and then never read. Every PC sample in a real +// profile reads gpu_src_status="no-module", and gate assertion 2 (a source +// line reached from a CPU stack) cannot be satisfied end to end by the product +// as shipped. TestStubDrivesPCSamplingToPprofWithoutAGPU therefore builds the +// store itself, from the CRCs the producer declared, and says so. +// +// This is the ONE hop between the transport and the labels, and both ends of +// it are built and tested. It is a wiring task, not a design gap. +// +// # Why a passing test rather than a failing one +// +// Same reason as gpu's TestGateGraphExecutionRefusalIsNotAssertableYet: the +// gate must not ship red, and it must not ship silently short of an assertion +// either. When the hop is wired, this test fails - by name, in the gate's own +// file - and the person wiring it deletes it and drops the injection from the +// end-to-end gate. +func TestGateTheCubinTransportDoesNotYetFeedTheModuleStore(t *testing.T) { + // The default sink a real Attach installs. + l, err := newCubinListener(Config{ShimPath: selfExe(t)}, nil) + require.NoError(t, err) + defer func() { _ = l.close() }() + + _, isPlaceholder := l.sink.(*memCubinStore) + assert.True(t, isPlaceholder, + "the cubin listener's default sink is no longer the placeholder memCubinStore (it is %T). "+ + "If that is gpu.ModuleStore, gate assertion 2 is now reachable end to end: delete this "+ + "test and drop the PC-sample injection from TestStubDrivesPCSamplingToPprofWithoutAGPU.", + l.sink) + + // And no caller can supply one, which is why the default is the whole + // story rather than merely a default. + typ := reflect.TypeOf(Config{}) + for i := range typ.NumField() { + name := strings.ToLower(typ.Field(i).Name) + assert.NotContains(t, name, "module", + "gpuprobe.Config grew a field naming a module store (%s); the hop may now be wired - see this test's doc comment", + typ.Field(i).Name) + assert.NotContains(t, name, "store", + "gpuprobe.Config grew a store field (%s); the hop may now be wired - see this test's doc comment", + typ.Field(i).Name) + } + t.Log("cubin transport -> gpu.ModuleStore: OUTSTANDING - the bytes arrive and stop at " + + "memCubinStore; see .superpowers/sdd/task-13-gate-report.md") +} diff --git a/gpuprobe/gate_test.go b/gpuprobe/gate_test.go index 7420d49..c3be386 100644 --- a/gpuprobe/gate_test.go +++ b/gpuprobe/gate_test.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "debug/elf" + "fmt" "os" "os/exec" "path/filepath" @@ -21,6 +22,9 @@ import ( "github.com/dpsoft/perf-agent/gpu" "github.com/dpsoft/perf-agent/gpuprobe" + "github.com/dpsoft/perf-agent/internal/cubin" + "github.com/dpsoft/perf-agent/internal/gpuabi" + pp "github.com/dpsoft/perf-agent/pprof" "github.com/dpsoft/perf-agent/symbolize" "github.com/dpsoft/perf-agent/unwind/ehcompile" ) @@ -1238,3 +1242,802 @@ func TestADlopenedProducerEnrollsBeforeItsFirstLaunch(t *testing.T) { // proving nothing, which is the failure this whole test exists to // correct; the DWARF assertion above is the derived one. } + +// --------------------------------------------------------------------------- +// Phase 6: the PC-sampling gate +// --------------------------------------------------------------------------- + +// gateCRCAbsent is a CRC no cubin is ever stored under, so a PC sample +// carrying it must read gpu_src_status="no-module". The two REAL fixture CRCs +// are not constants here: they are read out of the producer's own report and +// cross-checked against the bytes, so the number the store keys on is the +// number the producer put on the wire rather than one this test invented. See +// gateModuleCRCs. +const gateCRCAbsent uint64 = 0x6A7E0003 + +// TestStubDrivesPCSamplingToPprofWithoutAGPU is the end-to-end half of the +// Phase 6 phase gate (Task 13, assertions 1-5, 8, 9 and 12), driven by the +// same GPU-free producer TestStubDrivesThePipelineToPprofWithoutAGPU uses. +// +// It is a SECOND test rather than an extension of that one, deliberately. That +// test asserts an exact sampled count, reached-root == dwarf, zero abandoned, +// NoTables == 0 and a dozen other equalities that describe a run with PC +// sampling OFF - `require.Len(samples, len(snap.Executions))` among them, which +// is true precisely because the stub emits no PC samples there. Turning PC +// sampling on inside it would have meant weakening those. So the baseline stays +// exactly as it was and this runs the same producer in the PC-sampling +// configuration beside it. +// +// # What comes off the wire, and what does not +// +// Off the wire, from a real producer through a real uprobe_multi link: +// +// - 500 launches and 500 executions, 58 of the launches carrying a CPU stack +// walked through two -fomit-frame-pointer frames using this consumer's own +// compiled CFI, symbolized against the live process; +// - two real checked-in cubins, over the cubin channel, as sealed memfds +// passed by SCM_RIGHTS; +// - 64 PC-sample records, a stall-reason map, a config record and Tier A +// sampling windows. +// +// NOT off the wire, and this is a finding rather than a shortcut - see +// .superpowers/sdd/task-13-gate-report.md: +// +// The stub's PC records cannot be attributed to anything. Their cubin_crc is +// a pair of synthetic constants (shim/stub/stub.cc kStubCubinCRC = +// {0xC0FFEE01, 0xC0FFEE02}) unrelated to the cubins the same run delivers +// over the cubin channel, and their correlation is 0 in every tier. Tier B +// attribution runs crc -> module -> function name -> the execution's +// KernelName, and the stub's kernel names are "kernel_1111"/"kernel_2222" +// while the fixtures' only functions are the CUDA kernels they were compiled +// from. So neither join path can fire: every one of those 64 records is +// correctly counted as pending, and the gate asserts that exactly. +// +// Assertions 2, 3, 4 and 9 need a PC sample that DOES reach an execution, so +// the gate supplies those itself at Timeline.EmitPCSample - the same entry +// point the consumer calls - on correlations that a real, wire-delivered, +// stack-carrying launch is known to occupy. Everything downstream of that entry +// point is product code: the join, the module store's four-valued resolution, +// the projection's label set, the cardinality budget. What the injection skips +// is the consumer's decode arm for KIND_PC, and that is asserted separately and +// exactly by the 64 records above. +// +// # Assertion 11 is not here +// +// getcap on the gate binary is asserted in gate_compose_test.go, where it runs +// without capabilities. Asserting "this pipeline needs no cap_sys_admin" only +// on machines that hold enough privilege to run this test would be asserting it +// in the one place it cannot be checked usefully. +func TestStubDrivesPCSamplingToPprofWithoutAGPU(t *testing.T) { + if !hasGateCaps() { + t.Skip("needs CAP_BPF, CAP_PERFMON and CAP_CHECKPOINT_RESTORE " + + "(the last so blazesym can follow /proc//map_files/); " + + "sudo setcap cap_bpf,cap_perfmon,cap_checkpoint_restore+ep . " + + "gpu/gate_test.go and gpuprobe/gate_compose_test.go assert the same twelve " + + "points without privilege; this adds the end-to-end run") + } + built := filepath.Join("..", "shim", "perfagent-gpu-fpless") + requireBuilt(t, built) + requireFPLess(t, built) + stub := privateStubCopy(t, built) + + // The module store. It is built HERE, and filled after the run from the + // CRCs the PRODUCER declared, rather than by the consumer - and that is + // the second finding this gate records rather than papers over: + // + // The cubin listener's sink is gpuprobe's own bounded memCubinStore + // (gpuprobe/cubin.go). Nothing in gpuprobe, in gpu, or in + // cmd/gpu-cuda-profile ever hands a gpu.ModuleStore to gpuprobe.Config + // or to gpu.ProjectionConfig. The bytes cross the socket, are sealed, + // verified and stored - and stop there. As shipped, every PC sample in a + // real profile therefore reads gpu_src_status="no-module". + // + // CubinsReceived and snap.Modules are asserted below so the transport half + // is still proven end to end on this run, and the missing hop is one named + // gap rather than an invisible one. + lineInfo := readFixture(t, "single_lineinfo.cubin") + noLineInfo := readFixture(t, "single_nolineinfo.cubin") + store := gpu.NewModuleStore(gpu.ModuleStoreConfig{}) + lineIdx := fixtureSymIndex(t, lineInfo, "addOne") + noLineIdx := fixtureSymIndex(t, noLineInfo, "addOne") + + sym, err := symbolize.NewLocalSymbolizer() + require.NoError(t, err) + defer func() { _ = sym.Close() }() + + // PCSamplingSerialized, not the zero value: the tier is what decides + // whether an execution with no covering window reads "unknown" or "false", + // and a Timeline told nothing would answer "false" for every execution in + // this run - correctly, since nothing would then have been serialized, and + // uselessly, since the producer IS emitting windows. + timeline := gpu.NewTimeline(gpu.TimelineConfig{ + PCSampling: gpu.PCSamplingSerialized, + Modules: store, + }) + c, err := gpuprobe.Attach(gpuprobe.Config{ + ShimPath: stub, + Backend: gpu.GPUBackendID("stub"), + Sink: timeline, + Symbolizer: sym, + }) + require.NoError(t, err) + defer func() { _ = c.Close() }() + require.True(t, c.Stats().CubinsListening, + "the cubin channel did not bind, so no module can arrive and every source label would read no-module for a reason that has nothing to do with this phase: %q", + c.Stats().CubinsLastError) + + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { done <- c.Run(ctx) }() + + const ( + wantSampled = 58 // gpuabi.SampleSchedule(500, 8, DefaultSampleSeed) + wantPC = 64 + wantWindows = 4 + wantCubins = 2 + wantDropRecs = 4 // one gpu_dropped_v1 per drop class, from the stub + ) + cmd := exec.Command(stub, "500", "1000", "8", "10000") + cmd.Env = append(os.Environ(), + // The tier is the OUTER gate in the stub exactly as in the CUPTI + // adapter (shim/core/pctier.h): with it off, none of the four + // PC-sampling probes fires whatever the knobs below say. + "PERFAGENT_GPU_PC_SAMPLING=serialized", + "PERFAGENT_STUB_PC_SAMPLES="+strconv.Itoa(wantPC), + "PERFAGENT_STUB_SAMPLING_WINDOWS="+strconv.Itoa(wantWindows), + "PERFAGENT_STUB_CUBINS="+ + mustAbs(t, fixturePath("single_lineinfo.cubin"))+":"+ + mustAbs(t, fixturePath("single_nolineinfo.cubin")), + ) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + release, err := cmd.StdinPipe() + require.NoError(t, err) + require.NoError(t, cmd.Start()) + defer func() { _ = release.Close() }() + + // Hold the producer open until every sampled stack has been symbolized + // against its still-live /proc//maps, exactly as the baseline gate + // does. The cubins are waited for too: the stub offers them from its drain + // thread before its first launch, so they are the earliest thing to + // arrive, and a run that started asserting before they landed would report + // "no-module" for a scheduling reason. + deadline := time.Now().Add(20 * time.Second) + for { + st := c.Stats() + if st.SampledLaunches >= wantSampled && st.CubinsReceived >= wantCubins { + break + } + if time.Now().After(deadline) { + _ = release.Close() + _ = cmd.Process.Kill() + _ = cmd.Wait() + cancel() + <-done + t.Fatalf("timed out: sampled=%d/%d cubins=%d/%d. stats: %+v stderr: %s", + st.SampledLaunches, wantSampled, st.CubinsReceived, wantCubins, st, stderr.String()) + } + time.Sleep(5 * time.Millisecond) + } + stubPID := cmd.Process.Pid + require.NoError(t, release.Close()) + require.NoError(t, cmd.Wait(), "stdout: %s stderr: %s", stdout.String(), stderr.String()) + stubErr := stderr.String() + + // The CRC the producer declared for each fixture, from its own report, and + // the store keyed on those rather than on numbers this test chose. + // + // This is the offline half of hardware assertion 13 ("cuptiGetCubinCrc() + // over the received copy equals the PC records' cubinCrc"). The stub's + // crc is FNV-1a rather than CUPTI's unpublished polynomial, but the + // property under test is the same one and it is the one the join needs: + // ONE number identifies one set of bytes, and the same number reaches both + // ends. Recomputing it here over the checked-in fixture proves the number + // is a function of the bytes and not of the load order, the module id or + // the path. + crcs := gateModuleCRCs(t, stubErr) + require.Len(t, crcs, 2, "the producer reported %d captured modules, not 2: %s", len(crcs), stubErr) + gateCRCLineInfo, gateCRCNoLineInfo := crcs[0], crcs[1] + assert.Equal(t, stubCubinCRC(lineInfo), gateCRCLineInfo, + "the CRC the producer declared for single_lineinfo.cubin is not a content hash of those bytes, so it cannot identify them on the wire") + assert.Equal(t, stubCubinCRC(noLineInfo), gateCRCNoLineInfo, + "same, for single_nolineinfo.cubin") + assert.NotEqual(t, gateCRCLineInfo, gateCRCNoLineInfo, + "two different cubins collided on one CRC, which would make them indistinguishable to the store") + require.NoError(t, store.Put(gateCRCLineInfo, lineInfo)) + require.NoError(t, store.Put(gateCRCNoLineInfo, noLineInfo)) + + // The tail: PC batches, the stall map, the config record, the windows and + // the drop records are all flushed after the last launch, so they are the + // last things on the wire. + waitFor(t, 10*time.Second, func() bool { + st := c.Stats() + return st.PCSamplesDecoded >= wantPC && + st.SamplingWindowsDecoded > 0 && + st.ConfigsDecoded > 0 && + st.PendingStallSamples == 0 + }, func() string { + st := c.Stats() + return fmt.Sprintf("pc=%d/%d windows=%d configs=%d pending-stall=%d stderr: %s", + st.PCSamplesDecoded, wantPC, st.SamplingWindowsDecoded, st.ConfigsDecoded, + st.PendingStallSamples, stubErr) + }) + cancel() + <-done + + stats := c.Stats() + t.Logf("stub stderr:\n%s", stubErr) + t.Logf("pc sampling: pc=%d modules=%d stall-names=%d windows=%d(open=%d) configs=%d cubins=%d/%dB undecoded=%d", + stats.PCSamplesDecoded, stats.ModulesDecoded, stats.StallNamesLearned, + stats.SamplingWindowsDecoded, stats.SamplingWindowsOpen, stats.ConfigsDecoded, + stats.CubinsReceived, stats.CubinBytesReceived, stats.Undecoded) + + // ---- the transport, and the producer's own account of it -------------- + assert.Contains(t, stubErr, "pc_sampling=serialized", + "the producer did not take the tier it was handed, so nothing below describes Tier A") + assert.Contains(t, stubErr, "launch_dropped=0") + assert.Contains(t, stubErr, "exec_dropped=0") + assert.Zero(t, stats.SequenceGaps, "no batch may be lost silently") + assert.Zero(t, stats.Malformed, "reasons: %v", stats.DecodeFailures) + assert.Zero(t, stats.KernelDropped) + assert.Zero(t, stats.SinkRejected) + + // Assertion 12. Undecoded is zero for every kind THIS PHASE DECODES, and + // gpu_dropped_v1 is not one of them: it is decoded into batch.Drops and + // carried, and normalizing a drop class into an operator-visible number is + // the consumer task after this one (see Stats.Undecoded's own comment). The + // stub emits exactly one record per drop class when PC sampling is on, so + // this is an equality rather than a tolerance - a sixth undecoded record + // would mean a kind arrived that nothing on this side knows about, which is + // silent loss. + assert.Equal(t, uint64(wantDropRecs), stats.Undecoded, + "the only undecoded records in a PC-sampling stub run are the four gpu_dropped_v1 class records; anything else is a KIND_* added on one side of the wire and not the other") + assert.Equal(t, uint64(wantPC), stats.PCSamplesDecoded, + "every PC record the stub emitted must have been decoded, not carried") + assert.Equal(t, uint64(wantCubins), stats.CubinsReceived, + "both checked-in cubins must have crossed the cubin channel: err=%q", stats.CubinsLastError) + assert.Equal(t, uint64(2), stats.ModulesDecoded, + "gpu_module_load_v1 announces the load; the bytes travel separately, and both must arrive") + assert.Zero(t, stats.CubinsRejectedUnsealed, "a sealed memfd was refused: %q", stats.CubinsLastError) + assert.Zero(t, stats.CubinsRejectedTooLarge) + assert.Zero(t, stats.CubinsRejectedMalformed) + assert.Zero(t, stats.CubinsRejectedUnauthorized) + assert.Zero(t, stats.CubinsThrottled, "two offers cannot exhaust the cubin bucket") + // The isolation property, on a live run rather than in the unit test that + // proves it structurally (gate_compose_test.go assertion 10): real cubin + // traffic crossed while a real enrolment happened, and the enrolment's own + // counters are untouched. + assert.Zero(t, stats.UnwindEnrollThrottled, + "cubin traffic spent an enrolment admission token: %q", stats.UnwindEnrollLastError) + assert.Equal(t, uint64(1), stats.UnwindEnrollConfirmed) + assert.Zero(t, stats.StacksWalkedNoTables, + "a capture was walked with no CFI tables: enroll requests=%d confirmed=%d err=%q", + stats.UnwindEnrollRequests, stats.UnwindEnrollConfirmed, stats.UnwindEnrollLastError) + + // Stall names: the stub emits the map AFTER the batches on purpose, so + // these two zeros are the assertion that the consumer held the samples + // rather than rendering "stall#17" or dropping them. + assert.Zero(t, stats.StallNamesMissing, + "a PC sample carried a stall index the map never named; the label would then be empty") + assert.Zero(t, stats.PendingStallSamples, + "PC samples are still parked waiting for a stall name that has arrived") + // The gauge, not the counter: the stall map replays on the attach edge + // (spec §6.1's replay contract), so StallNamesLearned counts RECORDS and a + // replayed map legitimately doubles it. What must be exactly 8 is the + // number of distinct indexes the table ends up holding. + assert.Equal(t, 8, stats.KnownStallNames, + "the stub's synthetic GA102 table has 8 entries; a shortfall means a name was evicted or never interned") + assert.GreaterOrEqual(t, stats.StallNamesLearned, uint64(8)) + assert.Zero(t, stats.StallNamesEvicted, + "an 8-entry table cannot overflow the default bound; an eviction here costs PC samples their stall label") + + // Tier B's own signature, kept apart from the aggregate: every PC record + // in CONTINUOUS collection has correlation 0, so ZeroCorrelation stops + // being an anomaly signal for that population and this counter is what + // carries it instead. + assert.Equal(t, uint64(wantPC), stats.PCSamplesWithoutCorrelation, + "the stub emits correlation 0 on every PC record, in both tiers") + + // ---- the injected samples --------------------------------------------- + // + // See the doc comment: the stub cannot produce a PC record that attributes + // to anything, so assertions 2, 3, 4 and 9 are driven from records the gate + // emits into the same Timeline the consumer emits into, on correlations + // that a wire-delivered, stack-carrying launch occupies. + // + // SampleSchedule replays the shim's own sampler exactly (it is pinned + // against it by TestTheGoReplicaMatchesTheShimSampler), so these + // correlations are not guesses: launch ordinal N carries correlation N+1 + // and, if N is in the schedule, a CPU stack. + sched := gpuabi.SampleSchedule(500, 8, gpuabi.DefaultSampleSeed) + require.Equal(t, wantSampled, len(sched)) + corrOf := func(i int) gpu.CorrelationID { + return gpu.CorrelationID{ + Backend: gpu.GPUBackendID("stub"), + PID: uint32(stubPID), + Value: strconv.FormatUint(sched[i]+1, 10), + } + } + inject := func(corr gpu.CorrelationID, crc uint64, fnIndex uint32, pcOffset uint64) { + t.Helper() + require.NoError(t, timeline.EmitPCSample(gpu.GPUPCSample{ + Correlation: corr, + Module: gpu.ModuleRef{Backend: gpu.GPUBackendID("stub"), CRC: crc}, + FunctionIndex: fnIndex, + TimeNs: 1, + PCOffset: pcOffset, + StallReason: "long_scoreboard", + Count: 1, + })) + } + // One per gpu_src_status, each on its own stack-carrying execution. + inject(corrOf(0), gateCRCLineInfo, lineIdx, 0x10) // resolved + inject(corrOf(1), gateCRCNoLineInfo, noLineIdx, 0x10) // no-lineinfo + inject(corrOf(2), gateCRCAbsent, 0, 0x10) // no-module + inject(corrOf(3), gateCRCLineInfo, lineIdx, 0x180) // unmapped: past the function + // And a spray of distinct offsets for the cardinality cap. + const capSpray = 40 + for i := range capSpray { + inject(corrOf(4), gateCRCLineInfo, lineIdx, 0x1000+uint64(i)*16) + } + injected := 4 + capSpray + + snap := timeline.Snapshot() + require.Len(t, snap.Executions, 500, "500 launches + 500 execs, exactly, none lost") + + // The wire's own account of the same two modules: gpu_module_load_v1 + // announces THAT a module loaded, with its CRC and size, while the bytes + // travel the cubin channel. Both must name the same modules, or the + // announcement and the payload describe different things and the CRC join + // is meaningless. + gotCRCs := map[uint64]uint64{} + for _, m := range snap.Modules { + gotCRCs[m.Ref.CRC] = m.SizeBytes + } + assert.Equal(t, map[uint64]uint64{ + gateCRCLineInfo: uint64(len(lineInfo)), + gateCRCNoLineInfo: uint64(len(noLineInfo)), + }, gotCRCs, + "the decoded gpu_module_load_v1 records do not name the same (crc, size) pairs the producer reported for the bytes it sent") + + // ---- assertion 5: reconciliation -------------------------------------- + // + // Every PC record that reached the sink lands in exactly one of + // attributed-exact, attributed-kernel, still-pending (in either store) or + // evicted (from either store). The two pending stores are separate on + // purpose - a Tier B eviction storm must be distinguishable from a Tier A + // one - so both terms are in the identity. + accepted := uint64(wantPC) + uint64(injected) + assert.Equal(t, accepted, + snap.AttributedPCSamples+ + uint64(snap.PendingSamples)+snap.Dropped.EvictedPendingSamples+ + uint64(snap.PendingModuleSamples)+snap.Dropped.EvictedPendingModuleSamples, + "a PC sample went unaccounted for: attributed=%d pending=%d evicted=%d pending-module=%d evicted-module=%d of %d accepted", + snap.AttributedPCSamples, snap.PendingSamples, snap.Dropped.EvictedPendingSamples, + snap.PendingModuleSamples, snap.Dropped.EvictedPendingModuleSamples, accepted) + assert.Equal(t, snap.AttributedPCSamples, snap.PCJoin.AttributedTotal(), + "AttributedExact + AttributedKernel must account for every attributed sample: %+v", snap.PCJoin) + assert.Equal(t, uint64(injected), snap.PCJoin.AttributedExact, + "every injected sample carries a correlation a wire-delivered execution occupies, so all of them take the exact path") + // And the stub's own 64, which can attribute to nothing - see the doc + // comment. Asserted as an equality rather than tolerated as a shortfall: + // if the stub is ever given real CRCs and real kernel names this number + // changes, and the gate should say so rather than quietly pass. + assert.Equal(t, wantPC, snap.PendingModuleSamples, + "the stub's PC records carry synthetic CRCs and kernel names that no cubin can name, so every one of them must remain pending and counted - never attached to a plausible neighbour") + assert.Positive(t, snap.PCJoin.GroupsUnresolvedName, + "a group whose (crc, functionIndex) names nothing must be counted as such, not silently skipped") + assert.Equal(t, snap.PCJoin.GroupsExamined(), + snap.PCJoin.GroupsJoined+uint64(snap.PendingModuleGroups), + "every pending group must be joined or left pending for exactly one counted reason: %+v", snap.PCJoin) + + // ---- assertion 8: Tier A disclosure ----------------------------------- + assert.Equal(t, uint64(len(snap.Executions)), + snap.ExecutionsSerialized+snap.ExecutionsNotSerialized+snap.ExecutionsSerializationUnknown, + "the three gpu_serialized outcomes must partition the executions exactly") + assert.Positive(t, snap.SamplingWindowsReceived, + "Tier A was selected and the producer said it emitted windows, but none reached the disclosure store") + assert.Positive(t, snap.ExecutionsSerialized, + "the stub's bursts bracket about half its executions; not one was marked perturbed") + assert.Positive(t, snap.ExecutionsNotSerialized, + "not one execution fell in a proven gap between bursts, so \"false\" is unreachable on this run and the three-way split is not being exercised") + t.Logf("serialization: true=%d false=%d unknown=%d over %d windows (held=%d open=%d)", + snap.ExecutionsSerialized, snap.ExecutionsNotSerialized, snap.ExecutionsSerializationUnknown, + snap.SamplingWindowsReceived, snap.SamplingWindowsHeld, snap.SamplingWindowsOpen) + + // The second half of assertion 8, and the half that matters: Tier A + // selected with NO window arriving must read "unknown" on every execution + // and "false" on none. Driven off this run's own executions rather than + // off synthetic ones, so the population is identical and only the evidence + // differs. + t.Run("tier A with no windows is unknown, never false", func(t *testing.T) { + blind := gpu.NewTimeline(gpu.TimelineConfig{PCSampling: gpu.PCSamplingSerialized}) + for _, v := range snap.Executions { + require.NoError(t, blind.EmitExec(v.Exec)) + } + bs := blind.Snapshot() + require.Len(t, bs.Executions, len(snap.Executions)) + assert.Equal(t, uint64(len(bs.Executions)), bs.ExecutionsSerializationUnknown, + "Tier A ran and no window arrived, so nothing can be shown unperturbed") + assert.Zero(t, bs.ExecutionsNotSerialized, + "\"not perturbed\" when the truth is \"cannot tell\" is the one answer that must never be reachable by accident") + assert.Zero(t, bs.ExecutionsSerialized) + }) + + // Assertion 10b's third clause, on the same population: a window with + // end_ns == 0 is OPEN, not zero-length. Treating it as zero-length would + // mark a whole perturbed tail "false". + t.Run("an open window is unknown from its start, never false", func(t *testing.T) { + open := gpu.NewTimeline(gpu.TimelineConfig{PCSampling: gpu.PCSamplingSerialized}) + var first, last uint64 + for i, v := range snap.Executions { + if i == 0 || v.Exec.StartNs < first { + first = v.Exec.StartNs + } + if v.Exec.EndNs > last { + last = v.Exec.EndNs + } + } + require.Greater(t, last, first) + require.NoError(t, open.EmitSamplingWindow(gpu.GPUSamplingWindow{ + Backend: gpu.GPUBackendID("stub"), + PID: uint32(stubPID), + StartNs: first, + EndNs: 0, // the hard-exit shape: cuptiPCSamplingStop never ran + Mode: gpu.SamplingModeKernelSerialized, + })) + for _, v := range snap.Executions { + require.NoError(t, open.EmitExec(v.Exec)) + } + ws := open.Snapshot() + assert.Zero(t, ws.ExecutionsNotSerialized, + "an unterminated window read as zero-length would mark the whole perturbed tail \"false\"") + assert.Equal(t, uint64(len(ws.Executions)), + ws.ExecutionsSerialized+ws.ExecutionsSerializationUnknown, + "every execution at or after an open window's start is either perturbed or unknown, and nothing else") + }) + + // ---- the projection --------------------------------------------------- + // + // Projected TWICE over the SAME snapshot: once with the default budget, for + // the label assertions, and once with a small ceiling for assertion 9. + // Snapshot() is consuming, so the snapshot value is reused rather than + // retaken. + samples, projStats := gpu.ProjectExecutionsWith(snap, gpu.ProjectionConfig{Modules: store}) + require.NotEmpty(t, samples, "the gate is pprof samples, not counters") + assert.Zero(t, projStats.PCLabelsSuppressed, + "the default ceiling is nowhere near %d distinct offsets; a suppression here means the budget shrank", + projStats.DistinctPCLabels) + + // The walk over every projected sample: assertions 1, 2, 3 and 4, on real + // output rather than on a hand-built ExecutionView. + byStatus := map[string]int{} + byAttrib := map[string]int{} + var pcDerived, resolvedWithStack int + si := 0 + for _, view := range snap.Executions { + // ProjectExecutionsWith emits one sample per PC sample, or one for + // the execution itself when it carries none, in snapshot order. + n := max(1, len(view.PCSamples)) + for range n { + require.Less(t, si, len(samples)) + s := samples[si] + si++ + + // Assertion 1, in its exact form: the frames are the launch's own + // CPU stack, then the boundary marker, then the kernel - compared + // as a whole slice rather than scanned for forbidden substrings. + // Whole-slice comparison is strictly stronger (nothing may be + // inserted anywhere, not merely appended) and it has no false + // positives, which a substring scan would: several of the stub's + // stall reasons are spelt "wait", "barrier" and "membar", and libc + // frame names contain all three. + require.NotEmpty(t, view.Exec.KernelName, + "every execution carries an interned kernel name, so the kernel frame is never omitted") + var want []string + if view.Launch != nil && len(view.Launch.Launch.CPUStack) > 0 { + for _, f := range view.Launch.Launch.CPUStack { + want = append(want, f.Name) + } + want = append(want, gpu.FrameLaunch) + } else { + want = append(want, gpu.FrameLaunchUnsampled) + } + want = append(want, "[gpu:kernel:"+view.Exec.KernelName+"]") + require.Equal(t, want, frameNamesOf(s.Stack), + "frames are exhaustively the CPU stack, the boundary marker and the kernel; this sample's differ") + // The two frames this package synthesizes are the only ones it + // could smuggle per-sample detail into, so they take the substring + // scan the CPU frames cannot safely take. + for _, name := range want[len(want)-2:] { + for _, bad := range []string{"gpu:pc", "gpu:src", "gpu:stall", "long_scoreboard", "resolved", "0x"} { + assert.NotContains(t, name, bad, + "per-sample detail was promoted to a frame: %q", name) + } + } + + if len(view.PCSamples) == 0 { + assert.NotContains(t, s.Labels, "gpu_src_status", + "an execution with no PC samples has nothing to say about a source location") + continue + } + pcDerived++ + status := s.Labels["gpu_src_status"] + require.NotEmpty(t, status, + "gpu_src_status is unconditional on every PC-derived sample: an absent label reads as \"not sampled\"") + byStatus[status]++ + byAttrib[s.Labels["gpu_pc_attrib"]]++ + require.NotEmpty(t, s.Labels["gpu_pc_attrib"], "gpu_pc_attrib is unconditional too") + require.Contains(t, s.Labels, "gpu_serialized", + "gpu_serialized rides on every execution, PC-bearing or not") + + switch status { + case "resolved": + // Assertion 2: a source line reached from a CPU stack. + assert.Equal(t, "single.cu", s.Labels["gpu_src_file"], "the basename, never the build-host path") + assert.Equal(t, "addOne", s.Labels["gpu_src_func"]) + require.Contains(t, s.Labels, "gpu_src_line") + if view.Launch != nil && len(view.Launch.Launch.CPUStack) > 0 { + resolvedWithStack++ + if resolvedWithStack == 1 { + t.Logf("assertion 2: %v -> %s:%s (%s, attrib=%s)", + frameNamesOf(s.Stack), s.Labels["gpu_src_file"], s.Labels["gpu_src_line"], + s.Labels["gpu_stall"], s.Labels["gpu_pc_attrib"]) + } + } + default: + // Assertion 3, generalized to all three unresolvable statuses: + // no location is ever invented. + assert.NotContains(t, s.Labels, "gpu_src_file", "status=%s invented a file", status) + assert.NotContains(t, s.Labels, "gpu_src_line", "status=%s invented a line", status) + assert.NotContains(t, s.Labels, "gpu_src_func", "status=%s invented a function", status) + } + } + } + require.Equal(t, len(samples), si, "every projected sample must belong to an execution") + t.Logf("pc-derived samples: %d by status: %v by attrib: %v", pcDerived, byStatus, byAttrib) + + // Assertion 2, stated as a number rather than left implicit in the loop. + assert.Positive(t, resolvedWithStack, + "no sample carries BOTH a real CPU stack and gpu_src_status=resolved; that conjunction is the Phase 6 exit condition, and either half alone is not it") + + // Assertion 4: all four values reachable, each by the fixture that should + // produce it. + for _, want := range []string{"resolved", "no-lineinfo", "no-module", "unmapped"} { + assert.Positive(t, byStatus[want], + "gpu_src_status=%q is not reachable from this run: %v", want, byStatus) + } + // Assertion 3's own arithmetic: the -lineinfo fixture produced the + // resolved population, the no-lineinfo fixture produced exactly one + // no-lineinfo sample, and neither borrowed from the other. + assert.Equal(t, 1, byStatus["no-lineinfo"], + "exactly one sample was injected against the no-lineinfo fixture") + // One, not 65: the stub's own 64 records never reach an execution at all + // (see the doc comment), so they never project and never carry a label. + // The single no-module sample here is the injected one whose CRC no cubin + // was ever stored for - which is exactly the fixture that should produce + // that status. + assert.Equal(t, 1, byStatus["no-module"], + "only the injected absent-CRC sample can read no-module; the stub's records are pending and unprojected") + assert.Equal(t, 1, byStatus["resolved"]) + assert.Equal(t, 1+capSpray, byStatus["unmapped"], + "the injected past-the-function offset plus the %d cardinality-spray offsets, all past the line table's last address", capSpray) + assert.Equal(t, injected, pcDerived, + "every projected PC-derived sample must be one of the injected ones") + + // ---- assertion 9: the cardinality cap --------------------------------- + // + // Past the ceiling gpu_pc is dropped and counted, while gpu_stall and + // gpu_src_* survive untouched: they are coarser and more actionable, so the + // label that gives way is the numerous one rather than the useful one. + const ceiling = 8 + capped, capStats := gpu.ProjectExecutionsWith(snap, gpu.ProjectionConfig{ + Modules: store, + MaxDistinctPCLabels: ceiling, + }) + require.Len(t, capped, len(samples), "the cap changes labels, never the sample population") + distinct := map[string]bool{} + var suppressed uint64 + for i, s := range capped { + if _, isPC := s.Labels["gpu_src_status"]; !isPC { + continue + } + if pc, ok := s.Labels["gpu_pc"]; ok { + distinct[pc] = true + continue + } + suppressed++ + // Everything else must have survived. This is the half that a cap + // which simply stopped emitting labels would fail. + assert.NotEmpty(t, s.Labels["gpu_stall"], + "the cap dropped gpu_stall, which it must never touch (sample %d)", i) + assert.NotEmpty(t, s.Labels["gpu_src_status"], + "the cap dropped gpu_src_status, which is unconditional (sample %d)", i) + assert.NotEmpty(t, s.Labels["gpu_pc_attrib"], "the cap dropped gpu_pc_attrib (sample %d)", i) + assert.Equal(t, samples[i].Value, s.Value, + "a suppressed sample still carries its full share of the execution's duration") + } + assert.Positive(t, suppressed, + "a ceiling of %d over %d distinct injected offsets suppressed nothing, so this proves nothing", + ceiling, capSpray) + assert.Equal(t, suppressed, capStats.PCLabelsSuppressed, + "ProjectionPCLabelsSuppressed must equal the suppressions actually visible in the output, or the counter is decoration") + assert.Equal(t, uint64(len(distinct)), capStats.DistinctPCLabels) + assert.LessOrEqual(t, len(distinct), ceiling, "more distinct gpu_pc values were emitted than the ceiling allows") + assert.Equal(t, uint64(ceiling), capStats.PCLabelCap) + // And it is visible to the operator: a profile that silently lost its PC + // labels looks identical to one that never had any. + health := strings.Join(gpu.JoinHealthWith(snap, capStats), "\n") + assert.Contains(t, health, "gpu_pc", + "the suppression is not surfaced in joinhealth output:\n%s", health) + t.Logf("cardinality cap: distinct=%d cap=%d suppressed=%d", capStats.DistinctPCLabels, capStats.PCLabelCap, capStats.PCLabelsSuppressed) +} + +// waitFor polls until cond holds or the deadline passes, failing with what +// describe() reports rather than with a bare timeout. A producer that never +// emitted and a consumer that never drained look identical from a timeout. +func waitFor(t *testing.T, within time.Duration, cond func() bool, describe func() string) { + t.Helper() + deadline := time.Now().Add(within) + for !cond() { + if time.Now().After(deadline) { + t.Fatalf("timed out after %s: %s", within, describe()) + } + time.Sleep(5 * time.Millisecond) + } +} + +func frameNamesOf(frames []pp.Frame) []string { + out := make([]string, 0, len(frames)) + for _, f := range frames { + out = append(out, f.Name) + } + return out +} + +func fixturePath(name string) string { + return filepath.Join("..", "internal", "cubin", "testdata", name) +} + +func readFixture(t *testing.T, name string) []byte { + t.Helper() + b, err := os.ReadFile(fixturePath(name)) + require.NoError(t, err, "cubin fixture %s", name) + require.NotEmpty(t, b) + return b +} + +func mustAbs(t *testing.T, p string) string { + t.Helper() + abs, err := filepath.Abs(p) + require.NoError(t, err) + return abs +} + +// fixtureSymIndex reads the .symtab index the module store keys its +// functionIndex table on out of the fixture itself. Whether CUPTI's +// functionIndex IS that index is the design's premise and is measured on +// hardware (Task 6); nothing here depends on the answer, only on the store and +// the sample agreeing. +func fixtureSymIndex(t *testing.T, b []byte, fn string) uint32 { + t.Helper() + c, err := cubin.Parse(b) + require.NoError(t, err) + for _, f := range c.Functions() { + if f.Name == fn { + require.GreaterOrEqual(t, f.SymIndex, 0) + return uint32(f.SymIndex) + } + } + t.Fatalf("fixture has no function %q", fn) + return 0 +} + +// gateModuleCRCs pulls the CRC the producer declared for each captured module +// out of its own stderr, in capture order. +// +// From the producer rather than from a constant, so that the number the module +// store is keyed on is the number that was actually put on the wire. A +// constant would keep this gate green through a change to how the producer +// derives a CRC - which is exactly the change that would break the join on +// hardware. +func gateModuleCRCs(t *testing.T, stubErr string) []uint64 { + t.Helper() + re := regexp.MustCompile(`stub: module id=\d+ path=\S+ size=\d+ crc=0x([0-9a-f]{16}) captured=yes`) + var out []uint64 + for _, m := range re.FindAllStringSubmatch(stubErr, -1) { + v, err := strconv.ParseUint(m[1], 16, 64) + require.NoError(t, err) + out = append(out, v) + } + return out +} + +// stubCubinCRC is the Go replica of shim/stub/stub.cc's stub_cubin_crc: FNV-1a +// over the module bytes, with zero coerced to one because zero is the ABI's +// "no module". +// +// It exists so the CRC can be recomputed from the checked-in fixture rather +// than read back from the producer that produced it. Reading it back would +// assert only that the producer is self-consistent; recomputing it asserts +// that the identity the join runs on is a function of the BYTES, which is the +// property the whole content-addressed scheme rests on. +func stubCubinCRC(b []byte) uint64 { + h := uint64(1469598103934665603) + for _, c := range b { + h ^= uint64(c) + h *= 1099511628211 + } + if h == 0 { + return 1 + } + return h +} + +// TestGateTheStubsPCRecordsCannotAttributeToAnything pins the second reason +// TestStubDrivesPCSamplingToPprofWithoutAGPU injects PC samples of its own +// instead of using the producer's, and it runs WITHOUT capabilities so the +// reason is checkable on any machine. +// +// # The gap +// +// The stub emits real module loads carrying real checked-in cubins, and real +// PC-sample records. They are unrelated to each other: +// +// - the PC records' cubin_crc is one of two compile-time constants, +// kStubCubinCRC = {0xC0FFEE01, 0xC0FFEE02}, while the modules the same run +// delivers are keyed by a content hash of the fixture bytes (FNV-1a over +// the file, measured: 0x9d57accad01046eb for single_lineinfo.cubin). No +// cubin is ever stored under a 0xC0FFEE0n key, so every one of those +// records resolves as "no-module"; +// - their correlation is 0 in every tier, by design and correctly - that is +// what CONTINUOUS collection produces - so the exact-correlation path is +// unavailable to them; +// - Tier B attribution therefore runs crc -> module -> function name -> +// the execution's KernelName, and the stub's kernel names are +// "kernel_1111"/"kernel_2222" while the fixtures' only function is the +// CUDA kernel they were compiled from ("addOne"). No name can match. +// +// So neither join path can fire for a stub PC record, in either tier. That is +// not a bug in the pipeline - the pipeline correctly counts every one of them +// as pending, which the end-to-end gate asserts as an exact equality - but it +// does mean the producer cannot drive gate assertions 2, 3, 4 or 9, all of +// which need a PC sample that reaches an execution. +// +// Closing it is a small change to shim/stub/stub.cc: record the CRC each +// capture computed and use it on the PC records, and name the kernels after +// the fixtures' own functions. This branch is a test task and may not change +// the shim, so it is pinned here instead. +// +// # Why a passing test +// +// Same shape as the other two outstanding pins: when the stub is fixed, this +// fails by name and the person fixing it drops the injection from the +// end-to-end gate and asserts the real thing. +func TestGateTheStubsPCRecordsCannotAttributeToAnything(t *testing.T) { + src, err := os.ReadFile(filepath.Join("..", "shim", "stub", "stub.cc")) + require.NoError(t, err, "the producer's source must be readable; this test is about what it emits") + body := string(src) + + assert.Contains(t, body, "static const uint64_t kStubCubinCRC[] = {0xC0FFEE01ull, 0xC0FFEE02ull};", + "the stub's synthetic PC-record CRCs changed. If they now come from the CRC each capture "+ + "computed, gate assertions 2/3/4/9 may be drivable from the producer: drop the PC-sample "+ + "injection from TestStubDrivesPCSamplingToPprofWithoutAGPU and assert them off the wire.") + assert.Contains(t, body, "r.cubin_crc = kStubCubinCRC[i % 2];", + "the stub no longer keys its PC records on the synthetic constants - see above") + assert.Contains(t, body, `snprintf(namebuf, sizeof(namebuf), "kernel_%llx", (unsigned long long)kernel_id);`, + "the stub's kernel names changed. If they now name functions the checked-in cubins "+ + "actually contain, the Tier B module join can fire for its own records - see above.") + + // And the fixtures really do carry a different name, so the mismatch above + // is a fact about both ends rather than an assumption about one. + c, err := cubin.Parse(readFixture(t, "single_lineinfo.cubin")) + require.NoError(t, err) + for _, fn := range c.Functions() { + assert.NotContains(t, fn.Name, "kernel_", + "a fixture function is now named like a stub kernel; the join might match by accident, which is worse than not matching at all") + } + t.Log("stub PC records -> a cubin the agent holds: OUTSTANDING - synthetic CRCs and synthetic " + + "kernel names; see .superpowers/sdd/task-13-gate-report.md") +}