diff --git a/.superpowers/sdd/task-13-gate-report.md b/.superpowers/sdd/task-13-gate-report.md new file mode 100644 index 0000000..cd989bf --- /dev/null +++ b/.superpowers/sdd/task-13-gate-report.md @@ -0,0 +1,482 @@ +# Task 13 — The Phase 6 gate, GPU-free half (assertions 1–12) + +Branch `feat/phase6-gate`, one commit, test-only. No file under `gpu/`, `shim/` or +`bpf/` changed; `git diff --stat` on `gpuprobe/gate_test.go` is 803 insertions and +**zero deletions**, so no existing gate assertion was weakened. + +Verified on this machine: `CapEff: 0`, no GPU, no passwordless sudo. That shapes the +whole deliverable and is stated plainly below rather than glossed. + +## Where the gate lives, and why it is in three places + +| file | package | privilege | what | +| --- | --- | --- | --- | +| `gpu/gate_test.go` | `gpu` | none | `TestPhase6Gate` — assertions 1–9 and 10b | +| `gpuprobe/gate_compose_test.go` | `gpuprobe` | none | `TestPhase6GateConsumerHalf` — assertions 10, 10a, 11, 12 | +| `gpuprobe/gate_test.go` | `gpuprobe_test` | CAP_BPF + CAP_PERFMON + CAP_CHECKPOINT_RESTORE | `TestStubDrivesPCSamplingToPprofWithoutAGPU` — 1–5, 8, 9, 12 end to end | + +The plan writes the gate as an extension of `TestStubDrivesThePipelineToPprofWithoutAGPU`. +Written only there, the gate **skips on every machine without capabilities**, including +this one — and a twelve-point gate that cannot run is the most expensive instance of the +failure this project has hit nineteen times. So every assertion that is a statement about +the `gpu` or `gpuprobe` package is *also* asserted where it needs no privilege, and the +privileged test adds what only a real producer can add: a CPU stack walked by BPF through +two `-fomit-frame-pointer` frames, real cubins crossing a real socket, 500 real executions. + +The two unprivileged entry points **compose** rather than restate: they call the tests the +tasks already wrote, as sub-tests named for the gate assertion. Two assertions of one fact +drift apart; one assertion referenced from the gate does not. What composing buys is that +deleting or weakening any of them now fails *the gate*, by assertion number. + +`TestStubDrivesPCSamplingToPprofWithoutAGPU` is a **second** privileged test rather than an +edit to the baseline one. The baseline asserts `require.Len(samples, len(snap.Executions))`, +which is true precisely because that run emits no PC samples; turning PC sampling on inside +it would have meant weakening that and several others. + +--- + +## The twelve + +"Reused" means the gate calls an existing test. "New" means no test asserted the property. +"Ran" means it executed on this machine, unprivileged. "Compiled" means it type-checks and +is wired into the privileged gate, which cannot run here. + +### 1. Frames stop at the kernel + +**Asserts.** Frames are exhaustively ` → [gpu:launch]|[gpu:launch unsampled] → +[gpu:kernel:]`. No frame carries the PC, the stall reason, the source location or the +attribution quality. + +**Catches.** Promoting any per-sample detail to a frame, in any spelling. At PC-sampling +rates one frame per PC destroys aggregation and fragments the kernel's own block, which is +the §8 ruling the whole label design rests on. + +**Reused** — `TestProjectionAddsNoFrames`, `TestProjectionKeepsPCOutOfStackIdentity`. +**New end-to-end**: the privileged gate compares each sample's frames against the launch's +own `CPUStack` as a *whole slice*, so nothing may be inserted anywhere rather than merely +appended. Whole-slice comparison also avoids a false positive a substring scan would hit: +three of the stub's stall reasons are spelt `wait`, `barrier` and `membar`, and libc frame +names contain all three. + +**Ran** (unprivileged half) / **compiled** (end-to-end half). +**Mutation-checked**: appending `[gpu:src:mutant]` in `projectionFrames` fails assertions +1, 2 and 3 of `TestPhase6Gate`. + +### 2. A source line is reached from a CPU stack — the Phase 6 exit condition + +**Asserts.** One pprof sample carrying *both* a real CPU call path in its frames *and* +`gpu_src_status="resolved"` with `gpu_src_file`/`gpu_src_line` naming a real line of +`internal/cubin/testdata/single.cu`. The line number is checked against the `.cu` itself, +not against a constant, so a fixture rebuilt from edited source cannot leave it green. + +**Catches.** A projection that resolves source labels but loses the launch's frames, or the +reverse; a join that reaches the execution but drops `PCSamples` so nothing projects; a +store keyed on something the sample does not carry; a resolution naming a line the table +does not contain. + +**New.** Three tests each covered a segment and none the conjunction: +`TestTierBSampleReachesItsExecution` drives the join but its execution has no launch and +never projects; `TestProjectionEmitsAllFourSrcStatuses` reaches `resolved` from a hand-built +`ExecutionView` with no `Launch`; `TestProjectionAddsNoFrames` has the CPU stack but asserts +the *negative*. `TestGateASourceLineIsReachedFromACPUStack` drives a real `Timeline` so the +module join and the resolution both run for real. + +**Ran** (through the `gpu` package) / **compiled** (end to end). +**This is the assertion with the largest gap between what the gate proves and what the +product does — see "Findings" below.** + +### 3. No `-lineinfo`, no invention + +**Asserts.** Every no-lineinfo sample carries `gpu_src_status="no-lineinfo"` and no +`gpu_src_file`/`_line`/`_func`; the kernel frame is unchanged, so the resolvable and +unresolvable populations still aggregate at the same kernel. + +**Catches.** Synthesizing a nearest line or a function's first line. And, for the second +clause: any frame that varies with `gpu_src_status`, which would split one kernel's block +in two in a flame graph while every label assertion still passed. + +**Reused** for the labels — `TestProjectionEmitsAllFourSrcStatuses`. +**New** for the aggregation clause — `TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel`; +nothing asserted that the two populations share a stack. + +**Ran** / **compiled** (end to end, generalized there to all three unresolvable statuses). +**Mutation-checked**: making `Resolve` fall back to `Resolve(fn, 0)` when the PC is +uncovered fails assertions 3 and 4. + +### 4. All four `gpu_src_status` values reachable, each by the fixture that should produce it + +**Asserts.** `resolved` from the `-lineinfo` fixture, `no-lineinfo` from the no-lineinfo +fixture, `no-module` from a CRC no cubin was stored under, `unmapped` from a PC past the +function — in the store *and* in the labels; the enum is exhaustive and its zero value is +not a status. + +**Catches.** A fifth value; a silent default; the two "cannot resolve" statuses collapsing +into one, which is the difference between "recompile with `-lineinfo`" and "the compiler +emitted no line here". + +**Reused** — `TestModuleStoreAllFourStatusesAreReachable`, +`TestProjectionEmitsAllFourSrcStatuses`, `TestSrcStatusesIsExhaustiveAndStable`, +`TestSrcStatusZeroValueIsNotAStatus`. **Ran** / **compiled** (end to end, with exact +per-status counts). + +### 5. Reconciliation + +**Asserts.** Every PC record accepted by the sink lands in exactly one of attributed-exact, +attributed-kernel, still-pending (in either of the two pending stores) or evicted (from +either), and the counters sum to what was emitted. Plus the group-level identity: every +pending module group is joined or left pending for exactly one *counted* reason. + +**Catches.** A join that attaches samples without counting them (satisfies the sample +identity, reports a wrong per-tier split); a refusal path that returns without incrementing +any counter — the easiest and most invisible mistake available in that function. + +**Reused** — `TestConformancePCSampleReconciliationCoversPendingAndEvicted`, +`...CoversAttributedByKernel`, `TestTierBSamplesReconcile`, +`TestTierBJoinAccountsForEveryGroup`. **New end-to-end**: the identity over 64 wire records +plus 44 injected ones, with `PendingModuleSamples == 64` as an equality (see finding 1). +**Ran** / **compiled**. + +### 6. Tier B does not collapse + +**Asserts.** 10,000 samples across 50 distinct `(crc, functionIndex)` pairs attribute with +no loss to `EvictedPendingSamples`. + +**Catches.** Task 8a's pathology returning — a Tier B sample keyed on its (empty) +correlation value, collapsing a whole process onto `{backend, pid, ""}` and evicting +everything past `pendingSampleCap`. + +**Reused** — `TestTierBPCSamplesDoNotCollapseOntoOneKey`, +`TestTierBDistinctFunctionIndexSplitsGroups`. **Ran.** Not reachable end to end: the stub +emits 64 records, not 10,000, and none of them attributes at all. + +### 7. Ambiguity is marked in its own label + +**Asserts.** Two executions of one kernel in the horizon with one Tier B batch give +`gpu_pc_attrib="kernel-ambiguous"` **and leave `gpu_ambiguous` unset**. + +**Catches.** Reusing `ExecutionView.Ambiguous` for PC ambiguity, which would put +`gpu_join="exact" gpu_ambiguous="true"` on one sample — two unrelated facts on one boolean, +and `AmbiguousHeuristicMatchCount` no longer meaning what its name says. + +**Reused** — `TestTierBAmbiguityIsMarkedWithoutTouchingAmbiguous`, +`TestProjectionKernelAmbiguousNeverCoincidesWithGpuAmbiguous`. **Ran.** Not reachable end to +end: the stub's PC records cannot join through the module at all (finding 1). + +### 8. Tier A disclosure + +**Asserts.** Inside a window `"true"`, in a proven gap `"false"`; Tier A selected with no +window arriving gives `"unknown"` on every execution and `"false"` on none; the three +outcomes partition the executions exactly. + +**Catches.** The one answer that must never be reachable by accident — a profile reporting +"not perturbed" when it means "cannot tell". + +**Reused** — `TestSerializationMarksExecutionsOverlappingABurst`, +`TestSerializationIsFalseInAProvenGap`, `TestSerializationIsUnknownWhenNoWindowsArrived`, +`TestSerializationFalseIsOnlyEverReachedFromPositiveEvidence`, +`TestSerializedLabelIsUnconditionalAndHasThreeValues`. +**New end-to-end**: real windows from the producer bracket real executions, and the +"no windows" half is driven by re-emitting *this run's own* executions into a second Tier A +`Timeline` that never sees a window — same population, only the evidence differs. +**Ran** / **compiled**. + +### 9. Cardinality cap + +**Asserts.** Past the ceiling `gpu_pc` is suppressed and nothing else — `gpu_stall`, +`gpu_src_*` and `gpu_pc_attrib` survive, the sample keeps its full share of the execution's +duration — with `ProjectionPCLabelsSuppressed` **equal to the suppressions actually visible +in the output**, and the loss surfaced in `joinhealth`. + +**Catches.** A cap that drops the coarser, more actionable labels instead of the numerous +one; a suppression that is silent, since a profile that lost its PC labels looks identical +to one that never had any. + +**Reused** — `TestProjectionCapSuppressesGpuPCAndOnlyGpuPC`, +`TestProjectionCapIsSurfacedInJoinHealth`. **New end-to-end**: 40 distinct injected offsets +against a ceiling of 8, with the counter tied to the observed output rather than to a +hard-coded number. **Ran** / **compiled**. + +### 10. The cubin channel cannot touch enrolment + +**Asserts.** The isolation test runs *as part of the gate*, in **both** orders including +offers flooded *ahead of* an enrolment with the cubin listener's accept loop deliberately +wedged: `CubinsThrottled` non-zero, `UnwindEnrollThrottled` unchanged, the enrolment still +confirmed. Plus: the two admission buckets are separate objects with different numbers, the +two addresses are siblings and not one socket, and the enrolment handler performs **no read** +on the producer's connection — asserted behaviourally and structurally. + +**Catches.** Moving cubin traffic onto the enrolment listener, its goroutine, its accept +loop or its bucket. Any of those restores issue #49's ~38 % stack loss on a module-heavy +workload, silently, with only `UnwindEnrollThrottled` moving. + +**Reused** — `TestFloodingTheCubinChannelCannotStarveOrThrottleAnEnrolment`, +`TestTheCubinAdmissionBucketIsItsOwn`, +`TestTheCubinAddressIsASiblingOfTheRendezvousAndNotTheSameSocket`, +`TestAnEnrolmentCompletesWithNoReadOnThatConnection`. **Ran.** The privileged gate adds the +live form: real cubin traffic crossing while a real enrolment happens, with +`UnwindEnrollThrottled == 0` and `UnwindEnrollConfirmed == 1`. +**Mutation-checked**: sizing the cubin per-uid bucket at `enrollUIDBurst` fails +`assertion-10-the-buckets-are-separate-objects`. + +### 10a. Seals are enforced + +**Asserts.** A memfd missing each required seal *in turn* is rejected, counted in +`CubinsRejectedUnsealed`, and **never mapped**; a descriptor that is not a sealed memfd at +all (a pipe, an unsealed tmpfs file) is refused; the required set is spelt out. + +**Catches.** Dropping a seal — without `F_SEAL_SHRINK` a peer can `ftruncate` under our +`mmap` and SIGBUS the agent; without `F_SEAL_WRITE` the ELF mutates under the parser +mid-parse — and, worse, any fallback that maps it anyway. + +**Reused** — `TestEachRequiredSealMissingInTurnIsRejectedAndNeverMapped`, +`TestADescriptorThatIsNotASealedMemfdIsRefused`, `TestSealNamesAreSpeltOut`. **Ran.** + +### 10b. Out-of-scope conditions refuse rather than guess + +Three clauses, and they are not in the same state. + +- **Two device ids give `gpu_pc_attrib="kernel-multidevice"`** — **reused** + (`TestTierBMultiDeviceProcessIsMarked`, `TestMultiDeviceOutranksAmbiguity`), **ran**. +- **A window with `end_ns == 0` gives `"unknown"` on everything after it and `"false"` on + nothing** — **reused** + (`TestSerializationOpenWindowMakesEverythingFromItsStartUnknownAndNeverFalse`), **ran**; + the privileged gate also drives it over this run's own executions. +- **A graph execution makes Tier A refuse to start** — **NOT IMPLEMENTED BY THE PRODUCT.** + See finding 3. Pinned, not asserted, by + `TestGateGraphExecutionRefusalIsNotAssertableYet`, which **ran**. + +### 11. `getcap` on the gate binary shows no `cap_sys_admin` + +**Asserts.** From two independent directions, because either alone is escapable: the file +capabilities of the test binary itself (via `cap.GetFile`, the same +`security.capability` xattr `getcap` prints — no external tool needed, so it cannot degrade +to a skip on a container image without `getcap`), and this process's own Permitted set. A +separate test runs `getcap(8)` when present and requires it to agree. Vacuous under root, so +the process half is skipped there and said so, while the file half still runs. + +**Catches.** The two quiet one-liners that acquire the requirement: attaching with +`link.Uprobe` instead of `link.UprobeMulti` (the `perf_uprobe` PMU needs CAP_SYS_ADMIN, the +BPF link does not), and reading the producer's address space to follow +`gpu_module_load_v1.bytes_ptr` (needs CAP_SYS_PTRACE) — both of which work fine on a +developer box that runs tests as root and say nothing about why. + +**New.** No test asserted it anywhere; the standing Phase 1 assertion existed only as prose. +Composed alongside `TestEmbeddedProgramIsUprobeMulti`, which is *why* no CAP_SYS_ADMIN is +needed. **Ran.** + +### 12. `Stats.Undecoded` is zero for every kind this phase decodes + +**Asserts.** All five PC-sampling kinds (module, PC, stall map, sampling window, config) +have an `applyBatch` arm and are counted in `Records`, not `Undecoded`; a healthy Tier B run +leaves every new loss counter at zero; an unknown kind is still counted rather than dropped; +`kindMax` matches the embedded BPF object's `dropped` map; every cookie has a sized kind. + +**Catches.** A `KIND_*` added on one side of the wire and not the other — silent loss unless +counted — and, in the other direction, a decode arm removed, which would put a kind back +into the default arm while every other counter still read healthy. + +**Reused** — `TestTheFivePCSamplingKindsAreDecodedNotCountedUndecoded`, +`TestHealthyTierBRunLeavesEveryNewLossCounterAtZero`, +`TestUndecodedKindsAreCountedNotDropped`, `TestEmbeddedProgramIsUprobeMulti`, +`TestBPFSizesEveryKindCookieForInstalls`. **Ran.** + +**End to end this is `Undecoded == 4`, not zero, and that is correct.** `gpu_dropped_v1` is +*not* a kind this phase decodes: it is decoded into `batch.Drops` and carried, and +normalizing a drop class into an operator-visible number is the consumer task after Task 7 +(see `Stats.Undecoded`'s own comment). The stub emits exactly one record per drop class when +PC sampling is on, so the gate asserts the equality — a fifth undecoded record means a kind +arrived that nothing on this side knows about. **Compiled.** + +--- + +## Findings — three product gaps the gate surfaced + +All three are **reported, not silently worked around**, and each is pinned by a passing test +that **fails the moment the gap is closed**, so the gate is updated rather than left +claiming an assertion it never made. That is the shape issue #44 used and #45 inverted. + +### Finding 1 — the stub's PC records cannot attribute to anything, in either tier + +`shim/stub/stub.cc` emits real module loads carrying the real checked-in cubins, and real PC +records. They are unrelated: + +- the PC records' `cubin_crc` is one of two compile-time constants, + `kStubCubinCRC = {0xC0FFEE01, 0xC0FFEE02}`, while the modules the same run delivers are + keyed by a content hash of the fixture bytes — measured on this machine, + `0x9d57accad01046eb` for `single_lineinfo.cubin`. No cubin is ever stored under a + `0xC0FFEE0n` key; +- their `correlation` is 0 in every tier — correct, that is what CONTINUOUS collection + produces — so the exact-correlation path is unavailable to them; +- Tier B attribution runs `crc → module → function name → the execution's KernelName`, and + the stub's kernel names are `kernel_1111`/`kernel_2222` while the fixtures' only function + is the CUDA kernel they were compiled from (`addOne`). No name can match. + +So neither join path can fire, and the stub cannot drive assertions 2, 3, 4, 7 or 9. The +pipeline is not at fault — it correctly counts all 64 as pending, which the gate asserts as +an exact equality, and `TestTierBKernelNameMismatchStaysPending` already pins that samples +stay pending rather than being attached to a plausible neighbour. + +**Consequence for the gate.** `TestStubDrivesPCSamplingToPprofWithoutAGPU` injects 44 PC +samples of its own at `Timeline.EmitPCSample` — the same entry point the consumer calls — on +correlations that a wire-delivered, stack-carrying launch is known to occupy (replayed +exactly by `gpuabi.SampleSchedule`, which is pinned against the shim's own sampler). What is +injected is the *record*; everything downstream is product code — the join, the store's +four-valued resolution, the label set, the cardinality budget. What the injection skips is +the consumer's `KIND_PC` decode arm, and that is asserted separately and exactly by the 64 +wire records. + +**Fix** (a small change to `shim/stub/stub.cc`, out of scope on a test branch): record the +CRC each capture computed and use it on the PC records; name the kernels after the fixtures' +own functions. **Pinned by** `TestGateTheStubsPCRecordsCannotAttributeToAnything` +(unprivileged, ran). + +### Finding 2 — the cubin transport does not feed `gpu.ModuleStore`, so the product cannot resolve a source line at all + +Task 3 built the channel; Task 4 built the store. Nothing connects them: + +- `Attach` calls `newCubinListener(cfg, nil)`, and a nil sink becomes `memCubinStore` — a + bounded CRC→bytes map with no line table, no LRU and no `Resolve`. Its own comment says + "Task 4 replaces it"; +- `gpuprobe.Config` has no field by which a caller could supply one, and `gpu.ModuleStore` + does not satisfy `cubinSink` (`Put`/`HasCubin` versus `PutCubin`/`HasCubin`) even if it + had; +- `cmd/gpu-cuda-profile` builds neither: `gpu.NewTimeline(gpu.TimelineConfig{PCSampling: tier})` + with no `Modules`, and `gpu.ProjectExecutionsWith(snap, gpu.ProjectionConfig{})`. + +**On hardware today every cubin is received, sealed, verified, identity-checked, stored — +and then never read, and every PC sample in a real profile reads +`gpu_src_status="no-module"`.** Gate assertion 2, the Phase 6 exit condition, is therefore +not satisfiable by the product as shipped. This is one hop, with both ends built and tested; +it is a wiring task, not a design gap. It is also, per `ProjectionConfig.Modules`' own +comment, *designed* to be visible rather than silent — "no-module" on every sample points +straight at the missing store. + +**Consequence for the gate.** The privileged gate builds the `ModuleStore` itself and keys +it on the CRCs the **producer** declared (parsed from the producer's own report and +independently recomputed in Go over the checked-in fixture bytes), not on numbers the test +invented — so the identity the store keys on is the identity that went on the wire. It also +asserts that the decoded `gpu_module_load_v1` records name the same `(crc, size)` pairs. +That is the offline half of hardware assertion 13. + +**Pinned by** `TestGateTheCubinTransportDoesNotYetFeedTheModuleStore` (unprivileged, ran). + +### Finding 3 — Tier A does not refuse to start where CUDA graphs have been observed + +The plan's Task 10 and its out-of-scope section require: *"Tier A refuses to start in a +process where graph executions have been observed … The refusal is loud and counted, not a +silent downgrade to Tier B."* + +The wire signal exists — `internal/gpuabi.DropClassGraphExec` decodes and spells itself +`graph-exec`, and the stub emits one such record so the class is reachable from a test. The +refusal does not. **Nothing in `gpu/`, `gpuprobe/` or `cmd/` consumes that drop class**: the +consumer decodes `gpu_dropped_v1` into `batch.Drops` and stops (finding above), so there is +no counter, nothing on the `Snapshot`, no `joinhealth` anomaly, and no input by which +`PCSamplingRequest` could be told. Tier A starts happily in a graph-using process and +produces confident, exact-*looking* attribution of N kernels to one call site — which is the +plan's own finding 4, and the condition it says must be visible rather than merely out of +scope. Graph launches are the norm in inference serving. + +What *is* true today, and is asserted: the Tier A acknowledgement refusal an operator must +read before the tier can run names CUDA graphs, and so does the standing warning. + +**Pinned by** `TestGateGraphExecutionRefusalIsNotAssertableYet` (unprivileged, ran), which +uses reflection over `Snapshot`, `TimelineDropStats`, `PCJoinStats`, `TimelineConfig` and +`PCSamplingRequest` and fails the moment any of them grows a field naming graph executions. + +--- + +## What ran and what only compiled + +**Ran, unprivileged, on this machine** — `go test ./gpu/ ./gpuprobe/ ./internal/... -count=1` +and `-race -count=4`, both green: + +- assertions **1, 3, 4, 5, 6, 7, 8, 9, 10b** (two of three clauses) via `TestPhase6Gate`; +- assertion **2** via `TestGateASourceLineIsReachedFromACPUStack`, through a real `Timeline` + and a real `ModuleStore` over the checked-in `-lineinfo` fixture; +- assertions **10, 10a, 11, 12** via `TestPhase6GateConsumerHalf`; +- all three outstanding-gap pins. + +**Compiled only** — `TestStubDrivesPCSamplingToPprofWithoutAGPU`. `CapEff: 0` and no +passwordless sudo on this machine (`sudo -n true` → "a password is required"), so it skips +with the message that names the setcap line. It type-checks, vets and lints clean, and every +number in it is derived rather than guessed: + +- 58 sampled launches from `gpuabi.SampleSchedule(500, 8, DefaultSampleSeed)`, the same + constant the baseline gate uses; +- 64 PC records, 2 cubins, 8 stall reasons, 4 drop records, 4 bursts — all read out of + `shim/stub/stub.cc` and confirmed by **running the producer standalone** with the exact + environment the gate sets. That run printed `pc_sampling=serialized`, + `pc_samples=64 stall_reasons=8 cubins=2 functions=4 drop_classes=4`, and captured both + fixtures with the CRCs above; +- the FNV-1a CRC replica in the test was checked against the producer's own output for both + fixtures and matches exactly. + +What has *not* been executed is the attach, the walk, the symbolization and the assertions +that read `Stats` after a live run. Those need `cap_bpf,cap_perfmon,cap_checkpoint_restore`. + +**Mutation checks** (each applied, run, and reverted): + +| mutation | gate result | +| --- | --- | +| `projectionFrames` appends `[gpu:src:mutant]` | assertions 1, 2, 3 FAIL | +| `ModuleStore.Resolve` falls back to `Resolve(fn, 0)` for an uncovered PC | assertions 3, 4 FAIL | +| cubin per-uid admission bucket sized at `enrollUIDBurst` | assertion 10 FAIL | + +--- + +## Outstanding: assertions 13–16 (RTX 3090) + +None of these is attempted here; all four need the hardware. + +13. **`cuptiGetCubinCrc()` over the received copy equals the PC records' `cubinCrc`.** The + gate proves the *shape* of this offline — one number identifies one set of bytes and the + same number reaches both ends — against the stub's FNV-1a stand-in. CUPTI's polynomial + is unpublished and there is no CUDA toolkit on the agent path, so the real equality is + hardware-only. +14. **Tier B: a flame graph reaching a real line of `cuda_workload.cu` from a CPU stack, + through labels** — the Phase 6 exit condition on real hardware. Blocked additionally by + finding 2: the transport→store hop must be wired before this can pass on any machine. +15. **Tier A: `correlationId` non-zero on ≥ 99 % of PC records; `gpu_pc_attrib="exact"` on + the resulting samples; windows bracketing the executions that ran in them.** +16. **Overhead within the Task 12 thresholds, or the tier decision they dictate.** Task 12 + is active on `.worktrees/pc-overhead` and its numbers are not in the plan file yet. + +## The plan's own "cannot verify without hardware" list, restated + +Beyond assertions 13–16, and unchanged by this task: + +- whether `functionIndex` **is** the cubin's `.symtab` index — the finding-2 question of the + plan, and the trigger for `gpu_pc_sample_batch_v2`. Every test in this tree, including the + gate's, reads the index out of the fixture and asserts only that the store and the sample + use the same one consistently; +- whether `pcOffset` is function-relative in the sense the line table is; +- every rate and buffer-sizing number (the spike's 352 records for ~103 k samples drives all + of them); +- hardware-buffer overflow behaviour — `droppedSamples` / `hardwareBufferFull` under a + saturating workload; +- whether the collection mode can change between `Stop` and `Start` without a full + `Disable`/`Enable` — undocumented, and it decides whether a future tier switch is possible + at all; +- whether per-context enable holds when contexts are created lazily on other threads; +- whether the `MODULE_UNLOAD_STARTING` drain preserves PC uniqueness across a + load–unload–load cycle; +- whether a `cuptiFinalize` handler runs at all, and whether disabling per context inside it + errors. + +**Not verifiable at all, on hardware or otherwise:** MPS, and cross-process contention for +the per-device PC-sampling hardware. A process cannot observe either from inside itself. +Stated so a quiet profile is not read as a correct one. + +## Verification run + +``` +make -C shim && make -C shim test && make -C shim check-fpless \ + && make -C shim check-cubin-defer && make -C shim nvidia # all OK +go build ./... && go vet ./... # clean +go test ./gpu/ ./gpuprobe/ ./internal/... -count=1 # ok +go test ./gpu/ ./gpuprobe/ -race -count=4 # ok +~/go/bin/golangci-lint run --timeout=5m # 0 issues +``` + +`git diff --stat` for the one modified file: `gpuprobe/gate_test.go`, 803 insertions, +0 deletions. diff --git a/gpu/gate_test.go b/gpu/gate_test.go new file mode 100644 index 0000000..2df5138 --- /dev/null +++ b/gpu/gate_test.go @@ -0,0 +1,429 @@ +package gpu + +import ( + "os" + "reflect" + "strconv" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + pp "github.com/dpsoft/perf-agent/pprof" +) + +// TestPhase6Gate is the GPU-free half of the Phase 6 phase gate, assembled in +// the package whose behaviour it is about. +// +// # Why the gate lives in two places and why this half exists at all +// +// The plan's gate (Task 13, assertions 1-12) is written as an extension of +// gpuprobe's TestStubDrivesThePipelineToPprofWithoutAGPU, which needs CAP_BPF, +// CAP_PERFMON and CAP_CHECKPOINT_RESTORE to attach the uprobes and symbolize +// the producer's stacks. On a machine without those it SKIPS - and a gate that +// skips is a gate that cannot fail. Nineteen defects on this project have been +// checks that read green when things were worst; a twelve-point gate nobody can +// run is the most expensive available instance of that. +// +// So every assertion of the twelve that is a statement about the gpu package - +// which is nine of them - is also asserted HERE, where it needs no capability +// and runs on every `go test ./gpu/`. The privileged end-to-end gate in +// gpuprobe/gate_test.go adds what only a real producer can add: a real CPU +// stack walked by BPF, a real cubin crossing a real socket, real executions. +// +// # Why this composes rather than restates +// +// Every property below is already asserted by a task's own test. Restating it +// here would give the project two assertions of one fact, which is how two +// assertions of one fact drift apart. So this calls them. What that buys, over +// leaving them scattered: +// +// - deleting or weakening any one of them fails THE GATE, by name, with the +// assertion number attached, rather than removing a test nobody notices; +// - the twelve are enumerated in one place, so an assertion that has no test +// behind it is visible as a gap rather than as an absence; +// - the sub-test names carry the gate numbering, so a failure reads as +// "assertion 7 failed" instead of as a test name that has to be traced back +// to the plan. +// +// Two of the twelve are NOT compositions, because no test asserted them: +// assertion 2 (a source line reached from a CPU stack, through the join and the +// projection together) and assertion 3's aggregation clause. Both are written +// out below. +// +// Assertions 10, 10a, 11 and 12 are consumer-side and live in +// gpuprobe/gate_compose_test.go. Assertions 13-16 need an RTX 3090 and are +// stated as outstanding in .superpowers/sdd/task-13-gate-report.md. +func TestPhase6Gate(t *testing.T) { + // 1. Frames stop at the kernel. The §8 pin, and first for a reason: at PC + // sampling rates one frame per PC destroys aggregation and fragments the + // kernel's own block, so every other assertion here is about a label. + // Catches: promoting the PC offset, the stall reason, the source + // location or the attribution quality to a frame, in any spelling. + t.Run("assertion-01-frames-stop-at-the-kernel", TestProjectionAddsNoFrames) + t.Run("assertion-01-pc-is-not-stack-identity", TestProjectionKeepsPCOutOfStackIdentity) + + // 2. A source line is reached from a CPU stack. The Phase 6 exit + // condition, satisfied through labels rather than frames. New here - + // see the test's own comment for what no existing test covered. + t.Run("assertion-02-source-line-from-a-cpu-stack", TestGateASourceLineIsReachedFromACPUStack) + + // 3. No -lineinfo, no invention. + // Catches: synthesizing a nearest line, or a function's first line, for + // a module that carries no line table at all. + t.Run("assertion-03-no-lineinfo-no-invention", TestProjectionEmitsAllFourSrcStatuses) + t.Run("assertion-03-populations-aggregate-at-the-kernel", + TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel) + + // 4. All four gpu_src_status values reachable, each by the fixture that + // should produce it. Catches: a fifth value; a silent default; the two + // "cannot resolve" statuses collapsing into one. + t.Run("assertion-04-four-statuses-in-the-store", TestModuleStoreAllFourStatusesAreReachable) + t.Run("assertion-04-four-statuses-in-the-labels", TestProjectionEmitsAllFourSrcStatuses) + t.Run("assertion-04-status-enum-is-exhaustive", TestSrcStatusesIsExhaustiveAndStable) + t.Run("assertion-04-zero-value-is-not-a-status", TestSrcStatusZeroValueIsNotAStatus) + + // 5. Reconciliation: every emitted PC record lands in exactly one of + // attributed-exact, attributed-kernel, pending, evicted. + // Catches: a join that attaches samples without counting them; a + // refusal path that leaves a group pending without incrementing any + // counter - the easiest and most invisible mistake in that function. + t.Run("assertion-05-reconcile-pending-and-evicted", + TestConformancePCSampleReconciliationCoversPendingAndEvicted) + t.Run("assertion-05-reconcile-attributed-by-kernel", + TestConformancePCSampleReconciliationCoversAttributedByKernel) + t.Run("assertion-05-tier-b-samples-reconcile", TestTierBSamplesReconcile) + t.Run("assertion-05-join-accounts-for-every-group", TestTierBJoinAccountsForEveryGroup) + + // 6. Tier B does not collapse: 10,000 samples across 50 (crc, + // functionIndex) pairs attribute without loss to EvictedPendingSamples. + // Catches: the Task 8a pathology returning - a Tier B sample keyed on + // its (empty) correlation value, collapsing an entire process onto one + // pending entry and evicting the rest. + t.Run("assertion-06-tier-b-does-not-collapse", TestTierBPCSamplesDoNotCollapseOntoOneKey) + t.Run("assertion-06-distinct-function-indexes-split", TestTierBDistinctFunctionIndexSplitsGroups) + + // 7. Ambiguity is marked in its OWN label, and leaves gpu_ambiguous unset. + // Catches: reusing ExecutionView.Ambiguous for PC ambiguity, which would + // emit gpu_join="exact" gpu_ambiguous="true" on one sample - two + // unrelated facts on one boolean, and a counter that stops meaning what + // its name says. + t.Run("assertion-07-ambiguity-has-its-own-label", + TestTierBAmbiguityIsMarkedWithoutTouchingAmbiguous) + t.Run("assertion-07-never-coincides-with-gpu-ambiguous", + TestProjectionKernelAmbiguousNeverCoincidesWithGpuAmbiguous) + + // 8. Tier A disclosure: "true" inside a window, "false" outside, and + // "unknown" - never "false" - when Tier A ran and no window arrived. + // Catches: the one answer that must never be reachable by accident, a + // profile saying "not perturbed" when it means "cannot tell". + t.Run("assertion-08-true-inside-a-burst", TestSerializationMarksExecutionsOverlappingABurst) + t.Run("assertion-08-false-in-a-proven-gap", TestSerializationIsFalseInAProvenGap) + t.Run("assertion-08-unknown-when-no-window-arrived", TestSerializationIsUnknownWhenNoWindowsArrived) + t.Run("assertion-08-false-only-from-positive-evidence", + TestSerializationFalseIsOnlyEverReachedFromPositiveEvidence) + t.Run("assertion-08-label-is-unconditional-and-three-valued", + TestSerializedLabelIsUnconditionalAndHasThreeValues) + + // 9. Cardinality cap: past the ceiling gpu_pc is suppressed and NOTHING + // else, with an exact ProjectionPCLabelsSuppressed. + // Catches: a cap that also drops gpu_stall or gpu_src_* (the coarser, + // more actionable labels), and a suppression that is silent - a profile + // that lost its PC labels looks identical to one that never had any. + t.Run("assertion-09-cap-suppresses-gpu-pc-and-only-gpu-pc", + TestProjectionCapSuppressesGpuPCAndOnlyGpuPC) + t.Run("assertion-09-suppression-is-surfaced", TestProjectionCapIsSurfacedInJoinHealth) + + // 10b. Out-of-scope conditions refuse rather than guess. The graph clause + // is asserted at the level the product implements it - see the test. + t.Run("assertion-10b-two-devices-are-marked", TestTierBMultiDeviceProcessIsMarked) + t.Run("assertion-10b-multidevice-outranks-ambiguity", TestMultiDeviceOutranksAmbiguity) + t.Run("assertion-10b-open-window-is-unknown-never-false", + TestSerializationOpenWindowMakesEverythingFromItsStartUnknownAndNeverFalse) + t.Run("assertion-10b-tier-a-refusal-names-cuda-graphs", + TestSerializedIsRefusedWithoutAnExplicitAcknowledgement) + // The first clause of 10b - "a graph execution makes Tier A refuse to + // start" - is NOT implemented by the product. This pins that, and fails + // when it becomes implementable so the gate is updated rather than left + // claiming an assertion it never made. + t.Run("assertion-10b-graph-refusal-is-outstanding", + TestGateGraphExecutionRefusalIsNotAssertableYet) +} + +// gateCubinCRCs are the CRCs this file stores its fixtures under. The values +// are arbitrary - what matters is that the sample and the store agree, exactly +// as cubin_crc makes them agree on the wire. +const ( + gateCRCLineInfo uint64 = 0x6A7E0001 + gateCRCNoLineInfo uint64 = 0x6A7E0002 +) + +// fixtureSourceLines reads the .cu the cubin fixtures were built from, so an +// assertion about a source LINE can be checked against the source rather than +// against a number someone copied out of a run. +func fixtureSourceLines(t *testing.T, name string) []string { + t.Helper() + b, err := os.ReadFile(fixturePath(name)) + require.NoError(t, err, "fixture source %s", name) + return strings.Split(strings.TrimRight(string(b), "\n"), "\n") +} + +// TestGateASourceLineIsReachedFromACPUStack is gate assertion 2, and the Phase +// 6 exit condition: an instruction sampled inside a GPU kernel is reported at a +// named line of the CUDA source that produced it, on a sample whose frames are +// the CPU call path that launched the kernel. +// +// # Why this is not covered by anything that already exists +// +// Three tests each cover a segment and none covers the join of them: +// +// - TestTierBSampleReachesItsExecution drives a Tier B sample to its +// execution through the module, but the execution has no launch and +// therefore no CPU stack, and it never projects to a pprof sample; +// - TestProjectionEmitsAllFourSrcStatuses reaches "resolved" with real file +// and line, but from a hand-built ExecutionView with no Launch at all; +// - TestProjectionAddsNoFrames has a Launch with a CPU stack, but asserts the +// NEGATIVE - that nothing about the PC reached the frames. +// +// The exit condition is the conjunction: one pprof sample carrying BOTH. It is +// asserted here through a real Timeline, so the module join and the source +// resolution run for real rather than being assumed by a literal ExecutionView. +// +// The line is checked against internal/cubin/testdata/single.cu itself - not +// against a number copied out of a previous run - so "a real line of the +// fixture's source" is a fact about the fixture rather than a constant that +// could go stale with it. +// +// Mutations this catches: a projection that resolves source labels but loses +// the launch's frames (or vice versa); a join that reaches the execution but +// drops PCSamples so nothing projects; a store keyed on something the sample +// does not carry; a resolution that reports a line the cubin's table does not +// contain. +func TestGateASourceLineIsReachedFromACPUStack(t *testing.T) { + const pid = 4242 + b := fixture(t, "single_lineinfo.cubin") + store := NewModuleStore(ModuleStoreConfig{}) + require.NoError(t, store.Put(gateCRCLineInfo, b)) + fnIndex := symIndexOf(t, b, "addOne") + + tl := NewTimeline(TimelineConfig{Modules: store}) + + // A real CPU call path. The privileged half of the gate replaces these + // names with frames the DWARF walker produced from a live process; here + // what matters is that they are the LAUNCH's frames and that they survive + // to the projected sample unchanged. + corr := CorrelationID{Backend: BackendCUPTI, PID: pid, Value: "17"} + require.NoError(t, tl.EmitLaunch(GPUKernelLaunch{ + Correlation: corr, + KernelName: "addOne", + TimeNs: 10, + Launch: LaunchContext{ + PID: pid, + TimeNs: 10, + CPUStack: pp.FramesFromNames([]string{"main", "run_training_step", "cudaLaunchKernel"}), + SamplePeriod: 8, + }, + })) + + // Tier B: no correlation value on the sample. It reaches the execution + // through cubin_crc -> module -> function name, and nothing else. + require.NoError(t, tl.EmitPCSample(GPUPCSample{ + Correlation: CorrelationID{Backend: BackendCUPTI, PID: pid}, + Module: ModuleRef{Backend: BackendCUPTI, CRC: gateCRCLineInfo}, + FunctionIndex: fnIndex, + TimeNs: 20, + PCOffset: 0x10, + StallReason: "long_scoreboard", + Count: 1, + })) + require.NoError(t, tl.EmitExec(GPUKernelExec{ + Correlation: corr, + KernelName: "addOne", + StartNs: 30, + EndNs: 130, + })) + + snap := tl.Snapshot() + require.Len(t, snap.Executions, 1) + view := snap.Executions[0] + require.Len(t, view.PCSamples, 1, + "the sample never reached its execution, so nothing downstream can be the exit condition") + require.NotNil(t, view.Launch, "the execution lost the launch that carries the CPU stack") + + samples, _ := ProjectExecutionsWith(snap, ProjectionConfig{Modules: store}) + require.Len(t, samples, 1) + s := samples[0] + + // --- the CPU stack half. + names := frameNames(s.Stack) + assert.Equal(t, + []string{"main", "run_training_step", "cudaLaunchKernel", FrameLaunch, "[gpu:kernel:addOne]"}, + names, + "the sample's frames must be the launching CPU call path, the boundary marker and the kernel - nothing more and nothing less") + + // --- the source-line half. + require.Equal(t, "resolved", s.Labels["gpu_src_status"], + "the -lineinfo fixture's line table covers this pcOffset; anything else means the store never saw the module or never parsed it") + assert.Equal(t, "single.cu", s.Labels["gpu_src_file"], + "the basename, never the build-host path") + assert.Equal(t, "addOne", s.Labels["gpu_src_func"]) + require.Contains(t, s.Labels, "gpu_src_line") + + // The line names a real line of the fixture's own source. Read from the + // .cu rather than pinned as a constant: a fixture rebuilt from edited + // source would otherwise keep this assertion green while the label pointed + // somewhere else. + line, err := strconv.Atoi(s.Labels["gpu_src_line"]) + require.NoError(t, err) + src := fixtureSourceLines(t, "single.cu") + require.Positive(t, line) + require.LessOrEqual(t, line, len(src), + "gpu_src_line=%d is past the end of single.cu (%d lines): the label does not name a line of the source it claims", + line, len(src)) + body := strings.TrimSpace(src[line-1]) + assert.NotEmpty(t, body, "gpu_src_line names a blank line of single.cu") + t.Logf("assertion 2: %s -> %s:%s %q (stall=%s, pc=%s, attrib=%s)", + strings.Join(names, " -> "), s.Labels["gpu_src_file"], s.Labels["gpu_src_line"], + body, s.Labels["gpu_stall"], s.Labels["gpu_pc"], s.Labels["gpu_pc_attrib"]) + + // And the attribution quality is stated, not implied: one execution of this + // kernel was in the horizon, so the module join is not an inference about + // WHICH invocation. + assert.Equal(t, string(PCAttribKernel), s.Labels["gpu_pc_attrib"]) + assert.NotContains(t, s.Labels, "gpu_ambiguous", + "gpu_ambiguous means a heuristic LAUNCH join and nothing else; this join was exact") +} + +// TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel is the second half of +// gate assertion 3: "the kernel frame is unchanged, so the two populations +// still aggregate together at the kernel level". +// +// TestProjectionEmitsAllFourSrcStatuses already asserts that a no-lineinfo +// sample carries the explicit status and no location. What it does not assert - +// and what the gate's own wording asks for - is that the resolvable and +// unresolvable populations still SHARE A STACK. That is the property that +// decides whether a flame graph of a partially-built-with-lineinfo workload +// shows one kernel block or two, and it is not implied by the labels: a +// projection that appended the source location, or the status, or a +// "[gpu:src:unknown]" placeholder to the frames would satisfy every label +// assertion and split the block in half. +// +// Mutation this catches: any frame that varies with gpu_src_status. +func TestGateResolvedAndNoLineinfoAggregateAtTheSameKernel(t *testing.T) { + withInfo := fixture(t, "single_lineinfo.cubin") + noInfo := fixture(t, "single_nolineinfo.cubin") + st := NewModuleStore(ModuleStoreConfig{Capacity: 8}) + require.NoError(t, st.Put(gateCRCLineInfo, withInfo)) + require.NoError(t, st.Put(gateCRCNoLineInfo, noInfo)) + + // One execution of one kernel, sampled twice: once in a module built with + // -lineinfo and once in a module built without. On real hardware this is a + // process that links one library built each way. + view := pcView(PCAttribKernel, + pcSampleAt(gateCRCLineInfo, symIndexOf(t, withInfo, "addOne"), 0x10), + pcSampleAt(gateCRCNoLineInfo, symIndexOf(t, noInfo, "addOne"), 0x10), + ) + view.Launch = &GPUKernelLaunch{Launch: LaunchContext{ + CPUStack: pp.FramesFromNames([]string{"main"}), + }} + + samples, _ := ProjectExecutionsWith(Snapshot{Executions: []ExecutionView{view}}, + ProjectionConfig{Modules: st}) + require.Len(t, samples, 2) + + assert.Equal(t, "resolved", samples[0].Labels["gpu_src_status"]) + assert.Equal(t, "no-lineinfo", samples[1].Labels["gpu_src_status"]) + for _, forbidden := range []string{"gpu_src_file", "gpu_src_line", "gpu_src_func"} { + assert.NotContains(t, samples[1].Labels, forbidden, + "no-lineinfo must invent nothing: %s rode on a sample with no line table behind it", forbidden) + } + + assert.Equal(t, frameNames(samples[0].Stack), frameNames(samples[1].Stack), + "the resolvable and unresolvable populations must aggregate at the same kernel; a frame that varies with gpu_src_status splits the kernel's own block in two") + assert.Equal(t, []string{"main", FrameLaunch, "[gpu:kernel:addOne]"}, frameNames(samples[0].Stack)) +} + +// TestGateGraphExecutionRefusalIsNotAssertableYet is gate assertion 10b's +// FIRST clause - "a graph execution makes Tier A refuse to start" - pinned as +// an outstanding gap rather than asserted, because the product does not +// implement it. +// +// # What exists and what does not +// +// The wire signal exists: the plan's Task 6 gave the adapter a +// classGraphExec drop class, `internal/gpuabi.DropClassGraphExec` decodes it +// and spells it "graph-exec", and the stub emits one such record so the class +// is reachable from a test. The operator warning names CUDA graphs, and the +// Tier A acknowledgement refusal names them too. +// +// What does not exist is the REFUSAL the plan specifies: +// +// Tier A refuses to start in a process where graph executions have been +// observed, because Tier A's whole claim is exact launch attribution and a +// graph makes that claim false. The refusal is loud and counted, not a +// silent downgrade to Tier B. +// +// Nothing in gpu/, gpuprobe/ or cmd/ consumes DropClassGraphExec. The +// consumer decodes gpu_dropped_v1 into batch.Drops and stops there (see +// Stats.Undecoded's own comment, which says normalizing a drop class is the +// task after Task 7). So there is no counter for graph executions, nothing on +// the Snapshot that names them, no joinhealth anomaly, and no input by which +// PCSamplingRequest could be told about them. Tier A therefore starts happily +// in a graph-using process and produces confident, exact-LOOKING attribution +// of N kernels to one call site - which is finding 4 of the plan, and the +// condition it says must be visible rather than merely out of scope. +// +// # Why this is a passing test rather than a failing one +// +// It is the shape issue #44 used and #45 inverted: an assertion that pins the +// CURRENT state, with the note that fixing the defect must fail it. Writing +// the real assertion now would leave the gate red on a branch that is not +// allowed to change product behaviour; leaving nothing at all would let the +// gate ship claiming twelve assertions while one of them was never written. +// +// When Task 10's refusal lands, this test fails - by name, in the gate's own +// file - and the person landing it replaces it with the real assertion: +// a stub reporting a graph execution makes Tier A refuse to start, loudly and +// counted. +func TestGateGraphExecutionRefusalIsNotAssertableYet(t *testing.T) { + mentionsGraph := func(v any) []string { + var out []string + typ := reflect.TypeOf(v) + for i := range typ.NumField() { + if name := typ.Field(i).Name; strings.Contains(strings.ToLower(name), "graph") { + out = append(out, name) + } + } + return out + } + for _, tc := range []struct { + name string + v any + }{ + {"Snapshot", Snapshot{}}, + {"TimelineDropStats", TimelineDropStats{}}, + {"PCJoinStats", PCJoinStats{}}, + {"TimelineConfig", TimelineConfig{}}, + {"PCSamplingRequest", PCSamplingRequest{}}, + } { + assert.Empty(t, mentionsGraph(tc.v), + "%s now names graph executions (%v). Gate assertion 10b's first clause has become "+ + "assertable: replace this test with the real one - a stub reporting a graph "+ + "execution makes Tier A refuse to start, loudly and counted.", + tc.name, mentionsGraph(tc.v)) + } + + // The half that IS true today, so this test is not purely negative: the + // operator is told, in the refusal they must read before Tier A can run at + // all, that the tier is unavailable where graphs are in use. + _, err := PCSamplingRequest{Flag: "serialized"}.Select() + require.Error(t, err) + assert.Contains(t, err.Error(), "CUDA graphs", + "the only place the graph limitation is stated to an operator is the Tier A acknowledgement refusal; if that text loses it, nothing anywhere names the condition") + warning := strings.Join(PCSamplingStandingWarning(PCSamplingSerialized), "\n") + assert.Contains(t, warning, "CUDA GRAPHS", + "the standing warning must keep naming the third perturbation") + t.Log("gate assertion 10b, first clause: OUTSTANDING - see this test's doc comment and " + + ".superpowers/sdd/task-13-gate-report.md") +} diff --git a/gpuprobe/gate_compose_test.go b/gpuprobe/gate_compose_test.go new file mode 100644 index 0000000..45af8e7 --- /dev/null +++ b/gpuprobe/gate_compose_test.go @@ -0,0 +1,276 @@ +package gpuprobe + +import ( + "os" + "os/exec" + "reflect" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "kernel.org/pub/linux/libs/security/libcap/cap" +) + +// TestPhase6GateConsumerHalf is the consumer-side quarter of the Phase 6 phase +// gate: assertions 10, 10a, 11 and 12 of Task 13. +// +// It is a SECOND gate entry point, in package gpuprobe rather than +// gpuprobe_test, and both halves of that are load-bearing: +// +// - second, because TestStubDrivesThePipelineToPprofWithoutAGPU and the +// end-to-end PC-sampling gate beside it need CAP_BPF, CAP_PERFMON and +// CAP_CHECKPOINT_RESTORE and SKIP without them. Assertions 10, 10a, 11 and +// 12 need no capability at all - the cubin channel is an AF_UNIX socket and +// a memfd, the decode path is pure Go - so putting them behind the +// privileged skip would be putting four assertions behind a gate nobody +// without a GPU box can run; +// - in-package, because the tests these compose are in-package. Calling them +// is the point: the plan asks the gate to COMPOSE the assertions the tasks +// already made rather than to write second copies of them, and a second +// copy of an assertion is two assertions of one fact that drift apart. +// +// See gpu/gate_test.go for assertions 1-9 and 10b, and +// gpuprobe/gate_test.go for the privileged end-to-end run. +func TestPhase6GateConsumerHalf(t *testing.T) { + // 10. The cubin channel cannot touch enrolment, in BOTH orders - + // including offers flooded AHEAD of an enrolment, which is the + // direction a shared socket fails and the easy direction is not. + // CubinsThrottled non-zero while UnwindEnrollThrottled is unchanged, + // and the enrolment still succeeds. + // + // Catches: moving cubin offers onto the enrolment listener, its + // goroutine, its Accept loop or its admission bucket. Any of those + // restores issue #49's ~38% stack loss on a module-heavy workload, + // silently, with only UnwindEnrollThrottled moving. + t.Run("assertion-10-cubins-cannot-starve-or-throttle-enrolment", + TestFloodingTheCubinChannelCannotStarveOrThrottleAnEnrolment) + t.Run("assertion-10-the-buckets-are-separate-objects", TestTheCubinAdmissionBucketIsItsOwn) + t.Run("assertion-10-they-are-not-the-same-socket", + TestTheCubinAddressIsASiblingOfTheRendezvousAndNotTheSameSocket) + // The regression that keeps them apart forever. The enrolment handler must + // never read from its connection: a read would block until the producer's + // 2s budget expired, turning every rendezvous into a 2s stall ending in + // kEnrollError. Discriminating an offer from an enrolment on one socket + // REQUIRES such a read, so this is the assertion that makes the shared + // socket unbuildable rather than merely unbuilt. + t.Run("assertion-10-enrolment-performs-no-read", TestAnEnrolmentCompletesWithNoReadOnThatConnection) + + // 10a. Seals are enforced: a memfd missing ANY required seal is rejected, + // counted in CubinsRejectedUnsealed, and never mapped. + // + // Catches: dropping a seal from the required set, or - worse - a + // fallback that maps it anyway. Without F_SEAL_SHRINK a peer can + // ftruncate under our mmap and SIGBUS the agent; without F_SEAL_WRITE + // the ELF mutates under the parser mid-parse. Falling back is how a + // defended path becomes an undefended one. + t.Run("assertion-10a-each-missing-seal-is-rejected-and-never-mapped", + TestEachRequiredSealMissingInTurnIsRejectedAndNeverMapped) + t.Run("assertion-10a-a-plain-fd-is-not-a-sealed-memfd", TestADescriptorThatIsNotASealedMemfdIsRefused) + t.Run("assertion-10a-the-required-seals-are-spelt-out", TestSealNamesAreSpeltOut) + + // 11. The capability set does not grow: no cap_sys_admin, anywhere. + t.Run("assertion-11-no-cap-sys-admin", TestPhase6GateBinaryDoesNotAskForCapSysAdmin) + + // 12. Stats.Undecoded is zero for every kind this phase decodes. + // + // Catches: a KIND_* added on one side of the wire and not the other, + // which is silent loss unless counted - and, in the other direction, a + // decode arm removed, which would put a kind back into the default arm + // while every other counter still read healthy. + t.Run("assertion-12-the-five-pc-kinds-are-decoded", + TestTheFivePCSamplingKindsAreDecodedNotCountedUndecoded) + t.Run("assertion-12-a-healthy-tier-b-run-loses-nothing", + TestHealthyTierBRunLeavesEveryNewLossCounterAtZero) + t.Run("assertion-12-unknown-kinds-are-still-counted", TestUndecodedKindsAreCountedNotDropped) + t.Run("assertion-12-kindmax-matches-the-bpf-object", TestEmbeddedProgramIsUprobeMulti) + t.Run("assertion-12-every-cookie-has-a-sized-kind", TestBPFSizesEveryKindCookieForInstalls) + + // Not one of the twelve: the wiring gap that stops assertion 2 from being + // reachable by the shipping product. Pinned here so it fails the moment it + // is closed, rather than being remembered only in a report. + t.Run("outstanding-cubin-transport-does-not-feed-the-module-store", + TestGateTheCubinTransportDoesNotYetFeedTheModuleStore) +} + +// TestPhase6GateBinaryDoesNotAskForCapSysAdmin is gate assertion 11, the +// standing Phase 1 assertion: the capability set this pipeline runs under is +// cap_bpf,cap_perfmon,cap_checkpoint_restore and it does not grow. +// +// # Why this is worth a test rather than a README line +// +// CAP_SYS_ADMIN is the difference between an agent that can run as one pod +// among many and an agent that is effectively root on the node. Nothing about +// losing it is loud: the two ways to acquire the requirement are both quiet +// one-liners a reviewer would wave through - +// +// - attaching with link.Uprobe instead of link.UprobeMulti, which routes +// through the perf_uprobe PMU (measured: needs CAP_SYS_ADMIN, while the BPF +// link does not - see Attach's doc comment and +// TestEmbeddedProgramIsUprobeMulti, composed above); +// - reading the producer's address space to follow gpu_module_load_v1's +// bytes_ptr, which needs /proc//mem or process_vm_readv and therefore +// CAP_SYS_PTRACE, and which the whole cubin channel exists to avoid. +// +// Both would be found on a developer box that runs the tests as root, where +// everything works and nothing says why. So this asserts the negative from two +// independent directions, because either one alone is escapable: +// +// 1. the FILE capabilities of the test binary itself, read with getcap(8) - +// which is the plan's own wording, and which is the thing a CI job or a +// developer following the plan's setcap line actually installs; +// 2. this process's own Permitted set, which is where a capability acquired +// any other way (a setuid wrapper, an inherited ambient set) would show up. +// +// Running as root makes (2) vacuous - root has the full set - so it is skipped +// there rather than asserted falsely, and (1) still runs, because a file +// capability set naming cap_sys_admin is wrong whoever executes it. +func TestPhase6GateBinaryDoesNotAskForCapSysAdmin(t *testing.T) { + self, err := os.Executable() + require.NoError(t, err) + + // --- 1. the file capabilities of this very binary. + // + // cap.GetFile rather than shelling out to getcap(8): it reads the same + // security.capability xattr getcap prints, needs no external tool, and + // cannot be defeated by getcap being absent from a container image - which + // would otherwise turn this assertion into a skip on exactly the machines + // that most need it. + set, err := cap.GetFile(self) + switch { + case err != nil: + // No file capabilities at all. That is the ordinary state for an + // unprivileged `go test` run and for a sudo run, and it is the + // strongest possible form of "no cap_sys_admin". + t.Logf("no file capabilities on %s (%v): nothing to grant cap_sys_admin", self, err) + default: + for _, flag := range []cap.Flag{cap.Effective, cap.Permitted, cap.Inheritable} { + have, gerr := set.GetFlag(flag, cap.SYS_ADMIN) + require.NoError(t, gerr) + assert.False(t, have, + "the gate binary's file capabilities name cap_sys_admin in the %v set (%s): "+ + "this pipeline runs on cap_bpf,cap_perfmon,cap_checkpoint_restore and the "+ + "capability set does not grow. The two ways to acquire this requirement are "+ + "attaching through the perf_uprobe PMU instead of a uprobe_multi BPF link, "+ + "and reading the producer's address space to follow bytes_ptr", + flag, set) + } + t.Logf("file capabilities on %s: %s", self, set) + } + + // --- 2. this process's own set. + if os.Geteuid() == 0 { + t.Log("running as root: the process capability half of this assertion is vacuous and is skipped; " + + "the file-capability half above still ran") + return + } + proc := cap.GetProc() + require.NotNil(t, proc) + for _, flag := range []cap.Flag{cap.Effective, cap.Permitted} { + have, gerr := proc.GetFlag(flag, cap.SYS_ADMIN) + require.NoError(t, gerr) + assert.False(t, have, + "this process holds cap_sys_admin in the %v set (%s); the gate is supposed to prove "+ + "the pipeline works WITHOUT it, and a run that holds it cannot", + flag, proc) + } + t.Logf("process capabilities: %s", proc) +} + +// TestGetcapAgreesWithTheLibraryReading cross-checks the assertion above +// against the tool the plan's wording actually names. +// +// Assertion 11 is written as "getcap on the gate binary shows no +// cap_sys_admin". The test above reads the same xattr through libcap rather +// than through getcap(8), which is stricter in one direction (it cannot be +// skipped by the tool being missing) and weaker in another: if libcap's +// reading and getcap's ever disagreed, the assertion would be about something +// other than what an operator sees. So when getcap IS available, run it and +// require the two to agree. +// +// Skipped, not failed, when getcap is absent: the assertion it backs has +// already run through libcap, and a missing binutils-ish tool is not a defect +// in this pipeline. +func TestGetcapAgreesWithTheLibraryReading(t *testing.T) { + path, err := exec.LookPath("getcap") + if err != nil { + t.Skip("getcap(8) unavailable; TestPhase6GateBinaryDoesNotAskForCapSysAdmin " + + "reads the same xattr through libcap and has already run") + } + self, err := os.Executable() + require.NoError(t, err) + + out, err := exec.Command(path, self).CombinedOutput() + require.NoError(t, err, "getcap %s: %s", self, out) + text := string(out) + t.Logf("getcap %s -> %q", self, strings.TrimSpace(text)) + assert.NotContains(t, strings.ToLower(text), "cap_sys_admin", + "getcap names cap_sys_admin on the gate binary") +} + +// TestGateTheCubinTransportDoesNotYetFeedTheModuleStore pins the gap that +// stops the Phase 6 exit condition from being reachable by the shipping +// product, so that it is a named, self-invalidating fact rather than prose in +// a report nobody re-reads. +// +// # The gap +// +// Task 3 built the cubin channel and Task 4 built gpu.ModuleStore - the store +// that turns (cubin_crc, functionIndex, pcOffset) into a source line and is +// "the single place gpu_src_status is decided". Nothing connects them: +// +// - Attach calls newCubinListener(cfg, nil), and a nil sink becomes +// memCubinStore - a bounded map of CRC to bytes with no line table, no LRU +// and no Resolve. Its own comment says "Task 4 replaces it"; +// - gpuprobe.Config has no field by which a caller could supply one, and +// gpu.ModuleStore does not implement cubinSink (Put/HasCubin versus +// PutCubin/HasCubin) even if it had; +// - cmd/gpu-cuda-profile builds neither: it calls ProjectExecutionsWith with +// a zero ProjectionConfig and NewTimeline without Modules. +// +// So on hardware today, every cubin is received, sealed, verified, size- and +// identity-checked, stored - and then never read. Every PC sample in a real +// profile reads gpu_src_status="no-module", and gate assertion 2 (a source +// line reached from a CPU stack) cannot be satisfied end to end by the product +// as shipped. TestStubDrivesPCSamplingToPprofWithoutAGPU therefore builds the +// store itself, from the CRCs the producer declared, and says so. +// +// This is the ONE hop between the transport and the labels, and both ends of +// it are built and tested. It is a wiring task, not a design gap. +// +// # Why a passing test rather than a failing one +// +// Same reason as gpu's TestGateGraphExecutionRefusalIsNotAssertableYet: the +// gate must not ship red, and it must not ship silently short of an assertion +// either. When the hop is wired, this test fails - by name, in the gate's own +// file - and the person wiring it deletes it and drops the injection from the +// end-to-end gate. +func TestGateTheCubinTransportDoesNotYetFeedTheModuleStore(t *testing.T) { + // The default sink a real Attach installs. + l, err := newCubinListener(Config{ShimPath: selfExe(t)}, nil) + require.NoError(t, err) + defer func() { _ = l.close() }() + + _, isPlaceholder := l.sink.(*memCubinStore) + assert.True(t, isPlaceholder, + "the cubin listener's default sink is no longer the placeholder memCubinStore (it is %T). "+ + "If that is gpu.ModuleStore, gate assertion 2 is now reachable end to end: delete this "+ + "test and drop the PC-sample injection from TestStubDrivesPCSamplingToPprofWithoutAGPU.", + l.sink) + + // And no caller can supply one, which is why the default is the whole + // story rather than merely a default. + typ := reflect.TypeOf(Config{}) + for i := range typ.NumField() { + name := strings.ToLower(typ.Field(i).Name) + assert.NotContains(t, name, "module", + "gpuprobe.Config grew a field naming a module store (%s); the hop may now be wired - see this test's doc comment", + typ.Field(i).Name) + assert.NotContains(t, name, "store", + "gpuprobe.Config grew a store field (%s); the hop may now be wired - see this test's doc comment", + typ.Field(i).Name) + } + t.Log("cubin transport -> gpu.ModuleStore: OUTSTANDING - the bytes arrive and stop at " + + "memCubinStore; see .superpowers/sdd/task-13-gate-report.md") +} diff --git a/gpuprobe/gate_test.go b/gpuprobe/gate_test.go index 7420d49..c3be386 100644 --- a/gpuprobe/gate_test.go +++ b/gpuprobe/gate_test.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "debug/elf" + "fmt" "os" "os/exec" "path/filepath" @@ -21,6 +22,9 @@ import ( "github.com/dpsoft/perf-agent/gpu" "github.com/dpsoft/perf-agent/gpuprobe" + "github.com/dpsoft/perf-agent/internal/cubin" + "github.com/dpsoft/perf-agent/internal/gpuabi" + pp "github.com/dpsoft/perf-agent/pprof" "github.com/dpsoft/perf-agent/symbolize" "github.com/dpsoft/perf-agent/unwind/ehcompile" ) @@ -1238,3 +1242,802 @@ func TestADlopenedProducerEnrollsBeforeItsFirstLaunch(t *testing.T) { // proving nothing, which is the failure this whole test exists to // correct; the DWARF assertion above is the derived one. } + +// --------------------------------------------------------------------------- +// Phase 6: the PC-sampling gate +// --------------------------------------------------------------------------- + +// gateCRCAbsent is a CRC no cubin is ever stored under, so a PC sample +// carrying it must read gpu_src_status="no-module". The two REAL fixture CRCs +// are not constants here: they are read out of the producer's own report and +// cross-checked against the bytes, so the number the store keys on is the +// number the producer put on the wire rather than one this test invented. See +// gateModuleCRCs. +const gateCRCAbsent uint64 = 0x6A7E0003 + +// TestStubDrivesPCSamplingToPprofWithoutAGPU is the end-to-end half of the +// Phase 6 phase gate (Task 13, assertions 1-5, 8, 9 and 12), driven by the +// same GPU-free producer TestStubDrivesThePipelineToPprofWithoutAGPU uses. +// +// It is a SECOND test rather than an extension of that one, deliberately. That +// test asserts an exact sampled count, reached-root == dwarf, zero abandoned, +// NoTables == 0 and a dozen other equalities that describe a run with PC +// sampling OFF - `require.Len(samples, len(snap.Executions))` among them, which +// is true precisely because the stub emits no PC samples there. Turning PC +// sampling on inside it would have meant weakening those. So the baseline stays +// exactly as it was and this runs the same producer in the PC-sampling +// configuration beside it. +// +// # What comes off the wire, and what does not +// +// Off the wire, from a real producer through a real uprobe_multi link: +// +// - 500 launches and 500 executions, 58 of the launches carrying a CPU stack +// walked through two -fomit-frame-pointer frames using this consumer's own +// compiled CFI, symbolized against the live process; +// - two real checked-in cubins, over the cubin channel, as sealed memfds +// passed by SCM_RIGHTS; +// - 64 PC-sample records, a stall-reason map, a config record and Tier A +// sampling windows. +// +// NOT off the wire, and this is a finding rather than a shortcut - see +// .superpowers/sdd/task-13-gate-report.md: +// +// The stub's PC records cannot be attributed to anything. Their cubin_crc is +// a pair of synthetic constants (shim/stub/stub.cc kStubCubinCRC = +// {0xC0FFEE01, 0xC0FFEE02}) unrelated to the cubins the same run delivers +// over the cubin channel, and their correlation is 0 in every tier. Tier B +// attribution runs crc -> module -> function name -> the execution's +// KernelName, and the stub's kernel names are "kernel_1111"/"kernel_2222" +// while the fixtures' only functions are the CUDA kernels they were compiled +// from. So neither join path can fire: every one of those 64 records is +// correctly counted as pending, and the gate asserts that exactly. +// +// Assertions 2, 3, 4 and 9 need a PC sample that DOES reach an execution, so +// the gate supplies those itself at Timeline.EmitPCSample - the same entry +// point the consumer calls - on correlations that a real, wire-delivered, +// stack-carrying launch is known to occupy. Everything downstream of that entry +// point is product code: the join, the module store's four-valued resolution, +// the projection's label set, the cardinality budget. What the injection skips +// is the consumer's decode arm for KIND_PC, and that is asserted separately and +// exactly by the 64 records above. +// +// # Assertion 11 is not here +// +// getcap on the gate binary is asserted in gate_compose_test.go, where it runs +// without capabilities. Asserting "this pipeline needs no cap_sys_admin" only +// on machines that hold enough privilege to run this test would be asserting it +// in the one place it cannot be checked usefully. +func TestStubDrivesPCSamplingToPprofWithoutAGPU(t *testing.T) { + if !hasGateCaps() { + t.Skip("needs CAP_BPF, CAP_PERFMON and CAP_CHECKPOINT_RESTORE " + + "(the last so blazesym can follow /proc//map_files/); " + + "sudo setcap cap_bpf,cap_perfmon,cap_checkpoint_restore+ep . " + + "gpu/gate_test.go and gpuprobe/gate_compose_test.go assert the same twelve " + + "points without privilege; this adds the end-to-end run") + } + built := filepath.Join("..", "shim", "perfagent-gpu-fpless") + requireBuilt(t, built) + requireFPLess(t, built) + stub := privateStubCopy(t, built) + + // The module store. It is built HERE, and filled after the run from the + // CRCs the PRODUCER declared, rather than by the consumer - and that is + // the second finding this gate records rather than papers over: + // + // The cubin listener's sink is gpuprobe's own bounded memCubinStore + // (gpuprobe/cubin.go). Nothing in gpuprobe, in gpu, or in + // cmd/gpu-cuda-profile ever hands a gpu.ModuleStore to gpuprobe.Config + // or to gpu.ProjectionConfig. The bytes cross the socket, are sealed, + // verified and stored - and stop there. As shipped, every PC sample in a + // real profile therefore reads gpu_src_status="no-module". + // + // CubinsReceived and snap.Modules are asserted below so the transport half + // is still proven end to end on this run, and the missing hop is one named + // gap rather than an invisible one. + lineInfo := readFixture(t, "single_lineinfo.cubin") + noLineInfo := readFixture(t, "single_nolineinfo.cubin") + store := gpu.NewModuleStore(gpu.ModuleStoreConfig{}) + lineIdx := fixtureSymIndex(t, lineInfo, "addOne") + noLineIdx := fixtureSymIndex(t, noLineInfo, "addOne") + + sym, err := symbolize.NewLocalSymbolizer() + require.NoError(t, err) + defer func() { _ = sym.Close() }() + + // PCSamplingSerialized, not the zero value: the tier is what decides + // whether an execution with no covering window reads "unknown" or "false", + // and a Timeline told nothing would answer "false" for every execution in + // this run - correctly, since nothing would then have been serialized, and + // uselessly, since the producer IS emitting windows. + timeline := gpu.NewTimeline(gpu.TimelineConfig{ + PCSampling: gpu.PCSamplingSerialized, + Modules: store, + }) + c, err := gpuprobe.Attach(gpuprobe.Config{ + ShimPath: stub, + Backend: gpu.GPUBackendID("stub"), + Sink: timeline, + Symbolizer: sym, + }) + require.NoError(t, err) + defer func() { _ = c.Close() }() + require.True(t, c.Stats().CubinsListening, + "the cubin channel did not bind, so no module can arrive and every source label would read no-module for a reason that has nothing to do with this phase: %q", + c.Stats().CubinsLastError) + + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { done <- c.Run(ctx) }() + + const ( + wantSampled = 58 // gpuabi.SampleSchedule(500, 8, DefaultSampleSeed) + wantPC = 64 + wantWindows = 4 + wantCubins = 2 + wantDropRecs = 4 // one gpu_dropped_v1 per drop class, from the stub + ) + cmd := exec.Command(stub, "500", "1000", "8", "10000") + cmd.Env = append(os.Environ(), + // The tier is the OUTER gate in the stub exactly as in the CUPTI + // adapter (shim/core/pctier.h): with it off, none of the four + // PC-sampling probes fires whatever the knobs below say. + "PERFAGENT_GPU_PC_SAMPLING=serialized", + "PERFAGENT_STUB_PC_SAMPLES="+strconv.Itoa(wantPC), + "PERFAGENT_STUB_SAMPLING_WINDOWS="+strconv.Itoa(wantWindows), + "PERFAGENT_STUB_CUBINS="+ + mustAbs(t, fixturePath("single_lineinfo.cubin"))+":"+ + mustAbs(t, fixturePath("single_nolineinfo.cubin")), + ) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + release, err := cmd.StdinPipe() + require.NoError(t, err) + require.NoError(t, cmd.Start()) + defer func() { _ = release.Close() }() + + // Hold the producer open until every sampled stack has been symbolized + // against its still-live /proc//maps, exactly as the baseline gate + // does. The cubins are waited for too: the stub offers them from its drain + // thread before its first launch, so they are the earliest thing to + // arrive, and a run that started asserting before they landed would report + // "no-module" for a scheduling reason. + deadline := time.Now().Add(20 * time.Second) + for { + st := c.Stats() + if st.SampledLaunches >= wantSampled && st.CubinsReceived >= wantCubins { + break + } + if time.Now().After(deadline) { + _ = release.Close() + _ = cmd.Process.Kill() + _ = cmd.Wait() + cancel() + <-done + t.Fatalf("timed out: sampled=%d/%d cubins=%d/%d. stats: %+v stderr: %s", + st.SampledLaunches, wantSampled, st.CubinsReceived, wantCubins, st, stderr.String()) + } + time.Sleep(5 * time.Millisecond) + } + stubPID := cmd.Process.Pid + require.NoError(t, release.Close()) + require.NoError(t, cmd.Wait(), "stdout: %s stderr: %s", stdout.String(), stderr.String()) + stubErr := stderr.String() + + // The CRC the producer declared for each fixture, from its own report, and + // the store keyed on those rather than on numbers this test chose. + // + // This is the offline half of hardware assertion 13 ("cuptiGetCubinCrc() + // over the received copy equals the PC records' cubinCrc"). The stub's + // crc is FNV-1a rather than CUPTI's unpublished polynomial, but the + // property under test is the same one and it is the one the join needs: + // ONE number identifies one set of bytes, and the same number reaches both + // ends. Recomputing it here over the checked-in fixture proves the number + // is a function of the bytes and not of the load order, the module id or + // the path. + crcs := gateModuleCRCs(t, stubErr) + require.Len(t, crcs, 2, "the producer reported %d captured modules, not 2: %s", len(crcs), stubErr) + gateCRCLineInfo, gateCRCNoLineInfo := crcs[0], crcs[1] + assert.Equal(t, stubCubinCRC(lineInfo), gateCRCLineInfo, + "the CRC the producer declared for single_lineinfo.cubin is not a content hash of those bytes, so it cannot identify them on the wire") + assert.Equal(t, stubCubinCRC(noLineInfo), gateCRCNoLineInfo, + "same, for single_nolineinfo.cubin") + assert.NotEqual(t, gateCRCLineInfo, gateCRCNoLineInfo, + "two different cubins collided on one CRC, which would make them indistinguishable to the store") + require.NoError(t, store.Put(gateCRCLineInfo, lineInfo)) + require.NoError(t, store.Put(gateCRCNoLineInfo, noLineInfo)) + + // The tail: PC batches, the stall map, the config record, the windows and + // the drop records are all flushed after the last launch, so they are the + // last things on the wire. + waitFor(t, 10*time.Second, func() bool { + st := c.Stats() + return st.PCSamplesDecoded >= wantPC && + st.SamplingWindowsDecoded > 0 && + st.ConfigsDecoded > 0 && + st.PendingStallSamples == 0 + }, func() string { + st := c.Stats() + return fmt.Sprintf("pc=%d/%d windows=%d configs=%d pending-stall=%d stderr: %s", + st.PCSamplesDecoded, wantPC, st.SamplingWindowsDecoded, st.ConfigsDecoded, + st.PendingStallSamples, stubErr) + }) + cancel() + <-done + + stats := c.Stats() + t.Logf("stub stderr:\n%s", stubErr) + t.Logf("pc sampling: pc=%d modules=%d stall-names=%d windows=%d(open=%d) configs=%d cubins=%d/%dB undecoded=%d", + stats.PCSamplesDecoded, stats.ModulesDecoded, stats.StallNamesLearned, + stats.SamplingWindowsDecoded, stats.SamplingWindowsOpen, stats.ConfigsDecoded, + stats.CubinsReceived, stats.CubinBytesReceived, stats.Undecoded) + + // ---- the transport, and the producer's own account of it -------------- + assert.Contains(t, stubErr, "pc_sampling=serialized", + "the producer did not take the tier it was handed, so nothing below describes Tier A") + assert.Contains(t, stubErr, "launch_dropped=0") + assert.Contains(t, stubErr, "exec_dropped=0") + assert.Zero(t, stats.SequenceGaps, "no batch may be lost silently") + assert.Zero(t, stats.Malformed, "reasons: %v", stats.DecodeFailures) + assert.Zero(t, stats.KernelDropped) + assert.Zero(t, stats.SinkRejected) + + // Assertion 12. Undecoded is zero for every kind THIS PHASE DECODES, and + // gpu_dropped_v1 is not one of them: it is decoded into batch.Drops and + // carried, and normalizing a drop class into an operator-visible number is + // the consumer task after this one (see Stats.Undecoded's own comment). The + // stub emits exactly one record per drop class when PC sampling is on, so + // this is an equality rather than a tolerance - a sixth undecoded record + // would mean a kind arrived that nothing on this side knows about, which is + // silent loss. + assert.Equal(t, uint64(wantDropRecs), stats.Undecoded, + "the only undecoded records in a PC-sampling stub run are the four gpu_dropped_v1 class records; anything else is a KIND_* added on one side of the wire and not the other") + assert.Equal(t, uint64(wantPC), stats.PCSamplesDecoded, + "every PC record the stub emitted must have been decoded, not carried") + assert.Equal(t, uint64(wantCubins), stats.CubinsReceived, + "both checked-in cubins must have crossed the cubin channel: err=%q", stats.CubinsLastError) + assert.Equal(t, uint64(2), stats.ModulesDecoded, + "gpu_module_load_v1 announces the load; the bytes travel separately, and both must arrive") + assert.Zero(t, stats.CubinsRejectedUnsealed, "a sealed memfd was refused: %q", stats.CubinsLastError) + assert.Zero(t, stats.CubinsRejectedTooLarge) + assert.Zero(t, stats.CubinsRejectedMalformed) + assert.Zero(t, stats.CubinsRejectedUnauthorized) + assert.Zero(t, stats.CubinsThrottled, "two offers cannot exhaust the cubin bucket") + // The isolation property, on a live run rather than in the unit test that + // proves it structurally (gate_compose_test.go assertion 10): real cubin + // traffic crossed while a real enrolment happened, and the enrolment's own + // counters are untouched. + assert.Zero(t, stats.UnwindEnrollThrottled, + "cubin traffic spent an enrolment admission token: %q", stats.UnwindEnrollLastError) + assert.Equal(t, uint64(1), stats.UnwindEnrollConfirmed) + assert.Zero(t, stats.StacksWalkedNoTables, + "a capture was walked with no CFI tables: enroll requests=%d confirmed=%d err=%q", + stats.UnwindEnrollRequests, stats.UnwindEnrollConfirmed, stats.UnwindEnrollLastError) + + // Stall names: the stub emits the map AFTER the batches on purpose, so + // these two zeros are the assertion that the consumer held the samples + // rather than rendering "stall#17" or dropping them. + assert.Zero(t, stats.StallNamesMissing, + "a PC sample carried a stall index the map never named; the label would then be empty") + assert.Zero(t, stats.PendingStallSamples, + "PC samples are still parked waiting for a stall name that has arrived") + // The gauge, not the counter: the stall map replays on the attach edge + // (spec §6.1's replay contract), so StallNamesLearned counts RECORDS and a + // replayed map legitimately doubles it. What must be exactly 8 is the + // number of distinct indexes the table ends up holding. + assert.Equal(t, 8, stats.KnownStallNames, + "the stub's synthetic GA102 table has 8 entries; a shortfall means a name was evicted or never interned") + assert.GreaterOrEqual(t, stats.StallNamesLearned, uint64(8)) + assert.Zero(t, stats.StallNamesEvicted, + "an 8-entry table cannot overflow the default bound; an eviction here costs PC samples their stall label") + + // Tier B's own signature, kept apart from the aggregate: every PC record + // in CONTINUOUS collection has correlation 0, so ZeroCorrelation stops + // being an anomaly signal for that population and this counter is what + // carries it instead. + assert.Equal(t, uint64(wantPC), stats.PCSamplesWithoutCorrelation, + "the stub emits correlation 0 on every PC record, in both tiers") + + // ---- the injected samples --------------------------------------------- + // + // See the doc comment: the stub cannot produce a PC record that attributes + // to anything, so assertions 2, 3, 4 and 9 are driven from records the gate + // emits into the same Timeline the consumer emits into, on correlations + // that a wire-delivered, stack-carrying launch occupies. + // + // SampleSchedule replays the shim's own sampler exactly (it is pinned + // against it by TestTheGoReplicaMatchesTheShimSampler), so these + // correlations are not guesses: launch ordinal N carries correlation N+1 + // and, if N is in the schedule, a CPU stack. + sched := gpuabi.SampleSchedule(500, 8, gpuabi.DefaultSampleSeed) + require.Equal(t, wantSampled, len(sched)) + corrOf := func(i int) gpu.CorrelationID { + return gpu.CorrelationID{ + Backend: gpu.GPUBackendID("stub"), + PID: uint32(stubPID), + Value: strconv.FormatUint(sched[i]+1, 10), + } + } + inject := func(corr gpu.CorrelationID, crc uint64, fnIndex uint32, pcOffset uint64) { + t.Helper() + require.NoError(t, timeline.EmitPCSample(gpu.GPUPCSample{ + Correlation: corr, + Module: gpu.ModuleRef{Backend: gpu.GPUBackendID("stub"), CRC: crc}, + FunctionIndex: fnIndex, + TimeNs: 1, + PCOffset: pcOffset, + StallReason: "long_scoreboard", + Count: 1, + })) + } + // One per gpu_src_status, each on its own stack-carrying execution. + inject(corrOf(0), gateCRCLineInfo, lineIdx, 0x10) // resolved + inject(corrOf(1), gateCRCNoLineInfo, noLineIdx, 0x10) // no-lineinfo + inject(corrOf(2), gateCRCAbsent, 0, 0x10) // no-module + inject(corrOf(3), gateCRCLineInfo, lineIdx, 0x180) // unmapped: past the function + // And a spray of distinct offsets for the cardinality cap. + const capSpray = 40 + for i := range capSpray { + inject(corrOf(4), gateCRCLineInfo, lineIdx, 0x1000+uint64(i)*16) + } + injected := 4 + capSpray + + snap := timeline.Snapshot() + require.Len(t, snap.Executions, 500, "500 launches + 500 execs, exactly, none lost") + + // The wire's own account of the same two modules: gpu_module_load_v1 + // announces THAT a module loaded, with its CRC and size, while the bytes + // travel the cubin channel. Both must name the same modules, or the + // announcement and the payload describe different things and the CRC join + // is meaningless. + gotCRCs := map[uint64]uint64{} + for _, m := range snap.Modules { + gotCRCs[m.Ref.CRC] = m.SizeBytes + } + assert.Equal(t, map[uint64]uint64{ + gateCRCLineInfo: uint64(len(lineInfo)), + gateCRCNoLineInfo: uint64(len(noLineInfo)), + }, gotCRCs, + "the decoded gpu_module_load_v1 records do not name the same (crc, size) pairs the producer reported for the bytes it sent") + + // ---- assertion 5: reconciliation -------------------------------------- + // + // Every PC record that reached the sink lands in exactly one of + // attributed-exact, attributed-kernel, still-pending (in either store) or + // evicted (from either store). The two pending stores are separate on + // purpose - a Tier B eviction storm must be distinguishable from a Tier A + // one - so both terms are in the identity. + accepted := uint64(wantPC) + uint64(injected) + assert.Equal(t, accepted, + snap.AttributedPCSamples+ + uint64(snap.PendingSamples)+snap.Dropped.EvictedPendingSamples+ + uint64(snap.PendingModuleSamples)+snap.Dropped.EvictedPendingModuleSamples, + "a PC sample went unaccounted for: attributed=%d pending=%d evicted=%d pending-module=%d evicted-module=%d of %d accepted", + snap.AttributedPCSamples, snap.PendingSamples, snap.Dropped.EvictedPendingSamples, + snap.PendingModuleSamples, snap.Dropped.EvictedPendingModuleSamples, accepted) + assert.Equal(t, snap.AttributedPCSamples, snap.PCJoin.AttributedTotal(), + "AttributedExact + AttributedKernel must account for every attributed sample: %+v", snap.PCJoin) + assert.Equal(t, uint64(injected), snap.PCJoin.AttributedExact, + "every injected sample carries a correlation a wire-delivered execution occupies, so all of them take the exact path") + // And the stub's own 64, which can attribute to nothing - see the doc + // comment. Asserted as an equality rather than tolerated as a shortfall: + // if the stub is ever given real CRCs and real kernel names this number + // changes, and the gate should say so rather than quietly pass. + assert.Equal(t, wantPC, snap.PendingModuleSamples, + "the stub's PC records carry synthetic CRCs and kernel names that no cubin can name, so every one of them must remain pending and counted - never attached to a plausible neighbour") + assert.Positive(t, snap.PCJoin.GroupsUnresolvedName, + "a group whose (crc, functionIndex) names nothing must be counted as such, not silently skipped") + assert.Equal(t, snap.PCJoin.GroupsExamined(), + snap.PCJoin.GroupsJoined+uint64(snap.PendingModuleGroups), + "every pending group must be joined or left pending for exactly one counted reason: %+v", snap.PCJoin) + + // ---- assertion 8: Tier A disclosure ----------------------------------- + assert.Equal(t, uint64(len(snap.Executions)), + snap.ExecutionsSerialized+snap.ExecutionsNotSerialized+snap.ExecutionsSerializationUnknown, + "the three gpu_serialized outcomes must partition the executions exactly") + assert.Positive(t, snap.SamplingWindowsReceived, + "Tier A was selected and the producer said it emitted windows, but none reached the disclosure store") + assert.Positive(t, snap.ExecutionsSerialized, + "the stub's bursts bracket about half its executions; not one was marked perturbed") + assert.Positive(t, snap.ExecutionsNotSerialized, + "not one execution fell in a proven gap between bursts, so \"false\" is unreachable on this run and the three-way split is not being exercised") + t.Logf("serialization: true=%d false=%d unknown=%d over %d windows (held=%d open=%d)", + snap.ExecutionsSerialized, snap.ExecutionsNotSerialized, snap.ExecutionsSerializationUnknown, + snap.SamplingWindowsReceived, snap.SamplingWindowsHeld, snap.SamplingWindowsOpen) + + // The second half of assertion 8, and the half that matters: Tier A + // selected with NO window arriving must read "unknown" on every execution + // and "false" on none. Driven off this run's own executions rather than + // off synthetic ones, so the population is identical and only the evidence + // differs. + t.Run("tier A with no windows is unknown, never false", func(t *testing.T) { + blind := gpu.NewTimeline(gpu.TimelineConfig{PCSampling: gpu.PCSamplingSerialized}) + for _, v := range snap.Executions { + require.NoError(t, blind.EmitExec(v.Exec)) + } + bs := blind.Snapshot() + require.Len(t, bs.Executions, len(snap.Executions)) + assert.Equal(t, uint64(len(bs.Executions)), bs.ExecutionsSerializationUnknown, + "Tier A ran and no window arrived, so nothing can be shown unperturbed") + assert.Zero(t, bs.ExecutionsNotSerialized, + "\"not perturbed\" when the truth is \"cannot tell\" is the one answer that must never be reachable by accident") + assert.Zero(t, bs.ExecutionsSerialized) + }) + + // Assertion 10b's third clause, on the same population: a window with + // end_ns == 0 is OPEN, not zero-length. Treating it as zero-length would + // mark a whole perturbed tail "false". + t.Run("an open window is unknown from its start, never false", func(t *testing.T) { + open := gpu.NewTimeline(gpu.TimelineConfig{PCSampling: gpu.PCSamplingSerialized}) + var first, last uint64 + for i, v := range snap.Executions { + if i == 0 || v.Exec.StartNs < first { + first = v.Exec.StartNs + } + if v.Exec.EndNs > last { + last = v.Exec.EndNs + } + } + require.Greater(t, last, first) + require.NoError(t, open.EmitSamplingWindow(gpu.GPUSamplingWindow{ + Backend: gpu.GPUBackendID("stub"), + PID: uint32(stubPID), + StartNs: first, + EndNs: 0, // the hard-exit shape: cuptiPCSamplingStop never ran + Mode: gpu.SamplingModeKernelSerialized, + })) + for _, v := range snap.Executions { + require.NoError(t, open.EmitExec(v.Exec)) + } + ws := open.Snapshot() + assert.Zero(t, ws.ExecutionsNotSerialized, + "an unterminated window read as zero-length would mark the whole perturbed tail \"false\"") + assert.Equal(t, uint64(len(ws.Executions)), + ws.ExecutionsSerialized+ws.ExecutionsSerializationUnknown, + "every execution at or after an open window's start is either perturbed or unknown, and nothing else") + }) + + // ---- the projection --------------------------------------------------- + // + // Projected TWICE over the SAME snapshot: once with the default budget, for + // the label assertions, and once with a small ceiling for assertion 9. + // Snapshot() is consuming, so the snapshot value is reused rather than + // retaken. + samples, projStats := gpu.ProjectExecutionsWith(snap, gpu.ProjectionConfig{Modules: store}) + require.NotEmpty(t, samples, "the gate is pprof samples, not counters") + assert.Zero(t, projStats.PCLabelsSuppressed, + "the default ceiling is nowhere near %d distinct offsets; a suppression here means the budget shrank", + projStats.DistinctPCLabels) + + // The walk over every projected sample: assertions 1, 2, 3 and 4, on real + // output rather than on a hand-built ExecutionView. + byStatus := map[string]int{} + byAttrib := map[string]int{} + var pcDerived, resolvedWithStack int + si := 0 + for _, view := range snap.Executions { + // ProjectExecutionsWith emits one sample per PC sample, or one for + // the execution itself when it carries none, in snapshot order. + n := max(1, len(view.PCSamples)) + for range n { + require.Less(t, si, len(samples)) + s := samples[si] + si++ + + // Assertion 1, in its exact form: the frames are the launch's own + // CPU stack, then the boundary marker, then the kernel - compared + // as a whole slice rather than scanned for forbidden substrings. + // Whole-slice comparison is strictly stronger (nothing may be + // inserted anywhere, not merely appended) and it has no false + // positives, which a substring scan would: several of the stub's + // stall reasons are spelt "wait", "barrier" and "membar", and libc + // frame names contain all three. + require.NotEmpty(t, view.Exec.KernelName, + "every execution carries an interned kernel name, so the kernel frame is never omitted") + var want []string + if view.Launch != nil && len(view.Launch.Launch.CPUStack) > 0 { + for _, f := range view.Launch.Launch.CPUStack { + want = append(want, f.Name) + } + want = append(want, gpu.FrameLaunch) + } else { + want = append(want, gpu.FrameLaunchUnsampled) + } + want = append(want, "[gpu:kernel:"+view.Exec.KernelName+"]") + require.Equal(t, want, frameNamesOf(s.Stack), + "frames are exhaustively the CPU stack, the boundary marker and the kernel; this sample's differ") + // The two frames this package synthesizes are the only ones it + // could smuggle per-sample detail into, so they take the substring + // scan the CPU frames cannot safely take. + for _, name := range want[len(want)-2:] { + for _, bad := range []string{"gpu:pc", "gpu:src", "gpu:stall", "long_scoreboard", "resolved", "0x"} { + assert.NotContains(t, name, bad, + "per-sample detail was promoted to a frame: %q", name) + } + } + + if len(view.PCSamples) == 0 { + assert.NotContains(t, s.Labels, "gpu_src_status", + "an execution with no PC samples has nothing to say about a source location") + continue + } + pcDerived++ + status := s.Labels["gpu_src_status"] + require.NotEmpty(t, status, + "gpu_src_status is unconditional on every PC-derived sample: an absent label reads as \"not sampled\"") + byStatus[status]++ + byAttrib[s.Labels["gpu_pc_attrib"]]++ + require.NotEmpty(t, s.Labels["gpu_pc_attrib"], "gpu_pc_attrib is unconditional too") + require.Contains(t, s.Labels, "gpu_serialized", + "gpu_serialized rides on every execution, PC-bearing or not") + + switch status { + case "resolved": + // Assertion 2: a source line reached from a CPU stack. + assert.Equal(t, "single.cu", s.Labels["gpu_src_file"], "the basename, never the build-host path") + assert.Equal(t, "addOne", s.Labels["gpu_src_func"]) + require.Contains(t, s.Labels, "gpu_src_line") + if view.Launch != nil && len(view.Launch.Launch.CPUStack) > 0 { + resolvedWithStack++ + if resolvedWithStack == 1 { + t.Logf("assertion 2: %v -> %s:%s (%s, attrib=%s)", + frameNamesOf(s.Stack), s.Labels["gpu_src_file"], s.Labels["gpu_src_line"], + s.Labels["gpu_stall"], s.Labels["gpu_pc_attrib"]) + } + } + default: + // Assertion 3, generalized to all three unresolvable statuses: + // no location is ever invented. + assert.NotContains(t, s.Labels, "gpu_src_file", "status=%s invented a file", status) + assert.NotContains(t, s.Labels, "gpu_src_line", "status=%s invented a line", status) + assert.NotContains(t, s.Labels, "gpu_src_func", "status=%s invented a function", status) + } + } + } + require.Equal(t, len(samples), si, "every projected sample must belong to an execution") + t.Logf("pc-derived samples: %d by status: %v by attrib: %v", pcDerived, byStatus, byAttrib) + + // Assertion 2, stated as a number rather than left implicit in the loop. + assert.Positive(t, resolvedWithStack, + "no sample carries BOTH a real CPU stack and gpu_src_status=resolved; that conjunction is the Phase 6 exit condition, and either half alone is not it") + + // Assertion 4: all four values reachable, each by the fixture that should + // produce it. + for _, want := range []string{"resolved", "no-lineinfo", "no-module", "unmapped"} { + assert.Positive(t, byStatus[want], + "gpu_src_status=%q is not reachable from this run: %v", want, byStatus) + } + // Assertion 3's own arithmetic: the -lineinfo fixture produced the + // resolved population, the no-lineinfo fixture produced exactly one + // no-lineinfo sample, and neither borrowed from the other. + assert.Equal(t, 1, byStatus["no-lineinfo"], + "exactly one sample was injected against the no-lineinfo fixture") + // One, not 65: the stub's own 64 records never reach an execution at all + // (see the doc comment), so they never project and never carry a label. + // The single no-module sample here is the injected one whose CRC no cubin + // was ever stored for - which is exactly the fixture that should produce + // that status. + assert.Equal(t, 1, byStatus["no-module"], + "only the injected absent-CRC sample can read no-module; the stub's records are pending and unprojected") + assert.Equal(t, 1, byStatus["resolved"]) + assert.Equal(t, 1+capSpray, byStatus["unmapped"], + "the injected past-the-function offset plus the %d cardinality-spray offsets, all past the line table's last address", capSpray) + assert.Equal(t, injected, pcDerived, + "every projected PC-derived sample must be one of the injected ones") + + // ---- assertion 9: the cardinality cap --------------------------------- + // + // Past the ceiling gpu_pc is dropped and counted, while gpu_stall and + // gpu_src_* survive untouched: they are coarser and more actionable, so the + // label that gives way is the numerous one rather than the useful one. + const ceiling = 8 + capped, capStats := gpu.ProjectExecutionsWith(snap, gpu.ProjectionConfig{ + Modules: store, + MaxDistinctPCLabels: ceiling, + }) + require.Len(t, capped, len(samples), "the cap changes labels, never the sample population") + distinct := map[string]bool{} + var suppressed uint64 + for i, s := range capped { + if _, isPC := s.Labels["gpu_src_status"]; !isPC { + continue + } + if pc, ok := s.Labels["gpu_pc"]; ok { + distinct[pc] = true + continue + } + suppressed++ + // Everything else must have survived. This is the half that a cap + // which simply stopped emitting labels would fail. + assert.NotEmpty(t, s.Labels["gpu_stall"], + "the cap dropped gpu_stall, which it must never touch (sample %d)", i) + assert.NotEmpty(t, s.Labels["gpu_src_status"], + "the cap dropped gpu_src_status, which is unconditional (sample %d)", i) + assert.NotEmpty(t, s.Labels["gpu_pc_attrib"], "the cap dropped gpu_pc_attrib (sample %d)", i) + assert.Equal(t, samples[i].Value, s.Value, + "a suppressed sample still carries its full share of the execution's duration") + } + assert.Positive(t, suppressed, + "a ceiling of %d over %d distinct injected offsets suppressed nothing, so this proves nothing", + ceiling, capSpray) + assert.Equal(t, suppressed, capStats.PCLabelsSuppressed, + "ProjectionPCLabelsSuppressed must equal the suppressions actually visible in the output, or the counter is decoration") + assert.Equal(t, uint64(len(distinct)), capStats.DistinctPCLabels) + assert.LessOrEqual(t, len(distinct), ceiling, "more distinct gpu_pc values were emitted than the ceiling allows") + assert.Equal(t, uint64(ceiling), capStats.PCLabelCap) + // And it is visible to the operator: a profile that silently lost its PC + // labels looks identical to one that never had any. + health := strings.Join(gpu.JoinHealthWith(snap, capStats), "\n") + assert.Contains(t, health, "gpu_pc", + "the suppression is not surfaced in joinhealth output:\n%s", health) + t.Logf("cardinality cap: distinct=%d cap=%d suppressed=%d", capStats.DistinctPCLabels, capStats.PCLabelCap, capStats.PCLabelsSuppressed) +} + +// waitFor polls until cond holds or the deadline passes, failing with what +// describe() reports rather than with a bare timeout. A producer that never +// emitted and a consumer that never drained look identical from a timeout. +func waitFor(t *testing.T, within time.Duration, cond func() bool, describe func() string) { + t.Helper() + deadline := time.Now().Add(within) + for !cond() { + if time.Now().After(deadline) { + t.Fatalf("timed out after %s: %s", within, describe()) + } + time.Sleep(5 * time.Millisecond) + } +} + +func frameNamesOf(frames []pp.Frame) []string { + out := make([]string, 0, len(frames)) + for _, f := range frames { + out = append(out, f.Name) + } + return out +} + +func fixturePath(name string) string { + return filepath.Join("..", "internal", "cubin", "testdata", name) +} + +func readFixture(t *testing.T, name string) []byte { + t.Helper() + b, err := os.ReadFile(fixturePath(name)) + require.NoError(t, err, "cubin fixture %s", name) + require.NotEmpty(t, b) + return b +} + +func mustAbs(t *testing.T, p string) string { + t.Helper() + abs, err := filepath.Abs(p) + require.NoError(t, err) + return abs +} + +// fixtureSymIndex reads the .symtab index the module store keys its +// functionIndex table on out of the fixture itself. Whether CUPTI's +// functionIndex IS that index is the design's premise and is measured on +// hardware (Task 6); nothing here depends on the answer, only on the store and +// the sample agreeing. +func fixtureSymIndex(t *testing.T, b []byte, fn string) uint32 { + t.Helper() + c, err := cubin.Parse(b) + require.NoError(t, err) + for _, f := range c.Functions() { + if f.Name == fn { + require.GreaterOrEqual(t, f.SymIndex, 0) + return uint32(f.SymIndex) + } + } + t.Fatalf("fixture has no function %q", fn) + return 0 +} + +// gateModuleCRCs pulls the CRC the producer declared for each captured module +// out of its own stderr, in capture order. +// +// From the producer rather than from a constant, so that the number the module +// store is keyed on is the number that was actually put on the wire. A +// constant would keep this gate green through a change to how the producer +// derives a CRC - which is exactly the change that would break the join on +// hardware. +func gateModuleCRCs(t *testing.T, stubErr string) []uint64 { + t.Helper() + re := regexp.MustCompile(`stub: module id=\d+ path=\S+ size=\d+ crc=0x([0-9a-f]{16}) captured=yes`) + var out []uint64 + for _, m := range re.FindAllStringSubmatch(stubErr, -1) { + v, err := strconv.ParseUint(m[1], 16, 64) + require.NoError(t, err) + out = append(out, v) + } + return out +} + +// stubCubinCRC is the Go replica of shim/stub/stub.cc's stub_cubin_crc: FNV-1a +// over the module bytes, with zero coerced to one because zero is the ABI's +// "no module". +// +// It exists so the CRC can be recomputed from the checked-in fixture rather +// than read back from the producer that produced it. Reading it back would +// assert only that the producer is self-consistent; recomputing it asserts +// that the identity the join runs on is a function of the BYTES, which is the +// property the whole content-addressed scheme rests on. +func stubCubinCRC(b []byte) uint64 { + h := uint64(1469598103934665603) + for _, c := range b { + h ^= uint64(c) + h *= 1099511628211 + } + if h == 0 { + return 1 + } + return h +} + +// TestGateTheStubsPCRecordsCannotAttributeToAnything pins the second reason +// TestStubDrivesPCSamplingToPprofWithoutAGPU injects PC samples of its own +// instead of using the producer's, and it runs WITHOUT capabilities so the +// reason is checkable on any machine. +// +// # The gap +// +// The stub emits real module loads carrying real checked-in cubins, and real +// PC-sample records. They are unrelated to each other: +// +// - the PC records' cubin_crc is one of two compile-time constants, +// kStubCubinCRC = {0xC0FFEE01, 0xC0FFEE02}, while the modules the same run +// delivers are keyed by a content hash of the fixture bytes (FNV-1a over +// the file, measured: 0x9d57accad01046eb for single_lineinfo.cubin). No +// cubin is ever stored under a 0xC0FFEE0n key, so every one of those +// records resolves as "no-module"; +// - their correlation is 0 in every tier, by design and correctly - that is +// what CONTINUOUS collection produces - so the exact-correlation path is +// unavailable to them; +// - Tier B attribution therefore runs crc -> module -> function name -> +// the execution's KernelName, and the stub's kernel names are +// "kernel_1111"/"kernel_2222" while the fixtures' only function is the +// CUDA kernel they were compiled from ("addOne"). No name can match. +// +// So neither join path can fire for a stub PC record, in either tier. That is +// not a bug in the pipeline - the pipeline correctly counts every one of them +// as pending, which the end-to-end gate asserts as an exact equality - but it +// does mean the producer cannot drive gate assertions 2, 3, 4 or 9, all of +// which need a PC sample that reaches an execution. +// +// Closing it is a small change to shim/stub/stub.cc: record the CRC each +// capture computed and use it on the PC records, and name the kernels after +// the fixtures' own functions. This branch is a test task and may not change +// the shim, so it is pinned here instead. +// +// # Why a passing test +// +// Same shape as the other two outstanding pins: when the stub is fixed, this +// fails by name and the person fixing it drops the injection from the +// end-to-end gate and asserts the real thing. +func TestGateTheStubsPCRecordsCannotAttributeToAnything(t *testing.T) { + src, err := os.ReadFile(filepath.Join("..", "shim", "stub", "stub.cc")) + require.NoError(t, err, "the producer's source must be readable; this test is about what it emits") + body := string(src) + + assert.Contains(t, body, "static const uint64_t kStubCubinCRC[] = {0xC0FFEE01ull, 0xC0FFEE02ull};", + "the stub's synthetic PC-record CRCs changed. If they now come from the CRC each capture "+ + "computed, gate assertions 2/3/4/9 may be drivable from the producer: drop the PC-sample "+ + "injection from TestStubDrivesPCSamplingToPprofWithoutAGPU and assert them off the wire.") + assert.Contains(t, body, "r.cubin_crc = kStubCubinCRC[i % 2];", + "the stub no longer keys its PC records on the synthetic constants - see above") + assert.Contains(t, body, `snprintf(namebuf, sizeof(namebuf), "kernel_%llx", (unsigned long long)kernel_id);`, + "the stub's kernel names changed. If they now name functions the checked-in cubins "+ + "actually contain, the Tier B module join can fire for its own records - see above.") + + // And the fixtures really do carry a different name, so the mismatch above + // is a fact about both ends rather than an assumption about one. + c, err := cubin.Parse(readFixture(t, "single_lineinfo.cubin")) + require.NoError(t, err) + for _, fn := range c.Functions() { + assert.NotContains(t, fn.Name, "kernel_", + "a fixture function is now named like a stub kernel; the join might match by accident, which is worse than not matching at all") + } + t.Log("stub PC records -> a cubin the agent holds: OUTSTANDING - synthetic CRCs and synthetic " + + "kernel names; see .superpowers/sdd/task-13-gate-report.md") +}