Skip to content

gpu: Tier A KERNEL_SERIALIZED PC sampling, duty-cycled, and its disclosure - #89

Merged
dpsoft merged 1 commit into
mainfrom
feat/tier-a-serialized
Aug 25, 2026
Merged

gpu: Tier A KERNEL_SERIALIZED PC sampling, duty-cycled, and its disclosure#89
dpsoft merged 1 commit into
mainfrom
feat/tier-a-serialized

Conversation

@dpsoft

@dpsoft dpsoft commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Task 10 of the GPU PC sampling plan. Defaults off; tier selection is Task 11.

Configures COLLECTION_MODE = KERNEL_SERIALIZED with ENABLE_START_STOP_CONTROL, then duty-cycles via cuptiPCSamplingStart/Stopnot Enable/Disable, which tears down and rebuilds the whole configuration every burst. A burst controller in shim/core/ holds a target PC/stall-pair rate through a closed loop that is a pure function of (target, observed, elapsed), so it unit-tests against a fake clock with no GPU: it converges, never exceeds a maximum duty fraction whatever the observed rate does, and a zero observed rate does not drive the gap to zero.

The disclosure is the substance of this tier

Serializing kernels perturbs the very durations it measures. Every burst emits gpu_sampling_window_v1, and an execution intersecting any window is marked serialized — the window, not the set of sampled kernels, because every kernel that ran while a burst was open ran serialized whether it was sampled or not.

A burst reaches the wire twice — a protocol the plan implied but did not spell out

  • the open record goes out the instant cuptiPCSamplingStart succeeds, with end_ns = 0;
  • the closed record goes out on the stop, same start_ns, real end_ns.

Emitting only on the stop would lose the entire burst to a hard exit, and its executions would then read gpu_serialized="false""not perturbed" when the truth is "cannot tell", the one answer that must never be reachable by accident. With the open record already delivered, a SIGKILL leaves a window saying a burst opened at start_ns and never closed, and every execution from there on reads "unknown".

The consumer's store supersedes one way only: a closed record replaces an open one with the same start; an open record never replaces a closed one. Both delivery orders are tested, so a lossy transport cannot resurrect an open window over a closed one.

never degrades to

Three values: "true" (overlapped a burst — duration perturbed), "false" (Tier A ran, no window overlapped), "unknown" (Tier A was selected but no window arrived — dropped batch, late attach, sequence gap). The type's zero value is "unknown", and the JSON tag is deliberately not omitempty: an absent field reads as "not perturbed" to a consumer that does not know to check for absence.

In Tier B and with sampling off, "false" is unconditional and correct — nothing was ever serialized.

TimelineConfig.SerializedSampling is the agent's own configuration rather than something inferred from the wire, and that is the point: "Tier A was asked for and no window arrived" and "Tier A was never asked for" are different facts with different answers, and only the agent knows which holds.

The sum identity holds by construction

ExecutionsSerialized + ExecutionsNotSerialized + ExecutionsSerializationUnknown == len(Snapshot.Executions), exactly — the counters are incremented where the ExecutionView is built, before any of the join loop's four continues, so every execution is counted once on every path. Asserted in all 17 tests of gpu/serialization_test.go and now part of assertConformanceInvariants, which also asserts the negative that matters more: in the default harness configuration no execution may read "true" or "unknown".

CUDA graphs

Tier A refuses to start where graph executions have been observed — loudly and counted, never a silent downgrade. Its entire claim is exact launch attribution, and a graph launch makes that claim false while still looking exact, since N executions share one correlation.

Rebase note

Merged over Task 8b, which reworked the same Snapshot loop. The one structural conflict was the ExecutionView construction: both tasks add a field and a switch there. Both switches survive, both run before the continues, and 8b's gpu_pc_attrib presence invariant plus this task's sum identity now both run on every conformance scenario. check-cubin-defer still reports OK - all 5 deferrals do not compile.

Cannot verify — 13 items, every one needs the RTX 3090

Including the four the plan names: that correlationId is non-zero on ≥99% of PC records in this mode; that windows actually bracket the executions that ran in them; whether the collection mode can be changed between Stop and Start without a full Disable/Enable — undocumented, and it decides whether a runtime tier switch is possible at all; and the overhead (Task 12), which decides whether this tier ships.

One found while building, worth naming: g_exec_from_graph is set from CUpti_ActivityKernel12.graphId on the activity path, which arrives on the drain tick — up to 100 ms, and one or two bursts, after the first graph kernel ran. Those bursts' windows are emitted and their executions marked, so nothing is silent, but Tier A does run briefly in a graph-using process before refusing, and the size of that window is unmeasured.

…osure

Configures COLLECTION_MODE = KERNEL_SERIALIZED with
ENABLE_START_STOP_CONTROL, then duty-cycles via cuptiPCSamplingStart /
Stop - not Enable/Disable, which tears down and rebuilds the whole
configuration every burst. A burst controller in shim/core/ holds a
target PC/stall-pair rate through a closed loop that is a pure function
of (target, observed, elapsed), so it unit-tests against a fake clock.

Serializing kernels perturbs the durations it measures, so the tier's
substance is the disclosure. Every burst emits gpu_sampling_window_v1,
and an execution intersecting any window is marked serialized - the
window, not the set of sampled kernels, because every kernel that ran
while a burst was open ran serialized whether it was sampled or not.

A burst reaches the wire twice: an open record (end_ns = 0) the instant
Start succeeds, and a closed record with the same start_ns on Stop.
Emitting only on Stop would lose the whole burst to a hard exit, and its
executions would then read gpu_serialized="false" - "not perturbed" when
the truth is "cannot tell", the one answer that must never be reachable
by accident. The consumer's store supersedes one way only: a closed
record replaces an open one with the same start; an open record never
replaces a closed one. Both delivery orders are tested.

gpu_serialized has three values and "unknown" never degrades to "false".
The three outcome counters sum to len(Snapshot.Executions) by
construction - incremented where the ExecutionView is built, before any
of the join loop's four continues.

Tier A refuses to start where CUDA graphs have been observed, loudly and
counted rather than downgrading: its entire claim is exact launch
attribution, and a graph launch makes that claim false while still
looking exact.

Defaults off. Tier selection is Task 11.
@dpsoft
dpsoft force-pushed the feat/tier-a-serialized branch from 52d423b to d3dcb61 Compare August 25, 2026 23:26
@dpsoft
dpsoft merged commit 65b503f into main Aug 25, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant