gpu: Tier A KERNEL_SERIALIZED PC sampling, duty-cycled, and its disclosure - #89
Merged
Conversation
…osure Configures COLLECTION_MODE = KERNEL_SERIALIZED with ENABLE_START_STOP_CONTROL, then duty-cycles via cuptiPCSamplingStart / Stop - not Enable/Disable, which tears down and rebuilds the whole configuration every burst. A burst controller in shim/core/ holds a target PC/stall-pair rate through a closed loop that is a pure function of (target, observed, elapsed), so it unit-tests against a fake clock. Serializing kernels perturbs the durations it measures, so the tier's substance is the disclosure. Every burst emits gpu_sampling_window_v1, and an execution intersecting any window is marked serialized - the window, not the set of sampled kernels, because every kernel that ran while a burst was open ran serialized whether it was sampled or not. A burst reaches the wire twice: an open record (end_ns = 0) the instant Start succeeds, and a closed record with the same start_ns on Stop. Emitting only on Stop would lose the whole burst to a hard exit, and its executions would then read gpu_serialized="false" - "not perturbed" when the truth is "cannot tell", the one answer that must never be reachable by accident. The consumer's store supersedes one way only: a closed record replaces an open one with the same start; an open record never replaces a closed one. Both delivery orders are tested. gpu_serialized has three values and "unknown" never degrades to "false". The three outcome counters sum to len(Snapshot.Executions) by construction - incremented where the ExecutionView is built, before any of the join loop's four continues. Tier A refuses to start where CUDA graphs have been observed, loudly and counted rather than downgrading: its entire claim is exact launch attribution, and a graph launch makes that claim false while still looking exact. Defaults off. Tier selection is Task 11.
dpsoft
force-pushed
the
feat/tier-a-serialized
branch
from
August 25, 2026 23:26
52d423b to
d3dcb61
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 10 of the GPU PC sampling plan. Defaults off; tier selection is Task 11.
Configures
COLLECTION_MODE = KERNEL_SERIALIZEDwithENABLE_START_STOP_CONTROL, then duty-cycles viacuptiPCSamplingStart/Stop— notEnable/Disable, which tears down and rebuilds the whole configuration every burst. A burst controller inshim/core/holds a target PC/stall-pair rate through a closed loop that is a pure function of(target, observed, elapsed), so it unit-tests against a fake clock with no GPU: it converges, never exceeds a maximum duty fraction whatever the observed rate does, and a zero observed rate does not drive the gap to zero.The disclosure is the substance of this tier
Serializing kernels perturbs the very durations it measures. Every burst emits
gpu_sampling_window_v1, and an execution intersecting any window is marked serialized — the window, not the set of sampled kernels, because every kernel that ran while a burst was open ran serialized whether it was sampled or not.A burst reaches the wire twice — a protocol the plan implied but did not spell out
cuptiPCSamplingStartsucceeds, withend_ns = 0;start_ns, realend_ns.Emitting only on the stop would lose the entire burst to a hard exit, and its executions would then read
gpu_serialized="false"— "not perturbed" when the truth is "cannot tell", the one answer that must never be reachable by accident. With the open record already delivered, aSIGKILLleaves a window saying a burst opened atstart_nsand never closed, and every execution from there on reads"unknown".The consumer's store supersedes one way only: a closed record replaces an open one with the same start; an open record never replaces a closed one. Both delivery orders are tested, so a lossy transport cannot resurrect an open window over a closed one.
never degrades to
Three values:
"true"(overlapped a burst — duration perturbed),"false"(Tier A ran, no window overlapped),"unknown"(Tier A was selected but no window arrived — dropped batch, late attach, sequence gap). The type's zero value is"unknown", and the JSON tag is deliberately notomitempty: an absent field reads as "not perturbed" to a consumer that does not know to check for absence.In Tier B and with sampling off,
"false"is unconditional and correct — nothing was ever serialized.TimelineConfig.SerializedSamplingis the agent's own configuration rather than something inferred from the wire, and that is the point: "Tier A was asked for and no window arrived" and "Tier A was never asked for" are different facts with different answers, and only the agent knows which holds.The sum identity holds by construction
ExecutionsSerialized + ExecutionsNotSerialized + ExecutionsSerializationUnknown == len(Snapshot.Executions), exactly — the counters are incremented where theExecutionViewis built, before any of the join loop's fourcontinues, so every execution is counted once on every path. Asserted in all 17 tests ofgpu/serialization_test.goand now part ofassertConformanceInvariants, which also asserts the negative that matters more: in the default harness configuration no execution may read"true"or"unknown".CUDA graphs
Tier A refuses to start where graph executions have been observed — loudly and counted, never a silent downgrade. Its entire claim is exact launch attribution, and a graph launch makes that claim false while still looking exact, since N executions share one correlation.
Rebase note
Merged over Task 8b, which reworked the same
Snapshotloop. The one structural conflict was theExecutionViewconstruction: both tasks add a field and a switch there. Both switches survive, both run before thecontinues, and 8b'sgpu_pc_attribpresence invariant plus this task's sum identity now both run on every conformance scenario.check-cubin-deferstill reportsOK - all 5 deferrals do not compile.Cannot verify — 13 items, every one needs the RTX 3090
Including the four the plan names: that
correlationIdis non-zero on ≥99% of PC records in this mode; that windows actually bracket the executions that ran in them; whether the collection mode can be changed betweenStopandStartwithout a fullDisable/Enable— undocumented, and it decides whether a runtime tier switch is possible at all; and the overhead (Task 12), which decides whether this tier ships.One found while building, worth naming:
g_exec_from_graphis set fromCUpti_ActivityKernel12.graphIdon the activity path, which arrives on the drain tick — up to 100 ms, and one or two bursts, after the first graph kernel ran. Those bursts' windows are emitted and their executions marked, so nothing is silent, but Tier A does run briefly in a graph-using process before refusing, and the size of that window is unmeasured.