[haishuok0525]

The gap

Kernels launched by a HIP graph replay arrive with nothing indicating which graph or which node produced them. A trace can show that 6400 kernels ran, but not which of them are the same node across replays, so per-node cost cannot be separated and a node cannot be followed over time.

CUDA does not have this problem: CUPTI reports graphId and graphNodeId on the kernel activity record, kineto emits them via addGraphNodeTypedMetadata(), and torch's exporter consumes them. ROCm is the only backend missing the fields.

rocprofiler-sdk cannot supply them today. It puts no graph identity on the dispatch record, and its HIP graph tracing exposes only EXEC_CREATE, EXEC_DESTROY and EXEC_LAUNCH — there is no node-level operation. So the tool has to reconstruct the association at collection time.

How it works

This subscribes to ROCPROFILER_CALLBACK_TRACING_HIP_GRAPH and keeps a per-thread stack of in-flight graph launches, then registers an external correlation id request service for kernel dispatches and memory copies. When the SDK asks for an external id during a graph launch, the callback returns:

(graph exec id << 32) | ordinal of the dispatch within that launch

buffer_callback puts that on the row, and the fields are emitted as graph id and graph node id under RocmMetadataFields, reusing the CudaMetadataFields key strings.

This is not an invented heuristic — it is the reconstruction the SDK documents for this purpose, in callback_tracing.h alongside the graph callback payload:

Tools that want to associate kernel dispatches with their producing graph node should:

1. Subscribe to ROCPROFILER_CALLBACK_TRACING_HIP_GRAPH and maintain a per-thread stack of (graph_exec_id, node_counter) by handling EXEC_LAUNCH ENTER (push) and EXIT (pop).
2. Register an external correlation id request callback that, when invoked for KERNEL_DISPATCH / MEMORY_COPY, captures the current top-of-stack graph attribution into the external correlation id and increments the node_counter.
3. At record consumption time, extract the captured attribution from rocprofiler_correlation_id_t::external on the dispatch/copy record.

Kineto's own External id travels through t_externalIds and externalCorrelations_ rather than this slot, so the two do not interact. Non-graph dispatches leave the slot at zero and emit no metadata. An exec id too large to pack drops attribution with a warning rather than emitting an aliased id.

KINETO_ROCM_DISABLE_GRAPH_ATTRIBUTION opts out.

Why graph node id carries the packed value

Deliberately, rather than the bare ordinal, because that is the layout CUPTI uses and torch reads it back that way. _cuspy/_event_nodes.py recovers a launch's exec graph with "graph node id" >> 32, and _chrome_trace_export.py keys graph annotations on the whole value. That exporter is gated only on kineto_available(), so it is reachable on a ROCm build, and a bare per-replay ordinal would collide across execs and splice one graph's annotations onto another graph's nodes.

One difference is worth being explicit about: the low half is the position of the dispatch within the replay, where CUPTI reports an id assigned by CUDA in node-creation order. Both are stable identifiers within an exec, so graph node id remains a unique per-node key, but the ordinal follows dispatch order rather than creation order.

Execs that get updated

hipGraphExecUpdate replaces the nodes behind an existing hipGraphExec_t without changing the id the SDK reports for it. Left alone, an ordinal recorded before the update names a different node after it, and nothing in the trace says so — every ordinal silently covers two kernels. Unlike a scheduling artifact this is deterministic; it aliases on every run.

The graph domain cannot see the update, since it has no EXEC_UPDATE operation. So the exec is given a fresh attribution id when a hipGraphExecUpdate call is seen to return hipSuccess, drawn from the top of the packed field, away from the SDK's process-monotonic ids. An exec is also forgotten when it is created or destroyed, because HIP hands the same address straight back out to the next instantiate and the new graph would otherwise inherit the id of the one that used to live there.

The consequence worth flagging is that graph id stops matching the SDK's exec id for a graph that has been updated. That seemed like the better trade: it only diverges in the case where the SDK's id is genuinely ambiguous, and what the field has to be is a stable unique key, which is all torch's exporter recovers from it.

The common path pays nothing for this — an atomic load that stays false until some exec is actually updated.

Validation

All measured on MI355X / ROCm 10.0.0, rocprofiler-sdk 1.3, driving the real libkineto through its public API and reading the answers out of the saved chrome trace.

Every graph node runs a distinct kernel, so the node that produced a dispatch is identifiable from the kernel name in the trace, independently of the pair this PR assigns. The test is then whether any (graph id, graph node id) was ever seen with more than one kernel — if so, that ordinal aliased across nodes. Measured with #1564 applied, so its truncation cannot be mistaken for an attribution defect.

A 32-node graph replayed 200 times on a single GPU, so 6400 dispatches expected:

	attribution on	opt-out set
kernel events in trace	6400	6400
carrying both fields	6400	0
distinct graph ids	1	—
distinct node ids	32	—
occurrences per node id	200, for all 32	—

Sample value: graph id = 1, graph node id = 4294967296, which is (1 << 32) | 0.

Graph shape does not perturb the ordinal, which was the case I most expected to break:

topology	result
linear chain, 6 nodes x 200 replays	1200 attributed, 6 keys, 0 aliased
root -> 4 independent -> join, 200 replays	1200 attributed, 6 keys, 0 aliased (3 of 3 runs)
root -> 2 independent -> join, 300 replays	1200 attributed, 4 keys, 0 aliased

Each ordinal maps to exactly one kernel, the expected number of times, and the order matches node creation order even when four nodes have no ordering constraint between them. The reason is that hipGraphInstantiate flattens the DAG into a fixed submission sequence and the ordinal is assigned at submission, not at completion, so parallelism inside the graph does not reorder it. That makes the "dispatch order, not creation order" caveat above real but narrower than it sounds.

Exec update, a 4-node chain replayed 150 times, then hipGraphExecUpdate to a chain of four different kernels and 150 more replays:

	graph ids	keys	aliased
without the handling above	1	4	4 — every ordinal covers two kernels, 150 + 150
with it	2	8	0

Then destroying that exec and instantiating a third graph, which HIP placed at the same address:

	graph ids	keys	aliased
without forgetting destroyed execs	2	8	4 — the new graph inherits the updated one's id
with it	3	12	0

Multi-GPU is fine: 2 devices, one exec each, launched concurrently, 150 replays — 1800 attributed, 12 keys, 0 aliased, 2 agents.

Cost is negligible at realistic kernel sizes. 32000 dispatches, comparing a profiled run against a profiled run with attribution enabled, scaling the kernel's own duration:

kernel duration	profiled (ms)	with attribution (ms)	per dispatch	overhead
3.8 us	122.81	123.44	+19.7 ns	0.51%
6.9 us	220.19	220.31	+3.8 ns	0.05%
58.5 us	1870.95	1871.11	+5.0 ns	0.01%

The per-dispatch cost is tens of nanoseconds, so it only registers at all on a kernel small enough that the trace is dominated by launch overhead anyway.

RocmActivityProfilerTest passes, 11 of 11.

Known limitation: concurrent launches from many threads

One case is not fixed here, and I would rather have it in the PR than discovered later. One exec per thread, root -> 4 independent -> join, 100 replays per thread:

threads	attributed	distinct keys	aliased keys	ideal keys
1	600	6	0	6
2	1200	12	0	12
3	1800	18	0	18
4	2400	27	11	24
8	4798	70	64	48
16	9598	153	138	96

The onset at 4 threads is HIP's default GPU_MAX_HW_QUEUES=4, and raising it moves the threshold: at GPU_MAX_HW_QUEUES=8 both 4 and 8 threads are clean, and at 16 even 16 threads is far less affected.

When streams share a hardware queue, packet submission is no longer performed by the thread that called hipGraphLaunch, so a per-thread launch stack credits the dispatch to whatever launch that other thread happens to be inside. Note the second symptom in the key counts — a 6-node graph producing ordinals 6, 7, 8 — which is the same cause seen from the other end: the node counter is being advanced by dispatches belonging to another launch.

This is a property of the per-thread reconstruction the SDK prescribes, not of this implementation, so I do not think it can be fixed inside the callback as specified. Keying in-flight launches by stream rather than by thread looks like the right shape, since the ordinal sequence is really a property of the submission stream, but the external correlation request callback is not given the stream, so it would need something from the SDK side. Happy to take this upstream to rocprofiler-sdk if that is the preferred route.

Single-threaded replay, which is what torch.cuda.CUDAGraph does, is unaffected, and KINETO_ROCM_DISABLE_GRAPH_ATTRIBUTION turns the whole thing off.

Relationship to #1564

Independent changes; this one does not depend on #1564 to build or work. It neither reads nor modifies anything #1564 touches, the two branches merge cleanly, and the shape and concurrency results above were measured on a build without it.

It does depend on #1564 to be fully measured. 20 runs per variant of the 6400-dispatch workload:

variant	crashes	kernels (mean)	carrying node id	distinct node ids
main	18 / 20	3374 (53%)	0	0
main + #1564	0 / 20	6400 (100%)	0	0
main + this PR	16 / 20	3994 (62%)	3994	32
main + both	0 / 20	6400 (100%)	6400	32

On current main most runs either crash during teardown or publish a truncated trace, so the full 200-per-node picture is rarely visible. Worth noting that attribution is correct on whatever records do survive — all 32 distinct node ids still appear on a 62% trace — which is how the truncation is distinguishable from an attribution defect.

[pytorch-bot]

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

• ciflow/rocm

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

[ryanzhang22]

Thanks for the PR @haishuok0525, one concern about the limitation you pointed out in the summary.

Known limitation: concurrent launches from many threads

The table you have in the summary shows pretty heavy aliasing on keys even starting at just 4 threads, wouldn't lots of users run into this? I assume that would cause the metadata/visualization for the colliding graph nodes to be really confusing in the traces, and people wouldn't have a way of knowing that it's messed up.

IMO ideally this would be handled in rocprofiler-sdk to avoid adding fragile logic into the profiler, and if there isn't a way around the aliasing I'd rather leave the information out than present it incorrectly. Or, if this is critical to add ASAP for some use case, then I'd prefer for this to be opt-in with documentation about the limitations.