Skip to content

[Performance]: Reduce CPU busy-waiting at sampler host completion events #21358

Open

Description

@skhasawneh

Proposal to improve performance

Use blocking CUDA events at the four existing host-synchronized sampler
completion sites in the PyTorch executor: side-stream copy completion,
asynchronous-copy readiness, shared sampler completion, and intermediate
pipeline-stage completion.

The host must await GPU completion at these points. Creating those events with
torch.cuda.Event(blocking=True) allows the existing host synchronization to
sleep rather than busy-wait, without removing a synchronization or changing
the event record point, stream ordering, source snapshot, or sampling policy.

Report of performance regression

This is a source-level optimization proposal, not a newly measured regression.
No current-revision CPU, throughput, or latency improvement is claimed.

Misc discussion on performance

The proposed change is default-on at these four constructor sites, with no new
configuration flag. It does not cover every executor wait or serving mode.
Encoder readiness and GPU-only stream-ordering events remain unchanged.

Four regression cases are prepared but not run: main completion, side-stream
copy completion, asynchronous-copy snapshot lifetime, and the actual
intermediate-PP producer/host-wait wrapper. The PP case substitutes CUDA; it
does not establish distributed liveness. GPU and applicable PP validation are
still needed, including CPU/wake-latency tradeoffs.

Related: #6839 discusses logprobs/D2H sampler overhead, but is not the same
host-wait proposal and is not claimed fixed by this change.

Your current environment (if you think it is necessary)

Source review is based on commit
1a1ba9a0a3ba8fb155301e2c9fd5d362a013a4cc. No new runtime reproduction or
machine-specific performance result is attached.

Activity

  1. added
    Pytorch<NV>Pytorch backend related issues
    General perf<NV>Broad performance issues not specific to a particular component
    on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    General perf<NV>Broad performance issues not specific to a particular componentPytorch<NV>Pytorch backend related issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions