You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Performance]: Reduce CPU busy-waiting at sampler host completion events #21358
Use blocking CUDA events at the four existing host-synchronized sampler
completion sites in the PyTorch executor: side-stream copy completion,
asynchronous-copy readiness, shared sampler completion, and intermediate
pipeline-stage completion.
The host must await GPU completion at these points. Creating those events with torch.cuda.Event(blocking=True) allows the existing host synchronization to
sleep rather than busy-wait, without removing a synchronization or changing
the event record point, stream ordering, source snapshot, or sampling policy.
Report of performance regression
This is a source-level optimization proposal, not a newly measured regression.
No current-revision CPU, throughput, or latency improvement is claimed.
Misc discussion on performance
The proposed change is default-on at these four constructor sites, with no new
configuration flag. It does not cover every executor wait or serving mode.
Encoder readiness and GPU-only stream-ordering events remain unchanged.
Four regression cases are prepared but not run: main completion, side-stream
copy completion, asynchronous-copy snapshot lifetime, and the actual
intermediate-PP producer/host-wait wrapper. The PP case substitutes CUDA; it
does not establish distributed liveness. GPU and applicable PP validation are
still needed, including CPU/wake-latency tradeoffs.
Related: #6839 discusses logprobs/D2H sampler overhead, but is not the same
host-wait proposal and is not claimed fixed by this change.
Your current environment (if you think it is necessary)
Source review is based on commit 1a1ba9a0a3ba8fb155301e2c9fd5d362a013a4cc. No new runtime reproduction or
machine-specific performance result is attached.
Activity