Skip to content

io_uring/io-wq: apply the current cpuset to new workers - #336

Draft
rucoder wants to merge 1 commit into
lf-edge:eve-kernel-amd64-v6.18.63-genericfrom
rucoder:mikem/iowq-cpuset-6.18
Draft

rucoder wants to merge 1 commit into
lf-edge:eve-kernel-amd64-v6.18.63-genericfrom
rucoder:mikem/iowq-cpuset-6.18

Conversation

@rucoder

@rucoder rucoder commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

io_uring's io-wq workers started after a cgroup's cpuset changes, or after the task moves to another cgroup, run on the CPUs the cgroup had when the io-wq was created. Since 6.12 (84eacf177faa "io_uring/io-wq: inherit cpuset of cgroup in io worker", in 6.18 too) io_wq_create() copies the cpuset once and io_init_new_worker() gives every new worker that copy. cpuset re-masks the workers that exist; workers started later, and the replacement of an idle worker that cpuset moved, land on the old CPUs.

This is why lf-edge/eve#6335 gives aio=io_uring only to VMs whose non-vCPU threads never move. With this fix, QEMU's io_uring workers follow the cpuset like every other thread of the VM.

Root cause / mechanism

wq->cpu_mask mixed two things: the io-wq's own affinity (all CPUs, or IORING_REGISTER_IOWQ_AFF) and a snapshot of the cpuset. The fix keeps only the former in wq->cpu_mask and intersects it with the worker's cpuset when each worker starts, falling back to the cpuset when the two don't intersect (the rule __set_cpus_allowed_ptr() applies to a task's requested affinity). The idle worker test uses the same mask, and unregistering the affinity goes back to all CPUs rather than to the cpuset of that moment.

Cost: one cpuset_cpus_allowed() per worker start and per idle timeout, not per request. Measured with the function graph tracer: about 0.3-0.4 us against 57-62 us for a whole worker start (create_io_worker).

upstreaming

UPDATE: the finding was confirmed valid by maintainer

Validation

A QEMU/KVM test (8 CPUs, cgroup v2, lockdep and CPUMASK_OFFSTACK on) that blocks requests in io-wq workers around cpuset changes and checks every worker's affinity:

case without the fix with the fix
cpuset 2-3 -> 4-5, new workers 2-3 (FAIL) 4-5
idle worker retired after the change replacement on 2-3 (FAIL) stays on 4-5
cpuset grows 2-3 -> 2-5 2-3 (FAIL) 2-5
cgroup move 2-3 -> 4-5 2-3 (FAIL) 4-5
IORING_REGISTER_IOWQ_AFF 2-3, cpuset 3-6 / 4-5 / unregister 2-3 / 2-3 / stale (FAIL) 3 / 4-5 / cpuset
CPU hotplug pass pass

Same results on 6.18.55, 6.18.63 and mainline io_uring-7.4. New liburing test test/iowq-cpuset.c fails without the fix and passes with it; the whole liburing suite gives the same results with and without (the only differences, wq-aff and iowait, are flaky on both: 20 runs each). No lockdep or sleep-in-atomic reports.

Upstream status

The same commit is prepared for Jens Axboe's io_uring-7.4 (Cc stable), with the liburing test. Not posted yet; this PR carries it until it lands in stable 6.18.y.

Backport / other branches

Also affected: eve-kernel-arm64-v6.17.13-nvidia-gb10, eve-kernel-arm64-v7.0.14-nvidia-gb, eve-kernel-amd64-next. Not affected: eve-kernel-arm64-v6.1.155-generic (6.1 predates the bug).

🤖 Generated with Claude Code

Since commit 84eacf1 ("io_uring/io-wq: inherit cpuset of cgroup in
io worker"), io_wq_create() copies the cpuset of the task that creates
the io-wq into wq->cpu_mask, and io_init_new_worker() sets the affinity
of every new worker to that copy. The copy is never refreshed, but
workers are started for as long as the io-wq exists: whenever a request
has to block and no worker is free, while idle ones exit after
WORKER_IDLE_TIMEOUT.

When the cpuset of the cgroup changes, or the task moves to another
cgroup, cpuset updates the affinity of the workers that exist. Workers
started later get the stale mask, which set_cpus_allowed_ptr() does not
check against their cpuset, and run on CPUs their cgroup no longer has.
A worker that cpuset moved is also retired at its next idle timeout,
even when it is the last one, as it now runs outside wq->cpu_mask, and
its replacement starts on the old CPUs.

With a task in a cgroup whose cpuset.cpus goes from 2-3 to 4-5 while one
request is blocked in a worker, and three more blocking requests issued
afterwards:

  worker started before the change    Cpus_allowed_list: 4-5
  3 workers started after the change  Cpus_allowed_list: 2-3

So io-wq workers keep running on CPUs that were taken away from their
cgroup, for instance to give them to latency sensitive tasks.

Keep wq->cpu_mask for the affinity of the io-wq itself, all CPUs or the
mask set with IORING_REGISTER_IOWQ_AFF, and intersect it with the cpuset
of the worker when the worker starts, using the cpuset alone if the two
have no CPU in common. That is how the scheduler treats the affinity a
task requested when its cpuset changes, see __set_cpus_allowed_ptr().
Use the same mask in the idle timeout test of a worker, and when the
registered affinity is removed, go back to all CPUs rather than to the
cpuset of that moment. A mask being registered must still lie within
the cpuset, as required since commit 0997aa5 ("io_uring/io-wq: do
not allow pinning outside of cpuset").

Fixes: 84eacf1 ("io_uring/io-wq: inherit cpuset of cgroup in io worker")
Cc: stable@vger.kernel.org
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@rucoder
rucoder marked this pull request as draft November 12, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant