[Japabu]

Problem

The harness ran guests side by side with no regard for how many host CPUs they need. The four
virtsmpcase rows boot 8 vCPUs each, and a 12-wide run put them beside each other and beside every
other worktree's guests on a 14-core host. A guest whose vCPUs share host CPUs has them descheduled.
The kernel's FIFO ticket lock then hands over at the host's scheduling pace (#1160's captures), and
test_rs_counters_read goes quiet past GUEST_QUIET
(issues/a-counters-read-under-host-load-can-go-silent-for-15-s.md). Ticket locks are kept, as
decided on 2026-11-04, so the fix belongs where the oversubscription is made.

Change

• toyos_build::buildlock::guest_cpus: one flock per host CPU in the host store
  (~/.cache/toyos/locks/guest-cpus/cpu-N), so every worktree's harness on the host shares the
  budget. A guest holds one per vCPU for its life. A guest wider than the host takes all of them.
  • All or nothing: an attempt keeps what it locked only once it has every CPU it asked for, so
    no guest holds some CPUs while it waits for others, and two waiters cannot deadlock.
  • A queue: the head waiter holds queue while it waits, so narrower guests do not overtake a
    wide one forever. Nothing waits on queue while holding a CPU.
  • Holder death releases everything (flock), as for the build locks.
• QemuInstance::boot_with_options takes the CPUs after the image is built and before QEMU starts,
  so the wait is never charged to a guest's clocks, and holds them until Drop has reaped QEMU.
• GUEST_QUIET stays unscaled. Its doc comment said the kernel speaks every 10 s; that report was
  removed on 2026-10-04, and the comment now says what holds instead. ceiling_verdict's doc loses
  the same premise.

What this does not cover: QEMU guests started outside the harness (cargo run), and host load that
is not guests, such as builds.

Evidence

Logs, tallies and scripts are in the evidence branch: r1 in pr-1166/r1, r0 in pr-1166/r0.

Parallel load, the shape that silenced the rows. Each round ran two harness invocations of the
four 8-vCPU rows at once (64 vCPUs asked of 14 cores), beside other worktrees' work at host load
19 to 53. The arms were interleaved over four rounds, 32 rows each. The base is main at
65c77e6dd, which already carries #1160's one-copy read; it is this change reverted.

arm	rows silent	spans of counters_read (guest clock)
main	5 of 32	0.88 to 15.6 s; 16 of 27 over 3 s
this change	0 of 32	0.85 to 14.22 s; 4 of 32 over 2.4 s

Each budgeted guest waited its turn: the logs show waiting for the host CPUs (8 of 14, …) and
then acquired; two 8-vCPU guests cannot hold 16 of 14. The 14.22 s read ran with only its own guest
admitted. It is what non-guest load still does, and it is the new record's evidence.

The nightly's shape (r1). guest.yml runs cargo run -- --ci guest, which runs the suite with
--jobs 1 (build/ci.rs:133). A four-core runner never has two guests from the suite at once, so the
budget changes nothing there but a lock take per boot. The suite's time against the 60-minute guard
stays what it was. Measured in that shape on the dev host, --jobs 1 with TOYOS_HOST_CORES=4, over
the four 8-vCPU rows and libc_children, one run per arm:

• neither arm waited for CPUs once;
• the 8-vCPU rows took 9–16 s on main and 8–17 s with the budget.

The head run's libc_children failed in tty_interrupt, at the nested shell's line, after 207 s.
That is the recorded libc-childrens-tty-interrupt-can-stall-at-the-nested-shells-line, which did not
wait for CPUs here either.

What the budget does cost is local wide runs. On the 14-core dev host at 12 wide, the running guests' vCPUs now sum to
at most 14: one 8-vCPU guest and three 2-vCPU ones, or seven 2-vCPU ones. That cost is not measured.

Ambient load, one invocation. Six rounds before #1160 landed, at host load 18 to 36, 24 rows
per arm: main 7 silent, this change 1 silent.

Checks

• cargo test -p toyos-build --lib -- buildlock::tests::a_guest buildlock::tests::a_narrow: three
  new tests, green. Each waits on another thread's report with a bound, never on a sleep that has to
  pass.
• Negative controls, each a checked patch built and run, then reverted (controls/):
  • M1, an attempt that keeps the CPUs of a short take: red, a waiting guest held a CPU.
  • M2, the queue let go before the wait: red, because the wide waiter never holds the queue.
    Both controls ran before r1's probe retry; M1 leaks the CPU for good, so the retry still reds it.
• cargo run -- --ci host at 69f7f7292, the head: exit 0, 109 steps green (r1/host-ci.log.gz).
• The guest A/Bs above ran at ca16c59f2. Review r1 changes no admission for a guest no wider than its host on a host whose
  real core count is at least TOYOS_HOST_CORES, so they stand. The nightly-shape run is at the r1 code.
• The guest runs above, each invocation's log with its exit code.

Metal: none. The change is in the QEMU harness.

🤖 Generated with Claude Code

[Japabu]

Review of #1166 at 28a9812 (r0)

Two commits on 65c77e6dd: +172/-57. Production is about +60 lines (guest_cpus and GuestCpus in build/buildlock.rs, and the take and the field in tests/common/qemu.rs). Tests are about +80 (three host tests). Records: +24, -49.

Architecture

The harness that makes the oversubscription is the right owner. Putting the budget in the host store is the right scope, since the contention spans worktrees. Each CPU is one flock token, taken all-or-nothing under a head-of-line queue lock. That is a small and correct design, and it reuses #1145's poll loop, the 6 h CEILING, heartbeats and holder notes rather than adding a second wait mechanism. The design has one structural property worth calling out: only the queue holder ever runs a CPU attempt, so two attempts never run at once anywhere on the host. Transient partial takes therefore cannot interfere with each other, and a waiter holds no CPU. The record that is closed (a-counters-read-…) asked for this exit.

What I checked

• Deadlock across processes and worktrees. CPU waiters hold nothing except queue. CPU holders never wait on queue. guest_cpus runs after build_boot_image_with has dropped its build locks, so a build-lock holder never waits for CPUs and no cycle with the build locks exists. All-or-nothing rules out a partial-hold deadlock. No deadlock is possible as long as no thread asks for CPUs while it holds a live guest (see S1).

• Holder death. flock goes with the open file description on Linux and macOS, so a harness killed while it holds CPUs or waits in the queue strands nothing. QEMU follows through the tether's SIGHUP. Lock fds are O_CLOEXEC, so QEMU never inherits them. A stopped harness (SIGSTOP) holds its CPUs until other waiters hit the 6 h ceiling. The panic then names its pid, which is buildlock: one polled wait with a ceiling, and a would-be maker tries #1145's contract.

• A guest wider than the host. It is clamped to every CPU, so it runs alone and its wait terminates. It is still oversubscribed by itself (see R2).

• Starvation. A wide guest that holds queue cannot be overtaken. Before it reaches the head, queue is a free-for-all, and backoff favours newcomers: they poll at 1–32 ms while an older waiter polls at 50 ms. So "never overtaken" holds only once a guest is the head waiter. That is probabilistic fairness, not FIFO, and the module doc states it accurately. It is not a blocker.

• The wait and the guest's clocks. The take comes after the image is built and before spawn_and_wait_ready. wait_for_ready's start, FASTEST_BOOT_MS and every Liveness begin after the spawn. Only Outcome::elapsed (from run_task's mark) now includes the wait, and nothing judges by it. Confirmed.

• Release after reap. For a constructed QemuInstance, _cpus is the last field, so it drops after Drop::drop has killed and waited, on every exit path including unwinding. On the panics inside spawn_and_wait_ready before the instance exists, QEMU is never waited:

  • a boot timeout or a missing build line runs child.kill() and then panics;
  • the root_file_on failure panics with no kill at all, leaving only the tether's SIGHUP.

  cpus is the first binding, so it drops last, after the tether hangs QEMU up but before QEMU is reaped. The overlap is milliseconds with a dying QEMU, which is harmless (S2). The claim "until Drop has reaped" is exact only for constructed instances.

Required

R1. build/buildlock.rs:852-856: the new helper splits an existing doc comment.
fn guest was inserted between a_queued_exclusive_phase_goes_first and its doc comment. Rustdoc now gives guest both paragraphs ("flock alone would let a stream of builds starve … the gate on it."), and the test has lost its doc. Move the helpers above that comment.

R2. tests/common/qemu.rs:361-363: the new GUEST_QUIET premise is false where it matters most.
The comment says "every vCPU a guest runs has a host CPU of its own". That is false in two cases:

• A guest wider than the host. On the nightly guest runners (4 cores, per both lanes' instrument lines in run 41618038084), all four 8-vCPU virtsmpcase rows are 2x oversubscribed by themselves.
• Non-guest load. The PR's own 14.22 s read, and the new record, show this.

Two fixes are needed:

• State the exception in the comment, or delete the sentence.
• Record the wider-than-host case: the deleted record's exit ("admits a guest only where the host has a CPU for each of its vCPUs") is not met there by design. One sentence in issues/the-guest-cpu-budget-counts-guests-and-not-builds.md (or a renamed slug) is enough, so the residual is tracked rather than silently closed.

R3. tests/common/qemu.rs:219-222: host_cores' doc is now stale.
It says TOYOS_HOST_CORES "can only widen a liveness ceiling … a stale value costs a slower wedge report and never a false pass". The value now also sets the admission budget. A value above the real core count re-admits exactly the oversubscription this PR removes, and produces a STALL red instead of a slow wedge report. Two acceptable fixes:

• Update the doc.
• Better: pass guest_cpus min(host_cores(), available_parallelism), so the override can only narrow the budget, matching its "only ever downward" intent.

R4. build/buildlock.rs:888-890: the free-CPU assertion is nondeterministic, and breaks the module's own rule at :572-574.
The test asserts that cpu-3 is free while the waiting guest b keeps taking cpu-3 for a moment on every poll attempt (lock, push, len != want, drop, which writes the note). It also asserts while other lib tests fork children: tether's pre_exec spawns copy every descriptor until exec. A probe that lands in either window reds "a waiting guest held a CPU" on a correct tree.

Fix: retry the probe until one try_lock succeeds within a bound (e.g. 2 s). A transient hold then costs one retry. M1, which leaks the hold permanently, still reds.

R5. Missing evidence: suite wall time.
The evidence covers the four virtsmpcase rows only. On the nightly shape the budget turns the suite from up to 12 guests at once into:

• at most two 2-vCPU guests at once (default smp: 2, 4 cores);
• every 8-vCPU row alone.

guest.yml has a 60-minute wedge guard. The last nightly's tcg / suite took 41 min and kvm / suite 36 min (run 41618038084), and KVM guests, which are mostly idle, lose the most concurrency. The growth is neither measured nor stated, so the risk that the nightly lanes start hitting their timeout is open.

Cheapest sufficient check: the full guest suite (cargo test --test toyos, the --ci guest set) on both arms with TOYOS_HOST_CORES=4. That reproduces the runner's admission on the dev host. Interleave the arms, report suite elapsed per arm against the 60 min guard, and state the result in the PR. If it exceeds the guard, the guard or the width needs a decision before landing.

Suggestions (optional)

• S1. Two live guests in one thread can deadlock. qemu.rs:1540 says "one test may hold two instances at once". A thread that holds a guest's CPUs and asks for a second guest breaks "nothing waits on queue while holding a CPU":

  • if the two widths add up to more than the host, the thread waits on itself;
  • otherwise it can deadlock with another process's wide head waiter.

  Either way it is invisible until the 6 h ceiling, past CI's 60 min. No current test does this; I found none. A thread-local count of held GuestCpus, with a panic on a second take, would make it fail at once.

• S2. On the panic paths in spawn_and_wait_ready, add child.wait() after child.kill(), and kill on the root_file_on failure too. "Released after reap" (and NvmeClaim's same promise) then holds everywhere.

• S3. The queue's "unnoted" text, "a wider guest is waiting for host CPUs", is wrong whenever the head waiter is narrower or the same width. Say "another guest is waiting for host CPUs".

• S4. Each attempt runs create_dir_all and opens every cpu-N again, every 50 ms, from the head waiter. Opening the files once before poll and only try_locking inside the attempt is simpler and cheaper.

• S5. qemu.rs:486-487: the edited ceiling_verdict paragraph left one unwrapped line of about 130 columns.

• S6. Negative controls. M2 reds at the test's own precondition ("the wide guest never queued"), not at the overtake assertion (:925). A control where the narrow guest skips queue would exercise :925, though inspection already shows that it would red.

Evidence checked

• The evidence README, both control patches and their logs, and par.log: the head arm's per-row times grow (up to 82 s, against about 32 s on base). The full parallel and ambient logs were not opened.
• --ci host ran at ca16c59. The later commit touches only issues/, so the run still applies.

SEND BACK