[Japabu]

This PR is step 10 of issues/the-kernel-is-small-interrupts-post-and-threads-wait.md and stage 4 of issues/a-device-function-is-driven-from-its-arrival-to-its-end.md.

What changes

usbd is a drives row. It runs one process per xHCI controller (drives = ["pci:0c:03:30"], receives = ["disk", "input"]).

• Its sticks are disks offered to One block service and a driver per disk controller; input names each device; SCSI is its own crate (usbd step 10, PR 1 of 2) #962's block service on disk. It serves their sessions straight out of each client's region.
• Its keyboards and pointers report to the owner of input over input. Each bound HID collection is a source of its own.

The whole xHCI leaves the kernel. Deleted, not bridged:

• the kernel's xHCI driver, USB storage, block layer and GPT;
• keyboard.rs, mouse.rs and input_merge_test;
• partition claims and SYS_PARTITION_*;
• usb_gate, irq_ring, drain_irqs and the scheduler's Operation framework;
• sync::OwedLock, its loom test and the owed-fence-off feature;
• every xHCI and USB actuator, with the harness rows that only they served.

From the ABI: the Keyboard and Mouse device and file types, Partition, and the USB, block and partition inventory records. A reset now stops every claimed function's bus mastering and messages (pcidev::stop_before_reset), where it used to run the xHCI stop.

The machine's stop hands every USB device back (the orchestrator's ruling in review, which keeps the owner's run-18 ruling).

• After logkeeper's flush and every file server's sync, the supervisor tells each disk driver the machine stops (a stop pipe it closes). It waits up to DRIVERS_MS, 3 s, for each one to end.
• usbd then runs Hc::stop:
  • it resets every connected port while the controller runs, then waits TRSTRCY;
  • it halts the controller, runs HCRST, and unpowers its ports;
  • it exits 0 only if every port's reset finished and the controller halted and reset.
• nvme does a normal shutdown (CC.SHN).
• If no usbd is running at the stop (it died, or the boot deadline, the hard-lockup detector or a panic is what stops the machine), the ports are not reset. The device sees the platform reset alone, which S1 of the xHCI move into usbd: a FAT entry that grows its file is written a cache flush after its chain and data, held by a module test that stops the disk inside a flush epoch #901 §6 accepted. stick_survives_a_reset_under_load measures that case.

The driver (system/usbd/xhci, moved from lib/xhci)

• Tasks on one thread. Each task waits on one thing: the event that matches its TRB, a deadline, or a change another task made.
• A deadline never cancels. Command Abort takes back a command the controller still holds. Stop Endpoint plus Set TR Dequeue Pointer takes back a transfer. Dropping an operation the controller never answered is a panic.
• Events are read on each interrupt record. IMAN.IP is cleared before the ring is drained, and every drain writes ERDP with EHB, an empty one too (xHCI 1.2 §4.17.2, §5.5.2.3.3). Without this, an interrupt that found nothing would leave the next event unannounced.
• Taking the controller from firmware follows §4.22.1. usbd sets HC OS Owned and waits up to 1 s for HC BIOS Owned to clear (§4.22.1: the wait "should not exceed '1' second"). A controller firmware keeps is refused by name and left as found.
• Starting it follows iommu: lowest-first device addresses below each claim's stated limit; a claim masters only when its holder starts it #974's hand-over: usbd states its address limit (64 bits, or 32 where HCCPARAMS1.AC64 is clear), takes its grant, and starts bus mastering before it writes any address to the controller.
• Settling waits for every port's attach window: TSIGATT, 100 ms on USB 2 (USB 2.0 §7.1.7.3), and tPollingLFPSTimeout, 360 ms on USB 3 (USB 3.2 §7.5.4.3). A port with a device settles once that device is enumerated or given up on by name. Without this the T14's stick was offered after LOG and BOOT had been answered absent.
• A TD is put whole before the ring's Link, first TRB last, TD Size stated. usbd moves a client's region directly, so a data phase crossing 64 KiB is a TD of several TRBs (the kernel driver bounced every transfer through one TRB). Each TRB states the packets left after it (§4.11.2.4). A TD that would cross the Link starts on the ring's first TRB behind a Link TD (§4.11.7; "a TD Fragment shall not span segments", §4.11.5.1). Its first TRB is handed over last (§4.11.7.1). An early Link marks every slot it skips with the closing lap's cycle, so no slot two laps old is the controller's on the next lap (§4.9.2). A TD is at most half a ring, so the TD after an early Link never overwrites it.
• Every refusal it can name ends usbd by name, and diskserver treats a driver that ends before settling as unusable at once instead of waiting its bound.
• Host-tested against a stub xHC written from xHCI 1.2. The crate is generic over toyos-device-memory's traits. It has 110 tests, including the three the step's exit named and the one it owed: a stick whose bulk endpoint never answers leaves a keyboard on the same controller delivering every report.

HID and storage

• HID (lib/usbhid, moved from kernel/usbhid) binds each collection by its report descriptor, in report protocol:
  • arrays are read as indexes and bitmaps as keys;
  • HUT's reserved usages and 00–03 name no key;
  • a collection the reader cannot bind costs only its own reports;
  • absolute axes are scaled onto the record's range.
• Mass storage keeps its Bulk-Only round trip and its recovery ladder. When the ladder resets a disk's port, the disk is held for its return and given back only to a device with the same identity. A recovery whose CLEAR_FEATURE(HALT) fails takes back the transfer the ring was moved past, and does not panic.

Decisions

• A PCI matcher may name a place (pci:0c:03:30/1022:1639@03:00.3). Test images use it so that the AMD bench's boot controller is driven and its other controller is left to the claim tests. Shipped images name the class alone.
• The input port's holders are ps2server and usbd (HELD_ALONE holds a list now).
• Spare-controller tests claim the controller no row names. Their control is a claim on the held boot controller, which is refused AlreadyExists. This replaces xhci-leave.

How a metal run's log reaches readback

The path is: logkeeper → the LOG file server → diskserver's block session → usbd's disk on the boot stick (Bulk-Only Transport) → the stick's log partition. toyos-metal reads that partition back (--fat32-check).

The kernel holds no part of this path any more. Every metal judge reads /log off the stick, so each staged row is also the proof of this path on its bench. usbd_serves_the_boot_stick (new, rides TESTCASES on both benches) checks four things:

• the boot controller was handed over once, to usbd;
• its message was armed through a remapping entry of its own;
• its No-Op was answered;
• usbd says the stick is serving it as disk 0.

On the benches:

• T14 at 8a93782dd: all five rows pass, and the log reaches the stick through usbd.
• AMD at 8a93782dd:
  • usbload passed: stick_survives_a_reset_under_load passes on its readback, with 7320 KiB of the load's lines on the stick.
  • testcases failed: its /log stopped mid-append at about 16 s while tone_beside_stick_write wrote 2000 lines a second, and the boot ran on to an orderly stop. A probe boot (r9) reproduced it; a second probe (r10) did not. The cause is not found yet (see Risks).

Checks

check	revision	exit
cargo run -- --ci host	421c2b3	0, "87 step(s), all green"
cargo run -- --build-only, then --arch aarch64	421c2b3	0, 0
cargo test --test toyos-build (68 rows)	421c2b3	1: 67 passed

The one red is crypto_instrument_prints_its_table: QEMU drops console bytes (issues/qemu-drops-console-output-the-harness-is-slow-to-read.md). That stall has the same signature at the base and in an image with no usbd.

Negative controls, each applied as a patch, red, and restored (mut/ in both evidence trees):

• x1–x10, the driver. x1 was rerun on the final driver code and is red on the same assertion. x10 is the CLEAR_FEATURE failure left unreclaimed.
• i1–i5, the input link.
• h1–h10, the HID reader.
• k1: usbd drops every key; the USB key smoke stalls.
• l1: usbd's writes turned into reads; the stick holds no load line.
• s1: the supervisor tells no driver; usbd does not end before the kernel's stop.
• s2: the stop writes no port reset; red on the stub and in QEMU (exit 3).
• t1: the old settle rule; both attach-window tests red.
• d1: diskserver forgets a driver that ended before settling; red.
• f1: a controller firmware keeps is taken anyway; red.
• r1: TDs placed across the Link as before (Link with Chain); the stub refuses "a TD crosses the ring's Link".
• z1: TD Size left 0; the hand-counted TD Size test is red.
• b3: an early Link that marks nothing it skips; the ring walk takes slot 254 two laps old.
• b4: a TD bound of 255 TRBs; the longest-TD walk stops on the overwritten Link.
• b5: an r14 readback with a third 03:00.4 fault record; amdvi_usbd_drives_its_own_entry is red.

x8 (IP never cleared) is red on the stub's interrupt model. Under MSI or MSI-X, IP clears itself (§5.5.2.1).

Integration tests and where they run

• QEMU (x86-64 q35, nec-usb-xhci and qemu-xhci):
  • The handover, claim and refusals of a real QEMU xHC, which no host model stands in for: usbd_drives_the_spare, usbd_refused_without_amdvi_remapping, usbd_refused_without_smi_filter, amdvi_firmware_left, iommu_virtio_platform, a_drives_row_holds_one_process_per_function and the other testcases rows.
  • the_log_lands_on_the_boot_stick: a stick boot whose log volume is read back on the host, and whose stop under load is judged.
  • a_usb_key_reaches_the_console: a key from a real xHC reaches the owner of input.
• T14: usbd_serves_the_boot_stick, usbd_drives_the_type_c_controller, stick_survives_a_reset_under_load, tone_beside_a_stick_write, mask_windows.
• AMD bench: usbd_serves_the_boot_stick, amdvi_usbd_drives_its_own_entry, amdvi_translates_the_stick_controller, amdvi_remaps_the_stick_controller, stick_survives_a_reset_under_load, tone_beside_a_stick_write.
• No metal row stages a machine that cannot remap (the orchestrator's ruling). Without remapping every claim is refused, usbd's included, so that boot leaves no /log to judge. amdvi_claim_refused_without_remapping and claim_refused_without_remapping are deleted, and with them the kernel's iommu-no-remap actuator and the usbd_spare_unhanded binary. QEMU holds the refusal, by usbd_refused_without_amdvi_remapping and amdvi_firmware_left.

These rows exercise real controllers, their interrupt remapping and the stick that carries /log. Neither the stub nor QEMU can establish that.

Risks and gaps

• No USB at all without remapping. A machine with no IOMMU, or one that cannot remap interrupts (QEMU's AMD-Vi included), has no USB now: the claim is refused, by design. amdvi_firmware_left expects that refusal.
• The AMD stick stall, and which of three xHCI rules it was, is not separated. At 2000 log lines a second the AMD testcases boot's /log stopped mid-append: r8, r9, then 6 of 7 probe boots (r12, r13); main with only the tone row added, 0 of 2. Probes that report by exit codes the kernel records showed usbd's: a stick transfer unanswered, no event on the ring, the endpoint Running, and in 2 of 3 boots USBSTS HCH, HSE or HCE set. The multi-TRB TDs broke three xHCI rules (above). Since 0e0c37e follows them, 6 of 6 AMD testcases boots are clean (r14). That is a rate from 6/7 to 0/6, not a proof of which rule this controller needs.
• metaldevicecase is ridden by no row. Finding filed: issues/the-device-boot-is-flashed-by-no-metal-row.md.
• A usbd row without input binds HID until its first open answers NotGiven. That is a few control transfers per keyboard.
• RawKeyEvent still lives in sdk/abi. Finding filed: issues/the-key-record-the-surface-protocol-carries-is-defined-in-the-kernel-abi.md.
• A hot-plugged stick is routed from its own offer alone. Recorded with an owner: issues/a-disk-plugged-in-after-settling-is-routed-as-soon-as-it-is-offered.md.
• The stop is protected only on the orderly path. It does not cover a usbd that died, or a deadline, lockup or panic reset.

Metal

• T14 at 8a93782dd: usbd_serves_the_boot_stick, usbd_drives_the_type_c_controller, stick_survives_a_reset_under_load, tone_beside_a_stick_write, mask_windows: all PASS.
• AMD at 8a93782dd: stick_survives_a_reset_under_load PASS. The testcases boot is open (above).
• b07a9b1ac changes, from 8a93782dd, only the refusal of a controller firmware keeps. That path ran on neither bench: firmware released 03:00.3 in about 11 ms on every boot after iommu: lowest-first device addresses below each claim's stated limit; a claim masters only when its holder starts it #974.
• r14 at 0e0c37e66: T14 five rows PASS; AMD usbload PASS. AMD testcases six boots: all completed with clean FAT checks, and usbd_serves_the_boot_stick and tone_beside_a_stick_write PASS on all six. The three amdvi rows were red on every boot from judges, not the machine. This branch folded main's separate xhci-leave boot into testcases, so test_rs_usbd_spare_fault's staged fault met the no-fault rule of the translate and remap rows. And usbd's line now begins ENDED — . The fix is in the judges: the translate and remap rows refuse every fault but 03:00.4's, and amdvi_usbd_drives_its_own_entry requires exactly 03:00.4's two staged records. The six r14 readbacks re-judged pass the amdvi rows. A copy with a fault on 03:00.3 added is red on the no-fault rows, and one with a third 03:00.4 record is red on the spare's row.
• 421c2b32a changes the ring again (review r2, B3 and B4), so both benches are staged again there: AMD testcases six boots and usbload, and the T14's three images.

🤖 Generated with Claude Code

[Japabu]

Review of #971 at bab43d811 (diff origin/wt/toyos-usbd...origin/wt/toyos-usbd-2)

Architecture

The shape matches #901's accepted design. usbd owns each controller whole. HID goes to the owner of input over input, with a source and a gone per collection. Sticks are offered on disk and served out of each client's region through toyos-blockserve. The kernel stop is now generic over claimed functions (pcidev::stop_before_reset). toyos-xhci moved under system/usbd, where its one program lives. toyos-usbhid stays in lib/ as a pure decoder of device-chosen bytes.

The deletions are complete. git grep outside issues/ finds nothing for OwedLock, owed-fence, irq_ring, drain_irqs, usb_gate, poll_if_pending, port_work_pending, input_merge_test, DeviceType::Keyboard/Mouse, FileType::Keyboard/Mouse, DEADMAN, between_attempts, until_answered, probe_usb_disks, usb_storage, either selftest actuator, kernel/usbhid, lib/xhci or toyos-xhci-sim. What remains is the stale references listed under finding 5. Every deleted issue slug has no citation left.

Size: +8895/−20753 overall. The kernel is +197/−10115. Test files are roughly +2.8k/−3.0k.

Trust model: a machine without remapping now refuses the claim, and it says so twice: pcidev's NOT HANDED OVER — its interrupts would not be remapped and the supervisor's usbd was not started on PCI …. iommu_virtio_platform's no-unit arm and amdvi_firmware_left assert that refusal, and a-machine-without-an-iommu-refuses-every-claim.md records the consequence. That is consistent with root's isolation policy.

Interrupt handling is as claimed. take_events writes IMAN (IP W1C, IE kept) before draining, and writes ERDP|EHB on every drain, empty ones included (hc.rs:536-568). an_interrupt_that_finds_no_event_leaves_the_next_one_announced holds the EHB half, with x9 as its control.

The owed tests exist:

• a_stick_whose_bulk_endpoint_never_answers_leaves_a_keyboard_on_the_same_controller_delivering_every_report (tests.rs:279), with x1 as its control. See finding 3.
• The three renamed stub tests.
• The FAT append/sync test (fat.rs:808), which is already on main via 17ea529.
• toyos-input's release test, which came from One block service and a driver per disk controller; input names each device; SCSI is its own crate (usbd step 10, PR 1 of 2) #962's 27b75c3 and is unchanged here.

The crypto red's independence holds on the base run alone: base-e07354fb1-run.log reads STALLED: 973s at the base. no-usbd-run.log ends in a SIGTERM with no verdict on that row, so it shows nothing either way and should not be cited as a reproduction.

The place matcher (@03:00.3) is backed by #901 (claims by location; roast 8 on the AMD twins), so it is not test-only machinery.

Findings

1. BLOCKER: the approved stage's metal rows are deferred, the one guarding the deleted reset stop among them.

• Where: issues/the-kernel-is-small-interrupts-post-and-threads-wait.md:204-209.
• What S1 of the xHCI move into usbd: a FAT entry that grows its file is written a cache flush after its chain and data, held by a module test that stops the disk inside a flush epoch #901 decided: §9 puts stick_survives_a_reset_under_load, mask_windows under a stick write and tone_beside_a_stick_write "on the T14 and the AMD bench, at S5". Its table's S5 row lists "the metal rows in section 9 on both benches". This PR is S5.
• Why the reset row matters: S1 of the xHCI move into usbd: a FAT entry that grows its file is written a cache flush after its chain and data, held by a module test that stops the disk inside a flush epoch #901 roast 6 and roast 14 accept deleting the kernel's port-reset stop only with that row: "Without them, … the reset stop's deletion would be judged by no machine." That stop was the mechanism of the owner's ruling after run 18, "ToyOS … must not brick a USB device" (5ab414b).
• What happens without it: every metal boot now ends in a platform reset with no port reset, which is the condition run 18 bricked the stick under. Nothing reds if it happens again. The orchestrator is booting both benches on this right now.
• What is needed: either the row lands here, or the orchestrator rules the deferral explicitly, consulting the owner on the run-18 ruling. Recording the rows as "Left" does not substitute for a decision that changes the approved stage.

2. BLOCKER: one recovery path panics usbd, which drops every device on the controller.

• Where: system/usbd/xhci/src/usb.rs:949-950.
• What happens: when the CLEAR_FEATURE(HALT) fails, restart returns Ok(Err(())) without calling op.reclaimed() on held. Op::drop (hc.rs:774-786) then panics, because ending is false.
• How it is reached: from msc.rs:732, when a bulk transfer goes silent on an endpoint whose output context reads Halted. That happens if the halt's event was not yet announced, or if the device stalls or leaves (control_held answers Gone at once once leaving is set). The route is Reset Endpoint → Set TR Dequeue → ClearHalt.
• Why it is a defect: by that point Set TR Dequeue has moved the ring past the TRB, so the controller holds none of it, and the panic is wrong. It breaks the "a device that stops answering costs only itself" invariant the PR claims.
• Fix: reclaim held before that return. A stub case for this path (an endpoint Halted with its event unread, and CLEAR_FEATURE stalled) would be cheap if you want it held.

3. BLOCKER: the PR's account of the x1 control is false.

• What the PR says: "x1–x7 … ran before the IP/EHB change. That change touches only take_events and the stub's interrupter, which none of those paths reads."
• What x1 is: a mutation inside take_events (mut/x1-dispatch-awaits-the-stick.patch). It is the negative control of the owed test.
• Why the result may differ: with the old code an empty drain wrote no ERDP; the final code clears EHB on every drain. x1's red may now come from a different cause, or not at all.
• Cheapest check: rerun x1 against bab43d811. It is one host test, in seconds.

4. BLOCKER: the usbd_serves_the_boot_stick judge contradicts its own comment.

• Where: tests/toyos.rs:1195 says "Boot: complete and the log volume's mount are printed, not judged". tests/toyos.rs:4480 then calls log.must_say("fileserver: Log serving /log"), which judges it.
• What S1 of the xHCI move into usbd: a FAT entry that grows its file is written a cache flush after its chain and data, held by a module test that stops the disk inside a flush epoch #901 settled: its review round asked this row to hold the kernel's part only.
• Fix: make the comment and the code agree, one way or the other.

5. BLOCKER: records and comments this change made false, and dead code it left.

• The small-kernel track (issues/the-kernel-is-small-interrupts-post-and-threads-wait.md):
  • Lines 75-77 ("which the kernel drives until step 10; the kernel's USB storage, block layer …") are stale.
  • Lines 113-119 (step 4's "Left") and 142-144 (step 7, "which step 10 deletes") are stale.
  • Line 103's step tally is stale.
  • Step 10's "Done" at line 190 claims "every usb-kbd and usb-tablet go through usbd". That is false for every test image: their usbd rows receive no input, so HID binds and is dropped at NotGiven.
  • Step 10's /log evidence (line 203) drops the client test issues/nothing-reopens-a-diskserver-session-over-a-used-page.md owes, which the exit named and which is still open. List it under "Left".
  • Line 206 has trailing whitespace.
• issues/a-cut-bulk-only-command-does-not-brick-the-benchs-stick.md:95-104: "The present state" describes a standing guard and a reset path ("drove every port reset and took the power away") that this PR deletes. The table's survivals were all measured with that port reset in place. Say so, rather than only appending to the exit.
• lib/bcachefs/tests/integration.rs:1339 cites "kernel/CLAUDE.md's BudgetExpired rule", which this PR deletes from kernel/CLAUDE.md.
• kernel/src/drivers/pci.rs:103-105: the CONFIG_BYTES rationale ("the reset-time xHCI stop rebuilds one") is false now.
• kernel/src/drivers/pci.rs:328: links [Self::message], which is deleted.
• kernel/Cargo.toml:362-363: "The block path's half went with the kernel's xHCI." is history. Delete it.
• sdk/abi/src/syscall.rs:1276-1282: PciId::parse is now dead. Its doc names the deleted xhci-leave=, and its only caller is its own test (:2553-2560, the_xhci_leave_parameter_…). Delete both.
• tests/guest/rust-tests/src/bin/end_returns_claims.rs:2: still says "the function xhci-leave= leaves".
• system/usbd/xhci/Cargo.toml:2: names src/stub.rs; the stub is src/stub/mod.rs.
• Dead code:
  • Hc::change_waiting (hc.rs:491) has no caller.
  • Inputs::not_given() (input.rs:184) has no caller.
• New findings with no owner: issues/the-device-boot-is-flashed-by-no-metal-row.md and issues/the-key-record-the-surface-protocol-carries-is-defined-in-the-kernel-abi.md record evidence and an exit but no owner. Root's "evidence, owner and exit" asks for all three.

6. BLOCKER: USB HID is unverified on any non-stub controller.

• What is missing: this PR deletes the kernel's working USB keyboard path and usb_keyboard_rollover. No guest or metal row now delivers a report from a real xHC through usbd to an owner of input. QEMU is the one available independent implementation, and S1 of the xHCI move into usbd: a FAT entry that grows its file is written a cache flush after its chain and data, held by a module test that stops the disk inside a flush epoch #901 §8 names it as an oracle. The stub was written by the same agent, so it is not independent.
• Cheapest sufficient check: one smoke run. A QMP key on a usb-kbd, in an image whose usbd receives input, is seen by the owner. console_image_boots' image already gives usbd input.
• Alternative: reword the claims in finding 5, record the gap with an owner, and have the orchestrator accept it.

Suggestions

• main.rs:178, Config.input: endow::namespace().is_some() binds HID on every test-image usbd until the first open answers NotGiven. Deciding from the endowment itself at start would remove Inputs::not_given and the stated risk.
• hc.rs:409: recomputes grant_bytes() inline. Call it instead.
• hc.rs:805, the firmware handoff: spins spin_loop for up to 1 s. Polling is justified, since firmware raises nothing. Sleeping between reads would not burn the CPU.
• x8 (IP never cleared): my reading of xHCI 1.2 §5.5.2.1 is that IP clears by itself under MSI/MSI-X; please check that against the text. If so, x8 reds on the stub's interrupt model rather than on a hardware-visible defect. Clearing IP is still correct.
• README.md:37 and :197: they still describe the kernel calling the xHCI port machine and driving the stick. The primary checkout has uncommitted README edits, so coordinate before touching it.

SEND BACK

[Japabu]

Round 1 is answered on wt/toyos-usbd-2 up to 45b247594. Evidence: https://github.com/ToyOSOrg/evidence/tree/13597a46531bb0ee3250c03c075fdc980da2f478/pr2-r1

1. The stick protection (on the orchestrator's ruling), 9bdc96abf.

• The supervisor tells the disk drivers. Every disk-driver instance is endowed a stop pipe (toyos_quiesce::DRIVER_STOP). After logkeeper's flush and every file server's sync, the supervisor closes those pipes and waits for each driver to end, within DRIVERS_MS (3 s). A driver still running at the bound is named, and the machine stops without it. If the kernel refuses the stop, the drivers are started again.
• usbd stops the controller (Hc::stop, bounded by STOP_NS, 2 s):
  • it resets every connected port while the controller still runs, then waits TRSTRCY if a USB 2 device was reset;
  • it clears RS and waits for HCH, runs HCRST, and unpowers the ports where the controller has PPC;
  • its exit code is 0 only if every port's reset finished (PRC set after this write) and the controller halted and reset. The kernel always records the code, while usbd's own line may reach no log.
• nvme shuts down normally (CC.SHN, awaited on CSTS.SHST).
• If usbd died: its claim's release already reset the function (FLR or the D3hot round trip), and the kernel's stop clears Bus Master. Nothing resets the ports; the device sees the platform reset alone. The boot deadline, the hard-lockup detector and a panic are the same case, which S1 of the xHCI move into usbd: a FAT entry that grows its file is written a cache flush after its chain and data, held by a module test that stops the disk inside a flush epoch #901 §6 accepted.

How the protection is shown before the stick test runs:

• the_stop_resets_every_connected_port_then_halts_resets_and_unpowers_the_controller (stub). The stub leaves a stick command open, and the test checks every step read back.
• the_log_lands_on_the_boot_stick (QEMU, stick boot):
  • stick_load writes the stick, and the power button is pressed while it does;
  • the stick's log volume is read back on the host and judged with the metal verdict;
  • usbd must end with code 0 between the stop's request and the kernel's stop: record.
• Controls:
  • the supervisor telling no driver: red on the order;
  • the stop writing no port reset: red on the stub (reset 0 of 2) and in QEMU (exit 3);
  • usbd's writes turned into reads: red, with the same "no logkeeper file" signature the metal boots showed.

The three #901 §9 rows are in this PR, none deferred (e185b68cc, ecbb0ead5):

• stick_survives_a_reset_under_load boots testcases with stick_load alone. The boot deadline is 40 s, below the runner's bound, so the reset touches no USB register while the stick is being written. The judge needs DEADLINE_EXPIRED, the load's lines in /log, and boot.usbload.stick_secs against the bench's record. A host check covers the judge.
• tone_beside_a_stick_write rides testcases: the tone plays while a thread of the same process writes ≥1 MiB to the stick. Verdict: underruns=0.
• mask_windows also reads windows under stick_write (32 MiB) on windowscase. Judged against the record (boot.windowscase.stick_write.*).
• None of them has run.

2. d4459fc14: a failed CLEAR_FEATURE(HALT) reclaims the held transfer. a_halt_whose_clear_feature_stalls_costs_only_its_stick covers it: the stub halts the endpoint with its event lost and stalls CLEAR_FEATURE. With the fix reverted, the test panics at the Op drop (mut/x10).

3. x1 was rerun on the final driver code. It is red on the same assertion (collection 0: a Keyboard never said). The PR's earlier account was wrong to say IP/EHB touched no path x1 reads, and the body now says what the rerun showed.

4. The judge no longer requires the mount, as its comment and #901 say.

5. All of finding 5 is done (d4459fc14, f40887c0f):

• the track's stale lines, the step tally, step 10's HID claim, the client test now listed under Left, and the trailing whitespace;
• the cut-command issue's present state now says its table was measured with the old port reset in place;
• the bcachefs citation, CONFIG_BYTES, the message link and the Cargo history line;
• PciId::parse and its test, end_returns_claims' doc, and the stub path;
• change_waiting and not_given();
• owners on the new issues.

README.md is untouched. It needs two edits, which I have passed to the orchestrator:

• :37 lists "the xHCI port state machine" among kernel decisions; it is now usbd's, in userland.
• :197 says the kernel's own xHCI and USB storage drivers read the stick and append the log through their FAT32 write path. Now usbd drives the stick, diskserver and fileserver serve it, and logkeeper writes /log.

6. f4c34d89d: a_usb_key_reaches_the_console. It boots images/console.toml on Profile::MetalUsbKeyboard, types ls and Enter through QMP, and waits for the shell to spawn toybox. Control: usbd drops key records, and the row stalls (mut/k1).

Suggestions:

• Taken: grant_bytes (45b247594).
• x8: §5.5.2.1 does have IP clear itself once an MSI/MSI-X write completes. So x8 reds on the stub's interrupt model (INTx-like), not on a defect visible under MSI-X. Clearing IP stays, since it is correct for either mode.
• Config.input stays decided at the first open. Deciding it at start would need a lookup the namespace does not offer.
• The firmware handoff spin is not changed in this round.

Still open.

• The metal log failure. The QEMU stick boot carries the log, so its cause is still unknown. The raw partitions hold no logkeeper text at all: no file, no stray clusters. AMD's testcases boot shows dmafault=3. Diagnostic images (a kernel that keeps the storage path's program lines and records in the black box) are staged for one boot per bench, with the stop protection in.
• amdvi_claim_refused_without_remapping cannot be judged on metal any more. Without remapping there is no USB, so there is no /log. I'd propose deleting the metal row and keeping the QEMU refusal rows (usbd_refused_without_amdvi_remapping, amdvi_firmware_left). That is the orchestrator's call.

[Japabu]

Orchestrator, T14 rounds 4–6 (hashes verified): at fa0f434 usbd_serves_the_boot_stick, usbd_drives_the_type_c_controller, tone_beside_a_stick_write, mask_windows PASS (loads now write /log on the stick and judges count bytes read back off it); at 56bd104 stick_survives_a_reset_under_load PASS (deadline reset at 51.5 s mid-write; the FAT left the designed crash state the checker now accepts only on deadline boots). All five T14 rows pass. Landing waits on #974 (K's DMA hold-off/activation), since the AMD bench's boot stick is on 03:00.3.

[Japabu]

---

r2: focused review at 5461e947c (new since 8a93782dd: b07a9b1ac, merge 4ee2dcadf, 0e0c37e66, 5461e947c)

Round 1's blockers: resolved

• BLOCKER 1, resolved at b07a9b1ac. Handoff::Kept returns Refusal::FirmwareKeepsIt again, before any SMI write, halt or HCRST. Its doc quotes §4.22.1, Reset's "firmware has let go of it" is true again, and a_controller_firmware_keeps_is_refused_and_left_as_found is restored with control f1.
• BLOCKER 2, resolved. HANDOFF_NS now quotes "should not exceed '1' second". Both round-1 suggestions were taken as well: the AC64-clear stub case and the settle doc.

Findings

BLOCKER 3. system/usbd/xhci/src/hc.rs:274-281 (Ring::link), called early from :254-256: an early Link leaves TRBs from two laps back that the controller treats as its own on the next lap.

Before this change every lap wrote every slot from 0 to 254. Each slot therefore held the previous lap's cycle bit, which never matches the controller's next Consumer Cycle State (§4.9.2). An early Link at slot t now ends lap L (cycle c) without rewriting slots t+1..=254. Those slots keep lap L−1's cycle !c, and !c is exactly the CCS the controller holds on lap L+1. On lap L+1, once the driver's tail rests at some p in t+1..=254, the controller finishes the TD before p, reads slot p, finds the cycle bit matching, and executes a TRB that lap L−1 wrote.

A concrete sequence:

1. Lap A (cycle 1): 1-TRB TDs run to the end, so slots 0..254 hold cycle 1.
2. Lap B (cycle 0): 1-TRB TDs run to tail 253. A 3-TRB TD then fails 253 + 3 > 255, so link() writes a Link at 253 and slot 254 still holds lap A's cycle 1.
3. Lap C (cycle 1): TDs end exactly at tail 254. The controller, with CCS 1, executes lap A's slot 254.

The same applies after a Set TR Dequeue to ring.dequeue() at such a p (usb.rs:949). This sequence is reachable now:

• An unaligned MAX_REQUEST_BLOCKS (32 × 4 KiB) data phase is 3 TRBs (msc.rs pieces).
• A control transfer is Setup, Data and Status: 3 TRBs on EP0.

The consequence is a stale Normal TRB pointing at a buffer from two laps ago. On bulk-out that can be an old CBW, possibly a WRITE, or old data. The result is a Bulk-Only phase desync or stale data written to the stick, and if the buffer is unmapped, an IOMMU fault.

a_td_that_would_cross_the_rings_link_starts_behind_it cannot expose this. Its bulk-out TDs are 1 or 2 TRBs, so an early Link only ever happens at 254 and leaves no gap. The stub would execute such a stale TRB, because next_trb follows the cycle bit alone.

Fix: when link() is called with tail < RING_TRBS - 1, rewrite slots tail+1..RING_TRBS-1, the end Link included, so that they carry the closing lap's cycle (self.cycle before the toggle). Each early Link then invalidates whatever it skips, and the invariant "every slot holds a cycle other than the next lap's CCS" holds again.

Cheapest check: a host test of Ring::enqueue against a cycle-bit walker that consumes from the last dequeue with its CCS. Run 1- and 3-TRB TDs that park the tail inside a gap left by an early Link, and assert that the walker stops exactly at the tail after every TD. It is red today by the trace above. A stub case with unaligned 128 KiB writes would also show it, as stray events or a phase error.

BLOCKER 4. hc.rs:283-288 (Ring::room) and the assert at :253: the stated capacity is false once early Links exist. room() says that "only the operation's own length matters" and allows a TD of up to 255 TRBs. After an early Link at t, a TD of n > t TRBs overwrites slot t, the Link the controller still has to follow. The controller then stops on a TRB that is not its own, and the ring deadlocks. Every caller today is at most 3 TRBs, so nothing reaches this. But the contract and its assert promise what the ring cannot deliver.

Fix: bound a TD to (RING_TRBS - 1) / 2 TRBs, or any bound for which the early-Link condition t + n > 255 implies n ≤ t, and state that in room().

BLOCKER 5. tests/toyos.rs:4518-4529 (no_fault_but_the_spares) together with amdvi_usbd at :4415-4425: the 03:00.4 exemption is too broad. The no-fault rows now skip every DMA FAULT line on stream=03:00.4. amdvi_usbd only requires that one record names the aimed address and that one is access=none … IO_PAGE_FAULT. It refuses no others. So any further process-owned fault on 03:00.4 passes all three rows. That includes a fault from usbd driving the spare before the staged arm, or a second read past the grant. The r14 readbacks are deterministic: each of the six boots has exactly two 03:00.4 records, addr=0x50400000 access=read … streamfaults=1 and access=none … streamfaults=2.

Fix: have amdvi_usbd refuse any 03:00.4 record other than those two, for example records.len() == 2 with each matched, and reuse the rejudge-r14 control with a third 03:00.4 record added. owner=kernel lines are not hidden, since alive() still reads them as a death.

Verified, no finding

• TD Size (§4.11.2.4) follows the spec's formula: packets minus floor(bytes so far ÷ MPS), capped at 31, and 0 in the last TRB. The hand-counted cases are right.
• First TRB last is correct (§4.11.7.1). The first TRB is written with !cycle, the rest and any trailing Link are written, then publish, then the first control word with cycle.
• The early Link's own bits are correct. It carries the closing lap's cycle and TC, and no Chain bit, which is consistent with §4.11.7.
• The tail never rests at 255, so a 1-TRB TD, every command among them, never links early.
• dequeue() is correct, including after an early Link: tail 0, cycle toggled.
• Bulk MPS 0 is refused at binding.

Folding xhci-leave into testcases: acceptable

The separate boot existed only for the kernel xHCI's xhci-leave actuator. With the place matchers, usbd drives 03:00.3 and leaves 03:00.4 to the claim tests by configuration, so no boot-only property remains. The root rule on amortizing boots favours sharing. The staged faults stay in 03:00.4's own domain, with BME cleared on the first fault, and do not reach the stick's controller. The r14 readbacks show only those two records. With BLOCKER 5 fixed, sharing the boot costs no coverage.

Evidence

• Metal: the AMD and T14 runs at 0e0c37e66 are the orchestrator's report, and I did not re-read them. They establish that the stall is gone on these boots. They do not exercise BLOCKER 3 deterministically: it needs a 3-TRB TD that ends a lap one slot short, followed by a lap that rests in that slot.
• The rejudge: pr2/rejudge-r14/ shows 6 of 6 green at 5461e947c, and the control with an added 03:00.3 fault is red. That control does not cover a hidden 03:00.4 fault (BLOCKER 5).

SEND BACK

[Japabu]

Review r2, at 421c2b3:

• B3, fixed. Ring::link now writes the closing lap's cycle into every slot it skips, the ring's own Link included. hc::ring_tests::the_controller_takes_each_td_and_stops_at_the_tail walks the ring as the controller does, from its dequeue with its CCS and following Links. It runs over 20000 TDs of 1, 2 and 3 TRBs, starting with your lap (255 single TRBs, then 253, then a 3-TRB TD, then 254 single TRBs). After every TD it requires that the walk takes exactly that TD and stops at the tail, ring.dequeue(), which is also where a Set TR Dequeue resumes. Control b3, with the marking removed, is red: the walk takes slot 254 (offset 4064) two laps old.
• B4, fixed. Ring::room() is (RING_TRBS - 1) / 2 and says why. the_longest_td_leaves_the_early_link_before_it_whole puts room()-long TDs after every lead length. Control b4, at the old bound of 255, is red: the walk stops on the overwritten Link.
• B5, fixed. amdvi_usbd collects every 03:00.4 record, of any owner. It requires the aimed read (owner=process, access=read, at the job's address) and the foreign message (owner=process, access=none, IO_PAGE_FAULT), and no other. The six r14 readbacks pass. Control b5, run-1 with a third 03:00.4 record, is red: "03:00.4 has 3 fault records".

Checks at 421c2b3: --ci host 0 (87 green); --build-only x86_64 0 and aarch64 0; the guest suite 67/68, where the red is crypto, the known QEMU console stall. AMD testcases ×6 plus usbload and the T14's three images are staged for the orchestrator.

[Japabu]

---

r3 at 421c2b32a (only 421c2b32a: hc.rs, tests/toyos.rs)

All three r2 blockers are resolved.

• B3 (cycle state after an early Link): resolved. Before it writes an early Link, Ring::link now marks every slot from tail+1 to RING_TRBS-1, including the ring's own end Link, with the closing lap's cycle bit, so none of them matches the next lap's CCS (§4.9.2). On a normal close the range is empty. Overwriting slot 255's control word with a bare cycle bit is safe:

  • it is never the controller's on the next lap, because its cycle differs;
  • before any later lap can reach it, it is either rewritten as a Link (when the tail reaches 255, inside enqueue, before the first TRB is handed over) or marked again by the next early Link.

  the_controller_takes_each_td_and_stops_at_the_tail walks the ring the way the controller does: by cycle bit, following Links and toggling on TC. After every TD it requires that the walk takes exactly that TD and stops at the tail, and that dequeue() names the same place. Its first lengths are r2's trace: 255+253 single-TRB TDs, then one of 3 TRBs, then 254 singles. The tail therefore rests on slot 254 inside the gap, which is where the unmarked ring takes the two-laps-old TRB. After that come 20,000 seeded TDs of 1, 2 or 3 TRBs, with more than 20 early Links required. The test is deterministic and fits the defect.

• B4 (TD length bound): resolved. room() is (RING_TRBS-1)/2 = 127. An early Link happens when t + n > 255, so with n ≤ 127 it means t ≥ 129 > n and the TD after it cannot reach slot t. The doc now states that reason. the_longest_td_leaves_the_early_link_before_it_whole places 127-TRB TDs after every lead-in offset, so a Link the TD overwrote would stop the walk short. Its control is the old bound of 255.

• B5 (03:00.4 fault exemption): resolved. amdvi_usbd collects every DMA FAULT line on stream=03:00.4 with the same match no_fault_but_the_spares exempts, kernel-owned lines included. It requires that record count to be exactly 2:

  • a process-owned access=read at the aimed address;
  • a process-owned access=none … IO_PAGE_FAULT.

  An unstaged spare fault can no longer pass all three rows.

Evidence. I read the agents/usbd/pr2/metal-amd-r15 readbacks. The request names 421c2b32a. run-1 to run-6 each record exactly two DMA faults, both on 03:00.4, and every verdict says passed. run-ul records none. I did not re-read the T14 r15 readbacks, so that result is the orchestrator's report. The host tests' negative controls are the implementer's claim and I did not re-run them. By inspection, the B3 test's opening sequence reaches slot 254 two laps old without the marking, and the B4 test reaches an overwritten Link at a bound of 255.

No new findings. Nothing earlier is reopened.

LAND