[Japabu]

Status: draft, stacked on #935. Head 17e7277dc, built on wt/toyos-aperture-b at f9e44e0fb (main d7f269dbc merged). Until #935 lands this diff carries #935's too. Review round 1's items are answered (round-1 comment). Host CI, both image builds and four QEMU rows are green at the head. Metal at 330a75662: T14 4 passed, AMD 1 passed.

Stage 2 of issues/a-device-function-is-driven-from-its-arrival-to-its-end.md, "Functions arrive and leave". The architecture was ruled on this branch's design round: the kernel owns the slot's interrupt, arrival work runs on the device set's reader, the button means remove, one record per requester, and the _OSC grant rides stage B's declaration. Since #947, "nothing moves a BAR": main's track asked for place_unassigned to become one assignment inside the port's window with no probe, which this round does.

What changes

• The device set (toyos_abi::devset, object kind 16, SYS_DEVICE_SET 139, SYS_DEVICE_SET_READ 140, toyos::devset; 136 to 138 are left to the open memory PR, and the profile has 142 bins): a level-triggered snapshot of every PCI function the kernel hands out. It is a header (generation, settle time) followed by inventory Pci records in rising arrival order. It is minted from a SysCap carrying DEVICE, with READ and WAIT only, so the supervisor alone holds it. devset::decode refuses by name: no header, a wrong count, a non-PCI record, arrival 0, or a duplicate or out-of-order arrival.
• Records per requester (kernel/src/pcidev): up to 512 records, each written once and read lock-free, each carrying its live arrival. A returning device is a new arrival on the same record and reuses its emptied domain. Every call a claim of an older arrival makes answers Gone. A departed function's release asks the device nothing (tear_down_departed), and its retirement drops its BARs from Machine::decoded, its reset entry, parked grants and residue.
• Native PCIe hot-plug (kernel/src/pcidev/slot.rs, toyos_pci::hotplug):
  • A slot is armed from pcidev::declare, once stage B's declaration is held, below a root whose _OSC commit was granted native hot-plug and capability structure control (bits 0 and 4, PCI Firmware 3.3 §4.5.1). Every port says why its slot is or is not armed.
  • acpiserver commits both bits where the root's query offered both (osc::negotiate_root) and declares the grant; q35's default 0x1E mask gets no commit.
  • Native hot-plug fails closed (orchestrator's ruling, holding the owner's "refuse external"; unknown isolation is an error). No firmware signal says a port is internal: ACPI 6.5 defines none, and Microsoft's ExternalFacingPort _DSD property ("Kernel DMA Protection") marks only external ports. So acpiserver commits a root's native hot-plug only where its image passes --hot-plug-ports-internal. Even then it does not commit a root with a port marked external, or one whose _DSD it cannot read. Only the QEMU hot-plug test's image declares it, and a build host test holds that. Without a commit the kernel arms no slot, so no metal slot is armed until claims behind external ports are refused in code (the track's exit).
  • An armed port is the kernel's: the set lists it as kernel-driven, and a claim on it answers KernelDriven.
  • Slots arm from sys_acpi once the declaration has answered, outside the acpi claim's borrow.
  • The handler is bounded. Powering a slot, §6.6.1's 100 ms, and reading the functions below it run on the set's reader, serialised so that two readers never enumerate one arrival twice.
• An arrived function's BARs (replacing place_unassigned and behind_bridges):
  • Before the function is listed, each BAR it leaves unassigned is sized and given the lowest address inside its port's windows that no other function decodes in (hotplug::Windows, hotplug::assign, host-tested), whole 4 KiB pages, a 32-bit BAR below 4 GiB.
  • The address must lie inside the declared root bridges' memory. It is written once and kept where the register reads it back. Nothing reads through it.
  • The BARs enter Machine::decoded, so bars() and pages::of_bar treat them like firmware's.
• IOMMU:
  • VT-d binds a function with no context entry yet.
  • SMMUv3's unclaimable refuses only a function that arrived after the unit was programmed, whose IORT route was never read.
  • A fault from a function the fault table does not know is handed to pcidev.
• toyos-supervisor and the supervisor: as in round 1, and main's drive_where now reads the set, says what joined and left, and decides over the rows asked for.

Deleted: pcidev::FUNCTIONS and arrival_of, VT-d bind's "every function was enumerated" assertion, and the supervisor's PCI reading from the inventory.

The guest row and why it needs QEMU

a_function_arrives_and_leaves runs on Profile::HeadlessHotplugPort: q35, an empty pcie-root-port, and acpi-pci-hotplug-with-bridge-support=off, whose _OSC offers 0x1F.

• It plugs an edu over QMP. The test requires the kernel's line that its BAR 0 was assigned inside the port's window, then a pciholder.
• device_del presses the button: the claim answers Gone on its interrupt read, config read, BAR map and DMA grant; the holder ends with 0; the supervisor says the arrival left.
• A second plug is a new, larger arrival.

No bench has a hot-plug port the owner lets a driver claim behind, so physical arrival is QEMU-only. The slot's, the assignment's and the supervisor's decisions are host-tested.

Checks and evidence

ToyOSOrg/evidence pr-934 at 01fa36f6a for the rows through addf8e97b; that commit was published through the GitHub API after the clone was refused, which the owner is deciding on. Logs from 47dc0138f on are kept locally and not published.

check	revision	result
cargo run -- --ci host	17e7277dc	exit 0, 93 steps green
--build-only, x86-64 and aarch64	17e7277dc	exit 0, 0
a_function_arrives_and_leaves, a_drives_row_holds_one_process_per_function, virt_claim_lpi, the_log_lands_on_the_boot_stick	17e7277dc	exit 0, 4 passed
17 QEMU rows	addf8e97b	exit 0, 17 passed
controls: assignment dropped; no commit; external port committed anyway; hot-plug image without its declaration	c5f97d961, 47dc0138f, 330a75662	each red
metal	330a75662	T14: acpi_tables_loaded, claim_reuses_its_remapping_entry, usbd_serves_the_boot_stick, usbd_drives_the_type_c_controller passed; its root reads hot-plug ports are not declared internal, so native hot-plug is not committed and no port arms. AMD: amdvi_translates_the_stick_controller passed. The QEMU rows above agree since, and the changes after it are ABI numbering, an armed port's bookkeeping, where arming runs, and main's merge, none of which a slot-less metal boot reaches; no re-boot is needed.

Evidence through addf8e97b is ToyOSOrg/evidence pr-934 at 01fa36f6a, published through the GitHub API after the clone was refused, which the owner is deciding on; later logs are kept locally.

Round 1's stand-in-grant.patch evidence is superseded: the row now runs on stage B's real commit.

Risks and open points

• No machine but QEMU's test arms a hot-plug slot until claims behind external ports are refused in code; the staged T14 acpi_tables_loaded boot should show not declared internal for its root.
• acpiserver commits before the kernel knows whether a slot can be armed. On a machine whose units remap no interrupt, firmware gives hot-plug up and nothing serves it (recorded in the track).
• PCI root bridges are acpiserver's declaration: each root's _CRS windows and _OSC offer, held by the kernel; the loader's window reads go (PCI apertures stage B) #935's open decision (a kernel driver's domain before the declaration) also applies here: cargo run -- --arch aarch64 panics on its console.
• A surprise removal is the handler's path with no guest reaching it; 512 records is a hard bound; a departed claim's release relies on the domain and remapping while the card may still be powered (button case).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q

[Japabu]

Review of #934 (lifecycle stage 2): pushed 330a75662, plus local c65ba5dd5

I reviewed the pushed head 330a75662 and the local merge c65ba5dd5, which is not pushed yet. The merge takes #935's 3d6257080 (main d7f269dbc) and renumbers the ABI. Over #935's coming head the diff is 39 files, +2510/−270. About 600 of the added lines are tests:

• toyos_pci::hotplug, about 170;
• toyos-supervisor's set diff, about 80;
• toyos_abi::devset, about 65;
• acpiserver's _OSC and _DSD cases, about 90;
• the QEMU row and its profile, about 120.

The new kernel files are kernel/pci/src/hotplug.rs (pure, host-tested) and kernel/src/pcidev/slot.rs (about 500 lines). The deletions are pcidev::FUNCTIONS, arrival_of, VT-d bind's enumerated-only assertion and the supervisor's reading of the inventory for PCI.

Architecture

The code follows the ruled design in issues/a-device-function-is-driven-from-its-arrival-to-its-end.md:

• The device set is a level-triggered snapshot. It is minted from DEVICE with READ and WAIT only, and devset::decode validates it at the reader's boundary.
• Slot work is the reader's. The port's interrupt is the kernel's and is bounded: it reads and clears Slot Status, departs the slot's functions, makes at most one Slot Control write and posts. Powering, §6.6.1's wait and enumeration run in pcidev::snapshot on the set's reader, and the wait is a settle time in the header, not a spin.
• One record per requester, with its live arrival. Every Binding call checks here() and answers Gone. A departed release asks the device nothing, and retire drops its BARs, reset entry, retired grants and residue.
• place_unassigned is replaced as main's track asked. hotplug::assign finds the lowest page-clean address in the port's windows. place then requires the result to lie inside a declared root window. The BAR is written once and kept only if it reads back. Nothing is probed through it, and no firmware-assigned BAR moves.
• Slots are armed only from pcidev::declare, below a root whose _OSC commit was granted bits 0 and 4. acpiserver commits only where the query offered both. It needs the image's --hot-plug-ports-internal, and even then refuses a root with an ExternalFacingPort port or an unreadable _DSD.
• The fail-closed rule is attributed honestly: the orchestrator's, holding the owner's "Refuse external" (issues/the-iommu-refuses-nothing-yet.md). The T14 readback at 330a75662 shows it working: \_SB_.PC00's hot-plug ports are not declared internal, so native hot-plug is not committed, then each of 00:07.0, 00:07.2 and 00:1c.0 reads not armed: its root bridge's _OSC committed nothing.
• IOMMU. The SMMU's unclaimable refuses only streams missing from Live::enumerated. VT-d builds a context table for a late arrival, and its not-present entry stays blocked until a claim attaches it. AMD-Vi's device table is all-blocked by default (amdvi::build), so an unclaimed arrival reaches nothing on any unit.

Required

1. ABI numbers at the pushed head collide with main.

   • 330a75662 defines SYS_DEVICE_SET = 135 and ObjectKind::DeviceSet = 15. Main (kernel: a process is asked to quit, and one that never listens is killed #995) already has SYS_PROCESS_QUIT = 135 and ObjectKind::Notice = 15.
   • The local c65ba5dd5 moves the set to 139/140 and kind 16. It skips 136-138 for the open memory PR and bumps SYSCALL_PROFILE_BINS to 141. That is coherent against main, and ObjectKind::ALL's word-order assertion holds.
   • Push it. The PR body's "SYS_DEVICE_SET 135, SYS_DEVICE_SET_READ 136" and the handoff's "renumbered past main" are false at the pushed head, and correcting them is the orchestrator's job.

2. kernel/src/pcidev/slot.rs:128 with pcidev/mod.rs:555-570: an armed hot-plug port is reported as claimed by a process.

   • arm_one takes the port's held for the kernel and never records it in Machine::kernel_driven.
   • So inventory(), and with it the device set, lists the port as Driven::Claimed, which the ABI defines as "a process holds a claim on it". No process does.
   • A claim on the port also answers Owned instead of KernelDriven.
   • Fix: record the port as kernel-driven where it is armed. note_kernel_driver's meaning, "a claim on one of them would be two drivers on one device", is exactly this case.
   • Cheapest proof: extend a_function_arrives_and_leaves to read the port's driven in the supervisor's set, or say it in the slot's armed line.

3. Evidence at the head that will land.

   • c65ba5dd5 brings boot: the kernel hands the supervisor its images and parses no filesystem #994's boot path, KernelArgs and supervisor start into this branch, plus PCI root bridges are acpiserver's declaration: each root's _CRS windows and _OSC offer, held by the kernel; the loader's window reads go (PCI apertures stage B) #935's coming merge.
   • Nothing has run there: the 17 QEMU rows are at addf8e97b, and three rows plus --ci host are at 330a75662.
   • Rerun at the pushed head:
     • --ci host, and both --build-only;
     • a_function_arrives_and_leaves, a_drives_row_holds_one_process_per_function, virt_claim_lpi, and one usbd boot-stick row, because the supervisor's start order and its device-set read are on every boot.
   • The T14 and AMD readbacks at 330a75662 (metal934c: T14 4 passed, AMD 1 passed) can stand if those QEMU rows agree.

Suggestions

1. slot.rs:290-305: a slot can stall silently.
   • A powered, occupied slot whose link never reports up answers Next::Nothing with no settle time. Nothing logs it and nothing asks again, unless a Data Link Layer State Changed interrupt arrives. QEMU's port does not keep Link Changed Enable (the track's own measurement).
   • Root CLAUDE.md asks that waits expose stalled progress.
   • Fix: give such a slot a settle deadline and log the stall once it passes, with the bound cited from the specification.
2. pcidev/mod.rs:1465 against :1908: a contradicted doc.
   • The doc says "a function that left the device set is never asked anything".
   • note_fault now clears Bus Master Enable through the config space of a departed function that still faults. That is a sound defence for a button-removed card that is still powered.
   • The doc should say so. Note also that note_fault now clears BME for held boot functions too, which the fault table already does.
3. kernel/src/syscall/device.rs:101 with mod.rs:784: slot::arm runs inside the acpi claim's borrow.
   • It makes MSI-X entry claims, IOMMU message claims and config writes for every port, all inside claim.isa(..). object/device.rs:95-97 states the contract for that closure: "f does not wait".
   • Either confirm that messages::claim and isolates never wait on a unit's invalidation, or arm after the borrow returns, still on the declarer's thread.
4. slot.rs:51: the refresh lock is held across enumeration.
   • REFRESHING is a ticket spinlock held across a whole enumeration: up to 8 functions' BAR sizing, MACHINE and log!. That is bounded, but a second reader spins for it.
   • The supervisor's pci_records mints and reads a second set handle, so two readers in one process can meet.
   • Fine as it is. Worth a line in the lock's doc if it is kept.
5. build/build.rs:3106: a gate for a rule a reviewer can see. only_the_hot_plug_test_declares_its_ports_internal checks something visible in review, which root CLAUDE.md prefers to keep in instructions. I would keep it, because it holds a security ruling cheaply, but that is the orchestrator's call.
6. §6.6.1 readiness. The 100 ms after the link comes up is cited at READY_NANOS. Readiness Notifications (§6.23: DRS and FRS) remove it on ports that support them. That capability is unused and unrecorded; a line in the track's gaps would cover it.

Evidence checked

• metal934c/t14/judge-c.log: 4 passed (acpi_tables_loaded, claim_reuses_its_remapping_entry, usbd_serves_the_boot_stick, usbd_drives_the_type_c_controller). The kernel log's slot lines are quoted above.
• metal934c/amd/judge-d.log: amdvi_translates_the_stick_controller passed.
• I read every Functions arrive and leave: the device set, native PCIe hot-plug, Gone (lifecycle stage 2) #934 hunk over PCI root bridges are acpiserver's declaration: each root's _CRS windows and _OSC offer, held by the kernel; the loader's window reads go (PCI apertures stage B) #935's head and traced:
  • hotplug::next, lowest (it terminates, because at strictly rises, and it refuses wrap) and Windows::read (an unprogrammed zero window decodes as 0..1 MiB and is then refused by place's declared-window check);
  • arrive's record reuse, with retire always run before an Enumerate, because the first Enumerate visit only sets link_since;
  • the race between a departure and a concurrent release: tear_down makes no waits on the device, and the deferred retire runs on the next refresh;
  • the supervisor's per-address budget and the departed-arrival forgetting, both host-tested;
  • negotiate_root's commit path and external_facing's refusals.

SEND BACK on items 1 to 3. #935's open owner decision also applies here, as the PR body says.

[Japabu]

Round 1 at 17e7277dc (#935's f9e44e0fb and main d7f269dbc merged).

1. ABI numbering: object kind 16, SYS_DEVICE_SET 139, SYS_DEVICE_SET_READ 140. The numbers 136 to 138 stay free for the open memory PR (mm: the vm_* lifecycle calls, std and libc on them, and the hot paths the T14 measured (memory stage 3a) #1008).
   • The profile has 142 bins, not 141. At c65ba5dd5, 141 failed the const assertion SYS_DEVICE_SET_READ < SYSCALL_PROFILE_OTHER; nothing had built that commit.
   • That build also found that main's watched_readable (kernel: a process is asked to quit, and one that never listens is killed #995) needed the device set's arm.
   • The body is corrected.
2. An armed port is the kernel's. arm records it in kernel_driven, so the set lists it as kernel-driven and a claim on it answers KernelDriven. Its armed line says so, and the hot-plug row waits for that line.
3. Evidence at the head:
   • --ci host: exit 0, 93 steps green.
   • --build-only on x86-64 and aarch64: exit 0 each.
   • a_function_arrives_and_leaves, a_drives_row_holds_one_process_per_function, virt_claim_lpi and the_log_lands_on_the_boot_stick: 4 passed.
   • The boot-stick row also covers PCI root bridges are acpiserver's declaration: each root's _CRS windows and _OSC offer, held by the kernel; the loader's window reads go (PCI apertures stage B) #935's supervisor fix, which this branch merges: since boot: the kernel hands the supervisor its images and parses no filesystem #994 the acpi holder has to be picked out of the whole service list.

Suggestions:

• Taken:
  • 3: slots now arm in sys_acpi after claim.isa returns, on the declarer's thread. declare itself only locks and posts.
  • 2 and 4: docs corrected.
  • 5: the gate stays, as the orchestrator ruled.
  • 1 and 6 are recorded in the track's gaps: a slot whose link never comes up stalls silently, and Readiness Notifications are not read. Each has an exit.
• Not taken: 1's deadline itself. The bound needs a citation I have not read in the specification, so the gap's exit asks for it.

[Japabu]

Review of #934, round 2, at 17e7277dc

17e7277dc is pushed. It merges #935's f9e44e0fb and merges cleanly with origin/main c2fa26319.

Round 1 items

1. ABI numbers: resolved.
   • Object kind is 16 (main's Notice is 15), and the syscalls are 139 and 140 (main ends at SYS_PROCESS_QUIT 135; 136–138 are left to the open memory PR).
   • SYSCALL_PROFILE_BINS is 142, so SYSCALL_PROFILE_OTHER is 141. 1f8edda28 was needed: at c65ba5dd5 the bins were 141, which made OTHER 140 and failed the const assertion SYS_DEVICE_SET_READ < SYSCALL_PROFILE_OTHER. That head did not compile.
   • ObjectKind::ALL's word-order assertion holds.
   • The watched_readable arm for DeviceSet (17e7277dc) is required by main's exhaustive match from kernel: a process is asked to quit, and one that never listens is killed #995. A device set has no listener notice, so the empty arm is right.
   • The PR body's "135/136" still needs correcting (the orchestrator's job).
2. Armed port reported as claimed: resolved.
   • slot::arm records the port's requester in Machine::kernel_driven under the same MACHINE lock that marks it armed.
   • So inventory(), and with it the device set, now lists the port as Driven::Kernel, and a claim on it answers KernelDriven.
   • The armed line says the port is the kernel's, and a_function_arrives_and_leaves now waits for that line. The driven value itself is established by inspection of inventory(), which is sufficient.
3. Evidence at the head that will land: resolved for host and QEMU. The logs carry no revision line, but their timestamps (15:23–15:35) all fall after 17e7277dc (15:22:06):
   • lr-ci-host.log: EXIT=0, 93 steps green.
   • lr-build-x86.log and lr-build-arm.log: both EXIT=0.
   • lr-guests.log: 4 passed (a_function_arrives_and_leaves, a_drives_row_holds_one_process_per_function, virt_claim_lpi, the_log_lands_on_the_boot_stick).
   • The T14 and AMD readbacks at 330a75662 predate PCI root bridges are acpiserver's declaration: each root's _CRS windows and _OSC offer, held by the kernel; the loader's window reads go (PCI apertures stage B) #935's start-order fix, which changes every boot's order. Re-run them with PCI root bridges are acpiserver's declaration: each root's _CRS windows and _OSC offer, held by the kernel; the loader's window reads go (PCI apertures stage B) #935's.

Suggestions

• Arming outside the claim's borrow (suggestion 3): taken.
  • sys_acpi now calls pcidev::arm_slots() after the borrow returns, only when DECLARE answered Ok, and still on the declarer's thread.
  • Only one declaration can answer Ok, and arm skips ports already armed, so arming cannot run twice.
• The silent link stall and Readiness Notifications (suggestions 1 and 6): recorded as gaps in the track, each with its exit.
• The note_fault doc (suggestion 2): corrected. release's doc now names the one config access a departed function still gets.
• The refresh spinlock (suggestion 4): documented. REFRESHING's doc states its bound.

Verdict

No review blocker remains. #934 is stacked on #935, so it lands after #935 and after the same metal re-run. #935's item 53 applies here too.

LAND on the review's findings, after #935.