Skip to content

A claim hands each BAR over where firmware put it, in 4 KiB pages (Wi-Fi P1) - #947

Merged
Japabu merged 16 commits into
mainfrom
wt/toyos-p1
Oct 20, 2026
Merged

Japabu merged 16 commits into
mainfrom
wt/toyos-p1

Conversation

@Japabu

@Japabu Japabu commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

P1 of issues/there-is-no-wifi.md, with its prerequisite, 4 KiB user leaves for device windows.

Problem

A claim moved every BAR it handed over onto a 2 MiB slot of its own, because 2 MiB pages were the only ones the kernel mapped into a process. That move is why the AMD bench's xHCI 03:00.4 (behind 00:08.1, whose window bus 3 fills) and the AX210 behind its root port could not be claimed, and it was what pcidev's placement machinery existed for.

Resulting behaviour

  • User mappings take any run of 4 KiB-aligned pages (map_range, both arches): a 2 MiB leaf wherever a whole aligned frame lands on a 2 MiB boundary, a table of 4 KiB leaves for the rest. The choice is one pure function, toyos_userbound::leaves, host-tested. Unmapping a slot of 4 KiB leaves answers the table it unlinked (paging::Detached), which the caller drops inside its existing shootdown, since a sibling's paging-structure cache may still name it (SDM 3A §4.10.3); no extra shootdown is issued. x86-64 4 KiB leaves carry their own PAT bit (bit 7). Page-table ownership and the IOMMU domain layout are unchanged.
  • A claimed function's BARs are mapped at their firmware addresses, in the 4 KiB pages they cover. publish records every memory BAR; a claim hands one over only where no other function decodes inside its pages and they lie in the declared root-bridge memory (toyos_pci::pages, host-tested). Otherwise the BAR is withheld by name (BarPages, BarUnrouted) and the function still handed over. The MSI-X table's page goes with its BAR; the kernel's remapping-entry rule (AMD-Vi stage 6: interrupt remapping, and claims only where every unit filters SMIs #921) is untouched, and tear-down masks the entry at the table address arming read.
  • Bring-up chooses BARs before changing anything, then arms the interrupt last, so no refusal follows an armed message and the undo path is gone.
  • Measured page sharing: every boot logs pcidev: N memory BAR(s), M of them sharing a 4096-byte page with another function or starting inside one. q35: 11 to 14 BARs by machine shape, 0 sharing (boots). Metal at 5f4418f: T14 21 BARs, AMD 14, 0 sharing (T14, AMD).
  • msix-exclusive-bar=on is gone from build/qemu.rs and tests/common/qemu.rs: the NVMe guests drive QEMU's default controller through its table in BAR 0 (nvme_disk_boots and the diskserver boots green).

Deleted (measured)

48 files, +1074 −2134 against the merge base with main; kernel/lib/sdk/build +591 −1836, tests +349 −108, issues +134 −190. Gone: toyos_pci::placement (545 lines: free runs, reservations, behind_bridges), toyos_pci::probe (the "answers its own value" proof), toyos_pci::bridge (the bridge walk, 291), bar::placement, PciDevice::bridge/forwarded, and in pcidev place_bar(s), Source/Candidate, give_back, cut_already/was_cut/cut, alone_in_its_page, probe_dword, the 2 MiB runs and slot records (pcidev −620 +164), and the refusals only they raised; the x86 AddressSpace::pages map nothing wrote (#940 deletes it too, merged with main c0b1547).

Decisions

  • Unassigned BARs: none is assigned. Main had no path that did (place_bar refused BarUnassigned), and the hot-plugged endpoint that needs one is lifecycle stage 2's (wt/toyos-lifecycle-2), whose place_unassigned will need rewriting onto this: a small assignment inside the port's window, no probe.
  • Stage 3 of the process-memory track beyond device windows is not taken: device leaves own no frame, so no OWNED ledger or per-process count is needed, and a BAR's pages are in physical order, so a typed value straddling two of them is still one run (object_run's one-segment store and window_split's bound hold). What remains of stage 3 — the ledger, demand-zero stacks, 4 KiB TLS, straddle segments, the T14 floor — is restated in the issue.

What temporarily stops working

Nothing measured.

Bench round 1 (5f4418f, run by the orchestrator)

T14 3/3 PASS. AMD 2/3: amdvi_translates_the_stick_controller and msix_table_read_in_place PASS, and the first claim behind a bridge on metal handed 03:00.4's BAR 0 over in place (0xd0100000..0xd0200000) and usbd drove it. amdvi_usbd_drives_its_own_entry was red in end_returns_claims, which chose 1022:1639 by id on a bench that has two; it now chooses the spare as usbd_spare does and claims it by arrival (dc18514). Readbacks, judges and manifests: bench.

Review round 1

Fixed the two comment blockers (AArch64 Tables' ownership, survey's doc and unrouted line), took the shootdown suggestion above, dropped bar_map_behind_a_bridge's by-construction address comparison (the job's register read through the BAR behind the port is the claim), recorded P1's T14 NVMe exit as an Identify Controller completing through the in-place entry, recorded that P1's rows need no metal-row naming first, added the place_unassigned rewrite and Machine::decoded to lifecycle stage 2, and the nits. Declined: keeping Pages in Bound (one rounding helper, no defect), and refusing a claim whose MSI-X table BAR is unrouted (pre-existing; recorded nowhere new). All four bench images re-staged, since the unmap change is in every kernel.

Tests and evidence (ToyOSOrg/evidence pr-947, README)

  • Host: cargo run -- --ci host, exit 0, 86 steps, at ecab334 (log); earlier at 3ef1e64 (log). New host tests: toyos_pci::pages (in place, sub-page BAR, page sharing refused, off-page, wrap) and toyos_userbound::leaves (device windows on and off a 2 MiB boundary, whole frames, misaligned runs); segment test notes the straddle-in-one-run case.
  • Builds: --build-only x86-64 and aarch64, exit 0 at ecab334 (x86, a64) and at 3ef1e64 (x86, a64).
  • QEMU at ecab334: 11/11 claim rows exit 0 (log), the nine below plus usbd_refused_without_smi_filter and usbd_refused_without_amdvi_remapping. At 3ef1e64, 9/9 exit 0 (log): usbd_drives_the_spare, bar_map_again (now also runs msix_in_place: the NIC's armed MSI-X entry read back through its holder's mapping of BAR 1's table page equals what the kernel logged writing), bar_map_behind_a_bridge (rewritten: was "BAR placed inside the port's window"; now the NIC behind the port is handed a BAR in place and the job reads its register through it), lent_device_memory_refused, a_drives_row_holds_one_process_per_function, the_compositor_drives_the_virtio_gpu, iommu_virtio_platform, virt_claim_lpi, nvme_disk_boots. No row deleted. An earlier red run (log) found bar_map_again reading the NIC's first BAR, which is now BAR 1 (the table, withheld before because it read zero at the probe dword); the job names BAR 4.
  • Negative controls (log, r2): the page-sharing check narrowed to the BAR's own bytes reds a_page_another_function_decodes_in_is_refused_by_name; a short 2 MiB-aligned window mapped as a 2 MiB leaf reds two leaf tests; both restored green. The "ledger" control the brief named has no subject: no ledger was built (above).
  • Metal, re-staged at ecab334 (T14, AMD); first staging (T14, AMD): T14 usbd_drives_the_type_c_controller, claim_reuses_its_remapping_entry, new msix_table_read_in_place (AX210 and NVMe entries at BAR 0 + 0x2000 read through their holders' mappings); AMD amdvi_usbd_drives_its_own_entry (restored with its judge from c648c14), amdvi_translates_the_stick_controller, msix_table_read_in_place (the AX200 behind 00:02.2).

Round 5 (head 4c72cf3, main 0e6716f merged)

PciDevice::disable_msix and Msix::disabled deleted: their last callers were the refused-hand-over undo this PR removes and the HDA driver #950 moved to userland. Build-only both arches exit 0, the 11 QEMU claim rows exit 0, the AMD testcases-amd-xhci-leave image re-staged (round 5). --ci host exit 1 on one step unrelated to this branch: build/llvm.rs's environment test runs git under a fake SDKROOT, which on this host's Xcode now fails (log); the branch touches neither build/llvm.rs nor build/sysroot.rs, and the step was green at round 4.

Round 6 (head 33adc6b, main ffd8bb9 with #953 merged)

--ci host exit 0 (86 green); build-only both arches exit 0; 12 QEMU rows exit 0, #953's hda_plays_to_the_controller_that_reaches_an_output among them (round 6). All four bench images staged at this head; the AMD testcases boot adds hda_tone, hda_tone_resampled and hda_volume_step, whose controllers 03:00.1 and 03:00.6 sit behind the bus-3 bridge that P1 makes reachable. The round-5 macOS llvm red is #952.

Bench rounds 6 and 7 (33adc6b, e004248)

T14 3/3 PASS (readback), the first T14 boots of the leaf-table change. AMD: hda_tone, hda_tone_resampled, hda_volume_step PASS (the first AMD audio, through controllers behind the bus-3 bridge that P1 hands over in place), msix_table_read_in_place PASS, and amdvi_usbd_drives_its_own_entry PASS after e004248: the vector arm had expected usbd to settle, but the unit's refusal of the rewritten message puts the claim in refusal by design, so usbd ends on it; the arm now asserts that and the judge requires usbd's own the claim refused its interrupt record (Io) line, against which the round-7 readback was judged again: PASS (re-judge) (round 6, round 7).

amdvi_translates_the_stick_controller stays red on the AMD bench, for a recorded reason independent of this PR: 03:00.1 (the GPU's HDMI audio, reachable since P1) drops the high bits of its domain's addresses although it advertises 64OK, so its accesses fault and the row refuses any fault. Recorded as issues/a-function-that-truncates-its-dma-addresses-faults-in-its-domain.md; the domain-layout choice is with the owner.

Review round 2 (77ed37a)

Detached's doc says what holds (dropped only after a user-half shootdown); AArch64's must_use text names no caller. The vector arm asserts the refusal (above); the spare selection is one file both jobs include; until_ended takes &self. Build-only both arches, four QEMU rows and --ci host (86 green) exit 0 (round 8). Recorded: a metal job's failure names the T14 on every bench.

Issues

Filed: a-metal-jobs-failure-names-the-t14-on-every-bench (finding), a-function-that-truncates-its-dma-addresses-faults-in-its-domain (owner question). Closed: a-bar-behind-a-bridge-with-no-free-2-mib-of-its-windows-is-not-handed-over, no-boot-reaches-the-retry-that-undoes-a-placement-nothing-answered. Rewritten: P1 in there-is-no-wifi (built; bench exits owed; the T14 NVMe driven through its table stays owed, the disk being another system's), nothing-reds-when-the-bar-survey-drops-an-assigned-bar, no-boot-reaches-the-refusal-that-keeps-a-bar-read-inside-a-declared-window, process-memory-is-2-mib-pages... (device windows done, stage 3 remains), and citations in four others.

Risks

  • First 4 KiB user leaves on real hardware: PAT bit 7 on x86 (UC devices use PCD|PWT, so bit 7 is unused for them), and freeing a leaf table after its shootdown. Only the metal runs read either.
  • The AMD row is the first claim behind a bridge on metal.

🤖 Generated with Claude Code

Japabu and others added 7 commits October 15, 2026 07:42
…-Fi P1)

A claim used to move every BAR it handed over onto a 2 MiB slot of its own,
because 2 MiB pages were the only ones this kernel mapped into a process.
That move is why the AMD bench's xHCI 03:00.4 (behind 00:08.1, whose window
bus 3 fills) and the AX210 behind its root port could not be claimed, and
it is what pcidev's placement machinery existed for.

User mappings now take any run of 4 KiB-aligned pages: a 2 MiB leaf wherever
a whole aligned frame lands on a 2 MiB boundary, a table of 4 KiB leaves for
the rest (toyos_userbound::leaves, host-tested, shared by both
architectures). Unmapping a slot of 4 KiB leaves frees its table: on x86-64
after a user-half shootdown, since a sibling's paging-structure cache may
still name it (SDM Vol. 3A 4.10.3); on AArch64 after the broadcast TLBI by
ASID that already completes on every CPU. x86-64's 4 KiB leaves carry their
own PAT encoding (bit 7), and the AddressSpace's never-written `pages` map
is gone.

pcidev records every memory BAR at publish and hands a BAR over at that
address, in the pages it covers, only where no other function decodes inside
them and the pages lie in the root bridges' declared memory
(toyos_pci::pages, host-tested); otherwise the BAR is withheld by name and
the function still handed over. The register must still hold firmware's
address (BarMoved). Every BAR is chosen before anything on the function
changes, and the interrupt is armed last, so no refusal follows an armed
message and bring_up's undo path is gone. publish logs how many memory BARs
share a 4 KiB page with another function: the bench measurement.

Deleted: toyos_pci::placement (free runs, reservation, behind_bridges),
toyos_pci::probe (the "answers its own value" reference dword),
toyos_pci::bridge (the bridge walk), bar::placement, PciDevice::bridge and
::forwarded, and in pcidev place_bar(s), Source, Candidate, give_back,
cut_already, was_cut, cut, alone_in_its_page, probe_dword, the 2 MiB slot
runs and their records, and the refusals only they raised. MSI-X arming
answers its table's address, which is where tear_down masks it.

No BAR firmware left unassigned is assigned: main had no path that did
(place_bar refused BarUnassigned), and the hot-plugged endpoint that needs
one arrives with lifecycle stage 2, which brings its own assignment inside
the port's window.

Tests: bar_map_behind_a_bridge asserts the NIC's BAR is handed over at the
address enumeration read; msix_in_place reads the MSI-X entry the kernel
armed back through the holder's mapping of the table's page, in
bar_map_again's boot and as the msix_table_read_in_place metal row (the
T14's AX210 and NVMe, the AMD bench's AX200); amdvi_usbd_drives_its_own_entry
and its judge return from c648c14. The NVMe guests drop
msix-exclusive-bar=on and drive QEMU's default controller through its table
in BAR 0.

Issues: closes a-bar-behind-a-bridge-with-no-free-2-mib-... and
no-boot-reaches-the-retry-that-undoes-a-placement-nothing-answered; restates
P1 of there-is-no-wifi, the survey and refusal-reach issues and the
process-memory track, whose ledger, counts and straddle segments device
windows did not need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UzbDnM9m6nQWhbjGi6Lv1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UzbDnM9m6nQWhbjGi6Lv1
…on a 2 MiB boundary in the leaf test

build/kernelkeys.rs declared the x86-64 AddressSpace's `pages` map, which the
P1 commit deleted. The leaf test claimed the AX210's 0xbc300000 is a 2 MiB
boundary; it is 1 MiB off one, so the T14 NVMe's 16 KiB at 0xbce00000 is the
case a whole 2 MiB leaf would hand its neighbours over in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UzbDnM9m6nQWhbjGi6Lv1
…he common-config BAR

tests/common/qemu.rs still gave the guest NVMe msix-exclusive-bar=on, so the
guests drive its table in BAR 0 now as build/qemu.rs does. bar_map_again took
the virtio NIC's first nonzero BAR, which was BAR 4 only while the deleted
probe withheld BAR 1 (the MSI-X table reads zero at the probe's dword); it
names BAR 4 now. bar_map_behind_a_bridge judges every BAR handed over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UzbDnM9m6nQWhbjGi6Lv1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the AMD bench both 1022:1639 controllers are on the machine, so the
by-id lookup refused two matches (amdvi_usbd_drives_its_own_entry, 5f4418f).
The job now takes the one spare function the kernel does not drive, as
usbd_spare does, and claims it by arrival every round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #947 at 5f4418f37 (Wi-Fi P1: BARs in place, in 4 KiB pages)

Architectural assessment

This is the right simplification. A BAR stays where firmware put it, and that removes the job the whole placement subsystem existed for: free runs, reservations, the bridge walk, the "answers its own value" probe, give_back, the slot records, and the undo path in bring_up. Ownership is clear:

  • toyos_userbound::leaves is one pure decision that both architectures' map_range use.
  • toyos_pci::pages::of_bar is the pure page/sharing rule.
  • pcidev keeps only what the kernel must do: record firmware's BARs, refuse a claim without a declaration, map, and arm MSI-X.

No new ABI. No toyos-abi misuse (sdk/abi/src/pci.rs loses one stale clause). Leaving process-memory stage 3 untaken is justified: device leaves own no frames, and a BAR's pages are in physical order. The issue now says so accurately.

Line counts, measured with git diff --shortstat origin/main...HEAD:

  • kernel/lib/sdk/build: +538 −1784
  • tests: +306 −60
  • issues: +70 −188
  • total: +914 −2032, which matches the PR body.

Checked and correct

1. 4 KiB user mappings

  • Leaf choice. leaves (lib/userbound/src/leaves.rs:18-34) asserts that va is 2 MiB-aligned. A slot becomes a Block only when pa + offset is 2 MiB-aligned and the slot is a whole 2 MiB. So:
    • A run starting off a 2 MiB boundary is pages in every slot.
    • A run ending mid-frame has a pages tail.
    • A short run that is 2 MiB-aligned is pages. This is the negative-control case, and the guest also hits it: 01:00.0's BAR 1 is 4 KiB at 0x80000000 in guests3.log.
    • BARs of 2 MiB or more are size-aligned powers of two, so they are always Block.
  • Crossing 1 GiB. On AArch64 each slot calls tables.directory(va), and on x86 each slot calls ensure_table(va), so a run crossing a 1 GiB boundary builds its own directory.
  • Table ownership. Unchanged: both children Vecs.
  • Unmap ordering, x86 (kernel/src/arch/x86_64/paging.rs:589-600):
    • PDE cleared and discharged locally.
    • Futex revocation.
    • shootdown(User). This returns only once every CPU has flushed, using INVPCID type 3 or a CR3 reload, both of which invalidate paging-structure caches (SDM 3A §4.10.4.1).
    • Only then children.retain frees the table.
  • Unmap ordering, AArch64 (kernel/src/arch/aarch64/paging.rs:476-489):
    • set(i, 0).
    • tlbi! brackets the invalidation as dsb ishst; tlbi aside1is; dsb ish; isb.
    • Only then the table is freed.
  • Shootdown under the address-space lock. Safe: Lock's spin calls tlb::poll().
  • Cache attributes.
    • x86 UC is PCD|PWT at both levels (PAT index 3). WC moves its PAT bit from 12 to 7 at 4 KiB (pte_bits).
    • AArch64 typed() is the same at L2 and L3. User and kernel device mappings are both Device, and both UC on x86.
    • policy_at now decodes 4 KiB leaves instead of asserting Normal.
  • Kernel copies through a user pointer into a BAR. Still refused: FramePins::pin(.., Pmm) on x86, and through_direct_map on AArch64.
  • Callers of alloc_and_map. RAM callers all pass multiples of 2 MiB (PageAlloc, PIPE_SIZE, align_2m in shm/vm), so RAM mappings are unchanged.

2. BARs in place

  • Which BARs are recorded. publish records BARs only where memory_bar decodes. Unassigned (address 0) and I/O BARs are excluded. Prefetchable and 64-bit BARs are included, stepped as two slots.
  • Zero size. advertised_size never answers Ok(0), so of_bar's bytes > 0 assert cannot be reached by a device.
  • Sharing check. It covers every recorded memory BAR of every other requester: bridges' own BARs (bar_slots = 2) and kernel-driven functions included. Bridge windows have 1 MiB granularity, so a page never straddles two of them.
  • Root-window check. aperture::decode runs on the rounded pages.
  • Declaration state. Unknown and Refused refuse the claim before held is taken (claim, pcidev/mod.rs:736-748).
  • A withheld BAR. It answers InvalidArgument from bar_object, and bar_bytes[i] = 0 in the description. Each consumer then refuses by name: diskserver's bar_bytes <= DOORBELLS, and usbd's map_bar(0, 0). The function is still isolated in its domain, so handing it over without that BAR is safe.
  • Order in bring_up.
    • function_space is reused per function, as before.
    • bars() only reads.
    • enable_memory_space, then claim_msi, then arm.
    • Every refusal after claim_msi drops message before any entry is written.
    • No state survives that the old code did not also leave (memory decode on, the per-function domain).
  • Tear-down. It masks at the table address that arming read.

3. Deletions.

  • No code residue: grep over kernel, sdk, lib, build, tests and system for place_bar, alone_in_its_page, placement::, behind_bridges, probe_dword, MsixAt, msix-exclusive-bar, was_cut and cut_already finds nothing. The give_back hits are unrelated.
  • LEND_WINDOW stays. It is the IOVA lend window for DMA grants, not a BAR slot, and its arithmetic (pcidev/mod.rs:299-301) is untouched.
  • Nothing references the deleted issue paths.

4. Tests and evidence

  • The nine QEMU rows. Run at 3ef1e64c9. git diff --stat 3ef1e64c9 5f4418f37 equals git diff --stat 3b77b9b2f 979d4c539: three files, tcp conn.rs/rx.rs and one deleted issue. Nothing the claim rows judge.
  • Restored AMD row. amdvi_usbd and its row match c648c1439.
  • msix_in_place. It establishes something independent: the device's own table answers, through the holder's mapping, the address and data the kernel logged writing. In guests3.log, 00:03.0 BAR 1 + 0 reads 0xfee00098.
  • Negative controls. Both are real and named: the patch, red (exit 101 on the named test), reverted, green.
  • Host log. ci-host3.log ends Host: 86 step(s), all green, exit 0. The evidence checkout path is masked at the tip ($WORKTREE, $HOME).

Findings

BLOCKER

  • kernel/src/arch/aarch64/paging.rs:170: the comment "A root and every table under it, owned together and freed together" is now false. unmap (line 489) frees one child table on its own. Fix: state the actual contract, e.g. "freed with the space, or by unmap once its ASID is flushed", or drop the second half.
  • kernel/src/pcidev/mod.rs:585-586: the survey doc says it "says the answer once and counts nothing from it", but the function now computes and logs withheld. Related: line 605's Unrouted message still says "no declaration to read it by". pcidev no longer reads BARs; the concern now is handing them over. Fix: delete or correct both phrases.

Both are comment-only. They can ride the #940 merge the branch needs anyway, and need no integration rerun.

SUGGESTION

  • Second shootdown in x86 unmap (kernel/src/arch/x86_64/paging.rs:599): unmap now issues its own user-half shootdown for every slot of 4 KiB leaves. It does this under the address-space lock, and the caller's Unmapped drop then issues another. Unmapped already models "cleared but reachable until shot down". Handing the freed table to that value would keep one mechanism and one shootdown. Correct as written; this only removes duplication and cost.
  • bar_map_behind_a_bridge comparison (tests/toyos.rs:4649-4686): comparing the handed-over address with the bars=[ enumeration line is true by construction. bar_at is only ever assigned bar.base, and bars() already refuses BarMoved, so the two numbers are one register read twice. The test's real value is the job reading device_feature through BAR 4 behind the port. That part is exercised and is not trivial. Consider dropping the parse.
  • Discarded of_bar result (kernel/src/pcidev/mod.rs:890 and lines 778 and 1297): bars() throws away of_bar's Pages, then rounds bar_bytes again with pages::PAGE in the claim log and in bar_object. Keeping the Pages in Bound would leave one rounding and one grain.
  • Unrouted MSI-X table (kernel/src/pcidev/mod.rs:833-868): bring_up now knows each BAR's verdict before arming. If the MSI-X table's BAR is withheld BarUnrouted, the kernel's own arming write still goes to that address. This was already true before this PR, but refusing here is now one line.
  • P1's NVMe exit has no path (issues/there-is-no-wifi.md): "the T14's NVMe driven through that table, which no row does" names no way to meet it. An Identify Controller admin command, with its completion delivered through the in-place MSI-X entry, drives the controller through its table without touching the other system's data.
  • Records the deletion leaves stale: these are for the orchestrator, not a branch cleanup.
    • Lifecycle stage 2 (issues/a-device-function-is-driven-from-its-arrival-to-its-end.md:100, "BARs are placed inside its port's window") loses the mechanism it would have built on. Two facts are recorded only in the PR body: place_unassigned on wt/toyos-lifecycle-2 needs rewriting, and a hot-plugged function must also enter Machine::decoded for the sharing check to see it.
    • issues/an-aarch64-cargo-run-boot-hands-over-no-pci-claim.md:34 still says "The windows placement offers".
  • Metal-rows rule: the Wi-Fi track's "Metal rows" rule makes every stage with a T14 or AMD row depend first on a-metal-row-names-no-hardware-and-a-bench-without-it-reds.md, which is open. P1 adds msix_table_read_in_place and restores amdvi_usbd_drives_its_own_entry. Record whether that dependency was waived, or that the rows run filtered per bench.
  • Nits:
    • kernel/src/arch/x86_64/paging.rs:19: a const sits between use lines.
    • from_bits(pde, pat) names its argument pde but also takes PTEs.
    • tests/common/iommu.rs:56 states which mutation reds. That is inferred, not in the evidence.

PR description (orchestrator)

  • "q35 (guests): 12 BARs, 0 sharing" is presented as measured, but no published log contains the memory BAR(s) line. guests3.log prints only judged lines. Link the line or label the number.
  • The quoted log text leaves out "or starting inside one".
  • Everything else checked true: the counts, the deletion list, the evidence links and revisions, the scope of the issue edits, and the risks.

Verdict

SEND BACK, for the two comment blockers alone. They are trivial and need no rerun. Once fixed, LAND is pending the six bench rows the orchestrator is judging. Those rows are the only reading of 4 KiB user leaves and table freeing on real hardware, and of the first claim behind a bridge on metal.

Japabu and others added 4 commits October 15, 2026 12:50
… comments and records

x86 unmap issued its own user-half shootdown per slot of 4 KiB leaves, under
the address-space lock, before the caller's Unmapped issued another. unmap
now answers the table it unlinked as paging::Detached, which every
free_and_unmap caller drops inside its existing shootdown (Unmapped, or the
pipe and dlopen paths' explicit one). AArch64 answers the same type; its
tlbi already reached every CPU.

Comments: Tables' ownership (freed with the space or by an unmap), survey
counts what it withholds, its unrouted line says no claim is handed it, a
const out of the use block, from_bits' argument. bar_map_behind_a_bridge
drops a comparison bars() makes true by construction; the job's register read
through the BAR behind the port is the claim. iommu.rs no longer names an
unrun mutation.

Issues: P1's T14 NVMe exit is an Identify Controller completing through the
in-place entry; P1's rows need no metal-row naming first; lifecycle stage 2
records the place_unassigned rewrite and Machine::decoded; the aarch64 claim
issue says what windows are checked, not placed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The refused hand-over that called it is gone (P1's bring-up arms last), and
#950 moved the HDA driver that was its other caller to userland.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu and others added 3 commits October 20, 2026 21:03
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the AMD bench (33adc6b) the holder's rewritten MSI-X entry was refused
and recorded by the unit (access=none IO_PAGE_FAULT against 03:00.4), which
puts the claim in refusal as iommu/fault.rs states, so usbd's next interrupt
read answered Io and usbd ended by name. The arm expected usbd to settle, an
outcome never measured: 03:00.4 was unclaimable before P1. It now reads
every line usbd says until it ends, asserts no No-Op was announced and that
usbd failed; the judge reads that line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The AMD bench's 03:00.1 HDMI audio, reachable since P1, drops the high bits
of its domain's addresses although it advertises 64OK. The domain layout
choice is the owner's; this records the evidence and why
amdvi_translates_the_stick_controller stays red there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #947 at c6e5de7ec (follow-up to the review at 5f4418f37)

Scope: P1's commits after 5f4418f37, which are ecab33411, dc185149c, 4c72cf3f3, e00424828 and c6e5de7ec. Also the resolutions in the merges of main (e136dafeb, 824e0478a, f9b9d38fb, 33adc6b47), read with git show --remerge-diff, and the bench evidence at ToyOSOrg/evidence 47584ef78 (rounds 6 and 7). Settled points are not reopened.

Architectural assessment

paging::Detached is the simplification the earlier suggestion asked for.

  • x86 unmap no longer issues its own shootdown under the address-space lock.
  • The unlinked leaf table becomes a value. It is dropped by each caller's existing shootdown: Unmapped's drop, or the explicit one in the pipe and dlopen paths.
  • unmap became private, so this is the only way to reach it.
  • AArch64 returns the same type, so generic code sees one contract.

The vector arm's new expectation is the right contract. iommu/fault.rs states one policy: a fault recorded against a process-driven function clears its BME and puts the claim in refusal. A message the unit refuses is a recorded fault, and here the holder rewrote its own entry, so the refusal lands on the holder. Exempting interrupt faults would add a second classification to every backend. The old expectation ("usbd settles") contradicted the kernel's own stated contract and was never measured.

disable_msix/Msix::disabled deletion is correct. No caller remains (grep over kernel/), and its host test went with it.

The merges resolve cleanly.

Net change against the merge base: 46 files, +1038 −2120.

Area Added Removed
kernel/lib/sdk/build +592 −1836
tests +331 −94
issues +115 −190

The PR body's "40 files, +914 −2032" is stale.

Previous blockers

Both are resolved.

  • kernel/src/arch/aarch64/paging.rs:170-171 now says tables are freed with the space or by an unmap (Detached). That is true.
  • kernel/src/pcidev/mod.rs survey: the doc now says it counts what it withholds. The unrouted line now reads "no claim is handed it". Both are accurate.

Checked and correct

Detached ownership, x86. Every free_and_unmap caller drops it only after a user-half shootdown that runs with no address-space lock held:

  • MappedPages::unmap_from and release: the guard is a temporary of the let.
  • release_thread_mappings → Vec<Unmapped<…>>.
  • SharedMemObject::unmap_from and zero_handles.
  • sys_mmap replace, inside Unmapped, dropped after the closure.
  • sys_munmap.
  • revoke_pipe_maps: shootdown, then drop(detached).
  • The dlopen rollback: shootdown, then drop.

Unmapped::drop shoots down before ManuallyDrop::drop, so the table outlives the flush (SDM 3A §4.10.4.1 for paging-structure caches). swap_remove with expect replaces retain, so a table this space doesn't own now fails loudly.

end_returns_claims.

  • The job now selects by inventory and claims by Arrival every round. This fixes the two-match refusal on the AMD bench.
  • Round 7's amdvi_usbd_drives_its_own_entry PASS reads it (bench/round7/amd/judge-p17a.log).

The issue record a-function-that-truncates-its-dma-addresses-faults-in-its-domain.md.

  • Frontmatter is question/owner, which issues/README.md allows.
  • The cited evidence (b0148546c, an ancestor of 47584ef78) is byte-identical to round 6's testcases/kernel.log.
  • Lines 337–345 are what it says. The GCAP is 0x4001. Domain 4 maps 0x13200000.. at 0x400010000000. The faults are at 0x10000000, 0x10000400 and 0x10000410. Domain 5 (03:00.6) maps at the same 0x400010000000 and works.
  • system/hda/driver/src/controller.rs:373-378 writes both halves and refuses a high grant only without 64OK, as the record says.
  • kernel/src/iommu/window.rs:11,29 is the quarter-width rule.
  • "Bits dropped" is labelled as what the addresses show, not as a measured width. That is fine.

Independence of the red amdvi_translates_the_stick_controller. Established.

  • P1 touches nothing in kernel/src/iommu/ or system/hda/.
  • The faulting addresses are the domain's IOVA with bit 46 gone.
  • A second function at the same IOVA base in the same boot works.

P1 only makes 03:00.1 claimable. Landing it turns that AMD nightly row red until the owner chooses a layout. The PR body says so, and the decision belongs to the owner, as the record states.

Findings

BLOCKER

  1. kernel/src/arch/x86_64/paging.rs:859-861 (and the AArch64 must_use text at aarch64/paging.rs:612). The comment is false. It says the tables are "dropped only inside mm::Unmapped, after its shootdown". revoke_pipe_maps (process.rs:653-666) and the dlopen rollback (syscall/vm.rs:316-320) drop Detached outside any Unmapped, after their own explicit shootdown. That is correct, but this comment states the safety contract, and a reader checking those paths against it will think they are wrong. Fix: state the actual rule, e.g. "dropped only after a user-half shootdown". Do the same for the AArch64 must_use message, or delete that message, since on AArch64 dropping anywhere is safe. Comment-only; no rerun needed.

  2. tests/guest/rust-tests/src/usbd_spare.rs:296-297 with tests/toyos.rs:4345. The arm tolerates the refusal rather than asserting it.

    • What the job asserts: usbd ended non-zero within 30 s and printed no No-Op line.
    • What it prints regardless: "usbd ended on the claim's refusal". The judge then accepts that print as the evidence.
    • What slips through: a usbd that panics for any other reason after the unit's record. That covers a claim refused with the wrong error, or any of usbd's other eight panic/expect sites. Such a run passes while printing a false cause.
    • What the evidence already holds: usbd's own refusal line, on its stderr, so in the boot log and not in the job's pipe. It is at bench/round7/amd/testcases-amd-xhci-leave/kernel.log:1625: usbd: the claim refused its interrupt record (Io); nothing drives it from here. No other usbd in this boot can say it: the drive arm's usbd is killed, and the fault arm runs none.
    • Cheapest fix: in amdvi_usbd, add log.must_say("usbd: the claim refused its interrupt record (Io)"). Reword the job's print to what the job itself established ("usbd ended {status:?} and was never told of its No-Op"). The published round-7 readback already contains the line. Re-judging that readback, or a direct reading of the published log, is sufficient; no new boot is needed for a judge-only tightening.

SUGGESTION

  • tests/guest/rust-tests/src/bin/end_returns_claims.rs:55-71. spare() is a verbatim copy of usbd_spare::claim's selection, and its doc says it must choose "as usbd_spare chooses it". SPARES is copied too. Its comment (line 23-24, already stale on main) still lists two controllers for three entries. Including usbd_spare.rs by #[path], as usbd_spare_vector.rs does, would leave one rule and one list:
    • split out a spare(cap) -> (PciId, Arrival) that claim calls;
    • delete the copy and the stale comment.
  • usbd_spare.rs:140: until_ended takes &mut self but needs only &self.
  • Unrelated discovery, for the orchestrator to record rather than fix here: tests/common/metal.rs:494 hardcodes "exited {code} on the T14". Round 6's AMD judge therefore reported the AMD failure as "on the T14" (bench/round6/amd/judge-p16a.log:483).

PR description and evidence (orchestrator)

  • Required. "Bench rounds 6 and 7 (33adc6b, e004248): T14 3/3 PASS" has no published readback. 47584ef78 holds T14 bench results only under p1/bench/t14, at 5f4418f37. round6/stage-t14.log is staging only (exit 2, untouched), and bench/round6/ contains only amd/.
    • Why it matters: ecab33411 changed how every x86 user unmap frees its leaf tables, which is the PR's named metal risk.
    • Why it is not a coverage gap: AMD rounds 6 and 7 do exercise that code on x86 metal.
    • Fix: publish the T14 round-6 readback and judge, or relabel T14 3/3 as measured at 5f4418f37, before ecab33411, with AMD as the post-change x86 metal reading.
  • "40 files, +914 −2032 overall; kernel/lib/sdk/build +538 −1784" is stale. Current figures are in the table above.

Verdict

SEND BACK for two small blockers:

  • the false Detached safety comment;
  • the vector arm's judge, which must read usbd's own refusal line rather than the job's assertion of it.

Neither needs a new metal boot; the existing round-7 readback carries the line. Separately, the orchestrator should correct the T14 rounds-6/7 claim. Everything else in this round checks out. That includes the independence of the red amdvi_translates_the_stick_controller row, which the new issue records accurately.

…fusal

The x86 Detached doc said it is dropped only inside mm::Unmapped; the pipe
and dlopen paths drop it after their own shootdown, so it says a user-half
shootdown. AArch64's must_use names no caller it does not have.

amdvi_usbd now requires usbd's own "the claim refused its interrupt record
(Io)" line, and the job prints what it established: usbd exited and never
said its No-Op completed. The round-7 AMD readback re-judged against it:
PASS. The spare selection is one file (src/spare.rs) both jobs include, and
until_ended takes &self.

Records that a metal job's failure names the T14 on every bench.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Review of c6e5de7 addressed at 77ed37a:

  1. x86_64/paging.rs: Detached is "dropped only after a user-half shootdown", true of every caller (Unmapped, the pipe path, the dlopen rollback). aarch64/paging.rs: the must_use text no longer names mm::Unmapped.
  2. amdvi_usbd now requires usbd: the claim refused its interrupt record (Io); the job prints only what it established ("usbd exited … and never said its No-Op completed"). The round-7 AMD readback judged again with it, touching no machine: PASS (https://github.com/ToyOSOrg/evidence/blob/4a237d120f745ccef1f9794581d3b5450835864a/p1/round8/rejudge-amd-r7.log).
  3. T14 round-6 readback published and cited (https://github.com/ToyOSOrg/evidence/blob/4a237d120f745ccef1f9794581d3b5450835864a/p1/bench/round6/t14, serials masked). Counts: 48 files, +1074 −2134 against the merge base.
  • Optional taken: src/spare.rs shared by both jobs; until_ended(&self).
  • Recorded, not fixed: issues/a-metal-jobs-failure-names-the-t14-on-every-bench.md.

Checks at 77ed37a: build-only x86-64/aarch64 exit 0; usbd_drives_the_spare, both usbd_refused_*, bar_map_again exit 0; --ci host exit 0, 86 green.

@Japabu

Japabu commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator: the focused review's required items are resolved at 77ed37a (checked the diff c6e5de7..77ed37a): Detached comments on both arches now say "dropped only after a user-half shootdown"; amdvi_usbd now requires usbd's own the claim refused its interrupt record (Io) line, so the vector arm asserts the refusal's cause, and the round-7 AMD readback re-judges PASS; the T14 round-6 readback (3/3 at 33adc6b) is published and cited; counts updated. The "on the T14" judge wording is recorded as an issue. Bench: T14 3/3; AMD all rows but amdvi_translates_the_stick_controller, red for the recorded, independent DMA-truncation reason (owner question pending). Reviewed: LAND.

@Japabu
Japabu marked this pull request as ready for review October 20, 2026 20:17
@Japabu
Japabu added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit 4432fdf Oct 20, 2026
8 checks passed
@Japabu
Japabu deleted the wt/toyos-p1 branch October 20, 2026 20:46
Japabu added a commit that referenced this pull request Oct 22, 2026
Conflicts resolved against main's changes since 3b77b9b:

- #947 deleted kernel/pci/src/placement.rs and
  issues/no-boot-reaches-the-retry-that-undoes-a-placement-nothing-answered.md;
  the branch's hunks there (a doc comment and an import, and the issue's
  exit) were about the placement that no longer exists, so both stay
  deleted. issues/an-aarch64-cargo-run-boot-hands-over-no-pci-claim.md
  stays deleted: main's one-line edit restated how its windows are read,
  which this branch replaces.
- K1's IOVA allocator (6cb27ce): a domain's allowed addresses are still the
  unit's range less every reserved range, but the root bridges' windows now
  come per domain from the declaration (`Domains::prepare`), beside the
  backend's own reserved ranges (RMRRs, IVMDs, the interrupt range, AMD-Vi's
  special range, the ITS doorbell), which `avoid` keeps. No declaration, no
  domain. `OwnSpace::create` takes the declared windows; the IOMMU
  selftests, which run before any declaration on QEMU's root bus, pass none.
- pcidev: main's `publish` without the loader's list, and the declaration,
  its wait and its refusals as on the branch.
- KernelArgs: main's `slot_volume_guid` follows `microcode_len` at 456;
  472 bytes.
- The supervisor (27b75c3): the `acpi` claim's holder starts before the
  disk drivers, whose claims wait for its declaration.
- acpicase: main's nvme and usbd rows claim PCI functions on a boot whose
  only `acpi` holder is a job, so it keeps no storage rows; the probe's
  held-function arm takes the console function the kernel drives, the only
  function a kernel driver still holds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q
Japabu added a commit that referenced this pull request Oct 22, 2026
…2d60) into wt/toyos-lifecycle-2

Lifecycle stage 2 rebuilt on main and stage B.

- pcidev: stage B's module is the base. The per-requester records, the
  device set's generation and snapshot, `Gone` on an older arrival's claim,
  the departed release and retirement, and the slot's interrupt are this
  branch's, over main's BAR records (`Decoded`); `Machine::windows` and
  the placement state are gone with #947.
- An arrived function's BARs: `place_unassigned` and `behind_bridges` are
  gone. Each BAR the device leaves unassigned is given, before the
  function is listed, the lowest address inside its port's windows that no
  other function decodes in and that the declared root bridges' memory
  holds (`toyos_pci::hotplug::{Windows, assign}`, host-tested), written
  once and kept where the register reads it back, with no probe through
  it. Its BARs enter `Machine::decoded` and leave it on retirement.
  `bar::placement` stays for that one write, refusing only an I/O BAR and
  asserting the caller's own choice.
- The slots are armed by stage B's declaration, from `pcidev::declare`,
  below a root whose `_OSC` commit was granted native hot-plug and the
  capability structure. acpiserver commits both where its query offered
  both (`osc::negotiate_root`); the declaration carries the grant. The
  boot-time arm with no bridges is gone.
- A slot's port reads its own bus numbers and windows: main deleted
  `toyos_pci::bridge` and `PciDevice::bridge`. `PciDevice::header_type`
  tells a bridge below a slot.
- Two readers of the set never enumerate one arrival twice (`REFRESHING`).
- ABI: the device set's object kind is 15 and its syscalls 135 and 136,
  past main's; the static `IrqWatch` poll returns for its watch.
- supervisor: main's `drive_where` reads the device set, says what joined
  and left, and decides over the rows asked for.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant