Skip to content

vtd: every unit is handed over inside protected memory regions, and the T14 iGPU fault is measured - #1086

Merged
Japabu merged 4 commits into
mainfrom
igpu-fault
Oct 28, 2026
Merged

Japabu merged 4 commits into
mainfrom
igpu-fault

Conversation

@Japabu

@Japabu Japabu commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

One T14 shared-debug boot (7ef0a48) halted 1 ms after Boot: peripherals ready. The iGPU (00:02.0) read 0x9cea7000, inside its RMRR, with reason 6h (read permission). The identity domain grants that address. The same defect was first recorded on 2026-10-04 (issues/the-t14s-igpu-faults-read-permission-after-translation-goes-off-and-on-under-its-scanout.md).

Measured on the T14. The measurement branch igpu-fault-measure (never lands) ran 3 boots of 200 cycles of each kind, with fault events held:

route between two identical identity roots cycles with a fault records
root switched under translation 0 / 600 0
translation off, root, queued invalidation, on 22 / 600 94
  • Every record is 00:02.0, a read, reason 6h, inside the firmware framebuffer in data stolen memory.
  • Records land 546 to 598 µs after translation is back on.
  • The walk from memory is intact.
  • Intel's specification update 631123-034 (TGL001 to TGL075) lists no erratum.
  • The cause is unattributed. It stays an open defect that only the actuator reaches (the orchestrator's ruling).

Change

The keep-on branch is gone. VT-d gives no condition under which system software may switch firmware's tables under translation: §6.6 demands identical results, and nothing documented tells this kernel firmware's mappings.

Every unit is handed over with translation off, inside protected memory regions (§11.4.8), where the hand-over can be guarded:

  • Firmware's leftovers first. Protected memory firmware left on goes off first.
  • What the regions cover. The identity domains and roots are built before the regions are chosen. Each region is the largest stretch of the OS's RAM (usable or ACPI-reclaimable, never NVS), clear of every RMRR, that holds every page those tables came from and this kernel's image.
  • Programming. The same regions go on every unit, with alignment probed from bases and limits. The boot logs the regions, what they cover, and the functions mastering the bus as they go on.
  • Order. Arm, every unit's hand-over with register-based invalidation (§6.5.1), release, then each unit's queue and interrupt remapping. The invalidation queue is not a remapping structure (Appendix A, ECAP.C), so it may lie under the regions.

Unguarded otherwise, never refused. The boot logs the reason when:

  • a unit lacks the regions (QEMU's);
  • a unit refuses register-based invalidation (major version 6+, or IOTLB registers outside the window);
  • fewer units are programmed than described;
  • the tables or kernel fall outside the regions.

Such a machine invalidates through the queue before translation comes on, as main does.

The judge. iommu_firmware_left requires the arm line before translation first goes off and the release before the first unit is programmed, on VT-d machines whose units all have the regions. Its row rides a boot of its own, iommu-firmware-left, because a halt there is a known outcome. shared-debug no longer arms the actuator.

Records:

  • the hand-over issues are merged into a-unit-passes-dma-untranslated-outside-its-protected-memory-regions-while-it-is-handed-over.md, with four named exit cases;
  • the iGPU issue carries the measurement and labels the boot rate an estimate;
  • a-unit-this-kernel-does-not-program-… is updated;
  • the containment proposal is recorded on the IOMMU track.

Checks (c6a9c99)

  • cargo run -- --ci host lint: exit 0.
  • dmaguard host tests: exit 0.
  • QEMU, all PASS:
    • vtd_firmware_left: unguarded, queue route, actuator;
    • iommu_virtio_platform: shipping kernel;
    • hda_plays_to_the_controller_that_reaches_an_output;
    • a_drives_row_holds_one_process_per_function.
  • Judge control: on r3's T14 boot, which armed without the new arm line, iommu_firmware_left is red.
  • The full --ci host from r3 stands for the host suites. They do not compile vtd/; lint compiles the kernel.
  • Metal (T14, c6a9c99, 8 boots, each Boot: complete):
    • Every boot, guarded: each logs the regions (low 0x100000..0x8ef00000 high 0x100000000..0x45f800000, 1 MiB alignment), their coverage of the tables page (0x1a00000..0x1c00000) and the kernel image, the bus masters at arm, then arm and release. No hand-over is unguarded line appears.
    • iommu-firmware-left ×3: the judge passes, exit 0. No 00:02.0 fault.
    • shared-debug ×3: 9 rows pass; only the 5 shared-member reds that main shows.
    • testcases, the 26 rows that ride only it: 22 pass, plus the same 4 known reds as r3.
    • testcases, the whole boot: 33 rows pass and 225 of 225 members complete. Six rows fail: those 4, plus amdvi_usbd_drives_its_own_entry (AMD bench) and i219_messages_on_its_msi_vector. The i219 failure happens seconds after the release, and handles: only an owner ends, moves, duplicates or lends a handle #1080's boot at 141055d, without this code, failed it the same way.
    • The one DMA FAULT is the test-staged 00:0d.0 owner=process fault.
  • Reused T14 evidence, and what it does and doesn't show:
    • r2 and r3 shared-debug: the 5 shared-member reds are independent of this change, being identical on main 576bebf. r2's boots ran the deleted keep-on branch, so they say nothing about this head's hand-over.
    • r3 (902bffb): the regions armed and released on all four units with 1 MiB alignment, and no 00:02.0 fault in 3 boots.

Evidence: https://github.com/ToyOSOrg/evidence/tree/0a4bcfac482940b89dc7b122a33867f09591e97e

Halting (the implementer's reasoning, not an owner decision)

A fault at an address the domain maps means the unit contradicted tables shared by all four units, so it halts.

Risks

  • During a guarded hand-over, these stay reachable: RAM outside the two stretches, every RMRR and peer MMIO.
  • From ExitBootServices until the regions go on, firmware's T14 leaves everything reachable.
  • §11.4.8.1 marks the regions for deprecation.
  • Blocked DMA is never recorded. The bus-master line at arm is the audit.

🤖 Generated with Claude Code

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review r1: igpu-fault at 30a7071

Spec used: VT-d Architecture Specification Rev. 4.1 (D51397-016), the revision the narrowed issue cites. I checked it against Intel's published PDF: §6.6, §6.5.2.11, §8.1 Flags, §3.16, §11.4.4.1 (GCMD.TE/SRTP), §11.4.5 and Table 26.

Architectural assessment

The hand-over belongs in vtd/, and no ABI or userland boundary moves. On the core question my answer is no. §6.6 does not permit switching tables under translation on exactly these conditions. The branch rests on an inference about firmware's tables.

  • What §6.6 requires (Rev. 4.1 p. 6-42). After SRTP with ESRTPS clear: global invalidations of the context-cache, then the PASID-cache "if applicable", then the IOTLB, in that order. With TES=1, the new structures "must ... provide identical remapping results". TTM must not change while TES=1 (also §11.4.5).
  • The code's sequence is correct. It writes RTADDR, does SRTP, then sends invalidate_all: a global context-cache descriptor, then a global IOTLB descriptor with DR/DW, then a wait. §6.5.2.11 makes hardware run the IOTLB descriptor only after the context-cache descriptor ahead of it completes, so one batch with one wait gives the required order.
  • No PASID-cache invalidation is needed. Descriptor 0x7 is not valid in legacy mode (the descriptor-type table in §6.5.2), and the T14's unit0 reports SMTS=0 (ecap bit 43).
  • No drain is required by §6.6. The DR/DW bits already set are harmless; the T14 reports DRD/DWD.
  • The TTM=00 check is the right §11.4.5 guard.
  • What "identical" rests on.
    • §8.1 says only that platform-initiated DMA "is restricted to only reserved memory regions".
    • §3.16 says that "when system software enables DMA remapping" the RMRR devices' tables "are expected to" identity-map those regions. That expectation is placed on system software, not on firmware's own pre-boot tables. §3.16 also warns that "DMA virtual addresses can indeed overlap with the host physical addresses of reserved system memory".
    • So "firmware's tables are identity over its RMRRs" is the implementer's inference. The PR's risks section labels it as one; the code and issue text do not (finding 2).
  • The T14 cannot test that inference. The T14 firmware hands every unit over with TE off (gsts=0x40000000 in the fault boot). Under iommu-firmware-left, the "firmware" root is this kernel's own identity root, built by leave_on. The metal rows therefore test the hardware mechanism (a root switch while translating, with live iGPU scanout). They do not test the declared-branch premise. On every T14 boot without the actuator, this change does nothing. The PR body's "The T14 takes this path" should say "under iommu-firmware-left".
  • The fallback path is still correct.
    • TE goes off first.
    • GCMD.TE "disables only at deterministic transaction boundaries" (§11.4.4.1).
    • The invalidation now runs before SIRTP, which is harmless with TE off.
    • TE comes back on after both pointers are set.
    • vtd_firmware_left covers it on QEMU, and the mutation control shows that test can tell the two branches apart.

Size: +639/−62. Production vtd/mod.rs gains about 76 lines. About 370 lines are actuator-only measurement code (cycles.rs, fault holding, copy_root/walk, the row and the judge). The rest is a QEMU test, the judge change and the issue text.

Required findings

1. BLOCKER: kernel/src/arch/x86_64/vtd/mod.rs:163-166, 217, 261-265 (with dmar.rs:110-116). The condition is computed over a DMAR that was never completely validated.

regions_identity starts true and is only cleared by RMRRs the walk actually reaches. Four inputs get through:

  • (a) The walk can end early. It stops at a Malformed structure, and at the MAX_UNITS break (line 175). Either way regions_identity stays true. §8.1 has every DRHD listed before any RMRR, so a DMAR with more than 16 units examines no RMRR at all and still yields InFlight::Identical.
  • (b) A truncated RMRR passes. If its length is between 4 and 23 bytes, bounded reads base=limit=0. That counts as held, the RMRR has no scopes, and .all() is vacuously true.
  • (c) The scope kind is not checked. For a type-2 sub-hierarchy scope, enumerated checks only the bridge's own BDF, but the requesters are the devices below it.
  • (d) The RMRR segment is ignored.

Consequence: translation stays on through firmware's tables on a machine where the premise was never checked. This firmware input should fail closed at the boundary. Clear the flag unless:

  • the walk ended normally;
  • each RMRR's declared length covers its limit field;
  • base <= limit;
  • every scope is an endpoint on segment 0.

Adopting proposal (2) below would delete the condition altogether.

2. BLOCKER: comments and records that state the inference, or the wrong section, as specification fact.

  • dmar.rs:25. "DMA blocked until the OS programs the unit; honoured for free by this kernel's ordering" is false against §8.1. The flag reports that platform DMA is confined to RMRR ranges. The new code depends on exactly this definition, so it has to be correct.
  • mod.rs:271-275 (InFlight::Identical). "§3.16 has it map identity and read-write" attributes to the specification a guarantee it does not make (see above). The same claim appears in the issue at issues/a-unit-left-translating-passes-dma-untranslated-while-programmed.md:17-21 and in the new issue's "The change made". Say it is an inference about firmware's tables.
  • mod.rs:408-411. "RTADDR_REG.RTA may change while TES is set, §11.4.5" cites the wrong section. §11.4.5 says RTA and TTM take effect at SRTP and forbids changing TTM while TES=1. The permission to set the root pointer while translating is in §6.6.
  • mod.rs:54 (TRANSLATION_TABLE_MODE). The comment says only "the only one this kernel writes". It should give the actual reason for the check: TTM may not change under TES=1 when ESRTPS is clear (§6.6, §11.4.5).

3. BLOCKER: the reproducer's design and the new issue's exit can misattribute the result (cycles.rs:87-90; issues/the-t14s-igpu-faulted-...md:55-60).

  • Interleaving confounds the switch cycles. The faulting boot halted on its first fault, so nothing shows whether an off-on cycle leaves unit0 broken afterwards. If it does, the following switch cycles fault too. The exit then reads that outcome as "this change does not cure it". Cheapest fix: run all 200 switch cycles first, as a clean phase, then the 200 off-on cycles. Under the fix, the boot's own hand-over is already a switch, so nothing before the switch phase has turned translation off.
  • The exit covers only two outcomes. It must also say that zero faults in both phases is inconclusive.
  • The close condition leaves known debt. The issue closes once iommu_firmware_left passes. But if off-on is shown to fault, the kernel still turns translation off on every undeclared machine. Recording "the cause at Unit::hand_over" is a code comment, not a tracked defect. Either keep the defect open for that path, or remove the path (proposal 2).

4. BLOCKER for landing (the metal run is not affected): the measurement code must not land.

iommu-handover-cycles is exploratory, and the handoff says so. After attribution:

  • delete cycles.rs;
  • delete fault::{hold, index_of, Held, take_held, FECTL_MASK} and table::{copy_root, Walk, walk};
  • delete the actuator, its FLASHABLE entry, HANDOVER_CYCLES, the METAL row and the iommu_handover_cycles judge;
  • fold the free functions command/wait back into Unit, since only the cycles needed them;
  • keep read_record, which is a small clarity gain.

No durable claim needs this code:

  • The production path makes one switch per boot.
  • Every T14 shared-debug boot under iommu_firmware_left exercises that switch with live scanout.
  • Any fault there already halts loudly.
  • The off-on half measures a route the T14 no longer takes.

Under root policy, exploratory comparisons are evidence, not regression tests. Put the counts and lines in the evidence repo and the issue.

Evidence

  • Host CI. Reusing the host run (tree before vtd_firmware_left) is acceptable. Kernel code is byte-identical, and the later test edits are compiled by lint at the head (exit 0).
  • QEMU test. vtd_firmware_left passed at the head. Its mutation control (in_flight forced to Identical) went red, exit 1. That control shows the test separates the two branches.
  • Judge control. On the pre-fix readback the judge is red, as intended.
  • Statistics. The 1-in-46 versus 0-in-787 comparison is correctly labelled suggestive: 46/833 ≈ 0.055.
  • Metal. Not run yet. The hardware claim (switching under translation with live iGPU scanout does not fault on the T14) is unestablished until shared-debug and the cycles boot are read.

Suggestions

S1. Proposal (2): drop the TE-off path for legacy-mode roots. Engineering read: in favour, owner to approve the §6.6 departure.

  • §6.6 describes what happens when the tables are not identical: until the invalidations complete, hardware "may utilize the cached old ... or the new" structures. So an in-flight request gets firmware's translation or the identity domain's.
  • The TE-off route gives firmware's translation or none. "None" is unbounded passthrough, including peer MMIO and addresses above top. That is a wider exposure than the identity domain.
  • After the hand-over both routes end in the same identity root. A firmware DMA the identity domain refuses therefore faults microseconds later either way, so this proposal changes no fault outcome.
  • It deletes InFlight, regions_identity, enumerated, finding 1 and the judge's declared re-derivation, and QEMU would test the same path as the T14.
  • Two exceptions remain:
    • TTM≠00 (with ESRTPS clear) must still go off, under the owner's 2026-10-04 ruling;
    • the code site should state that §6.6's "identical" requirement is knowingly not met, and why.
  • If the cycles boot shows off-on faulting, this is the direct fix for undeclared machines too.

S2. Proposal (1): contain faults on unmapped addresses from undriven functions. Engineering read: sound in principle, but simpler than proposed, and not part of this PR.

  • It would not have changed this boot: the address was mapped, so it still halts.
  • No table walk is needed in the handler. An unclaimed function sits in the identity domain, which maps all of [0, top) with nothing carved out. So "unmapped" is just address >= top, and the handler stays bounded and lock-free.
  • The real change is in the ownership representation. Today Owner::Kernel merges functions the kernel drives with functions nobody drives. Splitting it gives "undriven" its own meaning.
  • Caveat for the owner: the iGPU is not truly undriven. The kernel's panic console draws into the firmware framebuffer, which the iGPU's scanout reads through unit0. Clearing BME blanks that console.

S3. Record a related exposure. mm::init runs before iommu::init and hands out EfiBootServicesData. From then until the switch (or TE off), a unit left translating walks firmware's tables in memory the pmm may already have reused. This applies to both routes. It also weakens the "identical" premise and the table-copy exit. Record it in a-unit-left-translating-...md.

S4. Smaller notes.

  • Cycles fidelity. The cycles' off-on omits the faulting sequence's QIE off/on with a fresh queue and the IRE/CFI toggles. A null off-on result therefore does not clear the original sequence.
  • Linux provenance. The narrowed issue's rewritten paragraph still names Linux's copy_translation_tables/drivers/iommu/intel/iommu.c as the model. Under the clean-room rule, provenance belongs in evidence, not the repository. The paragraph is being rewritten anyway, so drop it.
  • Judge gaps. iommu_firmware_left's declared does not mirror the kernel's handling of non-enumerated or malformed scopes. That gives a loud false red, never a false pass. Moot under S1.
  • PR description (for the orchestrator). "Decision recorded: ... halt" is the implementer's reasoning, not an owner decision. I agree with it: a fault at a mapped address means the unit contradicted tables shared by all four units.

SEND BACK

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

r1 response (head 5fb75fc)

  • B1: the declared branch now needs all of the following, else it fails closed:

    • a DMAR walk that ended normally (a malformed structure or the unit ceiling clears it);
    • every RMRR whole (len >= 24), with base <= limit, on segment 0;
    • every scope a type-1 endpoint that this kernel enumerates.

    See identity_mapped in vtd/mod.rs and Rmrr::whole in dmar.rs.

  • B2:

    • FLAG_DMA_CTRL_OPT_IN's comment now states what §8.1 says.
    • InFlight::Identical labels the firmware-tables claim as an inference, and says §3.16 sets that expectation for system software's tables.
    • hand_over cites §6.6 for the pointer set under translation.
    • TRANSLATION_TABLE_MODE gives the §6.6 and §11.4.5 TTM rule.
    • Both issues are corrected to match.
  • B3: the measurement runs every switch first, then every off-on. The exit covers three outcomes: switch faults, so the issue is re-scoped; only off-on faults, so it stays open for the remaining off path; neither faults, which is inconclusive and stays open. It closes only when no remaining path is known to fault.

  • B4: all measurement code is removed from the PR: cycles.rs, the fault-holding helpers, copy_root/walk, the actuator, its FLASHABLE entry, the row and the judge. command/wait are folded back into Unit, and read_record stays. The measurement is branch igpu-fault-measure (e3b1562).

  • S1: with the architect, via the orchestrator.

  • S2: recorded in the-iommu-refuses-nothing-yet.md, using address >= top, the Owner::Kernel split and the panic-console caveat.

  • S3: recorded in a-unit-left-translating-….

  • S4:

    • fidelity: noted in the issue;
    • Linux: dropped;
    • judge: left, since it can only red falsely and is moot under S1;
    • PR description: corrected.

Evidence: https://github.com/ToyOSOrg/evidence/tree/42a790b5eccac6e66a1f1911aa1c511b3fa6db07

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review r2: igpu-fault at 5fb75fc

Base: origin/main 717baf3, which is the merge base. The measurement branch, igpu-fault-measure at e3b1562, is the head plus 413 lines, all of them measurement. I read the head and the r1→head delta from git objects. Spec: VT-d Rev. 4.1 (D51397-016), §6.6, §8.1, §8.4, §3.16 and §11.4.4.1, re-read from Intel's PDF.

Architectural assessment

The r1 fixes are done as asked. They do not make the keep-on branch landable, and the spec does not support it.

The question that matters now is the one the architect raised: is it legitimate to keep translation on through tables firmware left behind? Here is what Rev. 4.1 actually says about a unit firmware left translating.

  • §11.4.4.1 (GCMD.TE). "There may be active DMA requests in the platform when software updates this field. Hardware must enable or disable remapping logic only at deterministic transaction boundaries, so that any in-flight transaction is either subject to remapping or not at all." On DRD/DWD parts, it also drains in-flight requests before reporting TES. This is the documented transition for a live unit:
    1. TE off;
    2. SRTP;
    3. the §6.6 global invalidations, in order;
    4. TE on.
  • §6.6. A root switch with TES=1 is allowed only where "software must ensure the structures referenced by the new root-table pointer provide identical remapping results as the structures referenced by the previous root-table pointer". The reason: "hardware may utilize the cached old paging structure entries or the new ... until the ... invalidations are completed". This obligation is on the software doing the switch, about the old structures. Software can meet it only if it knows those structures, either by reading firmware's root, context and second-level entries or because it wrote them itself.
  • §8.1 DMA_CTRL_PLATFORM_OPT_IN_FLAG. The flag reports which memory platform DMA targets ("restricted to only reserved memory regions"). The consequence it grants is that system software "may program DMA remapping hardware to block DMA outside of RMRR". It says nothing about how firmware's own tables translate. §8.4 adds only that pre-boot-only DMA targets are not RMRRs.
  • §3.16. The RMRR identity expectation applies "when system software enables DMA remapping". It is not stated of firmware's pre-boot tables, and the same section warns that DMA virtual addresses "can indeed overlap" reserved host-physical memory.

So nothing in the spec states that a firmware-left root is identity over its RMRRs.

The head never reads firmware's structures, so its Identical verdict meets §6.6's "must" only by assumption. The architect's failure mode is concrete. Today, firmware's in-flight DMA either completes untranslated (TE off) or translates by firmware's tables. Under the head, on a machine whose firmware does not identity-map an RMRR, that same DMA can fault against this kernel's tables. A fault at a mapped address halts, and an undriven function is Owner::Kernel. The head can therefore turn a booting machine into a halted one.

No machine runs the branch against real firmware tables.

  • The T14's firmware hands every unit over with TE off.
  • QEMU leaves the flag clear.
  • The branch runs only under the iommu-firmware-left actuator. There the "firmware" root is the kernel's own leave_on root, so the old and new structures are identical by construction.

That makes it production machinery whose only exercise is a test scenario, and whose premise in production is an inference.

What the spec supports for a firmware-left unit:

  • The default: the TE-off route hand_over already has.
  • Keeping TE on is supportable only after software has established identity over every in-flight requester and address. With the §8.1 flag, that set narrows to the RMRR ranges and the devices they name, but it still has to be read and checked from firmware's own structures. This kernel cannot do that today: mm::init reuses EfiBootServicesData before iommu::init, as the head's own S3 paragraph records. Making it possible would be a loader or mm ordering change, which is an owner-level decision and not this PR.

If the measurement shows off-on faulting on the T14, that is hardware departing from §11.4.4.1, or a defect in this kernel's sequence. Neither is fixed by keeping on through unread firmware tables. Two further points on the T14 itself:

  • the fault arises only under the actuator, since no real T14 boot leaves TE on;
  • the next authority to check would be Intel's 11th Gen specification update or errata.

Size. The head is +282/−43, of which about +150 is production kernel. Roughly 110 of those lines belong to the keep-on branch (InFlight, identity_mapped, Rmrr::whole, TRANSLATION_TABLE_MODE, SCOPE_ENDPOINT and the hand_over arm), along with the judge's declared clause. All of it goes with the branch. read_record stays, as settled in r1.

Previous blockers

  • B1 (fail closed): resolved in code, now moot.
    • The walk clears the flag on Malformed (mod.rs:236) and at the MAX_UNITS break (mod.rs:177).
    • A short tail (fewer than 4 bytes) cannot hold a structure, so ending the walk there loses nothing.
    • identity_mapped (mod.rs:264-273) requires all of: whole() (len >= 24); base <= limit; segment == SEGMENT_GROUP; and every scope Ok, type 1, single-hop and enumerated.
    • held (limit < top) supplies "inside the identity domain".
    • One residual gap: an RMRR with zero scopes passes vacuously. It is moot, because it goes with the branch.
  • Host test of the DMAR condition. No cheap host test is available today. DMAR parsing lives in the kernel crate (vtd/dmar.rs over drivers::acpi::Table, which wraps toyos_acpi::Table<DirectPhys>), and that crate has no host test target. IVRS and IORT, by contrast, live in lib/acpi, which has 136 host tests and already carries fixtures/qemu-11.1.1/dmar.bin. If a firmware-input predicate like this one were landing, I would require that host test. Root policy makes malformed input a unit test, and B1's four cases are exactly that. Getting it would mean moving the DMAR walk into lib/acpi. With the branch removed, nothing needs it now. Moving DMAR is an optional follow-up, not part of this PR.
  • B2 (wording): resolved.
    • dmar.rs:25-27 now matches §8.1.
    • InFlight::Identical labels the inference and places §3.16's expectation on system software.
    • hand_over cites §6.6 for the switch.
    • TRANSLATION_TABLE_MODE gives the ESRTPS/TTM rule (§6.6, §11.4.5), which I checked against the text.
    • Both issues match.
    • Most of this text goes with the branch.
  • B3 (reproducer order and the exit): resolved on the measure branch. cycles.rs:92-93 at e3b1562 runs cycles 0..200 as switch and 200..400 as off-on. Every unit ends each phase on its own root, because CYCLES is even. The exit lists all three outcomes, and "neither faults" is inconclusive. The measurement stays valid at e3b1562 whatever happens to the PR: at that revision the boot's own hand-over keeps TE on, so the switch phase follows no TE-off.
  • B4 (measurement code out of the head): resolved.
    • There is no diff from main in actuator.rs, table.rs or tests/metal.
    • No cycles, hold, take_held or copy_root remains.
    • command and wait are Unit methods again (mod.rs:364, 371).
    • read_record is kept.

Required findings

N1. BLOCKER: kernel/src/arch/x86_64/vtd/mod.rs:52-57, 165-168, 177, 220, 236, 251, 261-299, 415-447, 686; dmar.rs:189-193; tests/toyos.rs:5978-5979, 6004-6009, 6035. The keep-on branch switches roots under TES=1 without establishing §6.6's identical results.

  • Problem. The InFlight::Identical verdict comes from the §8.1 flag plus this kernel's own tables, never from firmware's structures. §8.1 constrains DMA targets, not firmware's translation, and §3.16 does not cover pre-boot tables. The §6.6 "must" is therefore unmet except by inference. The architect has ruled out exactly this.

  • Consequence. On a declaring machine whose firmware tables differ over an RMRR, in-flight firmware DMA faults against this kernel's tables, and the machine halts where the TE-off route would boot. No real machine exercises the branch. Its only exercise is the actuator, where identity holds by construction. That is production machinery shipped for a test scenario.

  • Fix. Delete the branch and everything that exists only for it:

    • InFlight and in_flight;
    • regions_identity and its three clears;
    • identity_mapped, SCOPE_ENDPOINT and Rmrr::whole;
    • TRANSLATION_TABLE_MODE;
    • hand_over's TE arm, so hand_over takes no argument again;
    • the judge's declared and kept.

    Then:

    • Keep the FLAG_DMA_CTRL_OPT_IN comment fix, which is now correct.
    • Revert invalidate_all to after SIRTP, or keep the move with the comment deleted. "At once on a unit still translating" (mod.rs:618-619) becomes false once no unit is still translating at that point. The same applies to the doc change on enable at mod.rs:573.
    • vtd_firmware_left may stay as QEMU coverage of the VT-d TE-off hand-over, alongside amdvi_firmware_left. Its doc and expected line must drop the declared/undeclared framing and the reason suffix, and MACHINE_TESTS's comment must be updated to match.

N2. BLOCKER: the records describe the keep-on as landed, and the new issue's exit presumes it is the remedy.

  • issues/the-t14s-igpu-faulted-…md:40-45. "Since pull request vtd: every unit is handed over inside protected memory regions, and the T14 iGPU fault is measured #1086, a unit … keeps translation on" is false once N1 is applied. Delete it.
  • Exit, second outcome (:62-65, "until that path no longer takes translation off"). This presumes the remedy is to stop turning TE off. That is the transition §11.4.4.1 documents for live DMA, and the one the architect has kept. Restate it: if only off-on faults, the cause is this kernel's sequence or a documented hardware erratum. The issue stays open until that cause is fixed, or until the erratum is cited with its workaround applied. The first outcome's "keeping translation on does not cure this" reads as a cure that is no longer proposed. Re-scope it to "the fault is not caused by translation going off".
  • issues/a-unit-left-translating-…md:16-26. Delete the "On that inference … keeps TE on" passage. Restore the requirement as the head's first sentence states it: keeping TE on needs firmware's structures read before the pmm reuses their memory. Keep the S3 exposure paragraph. Without the Linux citation (r1 S4), the restored text is the pre-PR paragraph.

Evidence

  • I accept reusing the r1 host CI run and the vtd_firmware_left pass for the r1 changes. The head's vtd_firmware_left PASS and lint (exit 0) are recorded at 5fb75fc.
  • After N1, vtd_firmware_left must be re-run at the new head, because its expected line changes. Host lint compiles the judge.
  • No metal claim is made. The measurement boots of e3b1562 remain the attribution evidence and are not affected by N1.

Suggestion

S1. Move the DMAR walk into lib/acpi beside IVRS and IORT, with host tests over the existing dmar.bin fixture and malformed variants. This is separate from this PR. It would put the kernel's VT-d firmware-input parsing under the same host tests as AMD's.

SEND BACK

Japabu and others added 3 commits October 28, 2026 13:12
…, and a reproducer for the T14's iGPU fault

One T14 shared-debug boot under iommu-firmware-left halted on a read by the
iGPU (00:02.0) at 0x9cea7000, inside the RMRR firmware reserves for it, with
reason 6h: a second-stage entry with R clear (VT-d 5.0 Table 30, LGN.3). The
identity domain maps that address read-write, so unit0 did not translate by
the tables it had been given. Under the actuator, and only there, unit0 goes
on, off and on while the firmware's scanout reads through it. The
orchestrator's T14 logs hold 787 boots with a single on and no iGPU fault,
and 46 cycled boots, one of which faulted. That points at translation going
off under live DMA, but does not establish it.

The hand-over now keeps translation on where §6.6 allows it. With
DMA_CTRL_PLATFORM_OPT_IN (§8.1), firmware's DMA reaches only its RMRRs, and
§3.16 has those mapped identity and read-write. Where the identity domain
holds every RMRR and every device they name is enumerated, this kernel's root
translates every request in flight as firmware's did. The root switch then
happens under translation (RTADDR.RTA may change while TES is set, §11.4.5),
with the global invalidation immediately after SRTP. The T14 declares the
flag. QEMU does not, so its unit still goes off first and says why; the new
vtd_firmware_left machine test holds that branch.

iommu-handover-cycles moves every unit between its root and a copy of it
200 times by each route: off-on and the switch under translation. Fault
events are held throughout, every record is counted against the cycle it
followed, and the first sixteen are printed with the walk the tables give
from memory. Its row, iommu_handover_cycles, runs only by name, on a boot
of its own. The iommu_firmware_left judge now requires, on a declaring
machine, that every unit kept translation on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… infers; the reproducer leaves the PR

The declared branch now holds only for a DMAR read to its end, and only where
every RMRR is whole, has base <= limit, is on segment 0 and names endpoints
alone. A walk stopped by a malformed structure or the unit ceiling leaves
regions unread. A short RMRR reads its fields as zero. A bridge scope's
requesters are the devices below it. Each of these now clears it.

The comments and records stop presenting the premise as specification. §8.1
reports platform DMA confined to RMRRs. §3.16 expects identity mappings of
system software's tables, not firmware's, so "firmware's tables translate
alike" is labelled an inference. The root-pointer permission is §6.6's, and
TTM's guard is §6.6 and §11.4.5. The DMA_CTRL_PLATFORM_OPT_IN constant said
something false of the flag.

The iommu-handover-cycles measurement moves to a branch of its own and does
not land. Its next run takes every switch first and every off-on after, so
off-on cannot leave the unit faulting into the switch phase. The issue's exit
covers all three outcomes, calls zero faults both ways inconclusive, and stays
open while a path known to fault takes translation off. The narrowed hand-over
issue drops its Linux citation and records the pmm exposure both routes share.
The IOMMU track records the containment proposal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… memory regions; the keep-on branch goes

VT-d gives system software no condition under which it may switch firmware's
tables for its own under translation. §6.6 asks for identical remapping
results, and nothing documented tells this kernel firmware's mappings.
DMA_CTRL_PLATFORM_OPT_IN (§8.1) says only where platform DMA goes. So the
keep-on branch and everything it alone used are deleted: InFlight,
regions_identity, identity_mapped, SCOPE_ENDPOINT, Rmrr::whole,
TRANSLATION_TABLE_MODE, and the judge's declared and kept.

Every hand-over now takes translation off. While it is off, a unit's
protected memory regions block DMA to the memory they cover (§11.4.8.1).
Where every unit reports PLMR and PHMR, vtd/pmr.rs programs one pair on all
of them, as §11.4.8.1 requires. They cover the largest stretch of firmware's
RAM from 16 MiB to 4 GiB and the largest above it, each clear of every RMRR
(kernel::dmaguard). The same alignment, probed and logged, applies to every
unit. The regions go on before the first unit goes off and are released once
the last translates. A unit without them logs that its hand-over is
unguarded; QEMU's is one.

Inside the regions, the hand-over invalidates through the unit's registers
(§6.5.1). The invalidation queue and its status write are listed apart from
the remapping structures the regions exempt (Appendix A, ECAP.C), so nothing
depends on the queue's memory being reachable. The queue and interrupt
remapping are programmed after the release. That makes Queue::invalidate_all
dead: the actuator's leave_on invalidates through the registers too. A unit
of version 6 or later refuses register-based invalidation (§11.4.6.1), so
plan leaves it unprogrammed, as it does a unit whose IOTLB registers lie
past its window.

The steps run in kernel::dmaguard::order, whose host test reads arm, then
every translate, then release, then every finish. It goes red with the arm
step removed.

Records:
- The two hand-over issues become one: a unit without protected memory
  regions passes DMA untranslated while it is handed over.
- The T14 iGPU fault is the existing 2026-10-04 issue, renamed because its
  claim is refused: no period of refused reads is needed. It carries #1086's
  measurement: on the T14, a switch under translation faulted in 0 of 600
  cycles, and translation off and back on in 22 of 600, the unit refusing
  reads its tables grant. Intel's 631123-034 lists no erratum for it. Its exit
  is this kernel's sequence fixed, or a documented erratum's workaround
  applied. This change does not cure it: under iommu-firmware-left the T14
  still takes translation off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu Japabu changed the title vtd: a unit left translating stays on where firmware declares its DMA, and a reproducer for the T14's iGPU fault vtd: every unit is handed over inside protected memory regions, and the T14 iGPU fault is measured Oct 28, 2026
@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

r2 response (head 902bffb)

N1: done, by the approved protected-memory round.

  • The keep-on branch and everything only it used are deleted: InFlight/in_flight, regions_identity, identity_mapped, SCOPE_ENDPOINT, Rmrr::whole, TRANSLATION_TABLE_MODE, and the judge's declared/kept.
  • invalidate_all is gone. The hand-over's invalidation is register-based, directly after SRTP inside the protected window, and Queue::invalidate_all is deleted.
  • vtd_firmware_left re-ran at the head: PASS. iommu_virtio_platform on the shipping kernel: PASS. It first caught a TABLES self-deadlock, now fixed.

N2:

  • No record describes keep-on as a cure.
  • The iGPU issue (the existing 2026-10-04 one, renamed because its claim is refused) carries the measurement: switch 0/600 cycles, off-on 22/600.
  • Its exit is "fix this kernel's sequence, or cite a documented erratum with its workaround applied". Intel's 631123-034 lists none (TGL001 to TGL075). The titles are in the evidence, with the document cited, not copied.
  • This change does not cure the fault under the actuator. The owner choice goes through the orchestrator.

Evidence: https://github.com/ToyOSOrg/evidence/tree/103ec6294d868d70ea94ae01674112ac6d552950

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review r3: igpu-fault at 902bffb

Base. origin/main 2a24fcf, which is the merge base.

Read.

  • The three commits and the full diff.
  • In full: vtd/mod.rs, vtd/pmr.rs, kernel/pure/dmaguard.rs and vtd/queue.rs.
  • The iommu_firmware_left judge and its row (tests/toyos.rs:1300-1306, 5983-6056), vtd_firmware_left, and the boot order in main.rs (iommu::init runs before any driver init).
  • The changed issues, and the evidence at 103ec629 (r3/).
  • The CAP, ECAP and VER of the T14's units, from r2's T14 kernel logs.

Spec. VT-d Rev. 4.1 (D51397-016), re-read from Intel's PDF: §3.4.4, §6.5, §6.5.1, §11.4.3 (ECAP.C, ADMS), the CAIG/IAIG encodings, §11.4.8.1–.5 and Appendix A.

Architectural assessment

Protected memory regions are the right mechanism for the T14, and the keep-on branch is fully gone.

  • §11.4.8.1 states the purpose exactly: "Protected memory regions may be used by software to securely initialize remapping structures in memory."
  • The replacement it points to is abort-DMA mode (§3.4.4). That needs ECAP.ADMS (bit 52), which is clear on all four T14 units. §3.4.4 also excludes it while a device an RMRR names is mastering the bus, and the iGPU is.
  • So the PMR route is the only documented one on this machine. The kernel owns it, which is correct.

Two structural problems remain:

  • Coverage is chosen by size, not by what the window must protect. Nothing ensures that the remapping structures, which are the regions' stated purpose, lie inside them (B1).
  • The hand-over runs through a step interpreter that exists to give a host test something to read (B4). It swaps move semantics for runtime expects.

The refusals added to plan remove the IOMMU from units this kernel programs today (B2).

Size.

  • Production is about +310 lines: pmr.rs 126, dmaguard.rs about 98 non-test lines (about 28 of them Step/order), mod.rs +101 net, queue.rs −14.
  • Tests are about +117 lines.
  • Issues come to about +8 net, from three merged into one plus the renamed iGPU record.
  • B4 deletes about 60 lines.

Answers to the questions put

PMEN protocol: correct.

  • Base and limit are written before EPM.
  • Every write of EPM waits for PRS to follow before the next, both ways, under a 1 s tripwire.
  • Base and limit are not touched while PRS is set, except for the firmware-left case in B3.

Alignment probe: correct for the bases.

  • Writing all 1s and reading back gives trailing_zeros = N+1.
  • PHMBASE bits at or above HAW read as 0, which does not change the result.
  • The limits are not probed (S1).

What the regions protect.

  • On a Major Version 2+ or Skylake-and-later unit, the regions block DMA to them "whether or not DMA remapping is enabled". The T14's units are ver 4.0 and 1.0 on Tiger Lake, so the second clause covers them.
  • The hardware's own accesses to remapping structures are exempt.
  • Blocked requests are neither recorded nor reported.
  • ECAP.C (§11.4.3) and Appendix A list the invalidation queue and its wait-status write apart from the remapping structures. So keeping the queue out of the window is the safe reading, and I agree with it.
  • pmr.rs:1-4 misstates the scope (B6).

Register-based invalidation with the queue off: valid.

  • §6.5.1 permits it only with queued invalidation disabled and the TTM in legacy mode, "irrespective of" TES. hand_over clears QIE first, after IQH == IQT.
  • The encodings check out:
    • CIRG/CAIG at 62:61/60:59;
    • IIRG/IAIG at 61:60/58:57;
    • DR/DW at 49/48;
    • IOTLB_REG at IRO+8.
  • The T14 has IRO=0x500 on all four units. All four report DRD and DWD set, so the ver-1.0 units honour the drains.
  • The §6.6 order holds: context cache, then IOTLB.
  • §6.5.2's precondition for later enabling the queue (no register request pending) is met by the waits.

Ordering against interrupt remapping and the queue: correct.

  • Every unit translates before the release.
  • The queue, IRTA/SIRTP, the IEC invalidation and IRE all come after it.

Can legitimate DMA be silently blocked in the window? I found none on the T14.

  • No ToyOS driver starts before iommu::init (main.rs:435). The stick, NVMe and USB are driven only after it.
  • The iGPU's scanout and GGTT live in BGSM..TOLUD. That range is both rmrr0 and reserved in the map, so no stretch can include it.
  • The T14 sets DMA_CTRL_PLATFORM_OPT_IN (flags=0x05). Firmware thereby declares that platform DMA at hand-over is confined to RMRRs (§8.1).
  • On a machine without that flag, the basis is only an inference: memory the OS owns after ExitBootServices is no legitimate DMA target. ACPI NVS is the exception, because it is firmware's memory and is_read_as_memory includes it (S5).
  • What would make a block visible is a device left mastering the bus. Nothing logs that (S7).

Is "largest stretch" enough for the protection claim? No, see B1.

Does the order test test behaviour? No, see B4.

Is the deadlock fix right? Yes.

  • identity_root(&mut TABLES.lock(), …, identity_domain(…)) would take TABLES inside its own argument.
  • Hoisting identity_domain out is correct, and the leave_on call site evaluates its argument before it locks.
  • iommu_virtio_platform's red-then-green is a real control. S4 would make the bug unrepresentable.

Is the merged issue's exit sound? Not as written, see B7.

The owner question needs correcting before it goes out; see the last section.

Required findings

B1. BLOCKER: vtd/pmr.rs:67-68, kernel/pure/dmaguard.rs:32-69, vtd/mod.rs:265-277. The regions are not shown to cover the structures they exist to protect.

  • Problem. The low region is the largest RMRR-free RAM stretch from 16 MiB, and the high region the largest above 4 GiB. But the hand-over's tables come from pmm::alloc_page (table.rs:52), wherever the pmm happens to put them. They are written inside the window: the identity domain, the four roots and their context tables are all built after Arm on the shipping path.

  • What the T14 shows. In r2's T14 log:

    • root=0x1a1c000, so the tables page is at about 26 MiB;
    • the kernel image is at 0x5e400000;
    • the pmm's records are at 0x100000, below the floor.

    Whether any of these lies inside the chosen low span is unobserved. The spans have never been logged on any machine.

  • Consequence. On the shipping T14, firmware leaves TE off, so DMA was already unrestricted from ExitBootServices to iommu::init. The one thing arming the regions buys there is that the tables the units are about to walk cannot be rewritten while they are built. If those tables fall outside the span, a device can redirect translation for the rest of the boot. The log would still say the hand-over was guarded, and iommu_firmware_left would pass.

  • Fix.

    • Take the tables page used for the hand-over before Arm, and choose or check the spans against it and the kernel image.
    • If they are not covered, log the hand-over as not protecting them, and do not report it as guarded.
    • Log the coverage, so the T14 run reads it.

    Choosing the stretch that contains these frames, rather than the largest one, is the simpler representation.

B2. BLOCKER: vtd/mod.rs:571-588, 616. The new plan refusals stop this kernel programming units it programs on main.

  • Problem. major >= 6 and "IRO past the window" now refuse the unit outright. translate invalidates through registers on every route, including the unguarded one, so a unit without regions (QEMU's, or a v6 one) gets the same refusal even though nothing on its route needs register invalidation.

  • Consequence. On a machine with a v6+ unit, the IOMMU is not programmed at all: no DMA isolation and no interrupt remapping. Main programs such a unit through the queue. §11.4.8.1 marks the regions for deprecation, so a v6 unit is the likeliest to have none and to be unguarded anyway. "No machine here has one" is not a reason to withhold a capability from metal.

  • Fix.

    • Use register-based invalidation where the unit supports it (v≤5, IOTLB registers inside the window).
    • Otherwise invalidate through the queue, programmed before TE as main does.
    • Treat a machine with any such unit as unguarded.

    If the architect explicitly approved refusing these units, record that as an issue with its owner and exit instead of a log line.

B3. BLOCKER: vtd/pmr.rs:56-63, 76-83. Protected memory that firmware left on is not handled.

  • Problem. §11.4.8.2–.5 forbid writing PLMBASE, PLMLIMIT, PHMBASE or PHMLIMIT while PRS is set. The probe and the programming write all four without reading PMEN.
  • Consequence. If firmware leaves EPM set, the writes may be ignored. The alignment is then derived from firmware's own base, and the log reports regions that are not in force. The hand-over exists to undo what firmware left on, and FIRMWARE_LEFT handles TE, CFI, IRE and QIE, but not this.
  • Fix. Read PMEN first. If PRS is set, log "was handed over with protected memory on; it goes off first" and set(false) before probing, in the same pattern as hand_over. About 5 lines.

B4. BLOCKER: kernel/pure/dmaguard.rs:71-98, 107-130; vtd/mod.rs:251, 278-289. The step interpreter is production machinery for a test that only restates it.

  • Problem. order() returns a fixed sequence, and its test asserts that same sequence. To make that possible, init turns moves into Vec<Option<_>> slots, guarded by four runtime expects ("translated twice", "finished twice", "armed without a guard", "release without a guard"). It also relies on translating[at] matching the order in which units were pushed.

  • What the test cannot see. It never sees the interpreter. Mapping Step::Release to arm() would leave it green. The arm-removed control shows only that order changed.

  • Why the premise is gone. The old exit asked for a host test "because no machine reaches the path". The T14 now reaches it on every boot.

  • Fix. Write the sequence directly:

    if let Some(g) = &guard { g.arm(); }
    let translating: Vec<_> = ready.into_iter().map(|(u, p)| translate(u, p, devices, &mut domains)).collect();
    if let Some(g) = guard { g.release(); }
    for t in translating { finish(t, devices, remap); }

    Moves make every unit translate once and finish once, and inspection establishes the order. Delete Step, order and their two tests. The real order is read on metal by the judge, once B5 is fixed. Keep largest and its tests: that is pure arithmetic over firmware input, and the right thing to host-test.

B5. BLOCKER: tests/toyos.rs:5991, 6040-6048. The judge is red on AMD-Vi and blind to a missing arm.

  • (a) Red on AMD-Vi. guarded is .all() over lines matching "iommu: unit" and " cap=". AMD-Vi lines read iommu: amdvi unit…, so on an AMD-Vi machine the set is empty and guarded is vacuously true. The judge then demands the arm and release lines and fails. The row's own comment (:1300-1304) says it judges AMD-Vi machines.
  • (b) Blind to a missing arm. "armed" matches iommu: protected memory for the hand-over on. That line is printed by pmr::program (pmr.rs:69-75), before Arm. With the arm step deleted, which is exactly B4's mutation, the judge stays green.
  • Fix.
    • Require at least one VT-d unit line before treating the machine as guarded.
    • Log in Guard::arm, and judge that line before the first "goes off first".
    • Judge the release before the first translating gsts= line. finish prints that line, after the release.

B6. BLOCKER: false, stale or unclear comments.

  • pmr.rs:1-4. "While a unit's translation is off, it blocks DMA to them" is wrong. On these units the regions block DMA whatever the translation state (§11.4.8.1), and that is why holding them until the last unit translates means anything. Say that.

  • dmaguard.rs:1-8.

    • The header runs on: "since a gap may decode as MMIO, above a floor the caller's platform sets" attaches the floor to the wrong clause.
    • "never more than that" is ambiguous.
    • "the order the hand-over takes" goes with B4.

    One sentence is enough here, because largest's doc already states the contract.

  • dmaguard.rs:126. "The T14's shape: RAM up to its graphics region" claims something nobody has checked. The T14's map has 76 usable entries (r2 pmm: line), and nothing shows it is one stretch from 1 MiB. Drop "The T14's".

  • mod.rs:97. "A unit past enable" refers to enable, which this change deletes.

  • mod.rs:798. "ECAP.QI; absent falls back to CCMD_REG/IOTLB_REG" is false: plan refuses such a unit. It was already wrong, but this change gives CCMD_REG its first use, which makes it actively misleading.

  • queue.rs:718-720. "Before the queue exists" is false under leave_on, which creates the queue first. "Before the queue is enabled" is accurate.

B7. BLOCKER: the records.

  • The merged issue's title (:7) says a unit without regions passes DMA untranslated. Its own body lists what the T14, which has them, still passes: RAM below 16 MiB, broken stretches, RMRRs and peer MMIO. Retitle it, for example "A unit passes DMA untranslated outside its protected memory regions while it is handed over".

  • Its exit (:53-60). "A host test or a machine reads each case" leaves "each case" undefined. Name the cases.

  • the-t14s-igpu-faults-…md:80-83 states "about one boot in 27" as fact.

    • The 22/600 rate was measured per cycle on the measurement branch: queued invalidation, no regions, no PRS drain.
    • Boot history on the pre-PR sequence is 1/46.
    • The head's own rate is unmeasured.

    Label it as an estimate.

  • issues/a-unit-this-kernel-does-not-program-keeps-what-firmware-left-on.md:10 names enable, which is deleted. This change also adds plan refusals, which widens the set of units that issue describes. Update it in this change.

Evidence

  • Host, accepted.

    • Lint (exit 0) and the dmaguard tests (exit 0) ran at the head's tree.
    • The full --ci host ran before two kernel-binary-only fixes. Its one red was kernel clippy on the since-deleted invalidate_all, and its FAILED lines are the loom controls.
    • Lint at the head compiles the kernel with warnings denied.
  • QEMU, accepted.

    • vtd_firmware_left and iommu_virtio_platform at the head exercise register-based invalidation on QEMU's unit (a refusal would assert) and the shipping path.
    • The deadlock red-then-green is a meaningful control.
  • Gap: the regions have never run. QEMU's unit has neither region (vtd_firmware_left asserts it). The probe, the spans, PRS and the release are therefore unexecuted on any machine. The staged image at 902bffb is invalidated by B1–B5, which change the kernel and the judge, so do not spend a T14 run on it. Re-stage after the fixes as one batch:

    • shared-debug ×3: the actuator route, the arm and release order, coverage of the tables, and the iGPU fault under the head's own sequence;
    • testcases ×1: the shipping route. Its rows that drive the stick, USB and NVMe after the release are the evidence that nothing stays blocked.

    This is required before landing.

  • Reuse. The PR's "Reused: r2's T14 shared-debug ×3" refers to boots of the deleted keep-on branch. Together with main 576bebf, they establish that the 5 shared-member reds are independent of this change, and nothing about this head's hand-over. The description should say that (for the orchestrator).

Suggestions

  • S1. Probe PLMLIMIT and PHMLIMIT as well (§11.4.8.3/.5 give each limit its own N), and take the coarser alignment. Two lines.
  • S2. When both spans are None, Arm turns on regions that cover nothing. Log that hand-over as unguarded.
  • S3. §11.4.8.1 makes the regions effective only when they are "consistent across all" units. Program them on every described unit that has them, not only on planned ones, or log the hand-over as unguarded when fewer units are ready than described.
  • S4. Have identity_domain take the held &mut Tables, so that one lock guard serves both builds and the self-deadlock cannot be written.
  • S5. Build the RAM set from pmm-usable memory plus ACPI reclaim, leaving out ACPI NVS, which is firmware's memory. Also reconsider the 16 MiB floor: the map already marks the legacy ranges and an enabled ISA hole as non-RAM, and the floor leaves the pmm's records unprotected.
  • S6. In the merged issue, record that on a machine whose firmware leaves TE off (the T14's real case), every function reaches all memory from ExitBootServices until Arm. That bounds what arming the regions on a shipping boot buys.
  • S7. At Arm, log the enumerated functions that have BME set. That makes "silently blocked" auditable on every boot, at the cost of one config-space read per function.

The owner question (for the orchestrator)

The question as framed needs four corrections:

  • "Hardware behaviour" is not established. The cause is unattributed: the issue's own exit is "fix this kernel's sequence, or cite an erratum". Ask whether to keep it recorded as an open defect that only the actuator reaches.
  • "1 in 27" is an estimate. It is the measurement branch's per-cycle rate for a different sequence, against a boot history of 1/46. The head's rate comes from the re-staged shared-debug boots.
  • State the practical cost. A halt kills the whole shared-debug boot. Roughly one nightly run in 27 to 46 then loses every row batched with it, not only iommu_firmware_left. A third option contains that: move the actuator row, or just unit0's part of it, into its own boot. A known destructive outcome is a valid reason for a separate boot.
  • Option 2 is weaker than stated.
    • Stopping the iGPU's bus mastering only when firmware left TE on would be production code that only the actuator ever reaches, the shape r2's N1 rejected.
    • Applied on every boot instead, it blanks the panel on shipping boots for no shipping benefit.
    • Either way it clears BME on a device an RMRR names while firmware's display engine is reading through it, which is the situation §3.4.4 warns about.
    • Whether the pipe recovers without a modeset is unverified, so "the panel blanks" may understate the cost. It needs a T14 recovery measurement before it is offered.

SEND BACK

…l, and a unit the window cannot serve is unguarded rather than refused

B1: the identity domains and every root are built before the regions are
chosen. Each region is the largest stretch of the OS's RAM, clear of the
RMRRs, that holds every 2 MiB page those tables were carved from and this
kernel's image, which iommu::init now takes. The boot logs what the regions
cover. Where they cannot hold all of it, it logs the hand-over unguarded
and does not arm them. The RAM is usable or ACPI-reclaimable, never NVS, and
the 16 MiB floor goes: the map marks the legacy ranges as non-RAM.

B2: plan refuses no unit for the hand-over's sake. A unit that cannot take
register-based invalidation (major version 6 or later, or IOTLB registers
outside the window) makes the machine's hand-over unguarded, and an
unguarded hand-over invalidates through the queue, enabled before
translation comes on, as main does. leave_on is main's again.

B3: protected memory firmware left on goes off before anything is probed or
programmed (§11.4.8.2 to §11.4.8.5).

B4: the step interpreter and its tests go; init runs arm, every translate,
release and every finish in sequence, the units moving through by value.
largest keeps its tests, now with the ranges a region must hold.

B5: Guard::arm logs, and the judge requires that line before translation
first goes off and the release before the first unit is programmed. It
treats a machine as guarded only when it has VT-d unit lines, so an AMD-Vi
machine is not. Judged on an r3 T14 boot, which armed without that line,
it is red.

B6 and B7: comments and records. The merged issue is renamed to what it
holds, a unit passing DMA untranslated outside its regions, and its exit
names its four cases. It records that a machine whose firmware leaves
translation off is open to every function from ExitBootServices until the
regions go on. The iGPU issue labels its rate an estimate.

Suggestions taken: the limits are probed with the bases (S1); a hand-over
with nothing covered or fewer units programmed than described is unguarded
(S2, S3); identity_domain takes the held tables, so the self-deadlock cannot
be written (S4); the RAM set as above (S5); the functions mastering the bus
are logged as the regions go on (S7).

The orchestrator's ruling: the iGPU fault stays an open defect only the
actuator reaches. Its row rides a boot of its own, iommu-firmware-left, so a
halt costs only that boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

r3 response (head c6a9c99)

  • B1: the identity domains and roots are built before the regions are chosen. Each region is the largest RMRR-free stretch of OS RAM that holds every tables page and the kernel image (iommu::init now takes the image range). The boot logs the coverage. If anything is uncovered, it logs the hand-over unguarded and does not arm.
  • B2: no refusals in plan. A unit that cannot take register-based invalidation makes the machine unguarded. An unguarded hand-over invalidates through the queue, enabled before TE as on main. leave_on is main's again.
  • B3: pmr::hand_over switches off protected memory firmware left on, before any probe or write.
  • B4: Step, order and their tests are deleted. init runs arm, every translate, release and every finish directly, by value. largest keeps its tests and gains the ranges a region must hold.
  • B5:
    • Guard::arm logs.
    • The judge requires arm < first translation-off and release < first translating gsts=.
    • It treats a machine as guarded only when it has VT-d unit lines.
    • Red on r3's T14 boot, which armed without the line.
  • B6:
    • the pmr.rs header now says the regions block DMA whatever the translation state on v2+/Skylake+;
    • the dmaguard header is one sentence;
    • "The T14's" is dropped from the test;
    • the Live doc no longer names enable;
    • the ECAP.QI doc and the queue.rs header are fixed.
  • B7:
    • the merged issue is renamed to its claim, and its exit names four cases;
    • the iGPU rate is labelled an estimate;
    • a-unit-this-kernel-does-not-program-… is updated.
  • Suggestions: S1 to S5 and S7 are taken. S6 is in the merged issue.
  • Owner question: per the orchestrator, the fault stays an open, unattributed defect that only the actuator reaches. The actuator row rides its own boot, and there is no bus-mastering stop.

Evidence: https://github.com/ToyOSOrg/evidence/tree/d900761951df99a25bdc1b3e7c4e73786b6be84b

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review r4: igpu-fault at c6a9c99

Base. origin/main 2a24fcf, the merge base.

Read.

  • The r3→head delta in full: vtd/mod.rs, pmr.rs, dmaguard.rs, queue.rs, table.rs, the iommu::init signature chain, the judge, the new FIRMWARE_LEFT arm and the issues.
  • Evidence d9007619, plus 90644380, which holds r3's T14 boots at 902bffb.

Size. The branch is now +829/−241 against main. Kernel and lib account for +577/−77; that includes dmaguard's host tests.

Assessment

The structure is now the right one.

  • Coverage. The regions are chosen to hold what they protect, and the boot falls back to unguarded, logged, when they cannot.
  • Invalidation. The mechanism is selected per machine, and no unit is refused.
  • Sequence. The hand-over runs directly and by value, so moves enforce "once each".
  • Isolation. The actuator's known halt is confined to its own boot, per the orchestrator's ruling.

No blocker remains in the code. Landing waits on the running T14 batch, with the pass criteria below.

Previous blockers

B1 (coverage): resolved.

  • init builds every domain and root before choosing the regions (mod.rs:305-313).
  • cover is every Tables page plus the kernel image (:346).
  • largest keeps only pieces that hold every cover range starting in its window (dmaguard.rs:50, 61-63).
  • program re-checks each range against both regions and goes unguarded, with the range logged, if one is uncovered (pmr.rs:689-693). A range that straddles 4 GiB fails closed.
  • On the guarded route nothing allocates from Tables between cover and the release:
    • translate takes its root by value and makes no queue;
    • the queue is made in finish, after the release.
  • New host tests cover the smaller part that holds the tables, cover ranges split across pieces, and cover under an RMRR.
  • At r3's head the T14 logged its low region as 0x1000000..0x8ef00000, which held the tables at about 26 MiB and the kernel at 0x5e400000. With the floor gone, it may now start lower.
  • Note. The tables are now written before the regions go on, so they sit in the ExitBootServices-to-Arm exposure the merged issue now records, alongside the kernel image. That is equivalent to r3, and better than r3's possibly uncovered tables.

B2 (refusals): resolved.

  • plan refuses nothing new.
  • invalidates_through_registers (mod.rs:412) feeds the guard's preconditions, and a unit that fails them makes the machine unguarded (pmr.rs:640-645).
  • The unguarded route makes the queue after hand_over and before SRTP, then invalidate_all, then TE (mod.rs:491-502). That is main's order, and it satisfies §6.5.2.
  • The guarded route enables the queue after the release, once every register request has completed.
  • leave_on is main's again.
  • QEMU now covers the queue route. The register route runs only on the T14, which takes it on every boot.

B3 (PRS left on by firmware): resolved.

  • pmr::hand_over runs on every ready unit before any probe or write (mod.rs:301-303). It reads PRS, logs, and switches the regions off through set, which waits for PRS.
  • On a unit without the regions, PMEN is read-only (§11.4.8.1), so the read is harmless.
  • Units plan refuses keep firmware's regions on. a-unit-this-kernel-does-not-program-… now says so.

B4 (step interpreter): resolved. Step, order, their tests and the four expects are gone. mod.rs:353-365 reads directly: arm, translate each, release, finish each.

B5 (judge): resolved.

  • guarded needs at least one VT-d unit line, so AMD-Vi's iommu: amdvi unit lines no longer make it vacuously true (toyos.rs:6000-6001).
  • "Armed" is now Guard::arm's own line (pmr.rs:147).
  • The order checked is arm, then the first translation-off, and the release before the first translating gsts=, which finish prints after the release.
  • The control boot from r3, which armed without the new line, goes red (exit 1).

B6 (comments): resolved.

  • The pmr.rs header now states the version-2/Skylake scope.
  • The dmaguard header is one line.
  • The test fixture no longer says "The T14's".
  • The Live, ECAP.QI and queue.rs header comments are now true.

B7 (records): resolved.

  • The merged issue's title states its claim, and its body matches the code: the four reasons for an unguarded boot, what a guarded boot leaves open, and the ExitBootServices window (S6).
  • Its exit names four cases, each to be closed by a documented mechanism.
  • The iGPU issue labels its rate as an estimate between 1 in 27 and 1 in 46, and records the separate boot.
  • The enable reference is updated, and the regions are added to what an unprogrammed unit keeps.
  • The deleted issue has no remaining references.

Suggestions taken

All verified:

  • S1: the limit probes;
  • S2: a region pair that covers none of the hand-over's ranges now falls back to unguarded;
  • S3: described versus programmed;
  • S4: one held &mut Tables;
  • S5: OS RAM only, no NVS, and no floor;
  • S7: the bus-master line before Arm.

Owner ruling

Implemented as ruled.

  • Its own boot. iommu_firmware_left rides FIRMWARE_LEFT: the actuator kernel, testcases.toml, and only iommu-firmware-left among its parameters.
  • Removed from the shared boot. iommu-firmware-left is out of SELFTEST_PARAMS.
  • Flashing. The parameter is still in FLASHABLE.
  • No other dependents. No other row or record depended on the actuator riding shared-debug.

Evidence

  • Accepted at c6a9c99:

    • lint, exit 0;
    • dmaguard host tests, exit 0;
    • QEMU, exit 0, for vtd_firmware_left (the unguarded queue route), iommu_virtio_platform (the shipping route) and the two claims rows;
    • the judge control, red as expected.
  • Required before landing: the running T14 batch at c6a9c99. The register route and the new coverage have no other machine. The batch should show:

    • iommu-firmware-left ×3:
      • each boot logs that the regions cover the tables and kernel, and protected memory on for 4 unit(s), with no hand-over is unguarded line;
      • the judge passes;
      • a boot halted on the 00:02.0 DMA FAULT is the recorded defect, so count it, but no other red is acceptable.
    • shared-debug ×3: only the 5 shared-member reds already shown on main 576bebf.
    • testcases and testcases-watchdog:
      • the same lines on the shipping route;
      • the row results as at r3: 22 pass, plus the same 4 known reds, which are independent of this change.

    An unguarded line on the T14 would make the issue's "the T14's four units are guarded" false. It would also undo B1's premise there, so that outcome would send the PR back.

Suggestions (optional)

  • S1. Make "no Tables allocation inside the window" an invariant rather than a reading of today's code. For example, Guard could keep the page count program saw, and release could assert it is unchanged.
  • S2. In the pmr.rs header and the coverage log line, "this kernel" means its image. The heap and CPU page tables are covered only if they happen to be. Say "this kernel's image".
  • S3. In dmaguard::largest, the parameter within reads like the window bounds. Something like hold would say what it is.
  • S4. When the kernel logged hand-over is unguarded on a machine whose units all report pmr=y, have the judge quote that reason rather than "they did not go on".
  • S5. The kernel image range now travels through generic iommu::init and the AArch64 SMMU signature, whose doc names VT-d, only for VT-d's use. If mm can give the image's extent, vtd could read it there and leave both signatures as they were.

LAND, provided the T14 batch meets the criteria above.

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator: landing at c6a9c99. Review r4 LAND, conditional on the T14 batch; the batch meets every condition. iommu-firmware-left ×3: covered, armed on 4 units, no unguarded line, judge passes, no 00:02.0 halt. shared-debug ×3: only main's 5 shared-member reds. testcases and testcases-watchdog: the same lines, 22 passing plus the known reds (each shown independent of this branch). Linux's NIC was ok after all 8 boots. The 00:02.0 off/on fault stays an open, unattributed defect that only the actuator reaches, in its own boot (orchestrator ruling).

@Japabu
Japabu marked this pull request as ready for review October 28, 2026 14:03
@Japabu
Japabu added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 9d818b3 Oct 28, 2026
10 checks passed
@Japabu
Japabu deleted the igpu-fault branch October 28, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant