M8: Passthrough, vIOMMU, confidential

The ninth milestone in spec/23-milestones.md. Device passthrough, virtual IOMMUs and confidential computing.

Passthrough

ruvm-hw-vfio with the legacy container and iommufd backends
Quirk tables, display passthrough, vfio migration
vfio-user client and server

vIOMMUs

intel-iommu with scalable mode, amd-iommu
smmuv3 including accelerated nested mode
virtio-iommu and riscv-iommu

Confidential computing

SEV, SEV-ES and SEV-SNP with IGVM and reset
TDX with guest_memfd
Arm CCA realms as far as Linux and QEMU support them
CET virtualization
swtpm backend with measured boot

Exit criterion

A physical NIC and an NVMe device work under vfio-pci on both backends, and a GPU passes through to a Windows guest with the driver loaded and a display.
vfio migration of a device that supports it works QEMU to ruvm and back.
SEV-SNP and TDX guests boot with attestation reports verified by the vendor tools.
The QEMU functional tests for intel-iommu, smmuv3 and virtio-iommu pass.

Depends on

M2 and M5.

Spec

spec/16-vfio-iommu-cxl.md, spec/19-security-and-confidential.md.

Landed #264, virtio-iommu on q35.

It ports the virtio-iommu device from QEMU 11.1 with its properties (boot-bypass, granule, aw-bits, reserved-regions) and errors, adds virtio-iommu-pci on q35 with the MSI reserved region, and builds the VIOT table. Virtio PCI devices with iommu_platform=on and the AHCI controller now do their DMA through a per function address space behind the vIOMMU.

What was compared on server2 with TCG, QEMU 11.1 against ruvm, booting Alpine 6.6 with -vga none -device virtio-iommu-pci -device virtio-blk-pci,drive=d0,iommu_platform=on,disable-legacy=on:

dmesg IOMMU lines are the same on both: VIOT of 0x58 bytes, translated default domain in lazy mode, 39 input address bits, page mask 0xfffffffffffff000, 00:02.0 in group 0 and 00:1f.2 in group 1, both DMA domains.
The VIOT table also matches QEMU's tests/data/acpi files byte for byte for q35 and arm virt in the unit tests.
DMA through the vIOMMU: md5 of /dev/vda is 3567f26cc5dd92ed39630707294a019a on both, a 4 MiB random write reads back intact, and there are no faults.
Eleven -device error cases print the same message on both.

Next is virtio-iommu on arm virt.

Landed #266, virtio-iommu on arm virt.

The board now takes -device virtio-iommu-pci with QEMU's pre_plug checks and the ITS or GICv2m doorbell as the reserved MSI region, writes the virtio_iommu DT node and iommu-map, and adds the VIOT after the IORT. Without an SMMU, PCI functions get their bus master address space at machine done, so a vIOMMU plugged after a device still covers it.

What was compared on server2 with TCG, QEMU 11.1 against ruvm, booting Alpine 6.6 aarch64 with -device virtio-iommu-pci -device virtio-blk-pci,drive=d0,iommu_platform=on,disable-legacy=on:

Three setups: DT with the ITS, DT with gic-version=2 and the GICv2m, and ACPI through edk2 with gic-version=3. The IOMMU dmesg lines are the same on both in each one, including the IORT (0x80 bytes) and VIOT (0x58 bytes) lines in the ACPI run.
DMA through the vIOMMU: md5 of /dev/vda matches (3567f26cc5dd92ed39630707294a019a), a 4 MiB random write reads back intact, no faults, and the virtio-blk MSIs reach the ITS and the GICv2m.
The DT matches QEMU's dumpdtb byte for byte, with and without default-bus-bypass-iommu, and the VIOT and DSDT match bios-tables-test's VIOT and DSDT.viot.
Seven -device error cases print the same message on both.

One thing left as is: ruvm's virt picks GICv3 by default where QEMU picks GICv2 under TCG, because ruvm has no GICv2 MADT yet, so the boots set gic-version on both sides.

Next is intel-iommu with scalable mode.

intel-iommu, part 1 of 3, landed in #270: legacy mode DMA remapping and the DMAR table, ported from QEMU 11.1's intel_iommu.c and x86-iommu.c.

What is in: the register file with QEMU's masks, CAP and ECAP from vtd_cap_init(), root and context lookup, the second stage walk, pass-through and device-IOTLB contexts, the context cache and IOTLB, register and queued invalidation with all descriptor types, the wait descriptor, fault recording and both interrupts, all of QEMU's properties with its realize checks in order, and build_dmar_q35() on q35. Interrupt remapping is not there yet, so intremap=auto resolves to off (QEMU turns it on under TCG) and intremap=on is an error.

What was compared against QEMU 11.1 on server2: Alpine 6.6 with intel_iommu=on, q35 under TCG, -device intel-iommu,intremap=off and a virtio-blk-pci with iommu_platform=on. The DMAR dmesg matches (cap 80d2008c222f0606 ecap f42, queued invalidation, no RMRR or ATSR), as do the IOMMU groups, the DMA-FQ domains, the md5 of the disk read through the IOMMU and the empty fault list. The DMAR table matches QEMU's expected DMAR.dmar byte for byte in a unit test, and the -device errors match for the cases I tried.

Next: interrupt remapping, then scalable mode.

Landed #274, interrupt remapping for intel-iommu. intremap now resolves like QEMU, so a plain -device intel-iommu under TCG has it on, and eim=on works too. The PCI MSIs and the IOAPIC go through the remapping table with QEMU's source ID checks and IR faults, while the HPET and the IOMMU's own interrupts go straight to the APICs as in QEMU.

Compared on server2 against QEMU 11.1 with Alpine 6.6 on q35 under TCG, -device intel-iommu and a virtio-blk behind it: the IOMMU dmesg matches (ecap f00f4a, IRQ remapping enabled in xapic mode, the IOAPIC under the DRHD), and so do the IOMMU groups, the DMA-FQ domains, the disk md5, the write and read back, and the IR-IO-APIC, IR-PCI-MSI and IR-PCI-MSIX lines in /proc/interrupts. No faults on either side. With eim=on both report ecap f00f5a. The DMAR table is 0x70 bytes on both and only its address differs, from the DSDT gap.

Next is scalable mode for intel-iommu.

Scalable mode for intel-iommu landed in #279, which finishes the intel-iommu port. scalable-mode=on now walks the PASID directory and table for PASID 0 and uses second stage tables, and fsts=on switches to first stage tables with the accessed and dirty bit updates. The PASID based IOTLB invalidation works by domain and PASID or by page, and svm=on brings the page request registers. What is still missing is translation for requests with a PASID and the ATS and page request paths behind svm.

Compared against QEMU 11.1 on server2 with Alpine 6.6 on q35 under TCG, intel_iommu=on,sm_on and a virtio-blk disk behind the IOMMU, once with scalable-mode=on and once with scalable-mode=on,fsts=on. The IOMMU dmesg lines are the same apart from the DMAR table address, including ecap 480080f00f4a and 880000f00f4a, queued invalidation, IRQ remapping in xapic mode and the three DMA-FQ groups. The DMAR table is 0x70 bytes on both. The disk md5 matches, a random write reads back on both, /proc/interrupts lists the same remapped vectors, and there are no DMAR faults on either side.

Next is smmuv3 on arm virt.

Merged #277, the first TPM slice: the ACPI side.

crates/firmware/src/acpi/tpm.rs builds the TPM2 table (TIS or CRB start method) and the TCPA table, both pointing at the etc/tpm/log area that firmware fills. It also emits the tpm-tis ISA device AML with the PPI methods (_DSM and the TPM2/TPM3 operation regions at 0xFED45000) and the CRB device. q35 adds the TIS window to the PCI0 _CRS and places the table after HPET, which is QEMU's order.

Q35Acpi has a new tpm field, so the tables are only built once a machine sets it. That wiring comes with the -tpmdev slice. The four TPM blobs in QEMU's bios-tables-test (DSDT and TPM2 for tis.tpm2, and DSDT and TCPA for tis.tpm12) match byte for byte, once the log pointer is patched to the 0x07ff0000 that SeaBIOS used when those blobs were made.

Next is the hw-tpm crate: the swtpm emulator backend over its control socket, plus the TIS register model.

Merged #281, the second TPM slice: the ruvm-hw-tpm crate now has the swtpm backend, the TIS register model and the PPI memory. No machine uses them yet.

The emulator backend follows tpm_emulator.c: it passes swtpm the data socket with SET_DATAFD over SCM_RIGHTS, probes the TPM version, checks the capability bits with QEMU's messages, and handles INIT, STOP, SET_BUFFERSIZE, SET_LOCALITY, CANCEL and the established flag. The TIS is a port of tpm_tis_common.c, covering the five localities, seizing, the STS state machine, the FIFO and the interrupt registers.

Against swtpm 0.7.3 on server2, the integration test sends TPM2_Startup through the TIS, reads SHA-256 PCRs 0 and 1, extends PCR 1 and sees it change. It also checks the TPM 1.2 probe. State migration is not covered yet.

Next: -tpmdev emulator and -device tpm-tis on q35, compared against QEMU 11.1 with swtpm.

Landed #283, which adds -device arm-smmuv3,primary-bus=pcie.0 on arm virt. The SMMUv3 model and iommu=smmuv3 were already in, so this covers the user-creatable device: QEMU's properties and their validation errors, the virt pre_plug, realize and plug checks, the platform bus placement at 0x0c000000 with SPIs 112 to 115, the DT node and iommu-map, and an IORT builder that handles several SMMUs with their own identifiers.

What was compared against QEMU 11.1:

The DTB for -M virt,gic-version=3 -cpu cortex-a57,pmu=off -device arm-smmuv3,primary-bus=pcie.0 matches QEMU's dump.
The IORT matches QEMU's IORT.smmuv3-dev expected file byte for byte.
Alpine booted on both with the SMMU and a virtio-blk behind it, over DT with GICv2, DT with GICv3 and ITS, and ACPI through edk2. The arm-smmu-v3 dmesg lines, the IOMMU group, the DMA-FQ domain, the vda md5 and a 4 MiB write and readback were the same, and over ACPI the guest's IORT hexdump was identical.

Known gaps: accel=on always fails with the KVM error, only one SMMU on pcie.0 is supported since there is no pxb-pcie, and virtio-iommu after the device is rejected where QEMU accepts it. Next up is amd-iommu.

#285 is merged: -tpmdev emulator and -device tpm-tis on q35. With swtpm behind it, an Ubuntu 6.8 guest booted through SeaBIOS reads the same SHA-1 and SHA-256 PCR banks as on QEMU 11.1, including the SeaBIOS measurements in PCRs 0 to 7 and the IMA boot_aggregate in PCR 10. The firmware event log, the TPM2 table and the DSDT are byte for byte the same. 12 of 18 error command lines match QEMU exactly; the others are passthrough, tpm-tis on microvm, and a case where QEMU aborts. Next is tpm-crb on q35.

#323 landed: vfio-pci MSI-X. hw-pci now has the MSI-X vector notifiers from hw/pci/msix.c, and hw-vfio follows vfio_msix_enable(), do_use and vfio_msix_disable(), including the no vector enable, the NORESIZE re-enable and the PBA that shows up while a masked vector has an interrupt pending.

Compared in the L1 guest with the e1000e on vfio-pci, using the same paced qtest script on QEMU 11.1 and ruvm. Entry 3 was set to write 0x4321 to 0x200000 and fired through IVAR and ICS: both wrote 0x4321, both showed PBA 0x8 after a fire on a masked entry and delivered it on unmask, both did the same with the function mask, and both went back to INTx after MSI-X was turned off. L1 showed vfio-msix[3] for both. The ruvm build from before the change wrote nothing. Apart from that, the second QEMU run and ruvm only differ on the GOTCL packet counter.

Next is DMA through the vIOMMU.

#326 landed: SMM on TCG. q35 now resolves smm=auto to on under TCG, as QEMU does. Each vCPU gets the SMRAM address space as its second address space, and a write to APM port 0xb2 raises an SMI. Two fixes were needed to boot the secure boot OVMF: SMI_LOCK in ICH9 now locks the global SMI enable bit, and the x86 translator keeps EIP 32 bits wide in 16-bit code.

Compared against QEMU 11.1 with a swtpm (PCRs, event log, FADT and ACPI checksums): SeaBIOS with SMM on and with smm=off, and the secure boot OVMF with SMM on, are identical. Plain OVMF differs only in two clocksource lines that depend on host timing. The secure boot OVMF with smm=off hangs on both. The pflash secure property is not ported yet.

#329 landed: the pflash secure property. With secure on, a flash write without secure attributes is dropped before it reaches the command logic, and a read returns the stored data, as in pflash_mem_write_with_attrs() and pflash_mem_read_with_attrs(). q35 takes it from -global driver=cfi.pflash01,property=secure,value=on, the way libvirt passes it, and the default stays off as in QEMU. Under KVM, ruvm refuses secure=on for now.

Compared against QEMU 11.1 with the secure boot OVMF: with secure off and on, the guest's busybox devmem probes of the variable store give identical output on both, and the variable store file after boot is byte-identical in both cases.

#330 landed: vfio-pci now works behind intel-iommu with caching-mode=on. ruvm-mem has IOMMU notifiers with register, unregister, notify_flag_changed and replay. intel-iommu has the shadow page walk and sends map and unmap events after context, domain, page and global invalidations and after TE and SRTP. It also refuses MAP notifiers and device assignment without caching mode, with QEMU's errors. On q35 the device address space switches between the nodmar alias and the IOMMU region, and the vfio container follows IOMMU sections like hw/vfio/listener.c, including the unmap of everything at teardown. vfio-pci with amd-iommu or virtio-iommu is still refused.

Compared on 2cc01b5 against QEMU 11.1, with an e1000e at 00:04.0 in L1 bound to vfio-pci and assigned to L2 with -device intel-iommu,caching-mode=on,intremap=off. qtest in L2 sets up a root table, a context entry and a 3 level table, turns on TE and has the e1000e send from IOVAs. Then it maps a second page with a domain invalidation and sends from it, and unmaps the first page with a domain invalidation and sends from it again. QEMU, ruvm and QEMU again gave the same 29 answers: TDH 1, 2 and 3 with DD set, and GSTS 0xc0000000. The L1 kernel logged the same DMAR read fault at 0x40001000, reason 0x06, for the unmapped page in every run. With caching-mode=off both print "Failed to set vIOMMU: Device assignment is not allowed without enabling caching-mode=on for Intel IOMMU." A second L1 boot with ruvm run first showed the same results. ruvm before this change refused the device at start.

Known gap: ruvm's intel-iommu does not print QEMU's error_report_once lines for a failed translation yet. The L2 here never translates a fault itself, so the compare does not cover it.

Next is mmap of the BARs.

#336 landed: -device pcie-root-port on Arm virt, with the functions behind it under the SMMUv3. Each function's IOMMU region works out its stream ID at each translation from the number its bus has then, as smmu_get_sid() does, so the stream IDs follow the firmware's bus numbering. The root port gets an IOMMU region too. The IORT root complex mapping covers the buses behind the root ports, and as in QEMU the ACPI tables are built again on the first fw_cfg read after each reset, so the IORT has the bus numbers the firmware assigned. The DSDT nests the functions behind a root port in the port's device.

Compared against QEMU 11.1 with Alpine linux-virt 6.12.110 and a virtio-blk behind a pcie-root-port, -M virt,iommu=smmuv3,gic-version=3 -smp 1, booted with a device tree and with edk2 and ACPI. In both boots the SMMU probe lines match, the IOMMU groups match (port and disk share a group with the device tree, the disk has its own with ACPI), and a sector written through the SMMU reads back with the same disk md5 before and after, in the guest and on the host image. The IORT is byte-identical (0xec bytes, IDs 0 to 0x1ff to the SMMU). The one difference is QEMU's default NIC at 00:01.0.

Not done yet: an id-less root port, a root port behind a root port, root ports together with virtio-iommu, and sbsa-ref. A first ruvm run with -smp 2 stalled in stop_machine on a heavily loaded server; I did not check whether main does the same.

Next is iommu_platform on virtio-mmio, then SMMUv3 notifiers and replay on the #330 API.

vfio-pci: mmap the BARs into the guest (#337)

#337 landed: vfio-pci now maps its BARs into the guest. The sparse mmap areas are read like vfio_setup_region_sparse_mmaps(), each one is a RAM device subregion of the trapped BAR, the MSI-X table is cut out unless the kernel says MSIX_MAPPABLE, and the dma-buf probe prints QEMU's warnings. INTx turns the mmaps off until a timer on the virtual clock finds it quiet, and x-no-mmap and x-intx-mmap-timeout-ms work as in QEMU.

Compared against QEMU 11.1 on the 7b2159e build, with an e1000e at 00:04.0 in L1 bound to vfio-pci and assigned to L2. With -S, QEMU and ruvm map the same three areas of the device fd, give the same 71 answers with and without x-no-mmap=on, and do 300 BAR reads with about 10 syscalls through the mmaps against about 300 trapped. In a guest that runs, both trap BAR reads after INTx, go back to the mmaps after the 1100 ms timeout, and stay trapped with x-intx-mmap-timeout-ms=0.

Known gaps: under TCG the mapped BARs go through IO callbacks rather than the TLB, there are no accelerator slots for them, and sub page BARs stay trapped.

About the -smp 2 stall mentioned with #336: it is not from #336. I ran the same device tree boot (Alpine linux-virt 6.12.110, -M virt,iommu=smmuv3,gic-version=3 -cpu max -m 1G -smp 2, release builds, one guest at a time on server2 at a load of about 12 to 28) on main before #336 (31dd93b), on d8ef5ac and on QEMU 11.1. 31dd93b cannot plug a device behind a root port, so the 31dd93b and d8ef5ac runs used a virtio-blk on the root bus with the same command line on both. The root port boot ran on d8ef5ac and QEMU. A run counted as hung once a CPU had been stuck 300 s.

31dd93b with the SMMU: 2 runs, 1 hung with both CPUs in multi_cpu_stop, the same signature as the first d8ef5ac run.
d8ef5ac with the SMMU: 3 runs (counting the first one), 2 hung, one in multi_cpu_stop and one with both CPUs idle and softirqs running but nothing else scheduled. The root port boot that did not hang finished with the right disk md5.
31dd93b without iommu=smmuv3: 3 runs, none hung.
QEMU 11.1, root port boot: finished in under a minute, no lockups.

Each run either finished in 2 to 10 minutes or hung early. In every hang it started before the guest probed the SMMU or any PCI driver (no arm-smmu-v3 line yet), at 27 to 50 s guest time. So it is a pre-existing ruvm TCG SMP hang under load. All three hangs had the SMMU on and none of the runs without it hung, but the samples are small and the runs without the SMMU happened at a lower load, so I have not shown that the SMMU matters. I have not looked for the cause.

vfio-pci: report from realize with the -device location, as QEMU does (…

#339 landed: vfio-pci realize now runs with the -device location pushed, as qdev_device_add() does in QEMU, so the dma-buf warning from #337 and the other lines realize prints start with "-device vfio-pci,...: ". In L1 on the 19fd2c7 build, QEMU 11.1 and ruvm print the same line byte for byte, "qemu-system-x86_64: -device vfio-pci,host=0000:00:04.0,addr=2: warning: VFIO dma-buf not supported in kernel, using mmap fallback, P2P DMA will not work", and give the same 71 answers in the part A compare of #337.

Next is hot reset or the device quirks.

#341 (e7c7db9) takes iommu_platform on the virtio-mmio devices of Arm virt, item 3 of the SMMUv3 list. As in QEMU 11.1, no IOMMU sits in front of virtio-mmio, so the device offers VIRTIO_F_IOMMU_PLATFORM and does its DMA in system memory, and the guest sees the bit only on a modern transport (-global virtio-mmio.force-legacy=false).

An Alpine guest with virtio-blk-device,iommu_platform=on showed the same feature bits as QEMU 11.1 on the legacy DT, modern DT and modern ACPI boots, and the same md5s before and after a write. Checks on ad204ac plus the diff are in the PR body.

Next is item 1, SMMUv3 notifiers and replay on the #330 API.

#343 (259922a) adds the SMMUv3 IOMMU notifiers on the #330 API, item 1 of the SMMUv3 list. As in QEMU 11.1, a notifier gets unmap events only: TLBI_NH_VA, TLBI_NH_VAA and TLBI_S2_IPA unmap their range, and TLBI_NH_ASID, TLBI_NSNH_ALL and TLBI_S12_VMALL unmap the whole range. MAP and DEVIOTLB_UNMAP notifiers are refused with the messages QEMU gives.

There is no SMMUv3 replay, also as in QEMU. hw/arm/smmuv3.c has no replay callback, and the only callers of replay are VFIO and vhost-vdpa, which both need MAP notifiers.

Nothing in ruvm registers an unmap-only notifier behind the SMMU yet, so three unit tests cover the events. The root port boot from #336 gave the same md5s and IORT bytes as QEMU 11.1 on DT and ACPI. Next are RMR, ATS, secure state and VMState. Accelerated nested mode and CMDQV come last, since they need host hardware.

The -smp 2 hang now has its own issue, #344, with the run table and backtraces. One correction to my earlier comment: in the hung 31dd93b run only CPU0 was in multi_cpu_stop. CPU1 was in a timer softirq from idle.

vfio-pci: hot reset when the device has no FLR, and the system reset …

#345 landed: vfio-pci resets a device without FLR the way QEMU 11.1 does. It tries FLR, then a bus or slot reset through VFIO_DEVICE_PCI_HOT_RESET with the group descriptors, and the PM reset last. On a system reset, vfio_device_reset_handler() runs before the devices reset and resets together the devices in use that share a bus. The realize-time reset is gone, since QEMU leaves that to the first system reset.

Compared in L1 against QEMU 11.1, with an e1000e behind a pcie-root-port assigned to L2 and a 0xcf9 reset under qtest. QEMU, ruvm and QEMU again make the same six ioctls on the device descriptor at start and after the reset: hot reset info ENOSPC then 0 from the reset handler, the same pair from the device reset, PCI_HOT_RESET ENOTTY in this L1, and the DEVICE_RESET fallback. RAL0 reads its reset value again afterwards in all three.

Still to do for vfio: the device quirks, the iommufd backend, display passthrough and vfio migration.

Navigation Menu

Repository navigation

M8: Passthrough, vIOMMU, confidential #9

Passthrough

vIOMMUs

Confidential computing

Exit criterion

Depends on

Spec

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

M8: Passthrough, vIOMMU, confidential #9

Description

Passthrough

vIOMMUs

Confidential computing

Exit criterion

Depends on

Spec

Activity

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

tamnd commented on Oct 13, 2026

17 remaining items

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 14, 2026

tamnd commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions