hw-vfio, intel-iommu: DMA through the vIOMMU for vfio-pci

[tamnd]

Part of #9.

vfio-pci can now sit behind intel-iommu. The device does DMA into its own address space, which holds the vIOMMU's region once the guest turns DMAR on, and the vfio container follows that region with an IOMMU notifier the way hw/vfio/listener.c does. When the guest maps a page and invalidates, the container maps the IOVA to the guest page in the host IOMMU, and when it takes the page away, the container unmaps it. This needs caching-mode=on, as in QEMU.

What is in it:

• ruvm-mem: IOMMU notifiers, IOMMUNotifier with memory_region_register_iommu_notifier(), memory_region_unregister_iommu_notifier(), notify_flag_changed and replay, and FlatRange::iommu() so a listener can find the region's callbacks.
• intel-iommu: the notifier side of hw/i386/intel_iommu.c. Each device has its IOVA tree, vtd_page_walk() sends map and unmap events, and vtd_sync_shadow_page_table() runs after context, domain, page and global invalidations, and after TE and SRTP. vtd_iommu_notify_flag_changed() refuses MAP without caching mode with QEMU's error, and vtd_dev_set_iommu_device() refuses assignment without caching mode, a second set and fsts=on.
• q35: the device address space switches between the nodmar alias and the IOMMU region as DMAR and pass through change, as vtd_switch_address_space() does.
• vfio-pci: the container handles IOMMU sections with vfio_iommu_map_notify(), with QEMU's checks for the target address space and the granularity and its map and unmap errors. A section of the whole 2^64 space is unmapped with VFIO_DMA_UNMAP_FLAG_ALL when the kernel has VFIO_UNMAP_ALL, or in two halves when it does not. realize calls pci_device_set_iommu_device() after the config setup and unsets it on a later failure, with the "Failed to set vIOMMU: " prefix.
• system: vfio-pci with intel-iommu is no longer refused. It still is with amd-iommu and virtio-iommu, which send no notifications yet.

Where it differs from QEMU:

• Notifications go out after the intel-iommu lock is dropped, in QEMU's order. The page walk itself runs under the lock.
• QEMU's IOMMU region has the interrupt window at 0xfee00000 as a subregion, so vfio sees two IOMMU sections. ruvm has one 2^64 section, so the unmap at teardown is one unmap of everything.
• ruvm's intel-iommu does not print QEMU's error_report_once() lines for a failed translation ("detected sspte permission error", "detected translation failure", "New fault is not recorded due to compression of faults"). The L2 here never hits them, and that is left for a later change.

Checked:

All on 2cc01b5, the branch with main merged in, on server2:

• clippy with -D warnings and --all-targets for ruvm-mem, ruvm-hw-iommu, ruvm-sys, ruvm-hw-pci, ruvm-hw-vfio, ruvm-machine-x86 and ruvm-system.
• clippy for x86_64-pc-windows-gnu and aarch64-apple-darwin.
• tests of ruvm-mem, ruvm-hw-iommu, ruvm-sys, ruvm-hw-pci, ruvm-hw-vfio and the ruvm-machine-x86 lib. Two new intel-iommu tests check the caching mode errors, and the map and unmap events on replay and domain invalidations.
• xtask unsafe-audit, provenance, layers and style.
• An L1 compare. L1 is QEMU 11.1 with intel-iommu,caching-mode=on and an e1000e at 00:04.0 bound to vfio-pci. L2 is run with -device intel-iommu,caching-mode=on,intremap=off -device vfio-pci,host=0000:00:04.0,addr=2 under qtest, as QEMU 11.1 (Q), ruvm (R), QEMU again (Q2) and ruvm before this change (P). Each batch waits for all of its answers. In L2, qtest writes a root table, a context entry for 00:02.0 in domain 1 and a 3 level table mapping IOVA 0x40000000 to the TX ring and 0x40001000 to a buffer, then sets SRTP, invalidates and sets TE. The e1000e then sends from IOVAs only.

Step	QEMU 11.1	ruvm	ruvm before
TX from IOVA 0x40001000	TDH 1, DD set	TDH 1, DD set	refused at start
map 0x40002000, domain invalidation, TX from it	TDH 2, DD set	TDH 2, DD set
unmap 0x40001000, domain invalidation, TX from it	L1 DMAR fault at 0x40001000, TDH 3	L1 DMAR fault at 0x40001000, TDH 3
caching-mode=off	Failed to set vIOMMU: Device assignment is not allowed without enabling caching-mode=on for Intel IOMMU.	same

All 29 answers are the same in Q, R and Q2. GSTS reads 0xc0000000 after TE and the fault status stays 0 in L2, since the fault for the unmapped IOVA happens in L1. The outer e1000e ignores the failed read, so TDH still moves to 3 and DD is set in both. For the unmapped IOVA, the L1 kernel logs the same fault in all three runs: "DMAR: [DMA Read NO_PASID] Request device [00:04.0] fault addr 0x40001000 [fault reason 0x06] PTE Read access is not set". The outer QEMU also prints its vtd lines for it, but with error_report_once(), so only the first run in an L1 boot shows them. A second L1 boot on the same build with R run first shows them during R, the same 29 answers for R and Q, the same L1 DMAR fault for both, and the same caching-mode=off error.