vfio-pci: hot reset when the device has no FLR, and the system reset handler

[tamnd]

Part of #9.

vfio-pci now resets a device the way QEMU 11.1 does when the device has no FLR. vfio_pci_reset() tries FLR first. Without it, it tries a bus or slot reset through VFIO_DEVICE_PCI_HOT_RESET, and falls back to the PM reset of VFIO_DEVICE_RESET only when nothing else worked. On a system reset, vfio_device_reset_handler() runs before any device is reset and resets together the devices in use that share a bus.

What is in it:

• ruvm-sys: VFIO_DEVICE_GET_PCI_HOT_RESET_INFO, which reports how many devices a hot reset takes along even when it fails with ENOSPC, and VFIO_DEVICE_PCI_HOT_RESET with the group descriptors as proof of ownership. The group descriptor is now reachable through AsFd.
• vfio-pci:
  • vfio_pci_pre_reset() and vfio_pci_post_reset(). The first stops interrupts and DMA, clears INTx Disable and brings the device to D0. The second turns INTx back on and clears the BARs.
  • vfio_pci_compute_needs_reset() and vfio_legacy_pci_hot_reset(), with QEMU's errors for a device with no reset mechanism and for a dependent group that is not owned.
  • A list of the realized devices, so a reset can find the other devices in use that it takes along.
• q35: early reset hooks, which run before any device is reset. That is where QEMU's qemu_register_reset() handlers run, since the system bus reset is registered only after every -device is realized. The first vfio-pci registers the vfio reset handler there.
• vfio-pci no longer resets the device at realize. QEMU doesn't, and the first system reset, which machine_done() runs once every device is plugged, resets it on both the TCG and KVM paths. ruvm has no device_add, so there is no hotplug path that would need its own reset after realize.

Checked:

On server2, on c64ade1, the branch with main up to #343 merged in:

• clippy with -D warnings and --all-targets for ruvm-mem, ruvm-hw-iommu, ruvm-sys, ruvm-hw-pci, ruvm-hw-vfio, ruvm-machine-x86, ruvm-machine-arm and ruvm-system.
• clippy for x86_64-pc-windows-gnu and aarch64-apple-darwin on the same crates.
• tests of ruvm-mem, ruvm-hw-iommu, ruvm-sys, ruvm-hw-pci and ruvm-hw-vfio, and the ruvm-machine-x86, ruvm-machine-arm and ruvm-system libs (40, 36 and 113). A new ruvm-sys test decodes the dependent devices of a hot reset info reply. ruvm-machine-x86 has 40 tests, not 43 as at ce49dd2, because machine-arm: migrate fw_cfg and the ACPI build state of virt #340 and machine-arm: migrate the ACPI GED of virt #342 removed three.
• xtask unsafe-audit, provenance, layers and style.
• An L1 compare on the release build of c64ade1. L1 is QEMU 11.1 with an e1000e behind a pcie-root-port at 01:00.0, bound to vfio-pci. L2 is run with -device vfio-pci,host=0000:01:00.0,addr=2 under qtest and strace, as QEMU 11.1, ruvm and QEMU 11.1 again, one at a time. qtest sets RAL0 of the e1000e, resets the system through 0xcf9, waits until BAR0 reads 0, maps BAR0 again and reads RAL0 and RAH0. The runs waited until the host load was under 16 and ran at a host load between 15 and 21.

The ioctls on the device descriptor are the same in all three, at start and again after the 0xcf9 reset:

Step	Ioctl	Result
reset handler	GET_PCI_HOT_RESET_INFO, no room	ENOSPC
	GET_PCI_HOT_RESET_INFO	0, no other device in use, so nothing to do
device reset	GET_PCI_HOT_RESET_INFO, no room	ENOSPC
	GET_PCI_HOT_RESET_INFO	0
	PCI_HOT_RESET	ENOTTY
	DEVICE_RESET	0

All the answers are the same in the three runs. RAL0 reads 0x12005452, reads back 0xa1b2c3d4 after the write, and reads 0x12005452 again after the reset, with RAH0 0x80005634. So the device really was reset. ruvm took two BAR0 polls to see the reset where QEMU took one.

Before this change, ruvm reset the device at realize too, which added four calls at start (ENOSPC, 0, ENOTTY, 0) before the sequence above. The compare that found it ran on 919a7c1.