NVIDIA: add Arm CCA Realm and RME device-assignment support

[ianm-nv]

Summary

This series adds Arm Confidential Compute Architecture (CCA) Realm support
to the NVIDIA QEMU 11.0 branch. It imports Linaro's QEMU CCA RFC v4, adapts
it to the QEMU 11.0 and NVIDIA kernel interfaces, adds Realm pause/resume,
and integrates Arm RME device assignment (RME-DA).

The NVIDIA follow-up commits validate Realm creation and image loading,
constrain shared-memory DMA, check RME-DA requests, and correct machine
configuration, object teardown, and DMA replay ordering. They also support
NVIDIA kernels whose live KVM_CAP_ARM_RMI number differs from the QEMU
build-time headers.

The series adds the downstream query-cca-capabilities QMP interface used
by NVIDIA libvirt to determine whether host KVM supports Realms.

Series layout

The branch contains 40 linear commits on top of
nvidia_unstable-11.0 (45aa1ef85148):

1. 24 Linaro QEMU CCA RFC v4 commits.
2. 2 Realm pause/resume commits by James Hunt.
3. 8 NVIDIA QEMU 11.0 integration and CPU-Realm correction commits.
4. 4 RME-DA commits by Arto Merilainen, with NVIDIA-maintained fixes folded
   into their final implementations.
5. 1 commit serializing guest-memory attribute changes and notifier replay.
6. 1 Realm DMA synchronization, mapping, and lifetime correction.

Provenance

• Source branch: https://gitlab.com/Linaro/cca-public/qemu/-/tree/upstream-v4
• RFC posting: https://lore.kernel.org/qemu-devel/20260903193611.1058589-1-mathieu.poirier@linaro.org/

The imported commits record their corresponding Linaro source hashes.
Commits requiring changes for QEMU 11.0 are identified as backports and
describe the relevant adaptation.

Main NVIDIA adaptations and fixes

• Discover the live KVM_CAP_ARM_RMI value through the NVIDIA KVM module
  parameter, validate it, cache it, and fall back to the build-time value
  when the parameter is unavailable.
• Align the Arm VM type encoding with the companion NVIDIA kernel and use
  VM-scoped KVM capability checks.
• Correct Realm PMU/debug-resource setup and register synchronization.
• Validate firmware, DTB, and boot-image loading. Snapshot loader data while
  valid, merge page-overlapping images, and bound Realm population to valid
  RAM ranges.
• Provide a stable Realm IPA layout and route only valid shared-memory DMA
  aliases through the Realm DMA address space.
• Add assigned-memory attribute handling and IOMMUFD VDEVICE support for
  RME-DA.
• Treat Realm Host Interface (RHI) and Trusted I/O (TIO) data as untrusted:
  validate lengths, offsets, mappings, residues, routing IDs, and
  device-memory ranges; serialize device access; and fail closed on rollback
  or registration errors.
• Reject unsupported Realm machine options instead of silently accepting
  them.
• Make Realm notifier/listener teardown idempotent, prevent deletion of an
  active RME object, and break the RmeGuest/AddressSpace reference cycle.
• Serialize guestmemfd attribute changes, listener notifications, and replay
  callbacks so DMA mappings cannot become stale during a concurrent memory
  conversion.
• Expose query-cca-capabilities using the response schema consumed by
  NVIDIA libvirt commit 00fe79fd234291db242a4c72c2d3373f8f3dcdb2
  on nvidia_unstable-11.9.

[nirmoy]

BaseOS Kernel Review

Note

🔄 Review in progress

Boro is reviewing this pull request. Results will appear here when ready.

🔍 Review artifacts

• 🟡 Boro review: in progress

Review metadata

• Reviewed head: 70e7658badd4
• Overall status: in progress

This comment is maintained by BaseOS Reviewer and updated when the GitHub watcher publishes a newer review.

[nvmochs]

@ianm-nv I reviewed this PR and confirmed that the issues I had mentioned previously in my cursory review a few weeks ago are now resolved. There are some new findings and a few "informational notes" (findings, but classified a bit differently due to recognition that this is beta code). I'm not sure how actionable these are at this point in time as none of them appear to functionally affect the "good path". Will let you review and then we can discuss how best to proceed.

---

Findings

1. [P1] Realm DMA conversion can succeed after a VFIO map/unmap failure

When a Realm page changes between private and shared, target/arm/kvm-rme.c:560 publishes an IOMMU MAP or UNMAP event. Through the registered notifier chain, this reaches hw/vfio/listener.c:156, which performs the actual VFIO/IOMMUFD mapping operation.

If that operation fails, VFIO reports the error but cannot return it because the generic IOMMU notifier interface returns void. Control eventually returns to target/arm/kvm-rme.c:598, which unconditionally returns zero. As a result,
kvm_convert_memory() finishes as though the device mapping update succeeded.

For a private-to-shared transition, the CPU-visible page can become shared while the assigned device has no corresponding DMA mapping. For a shared-to-private transition, an old device mapping can remain after QEMU considers device access revoked. This breaks the invariant that KVM attributes, QEMU’s RAM-discard bitmap, and IOMMUFD mappings describe the same sharing state.

Replay of existing shared pages uses the same notification mechanism, so a failed replay MAP can also be logged without preventing VFIO device setup from continuing.

2. [P2] Realm boot images cannot cross RAM MemoryRegion boundaries

QEMU can construct contiguous guest RAM from multiple NUMA memory backends:

Guest-visible RAM

  +-----------------------+-----------------------+
  | NUMA memdev 0         | NUMA memdev 1         |
  | MemoryRegion A        | MemoryRegion B        |
  +-----------------------+-----------------------+
                          ^
                    backend boundary

There is no address-space hole at this boundary, so a kernel, initrd, or ELF segment spanning it is still entirely within valid guest RAM. QEMU creates these adjacent backend regions in hw/core/numa.c:627, and Realm machine validation does not reject such a configuration.

The Realm loader records and page-aligns complete boot-image ranges. It can also coalesce overlapping or adjacent loader entries without considering the underlying RAM backend boundaries:

target/arm/kvm-rme.c:200

At VM activation, target/arm/kvm-rme.c:130 passes each complete range to kvm_convert_memory(). That function uses memory_region_find(), which returns only the portion contained in the first leaf MemoryRegion, and then explicitly rejects the range when the returned section is smaller than requested:

accel/kvm/kvm-all.c:3393

The resulting error propagates to the Realm run-state handler, which terminates QEMU during startup. The failure depends on image placement: the same image and memory backends can work or fail depending on whether an image happens to straddle one of the otherwise transparent NUMA boundaries.

3. [P2] Large zero-filled ELF segments create two full-size staging buffers

For an ELF PT_LOAD segment, QEMU distinguishes:

p_filesz: bytes physically present in the ELF file
p_memsz: complete segment size after loading

When p_memsz > p_filesz, the remainder is normally an implicit zero-filled BSS tail. rom_add_elf_program() deliberately retains only the file-backed portion and records the larger in-memory size, avoiding a separate p_memsz allocation:

hw/core/loader.c:1211

The Realm loader notification path materializes that sparse representation into a contiguous buffer. hw/core/loader.c:1300 first allocates and zeroes p_memsz bytes, then copies the p_filesz data into it. While that temporary buffer remains
live, target/arm/kvm-rme.c:324 allocates another p_memsz buffer and copies the complete temporary image into its retained Realm snapshot.

For a segment with 16 MiB of file data and a 1 GiB in-memory size, the notifier path temporarily holds two additional 1 GiB buffers. This is in addition to guest RAM and the mapped ELF file. Under a process or cgroup memory limit, an
otherwise valid ELF can therefore fail during Realm startup.

This applies only to direct ELF loading with a substantial p_memsz - p_filesz difference. Raw Arm Linux Image files and ELF segments without a meaningful zero-filled tail do not experience the duplicate full-size allocation.

Informational notes

4. The beta implementation does not provide RHI_DA_VERSION

The RHI-DA SMCCC filter starts at RHI_DA_FEATURES (0x4b), leaving RHI_DA_VERSION (0x4a) outside the range forwarded to QEMU:

target/arm/kvm.c:610

The RHI dispatcher likewise handles FEATURES, OBJECT_SIZE, OBJECT_READ, and the VDEV operations, but has no VERSION case:

target/arm/kvm.c:2139

Published DEN0148 beta1 describes RHI_DA_VERSION as the protocol-discovery operation. A guest following that sequence would receive SMCCC_RET_NOT_SUPPORTED and conclude that RHI-DA is unavailable.

The RFC v4 Linux guest used with this beta stack does not follow that sequence. It calls RHI_DA_FEATURES directly, which is forwarded and handled successfully. Consequently, the missing version call does not create a current functional problem and is noted only as relevant to future ABI alignment. See the Arm DEN0148 RHI specification (https://documentation-service.arm.com/static/6980b4be9135fd4e42bbf2ed?token=).

5. RHI_DA_OBJECT_READ uses source-offset semantics

The object-read handler maps the guest’s AArch64 argument registers as follows:

x2 / call->in[2] = object type
x3 / call->in[3] = guest buffer IPA
x4 / call->in[4] = maximum length
x5 / call->in[5] = offset

See target/arm/kvm.c:1987.

QEMU interprets x5 as an offset into the source object:

copy_len = MIN(max_len, object_size - offset);
memcpy(hva, object_data + offset, copy_len);

Under that interpretation, an offset of 128 means “skip the first 128 bytes of the object and place the remaining data at the beginning of the guest buffer.”

Published DEN0148 beta1 describes the field as an offset within the destination buffer. Under that interpretation, the complete object would be copied to guest_ipa + offset.

The NVIDIA kernel branch inspected implements the source-offset convention, and its current guest callers always set gprs[5] to zero. With a zero offset, the two interpretations produce exactly the same result. The difference is therefore latent and does not affect the currently inspected beta guest.

[NathanChenNVIDIA]

Hi @ianm-nv ,
Commit 701c144's description may need some details about why the 'backported from' tag was used, i.e. backporting the defines for KVM_VM_TYPE_ARM_PROTECTED and KVM_VM_TYPE_ARM_MASK