FCVM - MicroVM Manager Design Specification (Pluggable Hypervisor Backends: Firecracker + Cloud Hypervisor)
- Overview
- Requirements
- Architecture
- Core Components
- Networking
- Storage & Cloning
- VM Lifecycle
- Guest Agent
- CLI Interface
- Implementation Details
fcvm is a microVM manager designed to run Podman containers inside lightweight microVMs with lightning-fast cloning capabilities. It runs VMs behind a pluggable Hypervisor trait (src/hypervisor/) supporting two backends — Firecracker (default) and Cloud Hypervisor — selected with --hypervisor firecracker|cloud-hypervisor on fcvm podman run. It provides a simple CLI interface for spinning up isolated container environments with:
- Full-featured VMs: Filesystem access, outbound networking, port forwarding
- Fast cloning: Clone running VMs in <1s using snapshots and CoW disks
- Flexible networking: Both rootless and privileged modes
- Process lifetime binding: VM lifetime tied to controlling process
- Resource configuration: Configurable vCPU/memory with overcommit support
Target Platform: Linux only (requires KVM)
-
fcvm podman runCommand- Takes a Docker/Podman container image
- Spins up a microVM (Firecracker or Cloud Hypervisor, via
--hypervisor) running the container - Supports volume mounts via FUSE passthrough (host → guest)
- Supports port forwarding (host → guest)
- Process blocks until VM exits (hanging/foreground mode)
- VM dies when process is killed (lifetime binding)
-
fcvm execCommand- Execute commands in running VMs
- Supports running in guest OS or inside container (
-cflag) - Interactive mode with stdin forwarding (
-iflag) - TTY allocation for terminal apps (
-tflag)
-
fcvm snapshotCommandsfcvm snapshot create: Create snapshot from running VMfcvm snapshot serve: Start UFFD memory server for cloningfcvm snapshot run: Spawn clone from memory server- Lightning-fast clone startup (<1 second)
- Serves guest memory through a UFFD page fault handler; clones share clean pages only in minor mode (
--uffd-mode minor) - Creates independent VM with its own networking
-
Networking Modes
- Rootless: Works without root privileges using pasta (from passt)
- Privileged: Uses iptables + TAP for better performance
- Routed: IPv6 veth pairs with kernel routing at line rate (no userspace proxy)
- Port mapping:
[HOSTIP:]HOSTPORT:GUESTPORT[/PROTO]syntax - Support multiple ports, TCP/UDP protocols
-
Volume Mounting
- Map local directories to guest filesystem
- Support block devices, sshfs, and NFS modes
- Read-only and read-write mounts
-
Resource Configuration
- vCPU overcommit (more vCPUs than physical cores)
- Memory overcommit with balloon device
- Configurable memory ballooning
-
Snapshot & Clone
- Save VM state at "warm" checkpoint (after container ready)
- Fast restore from snapshot
- CoW disks for instant cloning
- Identity patching (MAC addresses, hostnames)
- Performance: Clone startup <1s
- Isolation: Full VM isolation via KVM (Firecracker or Cloud Hypervisor backends)
- Compatibility: Works with rootless Podman in guest
- Portability: Runs on bare metal or nested VMs (VM-in-VM)
- Reliability: Clean shutdown, resource cleanup
┌──────────────────────────────────────────────────────┐
│ fcvm CLI (Host) │
│ ┌────────────┐ ┌──────────────┐ ┌─────────────┐ │
│ │ Networking │ │ Hypervisor │ │ Storage & │ │
│ │ Manager │ │ Abstraction │ │ Snapshots │ │
│ └────────────┘ └──────────────┘ └─────────────┘ │
│ │ │ │ │
│ └────────────────┴─────────────────┘ │
│ │ │
└──────────────────────────┼────────────────────────────┘
│
▼
┌────────────────────────┐
│ Hypervisor Process │
│ (Firecracker / CH) │
│ (microVM) │
│ ┌──────────────────┐ │
│ │ Linux Kernel │ │
│ │ ┌────────────┐ │ │
│ │ │ fc-agent │ │ │
│ │ │ │ │ │ │
│ │ │ Podman │ │ │
│ │ │ │ │ │ │
│ │ │ Container │ │ │
│ │ └────────────┘ │ │
│ └──────────────────┘ │
└────────────────────────┘
-
fcvm CLI (Rust)
- Command-line interface
- Orchestrates VM lifecycle
- Manages networking, storage, snapshots
- Streams logs and handles signals
-
Hypervisor backend (External binary, pluggable via
--hypervisor)- Runs the microVM and manages VM resources (vCPU, memory, drives, network)
- Firecracker (default): REST API over a Unix socket; boot metadata via MMDS
- Cloud Hypervisor: REST API over the
--api-socketUnix socket (/api/v1/vm.create+vm.boot); boot metadata served over vsock (boot-plan port 4995, no MMDS) - Selected behind the
Hypervisortrait (src/hypervisor/); see Core Components for backend-specific details
-
fc-agent (Rust, runs in guest)
- Fetches container configuration (boot plan) via MMDS (Firecracker) or vsock (Cloud Hypervisor)
- Launches Podman with correct parameters
- Streams container logs to host via vsock
- Signals readiness to host
Location: fcvm/src/firecracker/api.rs
Provides Rust interface to Firecracker REST API over Unix socket using hyper + hyperlocal.
Key Functions:
set_boot_source()- Configure kernel + boot argsset_machine_config()- Set vCPU, memory, SMTadd_drive()- Attach rootfs and data disksadd_network_interface()- Setup networkingset_mmds_config()- Configure metadata serviceput_mmds()- Provide container plan to guestcreate_snapshot()- Save VM stateload_snapshot()- Restore from snapshotset_balloon()- Configure memory balloon
API Structures:
struct BootSource {
kernel_image_path: String,
initrd_path: Option<String>,
boot_args: Option<String>,
}
struct MachineConfig {
vcpu_count: u8,
mem_size_mib: u32,
smt: Option<bool>,
track_dirty_pages: Option<bool>,
}
struct Drive {
drive_id: String,
path_on_host: String,
is_root_device: bool,
is_read_only: bool,
}
struct NetworkInterface {
iface_id: String,
host_dev_name: String, // TAP device
guest_mac: Option<String>,
}Location: fcvm/src/firecracker/vm.rs
Manages Firecracker process lifecycle.
Responsibilities:
- Spawn Firecracker process with correct args
- Wait for API socket to be ready
- Stream stdout/stderr to tracing logs
- Handle graceful shutdown
- Clean up resources (socket, processes)
Key Functions:
impl VmManager {
async fn start(&mut self, firecracker_bin, config) -> Result<()>
async fn wait(&mut self) -> Result<ExitStatus>
async fn kill(&mut self) -> Result<()>
async fn stream_console(&self, console_path) -> Result<Receiver<String>>
fn client(&self) -> Result<&FirecrackerClient>
}Location: fcvm/src/network/
Three implementations based on execution mode.
Location: fc-agent/src/network.rs::publish_to_loopback
--publish delivers a packet to the guest's external interface. A guest service
listening on 0.0.0.0 receives it; one listening on 127.0.0.1 does not. fcvm
therefore DNATs each eligible published TCP port to 127.0.0.1:<port> inside the
guest when containment, route_localnet, and that port's DNAT setup succeed.
Port 12345 is ineligible because it is fcvm's egress proxy, and a TCP port also
given to --forward-localhost is rejected before boot. A setup failure leaves
the existing wildcard- and guest-address publication path unchanged and logs
that loopback-only listeners are not reachable.
host:P --publish--> guest eth0:P --DNAT--> 127.0.0.1:P --> the listener
Why unconditional. The rewrite breaks one case — a service bound to the
guest's own address (10.0.2.100:P), whose socket stops matching. Netfilter can
express the exception, -m socket --nowildcard -j RETURN, and the shipped guest
kernel cannot run it: Kata 6.12.47 has no xt_socket, and /lib/modules
holds the rootfs's module tree rather than the running kernel's, so no netfilter
match can be loaded at runtime. Inferring the exception in userspace (poll
/proc/net/tcp, add and remove the rule as listeners change) was written and
rejected — there is no safe moment to decide, because the address a service will
bind does not exist until it binds, so every such design has a window.
The exception is also close to unreachable here. The guest netns has exactly one
external interface with one IPv4 address (eth0 10.0.2.100/24, plus lo), so
binding it and binding the wildcard are equivalent; a specific bind exists to
EXCLUDE other interfaces and there are none. The hostname resolves to nothing
(/etc/hosts carries only 127.0.0.1 localhost), so InetAddress.getLocalHost()
throws rather than yielding the guest address. No listener in this repo binds a
specific non-loopback address.
Known limits: TCP only (a published UDP port to a loopback service stays
unreachable); 127.0.0.1 exactly, so a service on 127.0.1.1 — Debian's
hostname convention, and Akka/Pekko's default — is not matched; and a service
bound to 10.0.2.100:P loses --publish.
Two further details that are not obvious:
route_localnet=1is required. Without it the kernel treats a packet routed to127.0.0.0/8that arrived on a non-loopback interface as a martian source and drops it. fcvm already depends on the same switch host-side —src/network/portmap.rs::enable_route_localnet. It is set oneth0only, notall:allwould also cover interfaces a nested workload creates later.- This direction works where the reverse does not.
setup_localhost_forwarding(guest → host) documents that DNAT is unusable there, because the DNAT'd packet keeps its127.0.0.1source and pasta's L4 translation cannot carry a loopback source out through the TAP. Here only the destination becomes loopback; conntrack rewrites the reply's source back to the original destination, so nothing downstream ever sees a loopback source.
The container inherits this for free: fc-agent always runs it with
--network=host (fc-agent/src/container.rs:1203), so container loopback is
guest loopback.
Why it exists. Chromium's DevTools endpoint ignores
--remote-debugging-address: measured on chromium 151.0.7922.71 (Debian bookworm
arm64), with the flag confirmed in /proc/<pid>/cmdline, /proc/net/tcp shows
0100007F:2406 (127.0.0.1:9222) and nothing else. Before this, the only way in
was a userspace relay process running inside every clone, in the byte path — and
the direct DNAT removes that process and hop. A prior request-path A/B that
reported connection drops was withdrawn because its arms were not comparable;
it does not establish that the relay caused those failures.
route_localnet is per-interface, not per-port. Switching it on makes the
guest's entire loopback listener set reachable from the guest's network
segment, not merely the published ports — measurably: a frame addressed to
127.0.0.1:7777 for an unpublished, un-DNAT'd listener is answered with the
sysctl on and dropped with it off. That is strictly wider than what --publish
opened, and "the microVM guards ingress" does not justify it.
install_loopback_containment closes the gap:
iptables -I INPUT 1 -i eth0 -d 127.0.0.0/8 -m conntrack ! --ctstate DNAT -j DROP
Conntrack records whether a connection's destination was translated, so this
admits exactly the packets our PREROUTING rules rewrote and drops anything an
attacker addressed to 127.0.0.1 directly. xt_conntrack is built into the
Kata kernel — verified in the guest, unlike xt_socket. Scoped to -i eth0, so
locally-generated traffic (which re-enters on lo) is untouched, including the
egress proxy's nat OUTPUT REDIRECT. The rule is inserted at position 1 because
an earlier terminating ACCEPT would bypass an appended security rule.
12345— the transparent egress proxy (fc-agent/src/proxy.rs::PROXY_LISTEN_PORT) listens on guest127.0.0.1:12345and relays out through a host-sideTcpStream::connectthat validates no destination and sets no timeout. DNAT'ing it would hand external clients that relay, sopublish_to_loopbackrefuses the port and warns.- Any port also given to
--forward-localhost. That flag binds a relay on guest127.0.0.1:<port>pointing at the host, so publishing the same guest port makes the published port answer with the host's service instead of the guest's — a successful response from the wrong machine, not an error. Rejected at argument-parse time insrc/commands/podman/mod.rs.
Uses pasta (from the passt project) for L4 splice-based networking.
Features:
- No root privileges required
- Port forwarding via pasta CLI flags (
-tfor TCP,-ufor UDP) - Default guest IP:
10.0.2.100 - Default host IP:
10.0.2.2
Implementation:
struct PastaNetwork {
vm_id: String,
tap_device: String,
port_mappings: Vec<PortMapping>,
pasta_process: Option<Child>,
}
async fn setup() -> Result<NetworkConfig> {
// TAP device created by Firecracker
// pasta started after VM boots
// Port forwarding configured via -t/-u CLI flags
}Uses TAP devices + iptables for native performance.
Features:
- Requires root or CAP_NET_ADMIN
- Better performance than rootless
- Uses DNAT for port forwarding (scoped to veth IP)
- Network namespace isolation per VM
Implementation:
struct BridgedNetwork {
vm_id: String,
tap_device: String,
namespace_id: String,
host_veth: String, // veth_outer in host namespace
guest_veth: String, // veth_inner in VM namespace
guest_ip: String,
host_ip: String, // veth's host IP (used for port forwarding)
port_mappings: Vec<PortMapping>,
}
async fn setup() -> Result<NetworkConfig> {
create_namespace(namespace_id)
create_veth_pair(host_veth, guest_veth)
move_veth_to_namespace(guest_veth, namespace_id)
create_tap_device_in_namespace(tap_name, namespace_id)
for mapping in port_mappings {
// Scope DNAT to veth IP so same port works across VMs
setup_nat_rule(mapping, guest_ip, host_ip)
}
}NAT Rule Example (scoped to veth IP):
iptables -t nat -A PREROUTING -d 172.30.x.1 -p tcp --dport 8080 -j DNAT --to-destination 172.30.x.2:80Uses veth pairs + IPv6 routing for kernel line-rate networking without userspace proxies.
Features:
- Requires root and a host with a global IPv6 /64 subnet (or
--ipv6-prefixto specify one explicitly) - Native IPv6 routing through the kernel stack (no userspace L4 translation)
- Each VM gets a unique IPv6 derived from the host's /64 prefix, or the address
--ipv6-addressnames inside the prefix in use (not its address 0 or 1) - A VM leases its address before using it: an exclusive
flockon/run/fcvm/routed-ipv6/<address>.lock, held until its network is torn down and released by the kernel if the process dies. Of two VMs that ask for one address together, exactly one gets it. If the lease is free but the host already has a /128 route for the address, in any routing table, the route belongs to something other than a live fcvm VM (the host itself, a VM that died without cleanup, another program), so a given address is refused and a derived one is re-derived - Port forwarding via built-in TCP proxy (
setns+ tokio relay): a mapping listens on its HOSTIP, or on the VM's loopback IP (same allocation as rootless) when it names none - Parallel-safe: per-VM routes, proxy NDP, ip6tables rules
Implementation:
struct RoutedNetwork {
vm_id: String,
tap_device: String,
port_mappings: Vec<PortMapping>,
forward_localhost: Vec<u16>, // guest localhost:<port> -> host loopback:<port> relays (127.0.0.1, else ::1)
loopback_ip: Option<String>,
namespace_id: Option<String>,
host_veth: Option<String>,
vm_ipv6: Option<String>,
address_lease: Option<AddressLease>, // held until cleanup has removed everything naming vm_ipv6
default_iface: Option<String>,
proxy_handles: Vec<Relay>,
ipv6_prefix: Option<String>, // explicit /64 prefix (skips auto-detect + MASQUERADE)
ipv6_address: Option<Ipv6Addr>, // --ipv6-address, instead of a derived address
}
async fn setup() -> Result<NetworkConfig> {
self.preflight_check() // root, IPv6, ip6tables (skipped if --ipv6-prefix); rejects /udp publishes
detect_host_ipv6() // find /64 subnet (or /128 with on-link /64); skipped if --ipv6-prefix
generate_vm_ipv6(prefix, vm_id) // deterministic IPv6 from hash, or the --ipv6-address given
lease_address(vm_ipv6) // flock held for the VM's life; refused if held or routed
create_namespace(ns_name)
// Disable DAD inside namespace (accept_dad=0 on all + default)
create_veth_pair(host_veth, guest_veth)
// Suppress host veth link-local autogeneration (addr_gen_mode=1, keep_addr_on_down=1)
create_tap_in_ns(ns_name, tap)
connect_tap_to_veth(ns_name, tap, guest_veth) // bridge for L2
// Assign bridge IPs: 10.0.2.1/24 + fd00::1/64
// Host veth: enable forwarding, assign link-local with nodad (hard error on failure)
// Namespace: default IPv6 route via host veth link-local
// Host: /128 route to VM IPv6 via host veth
// Proxy NDP on default interface
// ip6tables MASQUERADE for outbound (skipped if --ipv6-prefix is set)
// TCP proxy port forwarding (setns + tokio relay): each mapping on its HOSTIP, else on the loopback IP
// --forward-localhost: 10.0.2.2/32 alias on bridge + TCP relays to host loopback ports
}Grammar: [HOSTIP:]HOSTPORT:GUESTPORT[/PROTO]
Examples:
8080:80 # TCP port 8080 → guest:80
127.0.0.1:8080:80 # Bind to localhost only
8080:80/udp # UDP protocol
0.0.0.0:53:53/udp # DNS forwarding
Parsing Logic (network/types.rs):
impl PortMapping {
pub fn parse(s: &str) -> Result<Self> {
// Split on ':'
// Extract optional host IP
// Extract protocol suffix (/tcp or /udp)
// Default to TCP if not specified
}
}Each VM has:
- Kernel: Shared across all VMs (read-only)
- Base rootfs: Shared base image with Podman + fc-agent
- CoW overlay: Per-VM writable layer (using btrfs reflinks)
- Volume mounts: Optional host directory mounts
/mnt/fcvm-btrfs/ # btrfs filesystem (CoW reflinks work here)
├── kernels/
│ ├── vmlinux.bin # Symlink to active kernel
│ └── vmlinux-{sha}.bin # Kernel (SHA of URL for cache key)
├── rootfs/
│ └── layer2-{sha}.raw # Base rootfs (~10GB, SHA of setup script)
├── initrd/
│ └── fc-agent-{sha}.initrd # fc-agent injection initrd (SHA of binary)
├── vm-disks/
│ └── vm-{id}/
│ └── disks/rootfs.raw # CoW reflink copy per VM
├── snapshots/
│ └── {snapshot-name}/
│ ├── vmstate.snap # VM memory snapshot
│ ├── disk.snap # Disk snapshot
│ └── config.json # VM configuration
├── state/ # VM state JSON files
├── image-cache/ # Container image layers (sha256:{digest}/)
└── cache/ # Downloaded cloud images
Goal: Share base rootfs across VMs, only store deltas per-VM.
Options:
-
overlayfs (preferred for simplicity)
mount -t overlay overlay \ -o lowerdir=/base/rootfs,upperdir=/vm/upper,workdir=/vm/work \ /vm/merged
-
btrfs reflinks (current implementation)
cp --reflink=always /mnt/fcvm-btrfs/rootfs/layer2-{sha}.raw /mnt/fcvm-btrfs/vm-disks/{id}/disks/rootfs.raw
Benefits:
- Instant cloning (no disk copy)
- Shared memory pages across VMs
- Fast snapshot restore
Memory Snapshot: Firecracker native format
{
"snapshot_path": "/snapshots/warm/disk.snap",
"mem_file_path": "/snapshots/warm/memory.snap",
"snapshot_type": "Full"
}Clone Process:
- Load snapshot via Firecracker API
- Create new CoW overlay disk
- Patch identity (MAC address, hostname, VM ID)
- Setup new networking (TAP device, ports)
- Resume VM
Identity Patching:
- Generate new MAC address
- Update hostname in guest
- Regenerate machine IDs
- Update MMDS with new config
fcvm snapshot create --disk-only captures only the disk (kind: DiskOnly):
sync → fsfreeze --freeze / over the exec vsock → btrfs reflink of the disk →
fsfreeze --unfreeze /. No vCPU pause, no memory/vmstate files. The podman
container store (btrfs.img loopback) is a file ON the rootfs, so freezing /
freezes it too — the capture is crash-consistent.
Clones of a disk-only tag cold-boot (snapshot run --snapshot <tag> dispatches
to the podman boot machinery with rootfs_override = captured disk). The guest
self-detects the situation via a provisioned marker (/var/lib/fcvm/provisioned,
written by fc-agent after first-boot provisioning): when present, fc-agent skips the
storage wipe and image import, re-mounts the existing storage loopback (never mkfs),
podman starts the captured fcvm-container (preserving its writable layer), and
regenerates per-machine identity (machine-id, SSH host keys). No policies, no MMDS
plumbing — the disk itself carries the signal.
Every VM is configured "reboot possible" up front: each lifecycle path builds a
launch plan (RebootSpec: firecracker binary/args, fully-resolved launch config,
boot args, vsock path, boot-plan transport) before the VM runs, and all paths boot
through one shared primitive, configure_and_boot_vm:
- initial
fcvm podman runboot - disk-only clone cold boot (via
prepare_vmwithrootfs_override) - in-place relaunch after a guest reboot (both the podman run loop and the snapshot-restore run loop)
Reboot detection: a systemd unit in the guest (fcvm-reboot-notify.service,
WantedBy=reboot.target — pulled in only on reboot, never poweroff/halt) runs
fc-agent --notify-reboot, which sends reboot on the vsock status port before
the guest resets. The host's status listener sets a flag and stays alive; when the
Firecracker child exits, the run loop sees the flag and relaunches in place —
same fcvm process/PID, same disk, same network namespace/holder, same listeners
and health monitor. The re-boot lands on the provisioned-marker path, so a reboot
is semantically identical to a disk-only clone cold boot. fc-agent's own
post-container shutdown uses poweroff -f, which never reaches reboot.target,
so normal termination is unaffected. After a relaunch the recorded snapshot parent
is cleared (fresh-boot memory must never be diff-snapshotted against the old
lineage).
Key Insight: pasta and Firecracker CANNOT share a TAP device (both need exclusive access). Solution: Use a Linux bridge (br0) for L2 forwarding between pasta and Firecracker inside a user namespace.
Architecture: pasta uses splice(2) zero-copy L4 translation for inbound port forwarding (host socket to namespace socket) and L2 TAP path for outbound VM traffic.
Topology:
Host │ User Namespace (unshare --user --net)
│
pasta <──────────────────┼── pasta0 ─┐
(L4 splice/TAP) │ │
│ br0 (10.0.2.1/24) ← namespace IP for health checks
│ │
│ tap-fc ─┘
│ │
│ ▼
│ Firecracker VM
│ eth0: 10.0.2.100
Why Bridge Instead of IP Forwarding?
- Bridge operates at L2 (MAC addresses) - preserves source MAC for proper ARP/NDP learning
- pasta expects traffic from specific MAC addresses for its internal NAT tables
- IP forwarding rewrites source MAC, breaking pasta's connection tracking
- Bridge also enables IPv6 with proper NDP neighbor discovery
Setup Sequence (3-phase with nsenter):
- Spawn holder process:
unshare --user --net -- sleep infinity(UID/GID mappings written externally) - Pre-pasta setup via nsenter: create Firecracker TAP device only
- Start pasta attached to holder's namespace (creates pasta0 TAP)
- Post-pasta setup via nsenter: create bridge, attach pasta0 + tap-fc, add namespace IP
- Run Firecracker via nsenter:
nsenter -t HOLDER_PID -U -n -- firecracker ... - Health checks via nsenter:
nsenter -t HOLDER_PID -U -n -- curl 10.0.2.100:80
Pre-Pasta Setup Script (Phase 2, executed via nsenter):
# Create TAP device for Firecracker (pasta creates its own TAP separately)
ip tuntap add tap-fc mode tap
ip link set tap-fc up
ip link set lo upPost-Pasta Bridge Script (Phase 4, executed via nsenter after pasta starts):
# Bring pasta0 up (pasta creates it but doesn't bring it up without --config-net)
ip link set pasta0 up
# Create L2 bridge — connects pasta0 and Firecracker TAP
ip link add br0 type bridge
ip link set br0 up
ip link set pasta0 master br0
ip link set tap-fc master br0
# Add IP to bridge for health checks
# This enables nsenter to route to guest via the 10.0.2.x subnet
ip addr add 10.0.2.1/24 dev br0
# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forwardPort Forwarding (unique loopback IPs):
# Each VM gets a unique loopback IP (127.x.y.z) for port forwarding
# No IP aliasing needed - Linux routes all 127.0.0.0/8 to loopback
# Port forwarding is configured via pasta CLI flags:
# -t <bind_addr>/<host_port>:<guest_port> for TCP
# -u <bind_addr>/<host_port>:<guest_port> for UDP
pasta \
--foreground \
--quiet \
-P <pid-file> \
--ns-ifname pasta0 \
-a 10.0.2.100 \
-n 255.255.255.0 \
-g 10.0.2.2 \
--no-dhcp \
-t 127.0.0.2/8080:80 \
-T none -U none \
<holder-pid>Traffic Flow (VM to Internet):
Guest (10.0.2.100) → tap-fc → br0 (L2) → pasta0 → pasta → Host → Internet
Traffic Flow (Health Check from namespace):
nsenter curl → br0 (10.0.2.1) → L2 forward → tap-fc → Guest (10.0.2.100:80)
Traffic Flow (Host to VM port forward):
Host (127.0.0.x:8080) → pasta → pasta0 → br0 (L2) → tap-fc → Guest (10.0.2.100:80)
IPv6 Support:
- pasta has native IPv6 support (no custom build needed)
- Guest uses fd00::2 (IPv6 gateway), fd00::100 (guest IPv6)
- Guest DNS uses host DNS servers directly (via
fcvm_dns=kernel cmdline parameter) - fc-agent sends gratuitous NDP NA at boot for MAC learning
- On snapshot restore, fc-agent re-sends NDP NA to teach new pasta process
Characteristics:
- No root required (runs entirely in user namespace)
- All VMs use same 10.0.2.x subnet (isolated by user namespace)
- Unique loopback IP per VM enables same port on multiple VMs
- Bridge-based L2 preserves MAC addresses for proper pasta ARP/NDP learning
- Namespace IP (10.0.2.1) enables health checks via nsenter
- IPv6 support with native pasta IPv6 forwarding
- Works in nested VMs and restricted environments
- Fully compatible with rootless Podman in guest
In rootless mode, outbound IPv4 TCP from the guest requires a transparent proxy because there is no NAT gateway. The egress proxy multiplexes all outbound TCP connections over a single vsock connection using a frame-based protocol.
Note: Routed mode does not use the egress proxy. All external traffic in routed mode goes natively through the kernel's IPv6 routing stack at line rate. IPv4 stays internal to the namespace (for health checks and port forwarding only).
Architecture:
Guest VM Host
───────── ────
App connects to 93.184.216.34:80
↓
iptables REDIRECT → proxy (127.0.0.1:12345)
↓
SO_ORIGINAL_DST → 93.184.216.34:80
↓
Assign stream_id, send OPEN frame
↓ ↓
Single persistent vsock (port 52000) ───→ UnixStream reader
↓ OPEN → spawn TCP to destination
↓ send OPEN_OK back
↓ (OPEN_OK received)
Bidirectional DATA frames ←──→ DATA frames relayed to/from TCP
↓
CLOSE frame when done ───→ Close TCP, cleanup
Frame Format (10-byte header):
stream_id(u32 LE): unique per TCP connectionframe_type(u8): OPEN=1, DATA=2, CLOSE=3, RST=4, OPEN_OK=5, OPEN_FAIL=6flags(u8): reservedpayload_len(u32 LE): payload length after header
Guest Side (fc-agent/src/proxy.rs):
- iptables REDIRECT captures outbound TCP (excluding local, link-local, MMDS)
- Single writer task serializes frame writes to vsock
- Reader task routes incoming frames to per-stream channels via
DashMap - Per-connection handler: accept → SO_ORIGINAL_DST → OPEN → relay DATA
Host Side (src/network/egress_proxy.rs):
- Accepts vsock connections from guest
- OPEN frames spawn TCP connections to real destinations
- DATA frames relayed bidirectionally between vsock and TCP
- CLOSE/RST frames trigger cleanup
Snapshot Restore: After VIRTIO_VSOCK_EVENT_TRANSPORT_RESET, a tokio::sync::Notify
signal breaks the stale mux session and triggers immediate vsock reconnection.
Topology:
┌─────────────────────────────────────────────────────────────────┐
│ Host Namespace │
│ ┌──────────────┐ veth pair ┌──────────────────┐ │
│ │ veth_outer │◄─────────────────────────►│ VM Namespace │ │
│ │ 172.30.x.1 │ │ (fcvm-vm-xxxxxx) │ │
│ └──────────────┘ │ │ │
│ │ veth_inner │ │
│ iptables DNAT (scoped to veth IP): │ 172.30.x.2 │ │
│ -d 172.30.x.1 --dport 8080 → 172.30.x.2 │ │ │ │
│ │ ▼ │ │
│ │ ┌──────────┐ │ │
│ │ │ TAP │ │ │
│ │ └────┬─────┘ │ │
│ │ │ │ │
│ │ ┌────▼─────┐ │ │
│ │ │Firecracker│ │ │
│ │ │eth0: │ │ │
│ │ │172.30.x.2 │ │ │
│ │ └───────────┘ │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Accessing port-forwarded services:
# Curl the veth's host IP (172.30.x.1), NOT localhost
curl http://172.30.x.1:8080
# Get the veth IP from VM state
fcvm ls --json | jq '.[0].config.network.host_ip'iptables Rules (from src/network/portmap.rs):
# DNAT for external traffic - scoped to veth's host IP to avoid port conflicts
# Each VM has unique veth IP (172.30.x.y) so same port works across VMs
iptables -t nat -A PREROUTING -d 172.30.x.1 -p tcp --dport 8080 -j DNAT --to-destination 172.30.x.2:80
# DNAT for localhost traffic (OUTPUT chain) - also scoped to veth IP
iptables -t nat -A OUTPUT -d 172.30.x.1 -p tcp --dport 8080 -j DNAT --to-destination 172.30.x.2:80
# MASQUERADE for outbound (guest → internet)
iptables -t nat -A POSTROUTING -s 172.30.x.0/30 -j MASQUERADEAccessing port-forwarded services:
# Curl the veth's host IP (172.30.x.1), NOT localhost
curl http://172.30.x.1:8080IP Allocation:
- Each VM gets unique /30 subnet:
172.30.{x}.{y}/30 - Veth host IP:
172.30.{x}.{y}(used for port forwarding) - Guest IP:
172.30.{x}.{y+1}
Key Insight: pasta's userspace L4 translation adds latency and doesn't scale to many parallel clones. Solution: Use veth pairs with native IPv6 routing through the kernel stack at line rate.
Topology:
┌─────────────────────────────────────────────────────────────────┐
│ Host Namespace │
│ ┌──────────────┐ veth pair ┌──────────────────┐ │
│ │ veth-host │◄─────────────────────────►│ Namespace │ │
│ │ (link-local) │ │ (fcvm-xxxxxxxx) │ │
│ └──────────────┘ │ │ │
│ │ │ veth-ns │ │
│ proxy NDP on eth0 │ │ │ │
│ ip6tables MASQUERADE │ ┌────▼───────┐ │ │
│ route: vm_ipv6/128 via veth-host │ │ br0 │ │ │
│ │ │ 10.0.2.1 │ │ │
│ │ │ fd00::1 │ │ │
│ │ └────┬───────┘ │ │
│ │ │ │ │
│ │ ┌────▼─────┐ │ │
│ │ │ TAP │ │ │
│ │ └────┬─────┘ │ │
│ │ │ │ │
│ │ ┌────▼─────┐ │ │
│ │ │Firecracker│ │ │
│ │ │eth0: │ │ │
│ │ │10.0.2.100 │ │ │
│ │ │vm_ipv6/128│ │ │
│ │ └───────────┘ │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Setup Sequence (16 steps):
- Preflight: verify root, global IPv6, ip6tables
- Detect host IPv6 /64 subnet (supports direct /64 or AWS-style /128 with on-link /64 route)
- Generate deterministic VM IPv6 from host prefix + hash of vm_id, or take
--ipv6-address; lease it (collision detection: the lease, then the host's /128 routes) - Create network namespace (
ip netns add fcvm-XXXX) - Disable IPv6 DAD inside namespace (
accept_dad=0on all + default, before any interface exists) - Create veth pair, move guest side to namespace
- Suppress host veth link-local autogeneration (
addr_gen_mode=1,keep_addr_on_down=1, before link-up) - Create TAP device in namespace, connect to bridge (br0) with veth
- Assign bridge IPs:
10.0.2.1/24(IPv4 gateway) +fd00::1/64(IPv6 gateway, nodad) - Bring up host veth, enable per-interface IPv6 forwarding
- Assign EUI-64 link-local to host veth with
nodad(hard error if nexthop address cannot be made usable) - Namespace default IPv6 route: via host veth link-local through bridge
- Host: route
vm_ipv6/128via host veth - Proxy NDP for vm_ipv6 on default interface (so network fabric routes to this host)
- ip6tables MASQUERADE on outbound interface (required for AWS source/dest check)
- TCP proxy port forwarding: each mapping on its HOSTIP, or on the VM's unique loopback IP (127.x.y.z) when it names none
Port Forwarding (built-in TCP proxy):
# Rust TCP proxy: bind on the host, connect inside namespace via setns(2)
# --publish 8080:80 listens on the VM's loopback IP (same allocation as rootless)
Host 127.0.0.2:8080 → tcp_proxy → setns(namespace) → connect 10.0.2.100:80
# --publish '[::]:80:80' listens on the address the mapping names
Host [::]:80 → tcp_proxy → setns(namespace) → connect 10.0.2.100:80
# Bidirectional relay via tokio::io::copy_bidirectional
An IPv6 listener is created with IPV6_V6ONLY off, so [::] accepts IPv4 and IPv6
clients whatever net.ipv6.bindv6only says. Every listener is bound before any relay
starts: a bind failure (address in use, address not on this host) fails the VM start with
the address and port in the error, and closes the listeners already bound. The loopback
IP is allocated, and reported as config.network.loopback_ip, only when a mapping names
no HOSTIP.
Traffic Flow (VM to Internet, IPv6):
Guest → TAP → br0 → veth-ns → veth-host → host kernel → ip6tables MASQUERADE → eth0 → Internet
Traffic Flow (Internet to VM, IPv6):
Internet → eth0 → proxy NDP → host kernel → route vm_ipv6/128 via veth-host → veth-ns → br0 → TAP → Guest
Traffic Flow (Host to VM port forward):
Host (127.0.0.x:8080) → tcp_proxy (setns) → 10.0.2.100:80 → br0 → TAP → Guest
Traffic Flow (Health check):
ip netns exec curl → br0 (10.0.2.1) → L2 forward → TAP → Guest (10.0.2.100:80)
IPv6 Addressing:
- Each VM gets a deterministic IPv6 from the host's /64 subnet (hash of vm_id)
- VM uses /128 prefix (fc-agent configures via boot parameter) to prevent on-link NDP for other subnet addresses
- fd00::1 on the bridge serves as the VM's IPv6 gateway
- Proxy NDP advertises the VM's IPv6 on the host's physical interface
Cleanup (on VM exit):
- Stop TCP proxy relays and wait for them, which closes their listeners and ends their connections (in-process, no external PIDs)
- Remove ip6tables MASQUERADE rule (scoped to vm_ipv6/128)
- Remove proxy NDP entry
- Remove host route (uses
devqualifier for parallel safety) - Delete veth pair (auto-deletes peer)
- Delete namespace
Characteristics:
- Requires root and global IPv6 on host
- Kernel line-rate IPv6 (no userspace proxy for traffic forwarding)
- Each VM gets unique /128 IPv6 — parallel clones route correctly without NAT
- IPv4 internal only (10.0.2.x for health checks, no external IPv4 routing)
- Port forwarding via built-in TCP proxy, on the mapping's HOSTIP or on the VM's loopback IP
- All resources per-VM: no shared state, clean parallel operation
┌─────────────────────────────────────────────────────────┐
│ 1. Parse CLI arguments │
│ - Image, vCPU, memory, ports, volumes, snapshot name │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 2. Detect execution mode (rootless/bridged/routed) │
│ - Check for root privileges │
│ - Check for /dev/kvm access │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 3. Setup networking │
│ - Create TAP device (bridged/routed) or prepare pasta│
│ - Parse port mappings │
│ - Generate MAC address │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 4. Prepare disks │
│ - Create CoW overlay from base rootfs │
│ - Setup volume mounts (block/sshfs/nfs) │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 5. Start Firecracker process │
│ - Spawn with Unix socket API │
│ - Wait for socket ready │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 6. Configure VM via API │
│ - set_boot_source (kernel) │
│ - set_machine_config (vCPU, memory) │
│ - add_drive (rootfs) │
│ - add_network_interface (TAP device) │
│ - set_mmds_config (metadata service) │
│ - put_mmds (container plan) │
│ - set_balloon (memory balloon if configured) │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 7. Start VM │
│ - put_action(InstanceStart) │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 8. Stream serial console logs │
│ - Open serial console device │
│ - Stream to stdout/file based on --logs flag │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 9. Wait for readiness (if --wait-ready specified) │
│ - vsock: Wait for guest connection │
│ - http: Poll HTTP endpoint │
│ - log: Search serial console for pattern │
│ - exec: Execute command in guest │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 10. Save snapshot (if --save-snapshot specified) │
│ - create_snapshot(memory + disk) │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 11. Setup signal handlers │
│ - SIGINT/SIGTERM → graceful shutdown │
│ - SIGCHLD → detect VM exit │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 12. Wait for VM exit or signal │
│ - Process blocks here (hanging mode) │
│ - VM lifetime = process lifetime │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 13. Cleanup │
│ - Kill Firecracker process │
│ - Remove TAP device │
│ - Remove NAT rules │
│ - Clean up temp files │
└─────────────────────────────────────────────────────────┘
Step 1: Create Snapshot (fcvm snapshot create)
┌─────────────────────────────────────────────────────────┐
│ 1. Pause the running VM │
│ - Firecracker API: pause │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 2. Create Firecracker snapshot │
│ - Snapshot memory to file │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 3. Copy disk via reflink (VM still paused) │
│ - Ensures memory/disk consistency │
│ - Reflink is O(1) — no pause time impact │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 4. Resume the original VM │
│ - VM continues running │
└─────────────────────────────────────────────────────────┘
Step 2: Start Memory Server (fcvm snapshot serve)
┌─────────────────────────────────────────────────────────┐
│ 1. Open the snapshot memory file │
│ - copy mode (default): map it read-only; each served │
│ page is copied into the clone's private memory │
│ - minor mode: copy it once into a sealed memfd that │
│ every clone maps; clean pages are shared │
│ - prefetch on (default): load the recorded working │
│ set; copy mode reads it into the page cache │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 2. Create Unix socket for clone connections │
│ - uffd-{snapshot}-{pid}-{pid_start_time}.sock │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 3. Register state in state manager │
│ - process_type: "serve" │
│ - snapshot_name │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 4. Wait for clone connections (async) │
│ - Handle UFFD page faults from clones │
│ - Serve memory pages on-demand │
└─────────────────────────────────────────────────────────┘
Step 3: Spawn Clone (fcvm snapshot run)
┌─────────────────────────────────────────────────────────┐
│ 1. Create CoW overlay disk (btrfs reflink) │
│ - cp --reflink=always (~1.5ms) │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 2. Setup new networking │
│ - Generate new MAC address │
│ - Create TAP device (bridged) or pasta (rootless) │
│ - Allocate loopback IP for health checks │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 3. Start Firecracker with UFFD backend │
│ - Connect to memory server's Unix socket │
│ - Firecracker fetches pages via UFFD on access │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 4. Load snapshot via Firecracker API │
│ - track_dirty_pages = !hugepages │
│ - resume_vm = false; the load retargets the TAP, │
│ fcvm patches the rootfs drive, then resumes the VM │
│ - Working-set replay starts at the UFFD handshake; │
│ the resume does not wait for it │
└────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 5. VM resumes (< 1 second total startup) │
│ - prefetch on (default): recorded working set │
│ populated in 2 MiB chunks, demand faults served │
│ between chunks │
│ - Other pages served on first touch │
│ - copy mode: every served page is private to clone │
│ - minor mode: clean pages shared via sealed memfd │
└─────────────────────────────────────────────────────────┘
Goal: VM dies when fcvm podman run process exits.
Implementation (using tokio::signal):
use tokio::signal::unix::{signal, SignalKind};
async fn main() -> Result<()> {
let mut sigterm = signal(SignalKind::terminate())?;
let mut sigint = signal(SignalKind::interrupt())?;
// Start VM
let mut vm = VmManager::new(...);
vm.start().await?;
// Wait for signal or VM exit
tokio::select! {
_ = sigterm.recv() => {
info!("received SIGTERM, shutting down");
vm.kill().await?;
}
_ = sigint.recv() => {
info!("received SIGINT, shutting down");
vm.kill().await?;
}
status = vm.wait() => {
info!("VM exited with status: {:?}", status);
}
}
// Cleanup
network.cleanup().await?;
Ok(())
}Graceful Shutdown:
- Receive SIGTERM/SIGINT
- Send shutdown signal to Firecracker
- Wait up to 10 seconds for graceful exit
- Force kill if timeout
- Clean up network resources
- Remove temporary files
Location: fc-agent/src/main.rs
Runs inside the microVM as a systemd service (both Firecracker and Cloud Hypervisor).
Responsibilities:
- Fetch container plan (boot plan) via MMDS (Firecracker) or vsock port 4995 (Cloud Hypervisor)
- Launch Podman with correct configuration
- Stream container logs to serial console
- Signal readiness to host (via vsock)
- Handle container lifecycle
Firecracker provides a metadata service accessible at http://169.254.169.254/.
Container Plan Format:
{
"image": "nginx:latest",
"env": {
"KEY": "VALUE",
"DB_HOST": "localhost"
},
"cmd": ["/bin/sh", "-c", "nginx -g 'daemon off;'"],
"volumes": [
{
"host": "/data",
"guest": "/mnt/data",
"readonly": false
}
],
"podman": {
"rootless": true,
"network": "host",
"privileged": false
},
"readiness": {
"mode": "http",
"url": "http://127.0.0.1:80/health"
},
"logs": {
"mode": "stream"
}
}#[tokio::main]
async fn main() -> Result<()> {
// 1. Fetch plan via MMDS (Firecracker) or vsock (Cloud Hypervisor)
let plan = fetch_plan().await?;
// 2. Build Podman command
let mut cmd = Command::new("podman");
cmd.arg("run").arg("--rm");
// Network mode
if plan.podman.network == "host" {
cmd.arg("--network=host");
}
// Environment variables
for (key, val) in plan.env {
cmd.arg("-e").arg(format!("{}={}", key, val));
}
// Volume mounts
for vol in plan.volumes {
let mount = if vol.readonly {
format!("{}:{}:ro", vol.guest, vol.guest)
} else {
format!("{}:{}", vol.guest, vol.guest)
};
cmd.arg("-v").arg(mount);
}
// Image
cmd.arg(&plan.image);
// Command override
if let Some(cmd_override) = plan.cmd {
cmd.args(cmd_override);
}
// 3. Spawn container
let mut child = cmd
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?;
// 4. Stream logs to serial console
stream_to_console(child.stdout.take(), "stdout").await;
stream_to_console(child.stderr.take(), "stderr").await;
// 5. Signal readiness (vsock or log marker)
if let Some(readiness) = plan.readiness {
signal_ready(readiness).await?;
}
// 6. Wait for container exit
let status = child.wait().await?;
eprintln!("[agent] container exited: {}", status);
Ok(())
}
// Actual implementation auto-detects transport via /proc/cmdline
// (fcvm_bootplan=vsock → vsock port 4995; otherwise → MMDS at 169.254.169.254)
async fn fetch_plan() -> Result<Plan> {
match detect_transport() {
Transport::Vsock => read_vsock_plan().await,
Transport::Mmds => {
loop {
match reqwest::get("http://169.254.169.254/").await {
Ok(resp) => return resp.json().await.context("parsing MMDS"),
Err(_) => tokio::time::sleep(Duration::from_millis(500)).await,
}
}
}
}
}The guest is configured to support rootless Podman:
-
User setup (
create-rootfs-debian.sh):# Create podman user useradd -m -s /bin/bash podman # Setup subuid/subgid ranges echo "podman:100000:65536" >> /etc/subuid echo "podman:100000:65536" >> /etc/subgid
-
Podman configuration:
# Enable unprivileged port binding sysctl -w net.ipv4.ip_unprivileged_port_start=0 # Use crun runtime (faster than runc) podman --runtime=crun
-
fc-agent runs as:
- Root: When
podman.rootless = false - podman user: When
podman.rootless = true
- Root: When
fcvm exec and podman run -it share one session implementation on each side. Its reference is podman exec: tests/test_exec_podman_parity.rs runs every case through podman exec, fcvm exec and fcvm exec --vm (one case, --privileged, has no --vm form) and requires the compared parts of each result to match: stdout bytes, stderr bytes, exit code, and whether the call returned. A case says which parts it compares, and why, where it leaves one out.
┌─────────────────────────────────────────────────────────────────────────┐
│ Host Process (fcvm) │
│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────────────┐│
│ │ stdin/stdout │────►│ Raw mode (-t on │────►│ exec_proto frames ││
│ │ stderr │◄────│ a terminal only) │◄────│ ││
│ └──────────────┘ └──────────────────┘ └──────────┬───────────┘│
│ │ │
└───────────────────────────────────────────────────────────┼────────────┘
│ vsock
▼
┌───────────────────────────────────────────────────────────────────────┐
│ Guest (fc-agent) │ │
│ ┌──────────────────┐ ┌──────────────┐ ┌──────────▼─────────┐ │
│ │ exec_proto │────►│ PTY (-t) or │────►│ Command, or the │ │
│ │ frames │◄────│ three pipes │◄────│ `podman exec` client│ │
│ └──────────────────┘ └──────────────┘ └────────────────────┘ │
└───────────────────────────────────────────────────────────────────────┘
| Flag | Meaning | Implementation |
|---|---|---|
-i |
Forward stdin | Host sends STDIN frames, then STDIN_EOF when its stdin ends |
-t |
Allocate PTY | Guest runs the command on a PTY sized like the host's terminal |
-it |
Both | Interactive shell with full terminal support |
| neither | Plain exec | Pipes for stdout and stderr, /dev/null on stdin |
-e, --env-file |
Environment | Resolved on the host into KEY=VALUE, files first so -e wins |
-w, -u |
Directory, user | podman exec applies them in a container; fc-agent applies them to a guest command |
--privileged |
Extended capabilities | Container only; a --vm command is already the guest's root |
-d |
Detach | Returns one identifier line and exit 0; the command keeps running. With -t it has a terminal of its own and no TERM, as under podman |
-t does not need a terminal on the host. With a pipe or file on stdin the guest still allocates the PTY and the host skips raw mode, as podman exec -t does.
The handshake is three text lines: the JSON ExecRequest, the server's ACK2, the client's GO. The ACK token names the protocol version. A client that reads the older ACK refuses the session before GO, so nothing runs on an agent that would not understand the frames below. After GO every mode speaks length-prefixed frames:
┌─────────┬─────────┬──────────────────┐
│ Type(1) │ Len(4) │ Payload(N) │
└─────────┴─────────┴──────────────────┘
Message Types:
DATA (0x01): Command stdout, or everything the PTY produced with-tEXIT (0x02): Command exit code (4 bytes, big-endian i32)ERROR (0x03): Error message stringSTDIN (0x04): Input from the hostSTDIN_EOF (0x05): The host's stdin ended (no payload)STDERR (0x06): Command stderr; never sent with-t, where the PTY merges both streamsRESIZE (0x07): The host terminal's size (rows and cols, big-endian u16 each)STDIN_WINDOW (0x08): Guest to host: this many more STDIN bytes may be sent (big-endian u32). The first one is the first frame of a session that forwards stdin
Frames carry raw bytes, so output is exact: no line splitting, no UTF-8 requirement, no added newline.
STDIN is flow-controlled by the guest, the way SSH does it. The host starts with no window. fc-agent grants 256 KiB when an -i session starts and grants back every chunk once the command has taken it, so the host sends exactly as fast as the command reads and the connection itself never backs up. This is what keeps the client-death kill (below) true with -i: measured on a VM, a host hangup does not reach the guest across a vsock connection that has backed up, and a command that ignored its stdin survived its killed client. With the window, fc-agent always has a read outstanding and sees the hangup at once. A host that sends beyond its window is treated as gone. Grants travel to the host on the same connection as the command's output, so a client whose output consumer has stalled also stops uploading stdin, as with SSH.
- A reader thread writes DATA to stdout and STDERR to stderr, and returns the EXIT code. A session that ends without one (connection closed, snapshot-pause orphan, not a frame) is an error, never an invented exit code. When the client's own stdout is gone, as in
fcvm exec ... | head -1, the client dies of SIGPIPE like any Unix tool. - The session socket has no write timeout, so a write that is slow for any reason never drops input. When the session ends the socket is shut down before the input thread is joined, which releases a write that is still blocked.
- An input thread polls stdin (with
-i, and only while the guest's window is open), a pipe the SIGWINCH handler writes to (with-tand a terminal on stdin), a pipe the reader thread pokes when a window grant arrives, and a wake pipe that ends the thread when the session is over. End of input, including a closed fd 0, becomes STDIN_EOF. - With
-tand a terminal on stdin,fcvm execputs the terminal's size in the request, so the PTY is sized before the command starts. Such a session also sends its size as the first frame, which is whatpodman run -itrelies on because it has no request. With-tand a pipe on stdin there is no size to send.
One async session per connection:
- Without
-tthe command gets three pipes and its own process group. With-tit gets a PTY as controlling terminal in a new session. - For a container exec the command is
podman execwith the same-iand-t. Its PTY starts in raw mode: the container's own PTY does the echo, line editing and signal keys, and fc-agent's PTY only carries bytes to it. - The session ends when the command exits. Whatever it wrote is forwarded first: a pipe is drained of the bytes it held at exit, and a PTY is read until it hangs up, for at most a second. A background process that inherited the output does not hold the session open, and what it writes afterwards is dropped.
- Frames from the host are read by one task and applied by another. A command that does not read its stdin blocks only the applier. The reader never stops, because the window bounds what the host may send, so it notices the host hanging up at once.
- STDIN_EOF closes the command's stdin. A PTY has no half-close, so with
-tit does nothing and Ctrl-D is the way to end input. - A command killed by signal N exits
128+N. One that cannot be found exits 127, and one that cannot be run exits 126. When fcvm itself fails (no such VM, no connection, a refused request) the client exits 125.
- Client death kills a guest command. When the host side of an
fcvm exec --vmdisappears, fc-agent kills the command's process group (#636), so a host-side timeout cannot leak a guest process. For a container exec the group is the guest'spodman execclient; the process inside the container outlives it, exactly as it does under podman. Exempt: a detached command (-d), and the VM's ownpodman run -itconsole, whose connection a snapshot drops and whose container a snapshot must not disturb. - Shell auto-detection.
fcvm exec -- bashon a terminal behaves as-it.podman execrequires the flags. -timplies--quiet. fcvm's own log shares stderr with the command, so a terminal session suppresses it.-don a guest command prints its pid. podman prints an exec session id. Both print one line and exit 0.- Old agents are refused. The handshake token names the protocol version. A VM started, or a snapshot taken, by an fcvm from before the framed protocol is refused before GO, with a message that says to restart or re-create it. Nothing runs. The
podman run -itconsole has no handshake. The stdin window is the first frame of every session that forwards stdin, so there the host reports an older fc-agent when output arrives before one. - Not offered:
--detach-keys(a session ends with its command),--preserve-fd,--preserve-fds,--cidfile,--latest,--no-session.
- Web terminal, closed tab. While 1 MiB of input waits for a program that is not reading it, the
fcvm servebridge stops reading the websocket, so a tab closed in that state is noticed only when the program reads again or exits. Before stdin was flow-controlled, any input the program left unread had this effect, and guest output stopped with it. - Job control: Background/foreground (
bg,fg) work within the container, but signals are not forwarded to the host.
Full CLI documentation with examples: See README.md
| Command | Purpose |
|---|---|
fcvm setup |
Download kernel, create rootfs (first-time setup, ~5-10 min) |
fcvm podman run |
Launch container in a microVM (Firecracker by default, or Cloud Hypervisor via --hypervisor cloud-hypervisor) |
fcvm exec |
Execute command in running VM/container |
fcvm balloon |
Print a VM's balloon target and size; with MIB, set the target first |
fcvm ls |
List running VMs |
fcvm snapshot create |
Create snapshot from running VM |
fcvm snapshot serve |
Start UFFD memory server for cloning |
fcvm snapshot run |
Spawn clone from memory server |
fcvm snapshots ls |
List stored snapshots (also: delete <NAME>, prune) |
-
Trailing arguments: Both
podman runandexecsupport trailing args after--:fcvm podman run --name test alpine:latest echo "hello" fcvm exec --name test -it -- sh -c "ls -la"
-
--namevs--pid: VM identification uses named flags (not positional):fcvm exec --name my-vm -- hostname # By name fcvm exec --pid 12345 -- hostname # By PID
Names are unique among live VMs. A launch writes its first state through
StateManager::claim_name, which holdsnames.lockin the state directory while it checks every state with that name and writes its own, so of two launches with one name exactly one proceeds. A state is live while its process runs (PID, start time, not a zombie); a state with the name whose process is gone is deleted by the claim. -
Interactive flags (
-i,-t): Match docker/podman semantics:-i: Keep stdin open-t: Allocate PTY-it: Both (interactive shell)
-
Network modes:
--network rootless(default, no sudo),--network bridged(sudo), or--network routed(sudo + IPv6)
Purpose: Create a snapshot from a running VM.
Usage:
fcvm snapshot create [--pid <PID> | <VM_NAME>] [--tag <TAG>]Options:
--pid <PID> fcvm process PID to snapshot
--tag <TAG> Snapshot name (defaults to VM name)
<VM_NAME> VM name to snapshot (alternative to --pid)
Examples:
# Create snapshot by PID
fcvm snapshot create --pid 12345 --tag my-snapshot
# Create snapshot by name
fcvm snapshot create my-vm --tag warm-nginxZero pages are holes: with the default Firecracker build ([firecracker] in rootfs-config.toml, used by every
kernel profile that does not set its own firecracker_repo), a Full snapshot written to a new memory file leaves
every all-zero 4 KiB page as a hole. The fork tests each page as it dumps, so memory.bin stores only the non-zero
pages and a hole reads back as zeros. With dirty tracking on, a diff from that build also holds the pages a discard
emptied (balloon inflation, free page reporting) as zeros: a discard is not a guest write, so KVM's dirty log never
records it, and the fork marks each discarded range itself. A diff writes every page the VM touched or a discard
emptied, zeros included, and its merge punches each run of its all-zero pages at least MERGE_MIN_PUNCH long,
measured within its 8 MiB merge window, as a hole, so memory a restored guest gave back in runs that long is a hole in
its snapshot. The nested arm64 profile builds its own NV2 fork, which writes every page of a Full snapshot that
holds data, leaves the pages the guest never wrote as holes without reading them (reading one would map it to the
zero page, and the guest's first write after the resume would then be a copy-on-write, #1264), and leaves discarded
pages out of a diff. A diff starts from a clone or copy of the parent's memory.bin,
which on btrfs shares the parent's extents, so a child keeps its parent's holes.
A snapshot of a restored VM: a VM that was itself restored from a snapshot is saved as a diff, written in place when the store allows it:
- In place. Before the pause, fcvm clones the parent's
memory.bininto the new snapshot with FICLONE (clone_base_for_in_place_diff,src/commands/common.rs), and Firecracker writes the dirty pages into that clone: every fork fcvm builds writes a diff into an existing memory file of the guest's size instead of truncating it. The extents the snapshot wrote are then the clone's only extents no other file shares, which FIEMAP reports (src/storage/extents.rs); Firecracker syncs the file before its create returns, so they are on disk when counted. The dirty-tracking check below counts them, stopping at its minimum, and a diff whose extents cannot be read counts as nothing written, so it is retried as Full. After the resumepunch_in_place_diffreads those of them long enough to hold a run to punch (most of a diff of scattered pages is not), punches their long zero runs as a merge does, and flushes the file. A clone costs one reference per extent of the parent: 31.3 s for a 128 GiB guest's memory file of 4.34 million extents. The check found its minimum in 0.17 s on that clone after a million scattered page writes, and a full walk of its extents took 7.1 s. - Merged. When the parent's
memory.binis not the guest's size (Firecracker would truncate it), the store cannot clone it (another filesystem, tmpfs), or its filesystem does not report shared extents, Firecracker writes the pages to a sparsememory.diff, fcvm copies the base snapshot'smemory.binafter the resume, andmerge_diff_snapshotcopies each data run of the diff to the same offset of that copy. On a 128 GiB guest that had dirtied 6.4 GB in 1,008,048 runs, the merge took 236 s, against 66 s for Firecracker's write of the diff.
The merge:
- Workers. One for each 256 MiB or 4,096 writes (
MERGE_BYTES_PER_WORKER,MERGE_WRITES_PER_WORKER), up to 16 (MERGE_WORKERS), because one reader waits for one read at a time: on a 128 GiB guest with 38.1 GiB touched in 2,328,992 runs, one worker took 1,646 s and 16 took 323 s. A first walk writes nothing and counts the runs. Each worker then takes the next 8 MiB of the file nobody has taken (MERGE_WINDOW), so they stay busy wherever in the file the diff's data lies. The calling thread is one of them, so a merge of at most 256 MiB in at most 4,096 writes starts no thread. A worker that fails stops the others before their next write. - Zero pages. A worker reads each piece of a run once. Each run of its all-zero 4 KiB pages (
GRANULE) at leastMERGE_MIN_PUNCH(128 KiB) long is punched as a hole (FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE), and the rest is written: the data, a shorter zero run with the data beside it, and a page the piece holds only part of. A run is measured inside its piece's window, so a zero run of 128 to 248 KiB that straddles a window boundary can be cut into two shorter runs and written. A short zero run costs more to punch than to write: on btrfs with compress-force=zstd, a merge of 1 GiB of alternating 4 KiB pages of data and zeros took 4.3 s with every zero page punched, 1.3 s with each zero page written by its own call, and 0.80 s with the zeros written in one call with the data beside them, which is what a run shorter than the minimum gets and what the merge did before it punched anything. Runs of 2 MiB took 0.48 s punched, against 0.76 s written in one call per piece. The holes never took more disk than the written zeros. The window is a multiple of the granule, so one worker alone writes or punches each page. A worker that fails stops the others between the runs of a piece as well as between pieces. A filesystem that refuses a punch with EOPNOTSUPP gets the zeros written, counted apart and logged once per merge, and is not asked again during that merge. Any other punch error fails the merge.zero_bytescounts the bytes of the punches that succeeded. They are holes on a filesystem of 4 KiB blocks, as btrfs is. A filesystem of larger blocks zeroes in place the part of a block a punch does not cover whole, so there some of them stay allocated. - Flush and cleanup. The merged file is flushed before the snapshot's
config.jsonis written, and a failed flush fails the snapshot. The diff is removed once it is merged. A failed merge removes the unfinished snapshot directory. create_snapshot_corelogsdiff merge completewith the diff's bytes and runs, the pieces written (writes), the holes punched (punches), the bytes they covered (zero_bytes), the bytes of zero runs written after a refusal (unpunched_zero_bytes), the workers and the duration.- An in-place diff logs
cloned the parent's memory filewith the clone's duration before the pause, or why it is merged instead, anddiff written in placeafter the resume with the bytes written (written_bytes), the extents they lie in (ranges), the bytes read to find zero runs (read_bytes), the punches,zero_bytes,unpunched_zero_bytesand the duration.
Purpose: Start a UFFD memory server for cloning.
Usage:
fcvm snapshot serve <SNAPSHOT_NAME> [--uffd-mode copy|minor] [--uffd-fault-around BYTES]
[--uffd-replay-threads COUNT]
[--uffd-prefetch on|off] [--uffd-prefetch-record-window SECS]The memory server:
- Loads the snapshot's memory file
- Listens for clone connections via Unix socket
- Serves memory pages on-demand via UFFD (userfaultfd)
- In copy mode (the default), copies each page a clone is served into that clone's private memory, so clones share no guest pages. In minor mode (
--uffd-mode minor), clones map one sealed memfd: a clean page has one physical copy, and a page a clone writes becomes private to that clone - Records each clone's restore working set and replays it into the next clone
- In copy mode, reads the recorded working set into the page cache when it starts and when a clone connects, so a restore does not replay it from disk one page at a time. On a 128 GiB guest with a cold cache, replay went from 94.5 s to 32.3 s, and to 22.3 s when the clone connected after the warm-up, and demand faults during replay from 992,535 to 562,321
Flags:
| Flag | Env var | Default | Effect |
|---|---|---|---|
--uffd-mode copy|minor |
FCVM_UFFD_MODE |
copy |
copy fills faults with UFFDIO_COPY (private per-clone pages); minor serves a sealed memfd with UFFDIO_CONTINUE (true page sharing) |
--uffd-fault-around BYTES |
FCVM_UFFD_FAULT_AROUND |
0 (off) |
Experimental. Copy mode only. A demand fault is served first exactly as without the option, then the rest of its granule is populated: BYTES aligned in snapshot file offsets and clipped to the region. A fault on a page the balloon gave back is answered with zeros and gets no fault-around. A fault on a hole in the memory file is answered with zeros and gets fault-around like any other. A given-back page or a hole inside a fault's granule is stepped over. 0, or a power of two above the host page size through 2097152. Each fault privately materialises the rest of its granule, bar pages the balloon gave back and holes, so memory per clone grows and clones per host drop. Measured once, on a 128 GiB guest at 64 KiB: the first real page after a restore went from 518.4 s to 336.2 s, restore to healthy went from 2m23s to 4m07s, and 17.5 million pages were installed beyond the demanded ones, 14.5 per fault. Limitation: with the option on, a page that fault-around installed is never recorded, so the recorded working set converges one demanded page per granule per clone, and replay never restores what fault-around would have installed. Record the working set with the option off. A non-zero value with --uffd-mode minor is an error |
--uffd-replay-threads COUNT |
FCVM_UFFD_REPLAY_THREADS |
8, or the host's core count if lower |
Copy mode only. The threads a clone's working-set replay copies on; the handler keeps planning, stepping over given-back pages and holes, and serving demand faults. 1 through 64. One thread took 15.3 s to replay a 128 GiB guest's 9.0 GiB recorded set of 1.00 M runs, during which the guest demand-faulted 479,129 pages replay had not reached |
--uffd-prefetch on|off |
FCVM_UFFD_PREFETCH |
on |
Working-set replay. on records faulted offsets to <memory.bin>.working-set, replays them into later clones, and in copy mode reads the recorded set into the page cache when the serve starts and when a clone connects; off is fully inert: no recording, no replay, no warm-up, no files |
--uffd-prefetch-record-window SECS |
FCVM_UFFD_PREFETCH_RECORD_WINDOW |
300 |
Seconds after a clone's UFFD handshake during which its demand faults are recorded into the working set. Later faults are served but not recorded. 0 records nothing; replay of an existing record is unaffected |
Example:
# Start memory server (blocks, keeps running)
fcvm snapshot serve my-snapshot
# Baseline arm for measurement: pure demand paging, nothing recorded
fcvm snapshot serve my-snapshot --uffd-prefetch offSee "Fault-Around" in AGENTS.md for what --uffd-fault-around trades and when to use it.
See "Working-Set Replay" in AGENTS.md for the fault-locality data behind the default and for
the invalidation/isolation rules.
Purpose: Spawn a clone VM from a running memory server.
Usage:
fcvm snapshot run --pid <SERVE_PID> [OPTIONS]
fcvm snapshot run --snapshot <NAME> [OPTIONS]Options:
--pid <SERVE_PID> Memory server PID (UFFD lazy-restore mode)
--snapshot <NAME> Snapshot name (no serve process needed)
(Either --pid or --snapshot is required; mutually exclusive)
--name <NAME> Clone VM name (auto-generated if not provided; unique among running VMs)
--ipv6-address <ADDR> Routed snapshots only: the clone's address, swapped in on restore
--exec <CMD> Execute command in container after clone is healthy
--publish <SPEC> [HOSTIP:]HOSTPORT:GUESTPORT[/PROTO], comma-separated or repeated.
The clone publishes these mappings instead of the snapshot's
Network mode, TTY, interactive flags, and --ipv6-prefix are inherited from the snapshot
metadata, and so are the port mappings unless --publish is given.
--publish moves the host side of a mapping (address and port). Its guest port and
protocol must be one the snapshot published: a guest restored from memory set up its
published ports when it booted (fc-agent's DNAT to loopback), and the restore does not
run that again. The rule is the same for a disk-only clone and for UDP, which would not
need it. A bridged clone listens on its veth address whatever HOSTIP says. A snapshot
taken of a clone records that clone's mappings, so a guest port the clone dropped is not
offered to clones of the new snapshot. The
choice is one function, clone_port_mappings in src/commands/snapshot.rs, which runs
right after the snapshot's config is loaded, so a refused mapping fails before a VM id,
state file, or network exists. The clone's state (config.port_mappings) records the
mappings it uses, a disk-only clone sets its network up from the same list, and no
snapshot key includes the flag. fcvm podman run restoring from its snapshot cache
passes its own --publish through the same path.
Examples:
# Spawn a clone (inherits network/ports from snapshot)
fcvm snapshot run --pid 12345 --name clone1
# Multiple clones in parallel
for i in {1..10}; do
fcvm snapshot run --pid 12345 --name clone$i &
done
wait # Lightning fast: all start in <1 second eachPurpose: List running memory servers.
fcvm snapshot lsPurpose: List running VMs.
Usage:
fcvm ls [--json] [--pid <PID>]Options:
--json Output in JSON format
--pid <PID> Filter by fcvm process PID
Example output:
NAME PID STATUS HEALTH NETWORK IMAGE
my-nginx 12345 running healthy bridged nginx:alpine
clone-1 12350 running healthy rootless (clone)
Purpose: Read or set the balloon target of a running VM.
Usage:
fcvm balloon (--pid <PID> | --name <NAME>) [MIB]It prints one JSON line, {"target_mib":96,"actual_mib":64}: the target the
device was asked to reach, and the size the guest has given it so far. With MIB
it sets the target first (PATCH /balloon) and prints right after, so the size
is still on its way. Refused before any request: a Cloud Hypervisor VM (fcvm's
client for it has no balloon call) and a target above the VM's memory. A VM
booted without --balloon has no device and cannot get one; whether it has
one is read from GET /vm/config. When Firecracker answers the set with a 400,
the error says what it refuses (a device the guest never activated, as with a
kernel that has no virtio balloon driver).
A set holds the VM's snapshot lock from its device check to its report. Every
snapshot of the VM holds that lock from its read of the balloon to its save, and
a pause does not keep a PATCH out (Firecracker applies it to a paused VM), so
without the lock a target could land in between and leave a snapshot whose
record and saved device disagree. Two sets cannot interleave either, so each
reports the target it set. A set waits up to 60 seconds for the lock, then fails
and says the target was not set: a snapshot of a large VM holds the lock for
minutes. A report without MIB changes nothing, takes no lock and does not wait.
A PATCH /balloon sent to the API socket by anything else is outside the lock.
A set while a workload initializes costs the run its startup snapshot. That
snapshot is named for the target its workload initialized under
(<key>-balloon<MIB>-startup), and a restore can set a target but cannot replay
the initialization. When the run turns healthy and its device is no longer at
the target the run started with, the workload initialized under a mix of two
targets and is the startup state of neither. The run takes no startup snapshot,
under either name, and logs why at info. The comparison runs under the per-VM
snapshot lock and before the VM is paused, so a set cannot land between it and
the save, and a VM whose snapshot is not taken is not disturbed. Later runs
restore the shared pre-start snapshot and make their own startup snapshot.
podman prepare fails instead, because the snapshot is what it was asked for.
The pre-start snapshot needs no such check: it is taken before the workload
starts and is shared between targets. A target that was changed and changed
back before the run turned healthy is not seen.
The command writes no state. What follows the new target, and what does not:
- The next snapshot of the VM does. A memory snapshot and a disk-only
capture read the device from the VMM, so they record the new target. A
clone of that snapshot records it in its state, and its relaunch after a
guest reboot attaches the device at it. The exception is
podman run's own startup snapshot, which is named for the target the run started with: a run whose device is at another target when it turns healthy does not take it (below). - The VM's own state does not.
config.balloon_mib(fcvm ls --json) keeps the target the VM booted with or, for a clone, the one its restore read. - The VM's own relaunch after a guest reboot does not. A cold-booted VM relaunches from its launch config and a restored clone from its state, both of which hold the earlier target. Firecracker has exited by then, so there is nothing to ask.
test_balloon_command_sets_and_reports_the_target covers the report, the set,
the unchanged state and the snapshot's record.
test_balloon_set_cannot_land_between_a_snapshots_read_and_its_save and
test_two_balloon_sets_each_report_their_own_target cover the lock, each with a
failpoint that holds the other side where the set must not land.
test_startup_snapshot_is_not_taken_after_the_balloon_target_was_changed covers
the startup snapshot, in the podman run loop and in the restore loop, with a
workload that starts only when the test lets it. podman prepare's refusal is
pinned by source, with the two loops' call sites, in
tests/test_balloon_call_sites.rs.
Purpose: Manage stored snapshots. A subcommand is required (ls, delete, or prune).
List snapshots:
fcvm snapshots ls [--json] [--filter <user|system>] [--image <NAME>] [--shared]Delete a specific snapshot:
fcvm snapshots delete <NAME> [--force]Prune snapshots (deletes system/auto-generated snapshots by default):
fcvm snapshots prune [--force] [--all] [--image <NAME>]Options:
ls:
--json Output in JSON format
--filter <user|system> Filter by snapshot type
--image <NAME> Filter by container image name
--shared Show accurate disk usage accounting for btrfs shared extents (slower)
delete:
--force, -f Force deletion without confirmation
prune:
--force, -f Force deletion without confirmation
--all Delete ALL snapshots (including user-created ones)
--image <NAME> Only delete snapshots matching this container image name
fcvm/
├── Cargo.toml # Workspace manifest
├── DESIGN.md # This document
├── README.md # User-facing documentation
├── Makefile # Build and test commands
├── Containerfile # Test container definition
│
├── src/ # Host CLI (fcvm binary)
│ ├── main.rs # Entry point
│ ├── lib.rs # Module exports
│ ├── paths.rs # Path utilities for btrfs layout
│ ├── health.rs # Health monitoring
│ │
│ ├── cli/ # Command-line parsing
│ │ ├── mod.rs
│ │ └── args.rs # Clap structures
│ │
│ ├── commands/ # CLI command implementations
│ │ ├── mod.rs
│ │ ├── common.rs # Shared utilities
│ │ ├── exec.rs # fcvm exec
│ │ ├── ls.rs # fcvm ls
│ │ ├── podman.rs # fcvm podman run
│ │ ├── setup.rs # fcvm setup
│ │ ├── snapshot.rs # fcvm snapshot {create,serve,run} + UFFD server
│ │ └── snapshots.rs # fcvm snapshots
│ │
│ ├── firecracker/ # Firecracker integration
│ │ ├── mod.rs
│ │ ├── api.rs # API client (hyper + hyperlocal)
│ │ └── vm.rs # VM manager
│ │
│ ├── network/ # Networking
│ │ ├── mod.rs
│ │ ├── bridged.rs # Bridged networking (iptables)
│ │ ├── pasta.rs # Rootless networking (pasta)
│ │ ├── routed.rs # Routed networking (IPv6 veth)
│ │ ├── namespace.rs # Network namespace management
│ │ ├── veth.rs # Veth pair management
│ │ ├── types.rs # Network types
│ │ ├── portmap.rs # Port mapping utilities
│ │ └── egress_proxy.rs # Host-side multiplexed egress proxy
│ │
│ ├── storage/ # Storage & snapshots
│ │ ├── mod.rs
│ │ ├── disk.rs # btrfs CoW disk management
│ │ ├── snapshot.rs # Snapshot management
│ │ └── volume.rs # Volume handling
│ │
│ ├── state/ # VM state management
│ │ ├── mod.rs
│ │ ├── types.rs # VmState, VmConfig
│ │ ├── manager.rs # StateManager (CRUD + loopback IPs)
│ │ └── utils.rs # State utilities
│ │
│ ├── uffd/ # UFFD memory server
│ │ ├── mod.rs
│ │ ├── server.rs # Userfaultfd page handler
│ │ ├── handler.rs # UffdHandler: spawns an external uffd_handler binary; nothing calls it and no crate builds that binary
│ │ ├── working_set.rs # Record/persist the restore working set
│ │ ├── prefetch.rs # Replay it into a restoring clone
│ │ └── warmup.rs # Read it into the page cache ahead of replay
│ │
│ ├── volume/ # FUSE volume handling
│ │ └── mod.rs # Host → guest filesystem mapping
│ │
│ └── setup/ # Setup utilities
│ ├── mod.rs
│ ├── kernel.rs # Kernel + firecracker setup/build
│ ├── pasta.rs # Pinned pasta build (upstream commit)
│ ├── rootfs.rs # Rootfs setup
│ └── storage.rs # btrfs storage setup
│
├── fc-agent/ # Guest agent crate
│ ├── Cargo.toml
│ └── src/
│ ├── main.rs # Entry point
│ ├── agent.rs # MMDS + Podman orchestration
│ ├── container.rs # Container lifecycle management
│ ├── exec.rs # Exec command handler
│ ├── mmds.rs # MMDS client + restore-epoch watcher
│ ├── mounts.rs # Mount setup (overlayfs, volumes)
│ ├── network.rs # ARP/NDP, TCP cleanup, localhost forwarding
│ ├── output.rs # Container log streaming via vsock
│ ├── proxy.rs # Guest-side multiplexed egress proxy
│ ├── restore.rs # Snapshot restore handler
│ ├── system.rs # System setup (sysctl, cgroups)
│ ├── tty.rs # TTY/PTY handling
│ ├── types.rs # Shared types (MMDS config)
│ └── vsock.rs # Vsock connection utilities
│
├── fuse-pipe/ # FUSE passthrough library
│ ├── Cargo.toml
│ ├── src/
│ │ ├── client/ # FUSE client (mounts in VM)
│ │ ├── server/ # Async server (runs on host)
│ │ ├── protocol/ # Wire protocol (request/response)
│ │ └── transport/ # vsock/Unix socket transport
│ ├── tests/ # Integration tests
│ └── benches/ # Performance benchmarks
│
└── tests/ # fcvm integration tests
├── common/mod.rs # Shared test utilities
├── test_sanity.rs # VM sanity tests (rootless + bridged)
├── test_snapshot_clone.rs # Snapshot/clone workflow tests
├── test_egress.rs # Egress proxy tests (rootless + bridged)
├── test_egress_stress.rs # Egress proxy stress tests
├── test_egress_proxy_bench.rs # 8000-connection benchmark
├── test_port_forward.rs # Port forwarding tests
├── test_rootless_ipv6.rs # IPv6 networking tests
├── test_exec.rs # Exec command tests
├── test_fuse_in_vm_matrix.rs # In-VM pjdfstest
├── test_localhost_image.rs
├── test_state_manager.rs
├── test_health_monitor.rs
└── ... # 35+ additional test files
fcvm (Cargo.toml):
[dependencies]
anyhow = "1"
clap = { version = "4", features = ["derive"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.11", features = ["json", "rustls-tls"] }
which = "6"
nix = { version = "0.29", features = ["user", "process", "signal", "ioctl", "net"] }
uuid = { version = "1", features = ["v4", "serde"] }
sha2 = "0.10"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
libc = "0.2"
hex = "0.4"
chrono = { version = "0.4", features = ["serde"] }
tempfile = "3"
rand = "0.8"
async-trait = "0.1"
hyper = { version = "0.14", features = ["client", "http1"] }
hyperlocal = "0.8"fc-agent (fc-agent/Cargo.toml):
[dependencies]
anyhow = "1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "process", "io-util"] }
reqwest = { version = "0.11", features = ["json", "rustls-tls"] }All builds are done via the root Makefile. Run make help for the complete target list (see also CLAUDE.md).
make build # Build fcvm + fc-agent
make test-root # Run all tests (requires sudo + KVM)
make help # Show available targetsAll fcvm data is stored under /mnt/fcvm-btrfs/ (btrfs filesystem for CoW reflinks).
Override with FCVM_BASE_DIR environment variable.
Layout (from src/paths.rs):
/mnt/fcvm-btrfs/
├── kernels/ # Kernel binaries
│ └── vmlinux-{sha}.bin
├── rootfs/ # Base rootfs images (contains /etc/fcvm-setup-complete marker)
│ └── layer2-{sha}.raw
├── initrd/ # fc-agent injection initrds
│ └── fc-agent-{sha}.initrd
├── vm-disks/ # Per-VM CoW disk copies
│ └── {vm-id}/disks/rootfs.raw
├── snapshots/ # Firecracker snapshots
├── state/ # VM state JSON files
│ └── {vm-id}.json
└── cache/ # Downloaded images and packages
├── ubuntu-24.04-arm64-{sha}.img # Cloud image cache
└── packages-{sha}/ # Downloaded .deb files
Rootfs Hash Calculation: The layer2-{sha}.raw name is computed from:
- Init script (embeds install + setup scripts)
- Kernel URL
- Download script (package list + Ubuntu codename)
This ensures automatic cache invalidation when any component changes.
VM State (/mnt/fcvm-btrfs/state/{vm-id}.json):
{
"schema_version": 1,
"vm_id": "vm-abc123...",
"name": "my-nginx",
"status": "running",
"health_status": "healthy",
"exit_code": null,
"pid": 12345,
"created_at": "2025-01-09T12:00:00Z",
"last_updated": "2025-01-09T12:00:05Z",
"config": {
"image": "nginx:alpine",
"vcpu": 2,
"memory_mib": 1024,
"network": {
"tap_device": "tap-abc123",
"guest_ip": "172.16.29.2",
"loopback_ip": "127.0.0.2"
},
"volumes": [],
"process_type": "vm",
"snapshot_name": null,
"serve_pid": null
}
}Strategy: Use anyhow::Result everywhere, with context.
Example:
use anyhow::{Context, Result, bail};
async fn setup_network() -> Result<NetworkConfig> {
create_tap_device("tap0")
.await
.context("creating TAP device for VM network")?;
add_to_bridge("tap0", "fcvmbr0")
.await
.context("adding TAP to bridge")?;
Ok(NetworkConfig { ... })
}User-facing errors:
// In main.rs
if let Err(e) = run().await {
eprintln!("Error: {:#}", e); // Pretty print error chain
std::process::exit(1);
}Setup (in main.rs):
use tracing_subscriber::{fmt, EnvFilter};
fn main() {
tracing_subscriber::fmt()
.with_env_filter(EnvFilter::from_default_env())
.with_target(false)
.init();
// ...
}Usage:
use tracing::{info, warn, error, debug};
info!(vm_id = %vm.id(), "starting VM");
warn!(tap = "tap0", "TAP device already exists");
error!(error = %e, "failed to start Firecracker");
debug!(config = ?config, "loaded configuration");Environment:
# Set log level
export RUST_LOG=fcvm=debug
# Run with debug logs
RUST_LOG=trace fcvm run nginx:latest
# Firecracker's own level for the per-VM firecracker.log: Error, Warning, Info (default) or Debug.
# Debug adds one line per vsock packet, which is gigabytes an hour under FUSE-over-vsock volumes.
FCVM_FIRECRACKER_LOG_LEVEL=Debug fcvm podman run nginx:alpineNetwork Mode Guards: The fcvm binary enforces proper network mode usage:
- Bridged without root: Fails with helpful error message suggesting
sudoor--network rootless - Rootless with root: Runs but prints warning that bridged would be faster
Test Isolation: All tests use unique resource names to enable parallel execution:
unique_names()helper generates timestamp+counter-based names- PID-based naming for additional uniqueness
- Automatic cleanup on test exit
Test Tier Organization (feature-gated):
test-unit: No feature flags, fast tests without VMstest-integration-fast:--features integration-fast,privileged-tests(quick VM tests <30s)test-root: All features includingintegration-slow(pjdfstest, slow VM tests)- Filter by name pattern:
make test-root FILTER=exec - Container configs:
CONTAINER_RUN_ROOTLESS(unit) andCONTAINER_RUN_ROOT(VM tests)
Test individual components in isolation:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_port_mapping() {
let pm = PortMapping::parse("8080:80").unwrap();
assert_eq!(pm.host_port, 8080);
assert_eq!(pm.guest_port, 80);
assert_eq!(pm.proto, Protocol::Tcp);
}
#[tokio::test]
async fn test_firecracker_client() {
// Mock Firecracker API
// Test API calls
}
}Test full workflows:
#!/bin/bash
# tests/integration/test_run.sh
# Test rootless mode (no sudo required)
fcvm podman run --name test-nginx --network rootless nginx:alpine &
PID=$!
sleep 5
kill $PID
# Test bridged mode with port forwarding (requires sudo for iptables/TAP)
sudo fcvm podman run --name web --network bridged --publish 8080:80 nginx:alpine &
PID=$!
sleep 5
curl http://localhost:8080 # Should return nginx page
kill $PID
# Test snapshot & clone (rootless with port forwarding)
fcvm podman run --name baseline --network rootless --publish 9090:80 nginx:alpine &
BASELINE_PID=$!
sleep 5 # Wait for VM to be healthy
# Create snapshot
fcvm snapshot create --pid $BASELINE_PID --tag warm
# Start memory server
fcvm snapshot serve warm &
SERVE_PID=$!
sleep 2
# Spawn clone (inherits network mode + port mappings from snapshot)
fcvm snapshot run --pid $SERVE_PID --name clone1 &
CLONE_PID=$!
sleep 2
curl http://localhost:9090 # Should return nginx page in <2s
kill $CLONE_PID $SERVE_PID $BASELINE_PIDNote: Network mode is set on the baseline VM: --network rootless (default, no root required) or --network bridged (iptables/TAP, requires sudo). Clones inherit the network mode and port mappings from the snapshot automatically.
The fuse-pipe library passes the pjdfstest POSIX compliance suite. Tests run via make test-root or make container-test-root.
Test Counts:
- 237 total test files in pjdfstest
- 54 skipped on Linux (FreeBSD/ZFS/UFS-specific)
- 183 real test files run
- 8789 assertions pass
Skipped Categories (via quick_exit() - outputs trivial "ok 1"):
| Category | Files | Skipped | Real | Reason |
|---|---|---|---|---|
| granular | 7 | 7 | 0 | FreeBSD extended ACLs only |
| open | 26 | 8 | 18 | FreeBSD-specific open behaviors |
| link | 18 | 6 | 12 | FreeBSD hardlink semantics |
| rename | 25 | 5 | 20 | FreeBSD rename edge cases |
| rmdir | 16 | 4 | 12 | FreeBSD rmdir behaviors |
| ftruncate | 15 | 3 | 12 | FreeBSD:UFS specific |
| mkdir | 13 | 3 | 10 | FreeBSD:UFS specific |
| mkfifo | 13 | 3 | 10 | FreeBSD:UFS specific |
| symlink | 13 | 3 | 10 | FreeBSD:UFS specific |
| truncate | 15 | 3 | 12 | FreeBSD:UFS specific |
| unlink | 15 | 3 | 12 | FreeBSD:UFS specific |
| chflags | 14 | 2 | 12 | Some UFS-specific flags |
| chmod | 13 | 2 | 11 | FreeBSD:ZFS specific |
| chown | 11 | 2 | 9 | FreeBSD:ZFS specific |
| mknod | 12 | 0 | 12 | All run |
| posix_fallocate | 1 | 0 | 1 | All run |
| utimensat | 10 | 0 | 10 | All run |
Skip mechanism: Tests check ${os}:${fs} and call quick_exit() for unsupported OS/filesystem combinations. This outputs TAP format 1..1 + ok 1 (trivial pass) rather than running real assertions.
Goal: <1 second from fcvm clone to ready
Breakdown:
- Snapshot load: ~200ms
- Network setup: ~100ms
- Identity patching: ~50ms
- VM resume: ~300ms
- Container ready: ~300ms
- Total: ~950ms
Optimizations:
- Pre-warmed snapshot (container already running)
- CoW disks (no disk copy)
- Shared memory pages
- Fast network setup (TAP device creation)
Memory:
- Base VM: ~100MB overhead
- Shared kernel + rootfs: ~200MB (shared across all VMs)
- Per-VM: Container memory + ~100MB overhead
Example: 10 nginx VMs
- Traditional VMs: 10 × 512MB = 5GB
- fcvm with cloning: 200MB (shared) + 10 × 150MB = 1.7GB
- Savings: ~66%
CPU:
- Support vCPU overcommit (e.g., 32 vCPUs on 8 cores)
- KVM handles scheduling efficiently
- Minimal overhead when VMs are idle
- VM-level isolation: Full hardware virtualization via KVM
- No shared kernel: Each VM has its own kernel
- No container escape: Podman runs inside VM, not on host
- No root required: Entire stack runs as regular user
- User namespaces: pasta uses user namespaces
- No privileged operations: No sudo, no CAP_NET_ADMIN
- Requires CAP_NET_ADMIN: For TAP/iptables setup
- Minimal privileges: Only for network setup, not VM execution
- Firecracker jailer: Can use jailer for additional sandboxing (future)
- Snapshot contains full VM state: Including memory (may have secrets)
- Encrypt snapshots: Option to encrypt at rest (future)
- Access control: Snapshots stored in user-owned directories
Every kernel in fcvm is delivered through an explicit profile. The shipped default, nested, and btrfs profiles are built from source and downloaded from content-addressed GitHub releases. User-defined profiles may instead use an archive URL.
A profile delivers a kernel in one of three ways:
- Source release (the shipped profiles): Downloads a published kernel or builds it locally from
kernel_version/kernel_repo - URL-based: Downloads a pre-built kernel archive
- Inherited: Uses the default profile's kernel, adding only runtime overrides (boot_args, firecracker_args, etc.)
# Custom kernel profile (build from source)
[kernel_profiles.minimal.amd64]
description = "Minimal kernel for fast boot"
kernel_version = "6.12"
kernel_repo = "your-org/your-kernel-repo"
build_inputs = ["kernel/minimal.conf", "kernel/patches/*.patch"]
kernel_sha = "0123456789ab" # required when publishing the profile
kernel_config = "kernel/minimal.conf"
patches_dir = "kernel/patches"
# firecracker_bin = "/usr/local/bin/firecracker-custom"
# firecracker_args = "--extra-flag"
# boot_args = "quiet"| Field | Required | Description |
|---|---|---|
kernel_url |
URL-based | URL to kernel archive (e.g., Kata release tarball) |
kernel_archive_path |
URL-based | Path within the archive to extract the kernel binary |
kernel_local_path |
No | Local filesystem path to kernel binary (overrides URL) |
kernel_version |
Custom | Kernel version (e.g., "6.18.3") |
kernel_repo |
Custom | GitHub repo for releases |
build_inputs |
Custom | Files to hash for kernel SHA (supports globs) |
kernel_sha |
Published custom | Verified 12-digit build-input hash used outside a source checkout |
base_config_url |
Custom | Base kernel .config URL (e.g., Firecracker's microvm config) |
kernel_config |
No | Kernel config fragment file path (applied on top of base) |
patches_dir |
No | Directory containing kernel patches |
firecracker_bin |
No | Custom Firecracker binary path |
firecracker_args |
No | Extra Firecracker CLI args |
boot_args |
No | Extra kernel boot parameters |
rootfs_type |
No | Root filesystem type: "ext4" (default) or "btrfs" (converts via btrfs-convert) |
nvidia |
No | Table pinning an NVIDIA driver and container toolkit; setup builds a driver image for the profile's kernel (see GPU Passthrough) |
- Config is source of truth: All kernel versions and build settings flow from
rootfs-config.toml - SHA computation: fcvm hashes all files matching
build_inputsand verifies a published profile'skernel_sha; installed binaries use that manifest value when source files are absent - Download first: Tries
kernel_reporeleases with tagkernel-{profile}-{version}-{arch}-{sha} - Build fallback: If download fails and
--build-kernelsis set, Rust generates build scripts on-the-fly - Config sync:
make buildsyncs embedded config to~/.config/fcvm/
The rootfs is built from rootfs-config.toml:
[base]
version = "24.04"
codename = "noble"
[packages]
runtime = ["podman", "crun", "fuse-overlayfs", "skopeo"]
fuse = ["fuse3"]
system = ["haveged", "chrony"]
debug = ["strace"]
[services]
enable = ["haveged", "chrony", "systemd-networkd"]
disable = ["snapd", "cloud-init"]
[files."/etc/myconfig"]
content = """
my custom config
"""After changing the config, run fcvm setup to rebuild the rootfs with the new SHA.
A container in an fcvm clone can use an NVIDIA GPU passed through to the VM
with VFIO. The golden snapshot is taken with Firecracker's PCI transport
(--enable-pci) and no GPU. Each clone restores from it, Firecracker attaches
the GPU to the paused VM (device_passthrough in the LoadSnapshot body), and
the guest makes it usable before the container starts. This section covers the
guest: the gpu kernel profile, the NVIDIA driver image, and fc-agent.
[kernel_profiles.gpu.{arm64,amd64}] builds the default kernel plus
kernel/gpu.conf (kernel/gpu-x86.conf):
| Option | Why |
|---|---|
CONFIG_PCI, CONFIG_PCI_MSI, CONFIG_PCI_HOST_GENERIC / CONFIG_PCI_MMCONFIG, CONFIG_ARM_GIC_V3_ITS |
Firecracker's PCI transport: ECAM config space and MSI-X only (no INTx). All are in Firecracker's base config; declaring them makes the build fail if a base-config change drops one. |
CONFIG_MODULES, CONFIG_MODULE_UNLOAD |
The NVIDIA driver is out of tree. |
CONFIG_FW_LOADER |
GSP firmware, which the open modules boot on Turing and later. |
CONFIG_MMU_NOTIFIER |
nvidia-uvm calls __mmu_notifier_register(). The option has no prompt upstream and nothing in this kernel selects it, so kernel/patches-gpu carries a one-line Kconfig patch giving it one. |
It is a separate profile, not the default, because module support and the
Kconfig patch would change the default kernel's release identity and every
default snapshot cache key for VMs that never see a GPU, and because GPU VMs
need the device-passthrough Firecracker build and --enable-pci, which the
profile carries (firecracker_branch = "agent/vfio-passthrough-hotplug"). No
release is published for it yet; make setup-gpu builds it locally.
[kernel_profiles.gpu.<arch>.nvidia] pins one driver version per architecture
(amd64 580.196.07, the R580 long-term branch; arm64 595.112.07, the first branch
whose cuda-checkpoint supports ARM CPUs; both support Turing T4/T4G) and one
nvidia-container-toolkit version (1.20.1), each by URL and SHA-256.
fcvm setup --kernel-profile gpu builds a read-only ext4 image
(src/setup/nvidia.rs) holding:
lib/modules/<release>/extra/{nvidia,nvidia-uvm}.ko, nvidia.params
lib/firmware/nvidia/<version>/gsp_{tu10x,ga10x}.bin
usr/lib/<triplet>/lib*.so.<version> and their soname links
usr/bin/{nvidia-smi,nvidia-persistenced,nvidia-debugdump,nvidia-cuda-mps-*}
usr/bin/{nvidia-ctk,nvidia-cdi-hook}
The build configures and builds the profile's kernel tree from the same
inputs as the kernel artifact (make <image> modules, so Module.symvers
exists and modpost checks every symbol), then builds the modules from the
driver's .run against it. Modules, libraries and firmware come from that one
.run, and the build refuses a module whose modinfo version is not the
pinned version or whose vermagic is not this kernel.
The table's module_type picks the modules:
module_type |
Sources | nvidia.ko parameters | GPU initialisation on a T4 |
|---|---|---|---|
open (default) |
kernel-open/ |
none | boots the GSP firmware: 1,113 to 1,156 ms |
proprietary |
kernel/ |
NVreg_EnableGpuFirmware=0 |
resource manager on the CPU: 576 to 623 ms |
The times are nvidia-persistenced's start in a gpu VM with the driver
unloaded and the GPU removed and rescanned, two rounds per variant, twice; the
proprietary modules with the GSP firmware left on took 1,111 to 1,223 ms. The
stage writes the parameters to nvidia.params beside the modules and fc-agent
passes them to insmod. The gpu profile uses proprietary on amd64 and on
arm64. On a g5g.metal (T4G) the proprietary modules took nvidia-persistenced's
start from 1,285 to 609 ms and gpu_ms from 1,823 to 1,026, medians of eight
restores each (see "arm64: g5g.metal" under Verified on hardware). With the GSP firmware
off the proprietary modules are allowed on Turing and Ampere GPUs only:
fcvm podman run --gpu refuses any other NVIDIA GPU before launch when the
profile builds them, and names module_type = "open" as the setting that runs
it. fcvm snapshot run --gpu restores the driver image the snapshot recorded,
which the current profile does not name, so it does not check; the guest's
attach fails closed for a GPU that driver cannot run.
The stage also lets nvidia.ko load with no GPU on the bus. nvidia_init_module
refuses an empty bus unless an NVSwitch is present, and all three refusals
test one local, is_nvswitch_present; the stage sets it to NV_TRUE in the
chosen sources' nv.c and fails the build if the declaration is not there
exactly once. The image is named
nvidia-<version>-<profile>-<kernel version>-<arch>-<key>.ext4 under
<assets_dir>/nvidia/, where the key hashes the kernel's identity, the pinned
table and the build stage. It is not in the rootfs: VMs without a GPU do not
carry it, and a driver bump does not rebuild the rootfs.
- Golden VM boot. The host attaches the driver image
(
crate::setup::ensure_nvidia_driver_image("gpu", allow_create)) as a read-only virtio-blk drive and names it in the container plan:"gpu_driver": {"device": "/dev/vdX"}. fc-agent mounts it read-only at/run/fcvm-gpu/driverwith the extra disks, before the pre-start snapshot, so every clone has it. It is not mounted into the container. Right after the mount, fc-agent pointsfirmware_class.pathat the image's firmware and loadsnvidia.koandnvidia-uvm.kowith no GPU present, so the snapshot carries the loaded driver. A failure there is logged and not fatal: the attach loads what is not loaded. The image path belongs in the snapshot cache key (it is content-addressed). It cannot be a pmem device: a restore with device passthrough refuses pmem. - Restore. The restore metadata document (the vsock
latestdocument and its MMDS mirror) carries"gpus": ["0000:01:00.0", ...], the host PCI addresses in attach order, and omits the key without GPUs. fc-agent only compares the count with what the guest bus shows; guest addresses are the VMM's. - Before the ACK. With a non-empty
gpus,handle_clone_restorestartsgpu::attachat the top of the restore sequence, concurrent with the network phases, and awaits it before publication.attach:- refuses if the container is already running (a clone of a snapshot taken after the container started cannot give the container a device);
- checks the driver image holds modules for the running kernel release;
- writes
/sys/bus/pci/rescanand waits up to 10s for that many NVIDIA display functions (vendor 0x10de, class 0x03xxxx); - points
firmware_class.pathat the image's firmware andinsmodsnvidia.koandnvidia-uvm.koif the boot did not load them (when it did, the rescan's synchronous probe binds the GPU to the loaded driver); - checks
/proc/driver/nvidia/gpuslists every GPU and creates/dev/nvidiactl,/dev/nvidia<minor>,/dev/nvidia-uvm{,-tools}; - starts
nvidia-persistencedfrom the image, which initialises the GPU and keeps it open, so later processes do not initialise it again; - zeroes every byte of each GPU's memory it can allocate with the image's
fcvm-gpu-scrub(see GPU memory between tenants) and fails if any GPU was not zeroed; - runs, together,
nvidia-smi -Lfrom the image, requiring one line per GPU, andnvidia-ctk cdi generate --driver-root=/run/fcvm-gpu/driverinto/var/run/cdi/nvidia.yaml. The spec cannot be generated before the GPU is there: it names each GPU by index and by UUID.
- Container start. The container starts after the restore is
acknowledged, as for any warm start, with
--device nvidia.com/gpu=all. The ACK's phase record carriesgpu_ms,gpusandgpu_steps, the milliseconds of each attach step (check_ms,rescan_ms,nvidia_ms,uvm_ms,nodes_ms,persistenced_ms,scrub_ms,smi_ms,cdi_ms; the last two overlap). fc-agent also logs each asgpu: step=<name> elapsed_ms=<ms>.
Any failure is fatal to the clone before the ACK: fc-agent prints
FATAL: clone restore failed closed ... restore phase gpu-attach: <cause> and
shuts the VM down with status 1. Each cause names what was expected and what
the guest saw: the PCI functions present after a rescan, the kernel release
the image was built for, the kernel log's NVRM lines after a failed insmod
or nvidia-smi.
A GPU goes from one VM to the next (a wall GPU lent to a visitor and given back, any pool of GPUs). NVIDIA documents that newly allocated memory is not cleared, and documents a scrub on reset only for H100 confidential computing. Two layers keep one VM's GPU memory from the next, each failing closed:
In the guest, on every restore. After nvidia-persistenced has
initialised the GPUs and before nvidia-smi -L, the CDI spec and the
container, fc-agent runs /run/fcvm-gpu/driver/usr/bin/fcvm-gpu-scrub
(src/setup/nvidia_gpu_scrub.c, compiled into the driver image by the image
stage). Per GPU it creates a CUDA context, allocates until the device refuses
2 MiB, memsets every allocation to zero, synchronises and frees. It loads the
image's libcuda.so.1 with dlopen, so the stage needs only gcc, and the
stage refuses a binary that needs a glibc newer than 2.34. A context costs
guest memory, so fc-agent passes --parallel N from MemAvailable (256 MiB
for the first context, 64 MiB for each further one), and the tool sets
CUDA_DEVICE_MAX_CONNECTIONS=1 and CUDA_MODULE_LOADING=LAZY, which cut a
context from about 230 to about 185 MiB. A failed scrub, a missing tool, or
fewer GPUs zeroed than the host attached fails the attach, so the clone is
never acknowledged. This layer holds even when the previous VM was killed and
nothing ran on the host in between.
On the host, between VMs. sudo fcvm gpu scrub <sbdf>...
(src/gpu/scrub.rs), for the fleet agent to run after a VM's Firecracker has
exited and before the GPU goes to another VM. On Turing, Ampere and Ada a GPU
clears its own memory after a secondary bus reset and reports when it has
finished; the scrub follows NVIDIA's gpu-admin-tools --reset-with-sbr --clear-memory for that:
- Refuse unless the GPU is NVIDIA, on vfio-pci, and its
reset_methodisbus. - Take the GPU's lock (
<sbdf>.lock, exclusively and without waiting) and open its VFIO group, and hold both until the outcome is written. A group has one opener, so no VM can open the GPU mid-scrub, and a GPU a VM still holds (its lease on the lock, or its VFIO group) is reportedbusyand left alone. Before touching the GPU, mark its record quarantined ("a scrub started and has not recorded its outcome"); a scrub that cannot write that mark stops there, and one that dies leaves the GPU refused. - Set
power/controltoon(vfio-pci lets an idle GPU suspend to D3, where BAR0 reads all ones) and enable memory decoding; both are restored after. - Check
PMC_BOOT_0names Turing, Ampere or Ada; set VBIOS scratch 22 (0x1458) to 1. - Reset through sysfs
reset, then require the scratch to read 0: the reset happened, so the clear-finished bit below is from this reset. - Wait up to 5 s each for boot (
0x118234 & 0x3ff == 0x3ff) and for the clear (0x100b20 & 1). A read of all ones or 0xbadfxxxx never counts as done.
Each GPU's record is /var/lib/fcvm/gpu/<sbdf>.json (FCVM_GPU_STATE_DIR):
scrubs, failures, quarantined, the last error and the last timings. A
failed scrub quarantines the GPU, and check_host_device refuses a
quarantined GPU, so podman run --gpu and snapshot run --gpu do not hand it
out until a later scrub of it succeeds. An unreadable record also refuses.
fcvm gpu scrub prints one JSON line per GPU with outcome scrubbed,
failed or busy and exits 1 unless every GPU was scrubbed; fcvm gpu status prints every record. Only root runs the scrub. Records are written to a
synced temporary file, renamed, and the directory synced.
The lease. podman run --gpu and snapshot run --gpu take a shared
flock on /var/lib/fcvm/gpu/<sbdf>.lock before the quarantine check, without
waiting, so a launch during a scrub is refused. snapshot run, the restore
that attaches the GPU, holds it until its VM has been torn down (the kernel
drops it if fcvm is killed), so a scrub cannot start between the check and
the VMM opening the GPU, nor while the VM runs. podman run releases its
lease after the check: its cold-booted VM has no GPU, and the restore that
attaches one takes its own. Each lease is its own open file, so one process
running several VMs holds one per VM. A rootless run cannot create the lock
file in the root-owned directory and is refused when it is missing; root
runs, the scrub and the fleet host's boot (fleet/ami/host/bind-vfio.sh)
create it, mode 0644.
Measuring it. make gpu-residue-test GPU=<sbdf> (scripts/gpu-residue/):
VM A fills all the memory it can allocate with 16-byte records (a 64-bit
marker and the offset) and then its container exits, or its fcvm is
SIGKILLed; VM B, restored on the same GPU, allocates as much with cudaMalloc,
cuMemAlloc or cudaMallocManaged, writes nothing, and counts the marker and
the non-zero words in a GPU kernel. A's fill reads its records back before it
reports, and each B scan then writes one marker into its own memory and fails
unless a second scan counts exactly one more. A same-vm case runs fill and
scan as two processes in one VM. HOST_CLEAR="sudo fcvm gpu scrub <sbdf>"
runs the host scrub between A and B.
Measured on 2026-10-23, g4dn.metal (T4, reset_method bus), driver
580.196.07, the default 1 GiB guest, with both module types:
- open modules (
kernel-open/, GSP firmware booted):cudaMemGetInfototal 15,636,037,632 bytes; B allocated 15,521,021,952 bytes (cudaMalloc,cuMemAlloc) and 15,456,010,240 (cudaMallocManaged); - proprietary modules with
NVreg_EnableGpuFirmware=0, the amd64 default since #1149 (in the guest,/proc/driver/nvidia/versionreads "NVIDIA UNIX x86_64 Kernel Module" and/proc/driver/nvidia/paramsEnableGpuFirmware: 0):cudaMemGetInfototal 15,828,320,256 bytes; B allocated 15,713,959,936 and 15,648,948,224 bytes, 184 MiB more, since no GSP firmware is resident. No scrub and host only ran 2cb12514 (#1149 merged, before the scrub); in-guest only, both, and the host scrub ran 65122f8c (#1158 merged). The box was a fresh one for this run alone, checked before and after each matrix: CPU 98.97 to 100 percent idle over 5 s, no process holding a VFIO group, no fcvm or Firecracker left between runs; the 1-minute load average read 6.33 before the first matrix (a build had finished 2 minutes earlier) and 1.04 to 2.45 after it.
Each layer ran the nine cases (same-vm, exit and kill, each with the three allocators). The host scrub runs between two VMs, so it acted on the six exit and kill cases; same-vm is one VM and has no host step:
| modules | layers | marker words in B | non-zero words in B | scrub time |
|---|---|---|---|---|
| open | none | 0 in all 9 | 0 in all 9 | none |
| open | host only | 0 in all 9 | 0 in all 9 | host 1.21 to 1.24 s per scrub, wall |
| open | in-guest only | 0 in all 9 | 0 in all 9 | scrub_ms 413 to 459 |
| open | both | 0 in all 9 | 0 in all 9 | scrub_ms 401 to 450, host as above |
| proprietary | none | 0 in all 9 | 0 in all 9 | none |
| proprietary | host only | 0 in all 9 | 0 in all 9 | host 1.22 to 1.24 s per scrub, wall |
| proprietary | in-guest only | 0 in all 9 | 0 in all 9 | scrub_ms 280 to 328 |
| proprietary | both | 0 in all 9 | 0 in all 9 | scrub_ms 282 to 322, host 1.21 to 1.24 s |
Nothing was found even without either layer: on this driver a later process of the same VM also gets zeroed memory, and the GPU runs its own clear after the bus reset VFIO gives it when a VM opens or releases it. Neither is a documented guarantee, and neither holds for a GPU or a host that resets another way, which is why both layers stay.
Costs:
- In the guest, one T4, open modules:
scrub_ms410 to 447 over six restores, andgpu_ms1,693 to 1,846 against 1,277 to 1,647 without the scrub. Inside the tool (347 to 401 ms):cuInit47 to 68, context 75 to 87, alloc 3 to 99, zeroing 15.5 GB 64 to 73, free 31 to 37, context destroy about 90. - In the guest, one T4, proprietary modules:
scrub_ms283 to 291 over six restores, andgpu_ms970 to 1,012 against 681 to 804 without the scrub (15 restores of 2cb12514). - Open modules, two T4s in one VM: 632 to 646 ms. Eight in the 1 GiB guest (two at a time):
2,609 to 2,674 ms,
gpu_ms11,798 to 11,834 against 9,249 on main. Before the contexts were bounded, eight at once OOM-killed the scrub and the guest. - On the host, open modules: 1,178 to 1,212 ms per T4 over 17 scrubs (reset 179 to 188, boot 95 to 103, clear 872 to 904); all eight GPUs at once, 1.23 s wall. gpu-admin-tools took 1.46 s for the same on one T4.
Which layer: the in-guest scrub is on every restore's critical path (0.3 to 0.4 s per GPU, and guest memory for a context) and holds whatever happened before the VM started. The host scrub is off the critical path (between tenants), confirms through the GPU's own registers that its memory was cleared, and quarantines a GPU that does not, but it runs only when the fleet agent runs it and only on Turing to Ada with a bus reset.
A restore ends at an empty GPU, so a model server in a clone loads its weights,
compiles and allocates again before it answers. fcvm snapshot create --gpu-warm
takes a snapshot of a VM that has its GPUs attached and its CUDA processes running,
with their GPU memory in the snapshot. Firecracker cannot save a passthrough
device, so the GPU state leaves the GPU first, through NVIDIA's cuda-checkpoint
(cuCheckpointProcess*, R550 and later; --device-map from R580), which the driver
image carries at usr/bin/cuda-checkpoint.
cuda-checkpoint supports ARM CPUs from NVIDIA driver 595 (its README, "595 Features"), which is why the arm64 gpu profile pins 595.112.07 (measured on arm64 below). If the source does not get its GPUs back after a snapshot that was taken, the snapshot is still recorded as the VM's diff parent before the failure is reported: the VMM has reset its dirty pages against it.
The sequence (crate::gpu::warm_gpu_snapshot, fc-agent/src/gpu_park.rs):
- Park (
fc-agent --gpu-parkover the exec channel). fc-agent finds every process with a/dev/nvidia*descriptor open, other than nvidia-persistenced, and refuses unless cuda-checkpoint reports each onerunning. It locks all of them, then checkpoints all of them: the driver copies each process's device memory into the process's own memory and closes its GPU handles. It stops nvidia-persistenced, checks that nothing holds an NVIDIA device, removes each GPU from the guest's PCI bus (/sys/bus/pci/devices/<addr>/remove) and writes/run/fcvm-gpu/parked.json(pids, GPU UUIDs, guest addresses). /run is a tmpfs, so the record is in the snapshot. A failure undoes every step taken. - Detach.
DELETE /device-passthrough/gpu<i>for each GPU (Firecracker fork branchagent/vfio-passthrough-hotplug): off the PCI and MMIO buses, device reset, BAR mappings and MSI-X routes removed, group out of the container; the container goes with the last device, which unpins guest memory. - Snapshot, as for any VM. Its metadata records
parked_gpus. - Re-attach each GPU to the source (
PUT /device-passthrough/gpu<i>after boot, which the fork now accepts) and unpark (fc-agent --gpu-unpark): the same bring-up a restore does (rescan, device nodes, nvidia-persistenced,nvidia-smi -L), thencuda-checkpoint --action restorefor each parked process and--action unlock. The source keeps its GPUs; its CUDA work is stopped from the lock to the unlock. Once the park succeeded, step 4 runs whatever steps 2 and 3 did, and a failure there is reported with the snapshot's own result.
A clone is restored with as many --gpu addresses as the snapshot parked
(snapshot run refuses any other count before it starts anything). The GPUs are
attached by the load as for any GPU restore, and the restore's GPU step finds the
parked record: instead of the launch gate (which refuses a GPU for a running
container) it brings the GPUs up and restores each process onto them, with
--device-map <snapshot UUID>=<clone UUID> when the clone has other GPUs than the
source (each list sorted). The kernel modules stay loaded through the park, so the
container's device nodes keep their numbers. The guest needs memory for the GPU
state it holds: a process's device memory is in guest RAM from the park on.
A GPU restore, warm or not, also loads through a UFFD server (snapshot serve, then
snapshot run --pid). Attaching the GPU pins all guest memory for DMA with a
long-term pin_user_pages_remote, whose page walk cannot wait for a userfaultfd
handler: on a page the server had not filled yet it failed with Bad address
(failed to add guest memory map into iommu table). The gpu profile's Firecracker
(fork branch agent/vfio-uffd-populate) runs madvise(MADV_POPULATE_WRITE) over
each memory region before the pin; the populate faults every page as user space
does, the server fills it, and the pin finds it present. Every page is then a
recorded fault, so the snapshot's working set holds all of guest memory and later
clones replay it. Measured on a g4dn.metal, 4 GiB guest, one snapshot: ACK 11.2 s
for the first clone through a serve (every page a demand fault), 3.4 s for clones
that replayed the recorded set (three runs, 3.37 to 3.43 s), against 3.9 s through
the File backend (three runs, 3.86 to 3.92 s). Warm llama.cpp clones through a
serve answered their first token in 3.75 and 4.13 s after the recording clone
(16.2 s), against 5.5 to 6.6 s through the File backend.
Measured on 2026-10-23 on a g4dn.metal (eight T4) with Qwen2.5-0.5B-Instruct and two
servers: vLLM 0.32.0 (vllm/vllm-openai:v0.32.0-cu129, fp16, TRITON_ATTN,
--gpu-memory-utilization 0.2, torch.compile and CUDA graphs on, 8 vCPU and 12 GiB
guests) and llama.cpp's CUDA server (ghcr.io/ggml-org/llama.cpp:server-cuda,
fp16 GGUF, 8 vCPU and 6 GiB guests). The weights are in the image. Time to first
token is from the start of the fcvm command to the first streamed token of a
16-token completion that the host retries every 20 ms. Each clone took the next of
the GPUs other than the source's, so every warm restore used --device-map. 20 runs
each, nearest-rank percentiles:
| server | path | p50 | p95 | min | max |
|---|---|---|---|---|---|
| vLLM | warm: snapshot run --snapshot <warm> --gpu X |
10.30 s | 10.50 s | 9.94 s | 12.26 s |
| vLLM | cold: cache-hit podman run --gpu X, server start, model load |
108.08 s | 111.16 s | 104.49 s | 113.02 s |
| llama.cpp | warm | 5.90 s | 6.00 s | 5.77 s | 6.09 s |
| llama.cpp | cold | 52.63 s | 54.18 s | 50.86 s | 54.20 s |
vLLM's warm restores (p50): load 6,926 ms (12 GiB pinned), GPU step 3,266 ms, of which bring-up 1,132 ms and cuda-checkpoint's restore of its two CUDA processes (the API server and the engine core, 2,896 MiB of device memory) 2,096 ms; ACK 10,268 ms. Its cold start: ACK 8.2 s, model load 4.9 s, engine init 52.4 s (15.9 s of it torch.compile, the rest profiling and CUDA graph capture), first token at about 106 s. Taking the vLLM snapshot took 33.9 s.
Every answer was its source's at temperature 0 (vLLM: " Paris. It was founded in 789
AD by Charlemagne,"; llama.cpp: " Paris. It is the largest city in Europe and the
second largest city in the"). In llama.cpp's warm restores (p50, min to max):
the snapshot load took 3,718 ms (3,594 to 3,841) and vLLM's 12 GiB guests' 6,926 ms:
the load attaches the GPU, which pins all guest memory for DMA, and pinning the File
backend's private mapping for writing copies every page; the GPU step 2,083 ms, of which the bring-up (rescan, nvidia-persistenced
initialising the GPU, nvidia-smi -L) 1,140 ms and cuda-checkpoint's restore and
unlock of 1,124 MiB of device memory 918 ms; the ACK 5,881 ms. In the cold path the
restore ACK came at 4.9 s, llama-server loaded the model in 1.8 s, then spent 31 s
before it reported ready (its warm-up decode), and the first 16 tokens took 12 s,
against 0.15 s in a warm clone.
Taking a warm snapshot of a running warm llama.cpp clone took 9.1 s: park 1,066 ms, detach 573 ms, snapshot 4,363 ms, attach 639 ms, unpark 2,466 ms. The clone kept answering afterwards, and a clone of that snapshot answered on yet another GPU.
Rechecked on 2026-10-25 after rebasing onto the proprietary NVIDIA modules and the in-guest memory scrub, which now runs in the warm restore's bring-up too (llama.cpp, 6 GiB guest, three restores on three GPUs other than the source's, every answer the source's): first token 6.56, 5.53 and 5.52 s; the snapshot took 10.2 s.
On arm64, measured on 2026-10-26 on a g5g.metal (two NVIDIA T4G, driver 595.112.07,
proprietary modules with the GSP firmware off), with the same llama.cpp server
(ghcr.io/ggml-org/llama.cpp:server-cuda, its arm64 image, the same fp16 GGUF in the
image, 8 vCPU and 6 GiB guests, rootless). The source ran on one T4G and every clone
on the other, so every warm restore used --device-map. The fcvm store was a btrfs
loop file on the box's EBS work volume, not local NVMe. 8 runs each, nearest-rank
percentiles, time to first token as above:
| server | path | p50 | p95 | min | max |
|---|---|---|---|---|---|
| llama.cpp | warm | 5.15 s | 6.46 s | 5.14 s | 6.46 s |
| llama.cpp | cold | 67.48 s | 68.46 s | 56.64 s | 68.46 s |
Every answer was the source's (" Paris. It is the largest city in Europe and the second largest city in the"), as on x86. The slowest warm run was the first after the snapshot. Warm restores (p50): snapshot load 3,441 ms, bring-up 842 ms, cuda-checkpoint's restore 750 ms, ACK 5,100 ms. Taking the snapshot took 46.7 s: park 1,201 ms, detach 9,806 ms, snapshot 33,231 ms, attach 720 ms, unpark 1,716 ms. The source answered the same afterwards, and again after the eight clones. The snapshot step writes the 6 GiB memory file to the EBS-backed store.
The detach was 17 times longer than on x86 (9.8 s against 573 ms) and grew with the
guest (6.65 s for 4 GiB) because it unmapped guest memory from the IOMMU. Dropping
the GPU's VFIO device took its group out of the container, and the kernel then
unmapped and unpinned all of guest memory. The g5g.metal's SMMUv3 has no range
invalidation (features 0x00120d4f at boot, bit 15 clear), so every 4 KiB page of
that unmap is one IOTLB invalidation command. bpftrace on a 6 GiB guest's detach
(9,883 ms): 9,495 ms in vfio_unmap_unpin, 9,176 ms of it in
__arm_smmu_tlb_inv_range, over 24,738 command lists; the three device resets took
576 ms. The arm64 gpu profile's Firecracker (#1317) now keeps a detached device's VFIO
device open, reset and with MSI-X disabled, so the container keeps its mappings,
and the attach after the snapshot reuses both instead of mapping guest memory again.
Measured on 2026-10-26 on a g5g.metal, rootless, a VM running sleep infinity with
one T4G, three warm snapshots per size, before and after:
| guest | detach before | detach after | attach before | attach after |
|---|---|---|---|---|
| 4 GiB | 6,678-6,683 ms | 182-191 ms | 611-623 ms | 198-200 ms |
| 6 GiB | 9,812-9,820 ms | 183-187 ms | 703-708 ms | 193-198 ms |
With the CUDA probe below (6 GiB guest, 1 GiB of device memory) the warm snapshot took park 936 ms, detach 190 ms, snapshot 14,084 ms, attach 196 ms, unpark 2,004 ms; the source and a clone on the other T4G kept counting with no mismatched word.
The device memory survives the move: a CUDA program that fills 1 GiB of device
memory with a position-derived pattern, increments a device-side counter every
200 ms and compares the whole buffer against the pattern on every step (4 GiB
guest) was snapshotted with --gpu-warm and restored eight times onto the other
T4G. Every clone resumed counting from the value at the park and found no
mismatched word, nvidia-smi in a clone showed the other T4G holding the
process's 1,128 MiB, and the source kept counting with no mismatch through the
snapshot and all eight clones.
On 2026-10-22, on a g4dn.metal (x86_64, eight NVIDIA T4, each bound to vfio-pci), with
the gpu profile built by make setup-gpu:
- A gpu-profile VM boots under
--enable-pci, mounts the driver image fromgpu_driver, takes the pre-start snapshot, and is restored with--gpu <sbdf>attached throughdevice_passthrough; in the containernvidia-smilists aTesla T4with driver 580.196.07. - The restore's
gpu_ms(rescan, module load, device nodes,nvidia-smi -L, CDI generation) was 3,337, 3,220, 3,028, 3,078 and 3,061 ms over five restores; the restore ACK came 4,157 to 4,569 ms after the load. - A cache-hit
fcvm podman run --gpu ... nvidia-smi -L, process start to exit, took 6.7 s; two at once, on two GPUs from one snapshot, each saw its own T4 (different UUIDs) and finished in 6.9 s together. - NVIDIA's CUDA vector-add sample (
nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0) passes in a gpu VM with the default guest memory, sonvidia-uvm, its device nodes and the CDI injection work for a CUDA process.
A second run on another g4dn.metal the same day:
- Rootless podman in the guest (
--user 1000:1000):nvidia-smi -Llists the T4 and the CUDA vector-add sample passes, as it does with root podman. - In the guest the T4 is
0000:00:06.0, class 0x030200, with BAR0 16 MiB, BAR1 256 MiB and BAR3 32 MiB assigned.dmesgshows the open kernel module 580.196.07 loading. - Without
nvidia-persistencedevery process that opens the GPU initialises it:nvidia-smi -Lin the container took 1.490, 1.436 and 1.447 s; with the daemon running, 0.031, 0.026 and 0.023 s. fc-agent now starts the daemon during the attach.
On 2026-10-23, on a g5g.metal (arm64, two NVIDIA T4G, each bound to vfio-pci), at
59f84996 with the gpu profile built by make setup-gpu, rootless fcvm and the default
1 GiB guest. Every step ran twice: with the profile's open modules, and with
module_type = "proprietary" added to [kernel_profiles.gpu.arm64.nvidia] (GSP
firmware off). Every step passed with both.
fcvm podman run --kernel-profile gpu --gpu <sbdf> ubuntu:24.04 nvidia-smi -L, on a snapshot miss and then on cache hits: the container listsNVIDIA T4Gwith its UUID.gpu_mson the miss was 1,611 (open) and 1,025 (proprietary); hits are in the table below.- The same with
--user 1000:1000, miss and hit: lists the T4G. nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0, as root and with--user 1000:1000, miss and hit:Test PASSED.- Two cache hits at once from one snapshot, one per T4G: each listed its own T4G (different UUIDs); 6.93 and 7.00 s (open), 6.17 and 6.37 s (proprietary), process start to exit.
- In the guest the T4G is
0000:00:06.0,[10de:1eb4], class 0x030200, behind the host bridge the FDT describes (pci-host-generic 70000000.pci, ECAM, no/sys/firmware/acpi), with BAR0 16 MiB, BAR1 256 MiB and BAR3 32 MiB assigned, as on the x86 T4. - MSI-X goes through the GIC ITS: the driver holds six vectors
(
ITS-PCI-MSIX-0000:00:06.0,msi_irqs28 to 33) whilenvidia-persistencedkeeps the GPU open, and they deliver. With the proprietary modules one vector counted 76 after the attach. Stopping the daemon released the vectors, so the counts started again when it was restarted: that vector read 10 after the restart and 39 afterfcvm-gpu-scrubran its CUDA work. With the open modules it counted 3 to 5. dmesgshows the aarch64 module loading ("NVIDIA UNIX Open Kernel Module for aarch64 580.196.07", or "NVIDIA UNIX aarch64 Kernel Module" withEnableGpuFirmware: 0) and three warnings, none of them a failure:of_irq_parse_pci: failed with rc=-22(the FDT's PCI node has nointerrupt-map; Firecracker's PCI transport has no INTx and the driver uses MSI-X),NVRM: Chipset not recognized (vendor ID 0x8086, device ID 0x0d57)with "has not been qualified on this platform" (the ID of Firecracker's emulated host bridge), and, with the open modules,objClInitPcieChipset: *** Unable to get PCI port handlesonce per GPU initialisation.nvidia-smi -Lin the container, three runs each: with the daemon 0.013 to 0.016 s (open) and 0.012 to 0.014 s (proprietary); with the daemon stopped 1.42 to 1.49 s (open) and 0.420 s (proprietary). Starting the daemon again took 1.12 s (open) and 0.35 s (proprietary).sudo fcvm gpu scrubon both T4Gs (reset_methodbus):scrubbed, 1.22 and 1.23 s (reset 192 to 214 ms, boot 77 to 98, clear 898 to 903).
Eight cache-hit relaunches of nvidia-smi -L on one T4G per module type, after one
warm-up, on a quiet box (mpstat 99.99 to 100 percent idle over 5 s before and after,
no other Firecracker). Columns as in the table under Attach time, medians with the
range in parentheses:
| modules | runs | relaunch (s) | ACK (ms) | gpu_ms | rescan | persistenced | scrub | smi / cdi |
|---|---|---|---|---|---|---|---|---|
| open | 8 | 5.50 (5.48-5.52) | 2,783 (2,782-2,804) | 1,823 (1,809-1,854) | 56 | 1,285 | 428 | 21 / 54 |
| proprietary, GSP off | 8 | 4.77 (4.74-4.82) | 2,063 (2,014-2,112) | 1,026 (1,008-1,045) | 55 | 609 | 306 | 19 / 54 |
Both load the driver at boot, so insmod at restore is 0 ms. After these runs the
arm64 gpu profile switched to the proprietary modules, as amd64 had.
Measured on a g4dn.metal (T4) on 2026-10-22: a cache-hit fcvm podman run --gpu <sbdf> ubuntu:24.04 nvidia-smi -L, relaunched in sequence on one GPU. "Relaunch" is process
start to exit; "ACK" is the host's ack_ms; the rest are the guest's gpu_ms and
gpu_steps. Medians, with the range in parentheses.
| runs | relaunch (s) | ACK (ms) | gpu_ms | insmod nvidia + uvm | rescan | persistenced | smi / cdi | |
|---|---|---|---|---|---|---|---|---|
| #1141 (daemon), open modules | 12 | 3.41 (3.33-3.48) | 2,370 | 1,362 (1,308-1,475) | 126 + 12 | 8 | 1,073 | 41 / 73, in sequence |
| + nvidia-smi and CDI together | 6 | 3.39 (3.30-3.41) | 2,356 | 1,318 (1,315-1,467) | 116 + 11 | 13 | 1,091 | 50 / 87, together |
| + driver loaded at boot | 12 | 3.38 (3.31-3.43) | 2,341 | 1,289 (1,233-1,405) | 0 | 65 | 1,140 | 48 / 85 |
| + proprietary modules, GSP off | 24 | 2.83 (2.75-2.93) | 1,791 | 724 (701-776) | 0 | 64 | 576 | 46 / 82 |
| the same without the boot load | 12 | 3.03 (2.90-3.16) | 1,987 | 951 (918-1,017) | 310 + 11 | 10 | 537 | 46 / 80 |
With the driver loaded at boot the rescan carries the PCI probe. The proprietary
nvidia.ko takes 280 to 340 ms to load at restore, so the boot load matters most with
it. Before #1141, nvidia-smi -L and nvidia-ctk cdi generate each initialised the
GPU, and gpu_ms was the 3,028 to 3,337 ms above.
The rest of the ACK is the host: Firecracker's /snapshot/load with the GPU in
device_passthrough took 955 ms in a debug-logged relaunch, almost all of it after
the load attached the PCI segment.
Not yet checked:
- MSI-X in use on x86_64: the idle guest had no MSI vectors allocated, since the driver
allocates them only while a process holds the GPU open. A CUDA kernel completing
implies interrupts work (Firecracker's PCI transport has no INTx), but no
/proc/interruptsreading was taken under load.
--map volumes are FUSE over vsock with AUTO_INVAL_DATA. Nothing tells the guest that the host changed a file. The guest finds out when it asks the server again, which it does once its 1 second attribute and entry timeouts have run out. What it does with the answer depends on how the map is mounted.
Read-only maps (--map HOST:GUEST:ro) are mounted read-only in the guest and without FUSE_WRITEBACK_CACHE. /proc/self/mountinfo in the guest shows the mount as ro, and a write from the guest OS or from the container fails with EROFS. Without the writeback cache the guest takes the size and mtime the server reports and drops its cached pages when either changed. A file the host rewrites in place, longer or shorter, or replaces by rename is read whole, with its new size and mtime, after the attribute timeout. That holds in a running VM and in a clone restored from a snapshot taken before the host changed the file.
A read-only map's VolumeServer answers OPEN and FLUSH with ENOSYS, fuse-backend-rs's zero-message open (PassthroughFs::for_volume). After the mount's first OPEN and FLUSH, which those replies switch off, the guest kernel sends no OPEN, FLUSH or RELEASE for it and keeps a file's pages from one open to the next. A read carries file handle 0, and the server opens the inode for it: about 7 us on a local btrfs, more where the host directory is itself on FUSE or NFS. A file read on every request (a credential, a config file) then costs a LOOKUP per path component and a GETATTR, each at most once a second, as the entry and attribute timeouts pass. A read-write map opens and closes through the server.
What that changes for a read-only map:
- A host change shows when the attribute timeout passes and the size or mtime differs, or, for a file replaced by rename, when the entry timeout passes. Before, every open dropped the cached pages. A rewrite in place that keeps both the size and the mtime is not seen while the pages stay cached.
- The host no longer checks permissions as the guest's caller when a file is opened, because nothing is opened. The guest kernel checks the mode bits (
default_permissions), so host POSIX ACL entries and LSM rules that deny one uid no longer apply. The server's own open for each READ is checked instead, so a host change that takes the server's access away fails the next READ, also in a file the guest already has open. - LSEEK answers ENOSYS, because fuse-backend-rs seeks only through a handle. The kernel then seeks by itself and takes every file on the mount for all data: SEEK_DATA and SEEK_HOLE stay correct, and a copy that skips holes reads a sparse file's holes as zeros.
The guest's kernel is what refuses the write. The host's VolumeServer does not check read_only and accepts writes on every volume, so a guest that remounts the volume read-write can write to the host directory (#1042).
A mount point inside a read-only map cannot be created by the guest, so it has to be a directory of the map's host directory already. The mount points are those of another --map, of a --disk, --disk-dir or --nfs whose guest path lies under the map, and /mnt/image-store, where fc-agent mounts the image disk of a localhost/ image in overlay image mode (the default mode). fcvm checks them on the host and fails with an error that names the map, what is mounted there and the directory the host lacks. The check runs:
- in every
fcvm podman runandfcvm podman prepare, before the kernel, the rootfs and the image are looked up and before the snapshot cache is consulted, so a refused run sets up nothing and a run that would have been a cache hit is checked too; - in
fcvm snapshot runof a disk-only snapshot, which cold-boots through the same code; - before each relaunch after a guest reboot, because the new fc-agent mounts the volumes again. A
podman runVM that fails it ends with the error. A restored clone that fails it logs the error and treats the reboot as the VM's exit.
fcvm snapshot run of a memory snapshot is not checked: the mounts are in the guest's memory and nothing is created. The check does not follow a symlink on the way to a mount point, which the guest resolves in its own namespace, and it cannot see a directory removed after it ran. In those cases fc-agent fails to create the mount point, the container exits with code 1, and fcvm's log has the cause on one line: [fc-agent] Error: mounting FUSE volumes: creating mount point: /mnt/outer/inner: Read-only file system (os error 30).
fc-agent mounts an outer map before a map inside it, whatever order the arguments are in. That order is among maps: disks and NFS shares are mounted after every map, and one whose guest path is above a map's covers that map. While a guest runs, the host must not replace or remove a directory that another mount sits on (the mount point of an inner map, a disk or an NFS share inside a map). The guest's kernel drops a mount whose mount point fails revalidation, so within the 1 second entry timeout the inner mount is gone from the guest's mount table, for a read-write outer map as well, and its path shows the outer map's new directory.
Read-write maps keep FUSE_WRITEBACK_CACHE unless the VM was booted with FCVM_NO_WRITEBACK_CACHE=1. With that cache the guest's kernel owns the size and mtime of a regular file it has cached and ignores the server's, because it may hold writes the server has not seen yet. The host must not change a file under a read-write map while a guest has it cached, unless the VM was booted with FCVM_NO_WRITEBACK_CACHE=1: a file the host rewrites longer is read cut off at the old length and keeps its old mtime, in the running VM and in clones restored from its snapshots (#1041).
Directory changes (new files, deletions) are subject to the kernel's directory entry cache TTL. A new file created on the host may not appear in guest readdir() until the cache expires.
There are no push notifications from host to guest. The guest discovers changes only on access. inotify/fanotify in the guest watches the FUSE mount, not the host filesystem, so host-side changes don't trigger guest notifications.
Potential fix: Use FUSE_NOTIFY_INVAL_INODE and FUSE_NOTIFY_INVAL_ENTRY — server-initiated invalidation notifications. The host VolumeServer would watch directories with inotify and push invalidations through the FUSE connection when files change. This is how production network filesystems (NFS, CIFS) handle it.
ARM64 FEAT_NV2 has architectural issues with cache coherency under double Stage 2 translation. The DSB SY kernel patch fixes this for vsock/FUSE data paths, but multi-vCPU L2 VMs still hit interrupt delivery issues (NETDEV WATCHDOG). L2 VMs are limited to single vCPU.
A snapshot of a VM with --map volumes holds the guest's FUSE mounts in its memory. A clone gets its own VolumeServers, and the guest's mounts reconnect to them over vsock without being mounted again. Each mount keeps the settings fc-agent made it with: read-only or read-write, with or without the writeback cache.
A snapshot made by an fc-agent from before read-only maps were mounted read-only keeps read-write, writeback-cached mounts for them in every clone, until the snapshot is made again. The automatic snapshot cache does that by itself: its key covers the path of the fc-agent initrd, which is named after the hash of the fc-agent binary, so a new fc-agent never restores an older one's snapshot. A snapshot made with fcvm snapshot create or fcvm podman prepare --tag has to be made again by hand.
Snapshot metadata records the source's kernel_profile, image_mode,
image_disk_path, and balloon_mib (all #[serde(default)], so old snapshots
still load):
- Custom kernel profiles: cold-boot clones and reboot plans resolve the recorded profile, so a btrfs/nested-profile disk boots with a kernel that can mount it. Snapshots created before these fields existed fall back to "default".
- Overlay/archive (localhost/) images: the recorded image device (content-addressed cache file) is re-attached on cold-boot clones and reboot relaunches, and the MMDS plan carries the mode so fc-agent re-mounts the additionalImageStore on a provisioned boot. The image layers stay reachable.
- Balloon (
--balloon): a VM's balloon target is recorded in its state and in every snapshot of it. A disk-only clone and a restored clone's relaunch after a guest reboot are cold boots given that target, and attach the device at it. A memory restore attaches nothing: the device comes back with the VMM state. This is tested on Firecracker only:test_disk_only_clone_keeps_the_balloonandtest_restored_clone_reboot_keeps_its_balloonread the target back from Firecracker'sGET /balloon/statistics. No Cloud Hypervisor VM was run with a balloon. From the code, a disk-only clone of one cold-boots under Cloud Hypervisor with the recorded target as theballoonsize in the VM config it is created from, and no test reads that device back. A restored Cloud Hypervisor clone has no reboot relaunch (a guest reboot ends it). On Firecracker the record follows the VMM, which is the authority on its device: the target can be changed on the VM's API socket after boot, and nothing writes that to the VM's state. The device is read fromGET /vm/config. A memory snapshot reads it inside the shared creator, with the VM paused and right before the save, sosnapshot createandpodman run's own pre-start and startup snapshots record the target the saved device holds. A disk-only capture has no pause and reads it under the per-VM snapshot lock. Every memory restore reads it before the resume, after it has set the caller's target if there is one, and the clone's state records what it reports (test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has). A disk-only clone has no restored VM to ask. It cold-boots from the snapshot's record, and attaches no balloon device when the snapshot has none. Cloud Hypervisor has no call to ask, so its snapshots and clones copy the record. - Free page reporting (
--free-page-reporting): the switch on the balloon device that makes the guest report the memory it has freed, which the VMM then discards. It needs--balloonand Firecracker and is refused with--hugepages, all before the snapshot cache is looked up. It is recorded beside the target (balloon_free_page_reportingin the VM's state and in snapshot metadata) and travels with it. A cold boot records what fcvm sent. A restore and every snapshot record whatGET /vm/configreports, from the same read that gives the target. A disk-only clone and a restored clone's relaunch after a guest reboot attach the device with the recorded switch (test_disk_only_clone_keeps_the_balloon,test_restored_clone_reboot_keeps_its_balloon), and a cold-booted VM's own relaunch attaches it from its launch config. The record says what the device offers, not what the guest negotiated. A memory restore cannot change the switch: Firecracker accepts it only when the device is attached before boot, and rebuilds a restored device from its saved feature bits. fcvm's minimum Firecracker is 1.14.0, the first whose balloon has the switch and whoseGET /vm/configreply says whether a device has it. A guest that does not accept the feature leaves the whole balloon device inactive in Firecracker, where the same guest without the switch has a working balloon. That happens withinit_on_free=1, with page poisoning, and with a balloon driver that has no free page reporting. A snapshot of such a VM would hand the dead device to every run restored from it, so the first cold boot of a VM with the switch checks that the guest brought the device up and fails the run if it did not (test_free_page_reporting_run_fails_when_the_guest_leaves_the_balloon_inactive). The check reads the balloon's statistics first. A balloon that holds memory was inflated by the guest's driver, and a device whose guest has sent its memory counters has served its statistics queue. Either shows an active device, which is sent nothing. A device with neither is sent, under the per-VM snapshot lock, aPATCH /balloonwith the target it already has, which Firecracker refuses for an inactive device. fcvm attaches every balloon with statistics every second, so an active device is past that state about a second after its guest's driver came up, and the check changes nothing in a running guest's balloon (test_snapshot_clone_free_page_reporting_reaches_the_page_serverreads the requests the source's Firecracker received).podman runmakes it when the guest's agent first reports in, which is before the pre-start snapshot, and on--no-snapshotruns too. It makes it again before a startup snapshot if the agent's ask never came.podman preparemakes it after its health wait and before its snapshot. Every memory snapshot repeats the test with the VM paused, so no snapshot holds an inactive device (test_snapshot_of_a_vm_with_an_inactive_reporting_balloon_is_refused). Three boots are not failed early: one whose agent's ask never reaches fcvm (the agent could not quiet the console, could not read the image digest, or could not send the ask) and that arms no startup snapshot, a cold-booted VM that reboots before its first ask, and a restored clone's relaunch after a guest reboot, which takes extra kernel arguments from its own process'sFCVM_BOOT_ARGSonly and not from the kernel profile. Such a VM runs with an inactive balloon,fcvm balloonrefuses a target for it, and a memory snapshot of it fails. A limit of the switch: Firecracker discards every range the guest reports, including memory the host holds, so memory the guest frees and uses again costs a first touch again. Measured in two runs on a host that is itself a virtual machine: writing 2 GiB for the first time took 22.0 s and 22.5 s, rewriting it right after freeing it took 2.3 s and 1.1 s, and rewriting it once the guest had reported it took 18.7 s and 19.4 s. Use the switch where giving memory back matters more than that. - Balloon target on a snapshot cache hit: the snapshot key says whether a
balloon device exists and not its target, so
podman runinvocations that differ only in the--balloonvalue share one pre-start snapshot. Whether the device exists stays in the key because one cannot be added to a restored VM. Whether it reports free pages (--free-page-reporting) is in the key for the same reason: a restore can turn it neither on nor off, so a run with the switch never restores a snapshot booted without it, nor the reverse. A cache hit sets its own target on the loaded, paused VM (PATCH /balloon) before the guest resumes, and its state records that target. A target that cannot be set ends the run, with no fallback to a cold boot. A target above--memis refused before the cache is looked up, so a hit and a miss refuse alike.fcvm snapshot runhas no flag for the target and restores the device as it was saved.test_balloon_target_honored_on_snapshot_cache_hitcovers it with one hit above the snapshot's target and one below. The startup snapshot (taken by a run with--startup-snapshotor--health-check, whichfcvm servepasses; a plain run takes and restores none, because every run restored from it shares the startup state the app made, such as keys, IDs and seeds) is not shared between targets. It holds a workload that initialised under the target of the run that made it (the memory the guest had, which a workload sizes itself by), and a restore can set a target but cannot replay that. So its name carries the target:<key>-balloon<MIB>-startup, and<key>-startupfor a run without--balloon, as before. A run looks up and saves only its own target's startup snapshot. A run at a target that has none restores the shared pre-start snapshot, sets its target, starts the workload under it and saves a startup snapshot for that target, so no cold boot is added.podman prepareinstalls a startup snapshot, and its content key carries the target the same way. A run whose targetfcvm balloonchanged before it turned healthy takes no startup snapshot (seefcvm balloon).test_startup_snapshot_is_per_balloon_targetcovers the lookup and both places that save. One limit:- The guest has to have activated the device. Firecracker refuses a target
for a device the guest never activated (a custom kernel without the
virtio balloon driver), so the first run of such a configuration
cold-boots and every later one, a cache hit, fails. On an NV2 profile the
first run fails too: a miss there tears its cold-booted VM down and
restores its own snapshot with the caller's target.
--no-snapshotruns it.
- The guest has to have activated the device. Firecracker refuses a target
for a device the guest never activated (a custom kernel without the
virtio balloon driver), so the first run of such a configuration
cold-boots and every later one, a cache hit, fails. On an NV2 profile the
first run fails too: a miss there tears its cold-booted VM down and
restores its own snapshot with the caller's target.
- Extra disks (
--disk-dir): intentionally fail-fast — disk-only capture rejects sources with extra disks, and reboot-in-place is disabled for restored clones that have them (a relaunch would silently drop the data disks). Re-attaching captured extra-disk images on cold boot is a follow-up.
-
Persistent volumes:
- Support Docker volumes API
- Persistent storage across clones
-
Custom networks:
- User-defined networks
- VM-to-VM communication
-
Resource limits:
- CPU pinning
- Memory limits (cgroups)
- I/O throttling
-
Metrics & monitoring:
- Prometheus exporter
- Real-time resource graphs
-
Snapshot encryption:
- Encrypt memory snapshots
- Key management
-
Jailer integration:
- Use Firecracker jailer for additional sandboxing
- chroot, cgroups, seccomp
-
Multi-host support:
- Durable workloads across pluggable compute and storage providers
- Stopped-only relocation between hosts
- See the durable remote workload design
-
GPU passthrough:
- vGPU support for ML workloads
-
Kubernetes integration:
- Run as CRI runtime
- Pod → Firecracker VM
- Firecracker: Lightweight VMM (Virtual Machine Monitor) from AWS
- microVM: Minimalistic virtual machine with fast boot times
- KVM: Kernel-based Virtual Machine, Linux's hypervisor
- MMDS: Micro Metadata Service, Firecracker's metadata API
- TAP device: Virtual network interface (TUN/TAP)
- pasta: L4 splice-based networking from the passt project for rootless containers
- CoW: Copy-on-Write, disk strategy for fast cloning
- iptables: Linux firewall/NAT configuration tool
- vsock: Virtual socket for host-guest communication
- Balloon device: Memory reclamation mechanism for VMs
Benchmarked on c6g.metal (64 ARM cores, 128GB RAM).
| Scenario | Time | Notes |
|---|---|---|
| Cold build (clean target) | 44s | ~12 parallel rustc processes |
| Incremental (touch main.rs) | 13s | Only recompiles fcvm |
| test-unit LIST (cold) | 24s | Compiles test binaries |
| test-unit LIST (warm) | 1.2s | No recompilation |
| Tool | Cold Build | Incremental | Verdict |
|---|---|---|---|
| Default (no tools) | 44s | 13.7s | Baseline |
| mold linker | 43s | 12.7s | ~1s savings, not worth config |
| sccache | 52s cold / 21s warm | 13s | Overhead > benefit for local dev |
Cargo parallelizes by crate, limited by the dependency graph:
- Early build: many leaf crates → high parallelism (11+ rustc)
- Late build: waiting on syn, tokio → low parallelism (1-3 rustc)
The 64 CPUs help within each crate (LLVM codegen), but crate-level parallelism is dependency-limited.
- Local dev: Use defaults. Incremental builds are fast (13s).
- CI: Consider sccache if rebuilding from scratch frequently.
- mold: Not worth it - linking is not the bottleneck.
Under parallel test load on ARM64, diff snapshots occasionally capture only ~94 KB of dirty pages instead of the expected ~37-43 MB. Restoring the merged snapshot kernel-panics with:
stack-protector: Kernel stack is corrupted in: do_idle
Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: do_idle
- Load pre-start snapshot with
track_dirty_pages: true - Resume VM — VM boots, container initializes until healthy
- Pause VM — Create diff snapshot
- Write the diff into a clone of the pre-start base, in place or by a merge onto a copy → startup snapshot
KVM ARM64's KVM_GET_DIRTY_LOG silently returns a nearly-empty bitmap. The diff snapshot
captures only device-emulation pages (virtio queue pages marked by Firecracker's internal
AtomicBitmap via mark_virtio_queue_memory_dirty()), while missing ALL guest OS memory
writes tracked by KVM's Stage-2 page tables.
CI data showing the failure:
| Round | bytes_merged | runs | Result |
|---|---|---|---|
| 07:54 | 37,834,752 | 2,824 | OK |
| 08:34 | 43,225,088 | 3,648 | OK |
| 15:20 | 40,157,184 | 3,039 | OK |
| 17:35 | 94,208 | 9 | KERNEL PANIC |
Both rootless AND routed snapshots created at the same time had EXACTLY 94,208 bytes — a systematic KVM dirty tracking failure, not random corruption.
Investigated the Firecracker source (ejc3/firecracker, branch bump-vsock-max-connections):
KVM_MEM_LOG_DIRTY_PAGESis correctly set on memory regions duringload_snapshot(viaKVM_SET_USER_MEMORY_REGIONinvstate/memory.rs)- No code path resets or discards the dirty bitmap between load and diff creation
- vCPU pause is properly synchronized before
KVM_GET_DIRTY_LOG - The ~94 KB corresponds to virtio queue pages marked by Firecracker's internal bitmap after the previous snapshot — these don't rely on KVM tracking
-
UFFD + dirty tracking interaction (highest suspicion): Pages populated via
UFFD_COPYduring snapshot restore may get Stage-2 mappings created without write-protection when dirty logging was enabled before the host PTE existed. Under load, delayed UFFD faults mean pages fault in after VM resumes, bypassing dirty tracking entirely. -
Stage-2 TLB stale entries: Incomplete TLB invalidation after enabling dirty logging under heavy load (IPI delays) allows writes through stale entries that bypass tracking.
-
Block mapping coalescing: KVM ARM64 can create 2 MB block mappings in Stage-2. If splitting into 4 KB write-protected pages is delayed, writes bypass per-page tracking.
In create_snapshot_core(), after creating the diff snapshot but before resuming the VM:
- Count what the diff wrote: the allocated blocks of a sparse
memory.diff(meta.blocks() * 512), or the unshared extents of an in-place diff's clone (unshared_bytes_up_to), read only as far as the minimum. Extents that cannot be read count as nothing written - If that is under
memory_bytes / 1024(0.1% of VM memory), the diff is corrupt - Remove the diff and retry as Full snapshot while VM is still paused. An in-place diff is
memory.binitself, and a Full snapshot leaves all-zero pages as holes, so its removal must succeed or the retry fails - Use the full snapshot directly, with no merge or punch pass
This is a detection + recovery approach because the bug is in KVM kernel code we don't control.
Additional defense:
- Post-resume liveness check: After restoring a snapshot, wait 200ms and
try_wait()to detect immediate kernel panics. Returns error so the snapshot is known to be bad.
Multiple clones from the same snapshot should share physical memory pages for read-only data. A large container VM may have 131 GB of guest memory, but most of it is identical across clones (kernel, application code, page cache). Only pages each clone writes to should be unique (Private_Dirty).
An earlier (2026-02-28) version of this table ranked the backends by per-clone RSS of the
firecracker process alone on a 131 GB guest, and called the File backend "Broken — KVM
CoW-copies pages into Private_Clean even for reads". Recomputing on a matched accounting
basis — PSS/RSS summed over every process belonging to a clone (fcvm + firecracker +
pasta + the unshare namespace holder), plus each backend's costs outside the clones —
showed that ranking was inverted: the File backend's big RSS is the same reclaimable page
cache counted once per clone, not per-clone private waste, and on whole-machine cost it is
the cheapest 4 KiB option, not a broken one. Measured, N=4 concurrent 1 GiB alpine/nginx
clones (aarch64, details in the next section):
| Backend | Σ clone PSS | Σ clone RSS | Unreclaimable footprint at N=4 (clones + server + backing) | Status |
|---|---|---|---|---|
File (MAP_PRIVATE on memory.bin) |
306 MiB | 743 MiB (double-counts page cache shared BETWEEN clones) | ≈ 160 MiB (only Private_Dirty; everything else is reclaimable page cache) | Works; clean pages shared via page cache; rejected by Firecracker for hugetlbfs snapshots |
| UFFD MISSING+COPY | 296 MiB | 344 MiB | ≈ 500 MiB (296 clone anon + the serve process's own faulted copy, 205 PSS) | Works; zero sharing between clones |
UFFD MINOR+CONTINUE (--uffd-mode minor) |
204 MiB | 355 MiB | ≈ 590 MiB (204 clone + 383 unevictable shmem memfd + 3.5 serve) | Works; true sharing via a sealed memfd mapped MAP_PRIVATE |
File backend: Firecracker maps memory.bin with MAP_PRIVATE | PROT_READ | PROT_WRITE.
Faulted-but-clean pages stay page-cache-backed: they show up as Shared_Clean/Private_Clean
in each mapping process (which is what the old measurement misread as per-clone cost), but
there is ONE physical, reclaimable copy regardless of clone count. Only written pages
(Private_Dirty, ~40 MiB/clone measured) are truly per-clone.
UFFD MISSING+COPY: Firecracker creates anonymous memory (MAP_PRIVATE | MAP_ANONYMOUS)
and registers it with UFFD in MISSING mode. On each fault the server UFFDIO_COPYs out of
memory.bin — every faulted page becomes unswappable private anon in that clone (~76
MiB/clone at idle), and the serve process itself faults the whole snapshot mmap in as it
copies (205 MiB PSS measured after serving 4 clones). With a balloon, a REMOVE event keeps
mmap_changing raised until the thread inside madvise runs again, and UFFDIO_COPY
returns EAGAIN with nothing copied until then. The handler parks such a fault and
retries it, bounded, exactly as the MINOR handler below does for UFFDIO_CONTINUE.
The handler installs nothing when it reads a REMOVE event: the kernel drops the pages only
after the event is read, so zeroing the range at that point is refused or undone. It
remembers the range, one bit per page, and the guest's next fault on a page of it is
answered with zeros and not with the snapshot's bytes. A fault that is on such a
page when it is first read is not recorded into the working set, and it gets no
fault-around. Replay and fault-around ask the set before each chunk and step over a page
in it, so a page in the set is filled only by the guest's own fault.
A hole in memory.bin is answered with zeros too. Once the server has bound its socket, it
maps the file's holes with SEEK_HOLE/SEEK_DATA on a thread of its own, so the build
reads no data and does not delay the bind (the walk costs about 1.7 us per data run on
btrfs, about 3.9 s for a file of 2.3 million runs). It publishes the map once, whole,
through a OnceLock: a fault sees either no map or all of it. Until then every fault is
served from the file, which reads a hole as zeros, so only the saving waits. A file without
holes costs one branch per fault. A memory file on FUSE is flushed with fdatasync before
the walk: fuse-pipe forwards SEEK_HOLE to the host, so in a nested L1 with the writeback
cache a page still dirty in the L1 would read as a hole. A local filesystem answers
SEEK_HOLE from its page cache and is not flushed, which saves seconds on a file just
written (3.27 s measured on btrfs). A fault on a hole is answered with zeros and recorded
into the working set like any other fault. Replay maps a run of recorded holes to the
kernel's zero page with one UFFDIO_ZEROPAGE, as a read fault on a hole is answered, and
gives an NV2 clone and hugetlb memory private copies of zeros instead. Fault-around steps
over holes, and still runs after a fault on one, since a page of zeros among data says
nothing about its neighbours.
The kind of zero answer depends on the fault. A read fault, on a hole or on a page the
balloon gave back, gets UFFDIO_ZEROPAGE, which maps the kernel's shared zero page
read-only: the read costs the clone no memory, and the guest's first write is a
copy-on-write fault the kernel resolves without the server, since a copy clone registers
MISSING faults only. A write fault (UFFD_PAGEFAULT_FLAG_WRITE, kept through a parked
retry) gets a private copy of zeros, mapped writable, so the write does not fault a second
time. Every fault of an NV2 clone gets a copy too: on arm64 the MMU-notifier invalidate of
that copy-on-write reaches kvm_nested_s2_unmap and drops every nested stage-2 mapping,
the cost that makes NV2 restores use UFFD. snapshot serve and the implicit server know a
snapshot's clones are NV2 from the Firecracker arguments they run with (--enable-nv2).
Hugetlb memory refuses UFFDIO_ZEROPAGE (EINVAL), so a clone whose page is larger than the
host's base page gets copies, and a clone whose UFFDIO_ZEROPAGE fails with EINVAL anyway
gets a copy for that fault and every later one.
Free page reporting on each transport (--free-page-reporting). In a copy-mode clone
each block the guest reports is one MADV_DONTNEED in Firecracker and one REMOVE event in
the handler, remembered as above
(test_snapshot_clone_free_page_reporting_reaches_the_page_server). Two limits follow from
the paragraph above. While a REMOVE is unread the kernel refuses populate calls for that
clone, so demand faults are parked, and one still refused after 2 s stops the clone. Replay
treats a refused chunk as speculation lost and leaves the rest of that recorded run to
demand faults, so reports that arrive while replay runs cost replay coverage, not data.
On the File backend, which every podman run cache hit uses except an NV2 guest and a run
with FCVM_FORCE_UFFD set, and in minor mode, Firecracker replaces each reported range
with fresh anonymous memory, the kernel zero-fills later touches, and the page server
hears nothing. Each isolated range adds mappings to the VMM. The worst case is guest
memory divided by the reported block size: 512 mappings per GiB at the guest's default of
2 MiB blocks. The block size can be lowered with page_reporting.page_reporting_order= in
the boot arguments or by a write inside the guest, and at single pages the worst case is
262,144 mappings per GiB. Past vm.max_map_count (65,530 by kernel default, 1,048,576 on
Ubuntu 24.04) the replacement mapping fails and Firecracker panics. fcvm does not set the
block size.
KSM: Disabled (/sys/kernel/mm/ksm/run=0). Firecracker doesn't mark guest memory
with MADV_MERGEABLE. Even if enabled, KSM is after-the-fact dedup with scanning overhead.
The mechanism is NOT MAP_SHARED. An earlier sketch in this document proposed mapping the
memfd MAP_SHARED; that is wrong and would let every clone's guest writes land in the golden
snapshot and in each other. The correct recipe, proven on kernel 6.18.3-fcvm and then
implemented, is:
snapshot serve --uffd-mode minor
memfd_create(MFD_CLOEXEC|MFD_ALLOW_SEALING [| MFD_HUGETLB|MFD_HUGE_2MB]) # one per snapshot
populate it once from memory.bin (all-zero pages left as holes on shmem)
fcntl(F_ADD_SEALS, F_SEAL_SEAL|F_SEAL_SHRINK|F_SEAL_GROW|F_SEAL_WRITE)
reopen O_RDONLY via /proc/self/fd — the ONLY fd clones ever see
on each clone connection: SCM_RIGHTS that sealed read-only fd to Firecracker
Firecracker (mem_backend.backend_type = "UffdMinor")
recv the memfd, mmap every guest region MAP_PRIVATE over it
(MAP_NORESERVE for shmem; hugetlbfs RESERVES the full range -> ENOMEM at
restore instead of SIGBUS at CoW when the pool is short)
create a userfaultfd via /dev/userfaultfd, register MODE_MINOR
send the usual mappings JSON + the uffd back
fcvm handler
minor fault -> UFFDIO_CONTINUE(addr, page_size, wake)
EEXIST -> page already mapped by a racing fault: success
EAGAIN -> mmap_changing is set: park the fault, drain the event
queue, retry (bounded); NEVER drop it — the faulting vCPU
sleeps until a CONTINUE wakes it
short mapped -> advance by the reported byte count, continue the rest
The seal + read-only reopen exist because kernel CoW only covers mapped writes: an
unsealed fd received over SCM_RIGHTS could pwrite(2)/ftruncate(2) the backing file — or
be reopened O_RDWR via /proc/self/fd — and corrupt the golden snapshot for every clone,
the same catastrophe the MAP_SHARED experiment demonstrated through a different door.
Sealing is verified to work for MFD_HUGETLB on this kernel; enforcement is tested by
uffd::server::tests::test_backing_memfd_is_sealed_* (pwrite EBADF on the handed-out fd,
EPERM via an O_RDWR reopen, ftruncate EPERM, mmap(MAP_SHARED, PROT_WRITE) EPERM — while
MAP_PRIVATE + MINOR + CONTINUE keep working off the sealed fd).
UFFDIO_CONTINUE into a non-shared VMA installs a READ-ONLY PTE onto the shared
page-cache folio (mm/userfaultfd.c: if (page_in_cache && !vm_shared) writable = false;
mm/hugetlb.c hugetlb_mfill_atomic_pte does the same for hugetlb). Reads across all clones
therefore hit one physical page and the first guest write takes an ordinary copy-on-write
fault into private memory. The snapshot stays pristine and reusable — enforced by
test_snapshot_clone_isolation_uffd_minor.
Two kernel facts shape the implementation:
- MINOR registration requires a hugetlb or shmem VMA (
vma_can_userfault()), so the backing file must be a memfd; a plain on-diskmemory.bincannot be registered MINOR. - All-zero pages are left as holes on shmem only.
shmem_faultallocates a hole's folio in the inode's page cache regardless ofVM_SHARED, so the first clone to touch it populates the shared copy and every later clone faults MINOR onto it.hugetlb_no_page()only adds to the page cache whenVM_MAYSHARE, so a hugetlb hole would become a private anon huge page per clone — hugetlb backings are therefore populated in full.
UFFDIO_CONTINUE refuses a hole, because the memfd has no page there to map, and a refused
populate gives up the rest of its run. The server keeps a map of the pages it left unwritten
(src/uffd/holes.rs), and each page in it is a candidate: the first clone to touch a hole
fills it in the memfd with no fault reaching the server. For a recorded run the map
marks, replay runs mincore over a read-only view of the memfd, maps the pages that are in
its page cache now, and steps over the rest. One mincore call answers a chunk, and replay
reuses the answer while it walks inside that chunk. A page that leaves the page cache (the
host swaps a filled page out, and UFFDIO_CONTINUE swaps it back in), or is filled after the
query, costs one demand fault. A MINOR fault on a filled hole is recorded like any other.
lseek(SEEK_DATA) on a private reopen of the memfd would answer the same question, but
shmem_file_llseek takes the inode lock exclusively, so every clone's queries would wait on
one another, while mincore needs only the view and the server's mm read lock.
Matched accounting basis: PSS/RSS summed over every process belonging to a clone — the
fcvm clone process, its firecracker child, the unshare --user --net holder, and pasta
(4 processes per clone) — plus the serve process measured after it had served all 4 clones.
| Config | Σ clone PSS | Σ clone RSS | 1-clone PSS | Serve-process PSS after N | Unevictable shmem | Marginal PSS per extra clone |
|---|---|---|---|---|---|---|
| UFFD MISSING+COPY | 296 MiB | 344 MiB | 76 MiB | 205 MiB | 0 | ~74 MiB |
File MAP_PRIVATE |
306 MiB | 743 MiB | 175 MiB | n/a | 0 | ~44 MiB (concave: page cache PSS spreads over clones) |
| UFFD MINOR+CONTINUE | 204 MiB | 355 MiB | 74 MiB | 3.5 MiB | 383 MiB | ~43 MiB |
Restore latency was identical across all three (0.91 s from snapshot run exec to healthy
for the first clone; 3 more concurrently in another 0.93 s).
Reading the table:
- MINOR cuts the clones' own PSS by ~31% versus both existing backends and all but eliminates the serve process (3.5 vs 205 MiB PSS — COPY's server faults in every page it ever copied).
- It pays a fixed 383 MiB of unevictable shmem for the memfd (63% of this snapshot's pages were all-zero and became holes, so 1024 MiB of guest memory cost 383 MiB resident). On a host with no swap that memory cannot be reclaimed, unlike the page cache the other two backends use.
- Against UFFD COPY the total-unreclaimable crossover is ≈7 clones (fixed 386 vs 205 MiB, marginal ~43 vs ~74 MiB).
- Against the file backend MINOR never pays off for 4 KiB guests on this metric: File's unreclaimable footprint is only its Private_Dirty (~40 MiB/clone, no fixed cost) because it shares clean pages through reclaimable page cache for free. Use MINOR when you need hugepages, lazy/remote page fetch, or per-page interception — not as a blanket replacement.
Firecracker rejects the File backend for hugetlbfs snapshots, so a --hugepages clone must
use UFFD, and hugetlb pages are neither swappable nor reclaimable — exhaustion is not "slow",
it is ENOMEM or SIGBUS. Two facts (both measured on this host) shape the design:
- 2 MiB CoW granularity privatises most of an active guest fast: a restored Ubuntu+systemd+podman guest wrote into ~70% of its 2 MiB regions within seconds of restore. Overcommitting the pool against "clones won't write much" is therefore a mirage.
- With
MAP_NORESERVE(the first implementation), pool exhaustion surfaced as a SIGBUS on a copy-on-write fault of a running guest: 4 concurrent 512 MiB MINOR clones on a 512-page pool all died mid-boot before fcvm even recorded their PIDs. Reproduced deterministically in a C harness (MAP_NORESERVEmapping + exhausted pool → child killed by SIGBUS).
The fix is kernel-enforced admission, not hope: hugetlb MINOR mappings now reserve their
full guest size at restore (the fork's UffdMinor backend maps the memfd MAP_PRIVATE
without MAP_NORESERVE; HugePages_Rsvd jumps by guest_size/2MiB per clone). An unservable
clone fails its mmap with a clean ENOMEM — surfaced as a restore error — and an admitted
clone's CoW always draws from its own reservation and can never SIGBUS. fcvm additionally
preflights the pool before spawning any hugepage UFFD clone (preflight_clone_hugepages) so
the operator sees "raise vm.nr_hugepages" instead of a dead process; that check is advisory
(concurrent spawns race), the kernel reservation is the authoritative gate.
The capacity contract is therefore explicit: a pool of P pages serves
floor((P - guest_pages) / guest_pages) MINOR clones (one guest_size for the fully-populated
memfd, one reserved per clone). What MINOR buys over COPY within that contract is not a
bigger N — COPY needs the same N × guest_size worst case, just unenforced — but:
Measured 2026-08-07, 4 concurrent 512 MiB alpine/nginx clones (aarch64, both runs restore to healthy in 0.93 s):
| Pool | Mode | Outcome | HugePages actually allocated | Reserved (admission contract) | Serve RSS |
|---|---|---|---|---|---|
| 512 (1 GiB) | copy | all 4 clones dead: pool 512→0 free mid-boot, UFFDIO_COPY ENOMEM, never healthy |
512 (everything) | none — unenforced | 398 MiB |
| 512 (1 GiB) | minor | clone 0 healthy and stays healthy (its CoW draws on its own reservation); clones 1-3 fail before the guest exists — kernel ENOMEM at restore mmap when racing spawns, or fcvm's preflight error ("hugepage pool too small … Raise vm.nr_hugepages") when the shortfall is visible at spawn |
256 memfd + survivor's CoW | 256/clone | 10 MiB |
| 1536 (3 GiB) | copy | 4/4 healthy, isolation OK | 766 (~383 MiB/clone touched: reads AND writes privatise) | none | 398 MiB |
| 1536 (3 GiB) | minor | 4/4 healthy, isolation OK | 946 (256 memfd + ~345 MiB/clone written; ~28 MiB/clone visibly Shared_Hugetlb) |
1280 (memfd + 4×256) | 10 MiB |
Reading it honestly: at N=4 with a mostly-written idle guest, MINOR's allocated draw (946) is not lower than COPY's (766) — the fully-populated memfd outweighs the small read-shared set, and 2 MiB CoW granularity privatises ~70% of the guest either way. What MINOR buys is:
- no SIGBUS, ever, and deterministic admission: COPY-mode admission is unenforceable
(anonymous
MAP_NORESERVEmemory); when the pool runs out,UFFDIO_COPYfails and running guests die (the pool-512 row). MINOR clones that start are guaranteed to finish, and clones that cannot be served fail up front with an explanation. - reads are shared: COPY privatises every page a guest touches; MINOR only what it writes. The gap grows with read-heavy guests; it is ~150 MiB of the pool here.
- the serve process costs almost nothing (10 vs 398 MiB RSS: it drops its populate mapping; the memfd pages are page cache of the hugetlbfs inode).
- Firecracker Documentation
- Firecracker API Specification
- Podman Documentation
- passt/pasta
- iptables Documentation
- KVM Documentation
- Linux UFFD Documentation
End of Design Specification
Version: 2.4 Date: 2026-02-28 Author: fcvm project