Activity

1. ejc3

   added 2 commits that reference this issue on Aug 10, 2026

   Honor FILTER in _test-unit

   ...

   73668a9

   Close the zero-check hole for ci.yml's ignored paths

   ...

   d927aa0
2. ejc3 commented on Aug 10, 2026

   

   ejc3

   on Aug 10, 2026

   OwnerAuthor

   More actions

   Session update — 2026-08-10

   Seven of the PRs listed above are merged, the #751 blocker is root-caused and fixed, and the ARM teardown item has its evidence machinery landed. A new production bug was found and proven along the way. Details below; every claim here has an artifact behind it.

   Merged to main

   PR	what
   #784	actions/checkout 7
   #779	readiness interleavings + VM diagnostics
   #789	teardown evidence: guard stack capture, TASK_KILLABLE control, dispatchable RED harness, in-job D-state watchdog
   #744	UFFD working-set replay
   #751	kernel repin 6.18.3 -> 6.18.44
   #788	Firecracker full-commit pin, fail-closed
   #787	one Summary check per PR
   (direct)	_test-unit FILTER defect

   #751 was not a timer bug — the whole 7.x line wedges nested guests

   A/B/A on one c7gd.metal box, holding host kernel (6.18.3-fcvm with the DSB patches), Firecracker build, rootfs and test fixed, changing only the L1 guest kernel:

   L1 kernel	test_nested_run_fcvm_inside_vm
   6.18.3	GREEN, 57s
   7.0.14	RED
   7.1.7 (the old pin)	RED — reproduces run 31349006120 exactly
   7.1.8	RED
   6.18.44	GREEN, both snapshot modes

   The L2 kernel boots in about a second and keeps running for the full 600s — entropy reseeds on schedule, its vsock muxer answers every probe — while L2 userspace never reaches the container, stopping at the first burst of virtio-blk reads against the FUSE-backed root disk. L1's FUSE path is not the constraint: with no L2 running, the same file through the same mount sustains 1.6 GB in 7.2s.

   Two things that had to be fixed before this was even visible, both in #751: the FUSE remap_file_range patch's second hunk targeted a fuse_file_operations block that only exists in 7.x, so it failed to apply and took the build down; and RUST_LOG does not cross the exec boundary, so a full 600s wedge had been capturing 262 bytes of health-monitor warnings instead of the L2 console.

   New: fcvm reports a VM ready when the guest is not answering

   Found by running the Chromium shared-nothing benchmark. This is a production bug, not a benchmark artifact.

   In src/network/pasta.rs the readiness loop issues an ARP-triggering ping and records ping.status.success() — but only logs it. The branch that returns success is if neighbor_is_resolved(...), so a resolved neighbour entry alone is accepted. wait_for_port_forwarding_until then "verifies published ports" with a TcpStream::connect to the host-side address; pasta is a userspace stack whose listener accepts regardless of guest state, and in the captured failure that completed 95 microseconds later. fcvm then logs VM lifecycle ready.

   Two independent 809-request runs, one clone restored per request, strictly sequential, loadavg 0.43–0.67:

   backend	ping unanswered	failed	rate	ping answered	failed	rate
   uffd	5	3	60.0%	803	0	0.000%
   file	8	7	87.5%	800	0	0.000%

   1,603 clones whose ping was answered: zero failures. The failures are TimeoutError at CDP stage connect and ConnectionResetError at stage enable, taking 104–120s against a 0.24–0.55s norm — hard hangs to the client's own deadline. Fix in flight: require positive evidence the guest answered, keep the existing retry-until-deadline, and fail fast with a named error instead of hanging.

   Ruled out, each with evidence rather than argument: host load (idle); endpoint collision between concurrent clones (requests are strictly sequential — each starts the millisecond the previous ends); UFFD service starvation (the file backend, which never touches the UFFD serve, fails at a higher rate); and the 13x Firecracker log volume in failing clones (451 extra lines are vsock muxer chatter from the health monitor polling for 105s — an effect of the duration, not a cause).

   Still open, and stated as open: why the ping goes unanswered for ~0.8% of clones. The guest answers ARP (the neighbour resolves) but not ICMP within 200ms, and never recovers across 100+ seconds while vsock keeps working — so it is crawling on the IP path, not dead. Because vsock exec still works during the failure, the guest is interrogable exactly when it is broken; failure-time capture of the guest's own ip neigh/ss -ltnp/dmesg plus a healthy-clone control is in flight, since today the harness tears the clone down and the evidence dies with it.

   The withdrawn CDP A/B: one question closed, the gate still not met

   REVIEW.md recorded the WsClosed root cause as undetermined and called the _recv_until coalesced-bytes fix "a variable removed, NOT a diagnosis." Across 104 CDP requests on the UFFD run there are zero WsClosed records, against 6/60 = 10% before. That fix was the diagnosis.

   The availability gate (>=200 CDP requests per backend at 0 failures) is not met: 3/809 uffd, 7/809 file. But every one of those failures is now attributed to the readiness bug above rather than unexplained.

   Benchmark suite defects (#790)

   • PERFORMANCE.md documents four make targets that do not exist — bench-quick, bench-throughput, bench-operations, bench-protocol, all No rule to make target, exit 2.
   • single_op/getattr and single_op/lookup stat one fixed path in a loop, so FUSE's 1s attr_timeout serves them and the daemon is never contacted: 1.06µs against a 1.02µs host baseline, published as "metadata ops have ~5% overhead", while every op that reaches the server costs 40–56x.
   • make bench root-owns target/criterion via the privileged runner, so the unprivileged protocol suite fails every write with Permission denied — it prints timings and persists nothing, so criterion can never hold a baseline for it — and the same tree trips _test-root.

   Not yet done and deliberately not guessed: PERFORMANCE.md's parallel-write table needs re-measuring. The host baseline alone came in at 863ms against a documented 161ms, and every reader count >=4 now sits at that same host floor.

   Environment traps, now recorded in bench/chromium/AGENTS.md

   Each cost a full cycle and each reports a cause that is not the cause. Rootless podman with no login session fails as default OCI runtime "crun" not found while /usr/bin/crun exists — the real error is mkdir /run/user/1000: permission denied, fixed with loginctl enable-linger. Building the image under one identity and running fcvm under another puts them in two podman stores, which the golden phase's identity guard correctly rejects in a message that reads like a corrupt cache. And build succeeding says nothing about fcvm setup having run.

   Corrections to this issue's earlier text

   • "Repin deployable kernels 6.18.3 -> 6.18.44 (7.x wedges nested guests) #751 nested local state: validate 3999e66 with both snapshot modes" — the branch is not the fix; the kernel pin was. Both modes pass on 6.18.44.
   • The ARM teardown item's "production root fix" is not achievable as stated: an uninterruptible kernel wait cannot be made reapable from userspace. What Capture and reproduce the SIGKILL-survivor teardown wedge #789 lands instead is bounded detection, a deterministic reproduction harness, and — the load-bearing part — an in-job watchdog that streams evidence into the live job log, because the runner is ephemeral and dies with its evidence every time.

   Still open

   • prepare + bench/chromium: prepare writes a content-keyed system cache snapshot, but the benchmark needs a caller-named tag it can snapshot serve per matrix arm, a User type that prune will not delete, and --force. All three in flight; the golden phase then collapses onto prepare.
   • Bump actions/setup-node from 6 to 7 #785 (setup-node 7) now gets real checks after the Give every PR exactly one Summary check #787 rebase; Let Summary pass when the VM matrix is skipped #791 fixes the Summary regression that exposed.
   • Summary still needs to be made a required check on main — the point of Give every PR exactly one Summary check #787.
3. ejc3 commented on Aug 10, 2026

   

   ejc3

   on Aug 10, 2026

   OwnerAuthor

   More actions

   Addendum — attribution result, a refuted fix, and WIP verdicts

   The 7.x wedge is not our FUSE patch

   The control named in the previous comment has now run properly: 7.1.8 built with every arm64 patch except 0001-fuse-add-remap_file_range-support.patch. Verdict RED, INNER_RC=1.

   Verified before believing it, because three earlier attempts each produced something that looked like a result and was not:

   • the control tree genuinely lacks the patch (fuse_remap_file_range = 0, .remap_file_range = 0) while still carrying dsb(sy) in nested.c;
   • the config in effect at run time was 7.1.8 + patches-noremap;
   • the kernel the L1 actually booted, read back from the run log, was vmlinux-nested-7.1.8-aarch64-a720f870d4d2.bin.

   The three false starts are worth recording because each is a reusable trap: a config silently regenerated back to the baseline (the probe script's own fcvm setup --generate-config --force overwrites the pin at the top of every run); strings on a compressed vmlinux reporting 0 occurrences of the patched symbol for both kernels, an instrument that cannot distinguish them; and reading INNER_RC=0 without checking which kernel produced it.

   So the FUSE patch is cleared. The control still carries the other arm64 patches (NV2 DSB, mmfr4-override, psci-debug, wfx-stopped-exit), and a fully unpatched 7.1.8 cannot practically be tested — without mmfr4-override the L1 does not advertise NV2, so there is no L2 to test. Defensible statement: the wedge survives removal of the FUSE patch, reproduces on 7.0.14/7.1.7/7.1.8, and has never appeared on 6.18.x. Upstream is now the stronger candidate, not a proven one. This control is n=1; it carries weight because the wedge has appeared on every 7.x attempt and never on 6.18.x.

   The readiness fix was refuted by CI, and pointed at the right design

   PR #792 required the readiness ping to be answered. test_clone_port_forward_rootless then failed on ARM:

   [fc-agent] ARP cache flushed
   [fc-agent] gateway 10.0.2.2 ARP resolved
   ERROR fcvm: port forwarding verification failed after snapshot restore:
               ping exceeded pasta's readiness deadline
   

   The guest's network came up — fc-agent flushed ARP and resolved the gateway. What did not arrive was an ICMP echo reply, because in rootless mode pasta is a userspace stack that does not reliably forward ICMP to the guest. The comment that was overridden said exactly this. The 803/808 statistic behind the change came from bridged clones and was generalised to pasta rootless without checking.

   The defect is still real — readiness declared from a neighbour entry with no proof the guest answered is what produces the 100 s hangs — but ICMP is the wrong instrument. The correct signal already exists and the guest already computes it: fc-agent knows when post-restore networking is up and simply does not tell fcvm. A readiness ACK over the vsock channel fcvm already holds is protocol-independent and works in every network mode. That is the "restore readiness/ACK flow" carried by agent/wip-issue776-snapshot-network-20260810, so the requirement belongs there. #792 is a draft; its GuestProbe seam and tests are worth keeping, since making the decision unit-testable is what let CI's failure be attributed in one read.

   WIP branch verdicts

   Verified by content, not by branch name. Four branches fully contained in main were deleted (vm-lifecycle-fuzz, sleep-signals, teardown-pasta-pdeathsig, uffd-minor-continue).

   branch	verdict	why
   agent/pr751-nested-data	DELETE	Premise gone and actively harmful: its invariant test asserts deployable profiles must pin 7.1.7, one of the three kernels proven to wedge. Every non-kernel file is byte-identical to main.
   release-uffd-slot-on-peer-exit	DELETE	Bug does not exist on main; 134 of 155 lines already there verbatim, including the test. A latent leak, and closed — not a candidate for the CDP wedge.
   uffd-fail-closed	DELETE	Landed and strengthened. Its docs are now factually wrong (says cap 64; main is 256) and argue a premise main corrected from corruption to wedge.
   issue776-scale-harness	EXTRACT	Measures what bench/chromium structurally cannot: main's reqbench.py is closed-loop, this drives an open-loop arrival rate and deconfounds FILE-vs-UFFD within a rate interval.
   agent/archive-loopback-benchmark-20260810	EXTRACT	Carries an unlanded fix for a live bug on main: cleanup_vm moves the health-monitor JoinHandle into a tokio::select!, so the timeout arm drops it — which detaches rather than cancels, leaving the monitor running while cleanup destroys its netns. Same class as the wait_for_stderr_eof defect review caught today, second location.
   agent/archive-benchmark-state-20260810	EXTRACT	Only copy of the instrument that produced the numbers working_set.rs:5-13 asserts; nothing on main can re-derive them.
   agent/archive-clone-overlap-runner-20260810	EXTRACT	One Makefile target; nextest pins those suites to max-threads = 1 so one process cannot produce that interleaving.
   agent/archive-pasta-stderr-prototype-20260810	EXTRACT	Two tests closing a real hole — main's only test covers the timeout branch, so an immediate-return would still pass.
   agent/archive-pr778-red-test-20260810	DELETE	Main's copy is a strict superset.
   agent/archive-uffd-ws-v2-20260810	DELETE	Main is a strict superset; zero branch-only test names.
   agent/archive-uffd-ws-rework-20260810	DELETE	Known-invalid: SO_PEERCRED+pidfd_open(pid), which main rejects as PID-recycling-unsafe.
   agent/archive-uffd-profile-20260810	DELETE after one doc line	Self-labeled DO NOT COMMIT.

   No branch merits landing wholesale. Deletions are being held until the extractions land, so nothing is dropped before it is salvaged.

   Independent finding: FCVM_UFFD_MAX_CLONES is documented nowhere outside src/ while twelve other FCVM_* vars are.

   Process failure worth recording

   PR #790 was merged without reading its review comments, which this repo's rules make mandatory. Reading them afterwards found four P1s, all legitimate — including that make bench-quick cannot run at all, because criterion flags land before Cargo's -- separator. A broken documented target shipped inside the PR whose purpose was fixing broken documented targets, and it shipped because it was verified with make -n instead of being run. All five findings are being fixed with the requirement that each is proven by executing the target.
4. ejc3 commented on Aug 10, 2026

   

   ejc3

   on Aug 10, 2026

   OwnerAuthor

   More actions

   Final state for handoff

   Summary is now a required status check on main (strict=false, contexts=["Summary"]), with every other protection setting preserved — enforce_admins false, force-push and deletions still blocked. Enabled only after #791 landed and #785 demonstrated a green Summary on a docs-only PR, which is the class that would otherwise have been blocked outright.

   Note what this does and does not buy: it closes the hole where a PR could sit CLEAN with zero executed checks. It does not stop a direct push to main by the repo owner, because enforce_admins is off — so the push.default=upstream trap that put one commit straight onto main today is still open. Turning on "include administrators" would close it at the cost of hand-pushing emergency fixes.

   Merged to main today (10)

   #784, the _test-unit FILTER fix, #779, #789, #744, #751, #788, #787, #791, #785.

   Open PRs (6)

   PR	what	state
   #793	Capture guest and host state when a CDP request fails	ready
   #794	Fix five review findings on the merged benchmark changes	ready
   #795	Salvage the unlanded work from the archived WIP branches	ready
   #796	fcvm podman prepare with a nameable, keepable snapshot	ready
   #797	Proven TCP generation boundary for memory snapshots	ready
   #792	Require the guest to answer before declaring readiness	draft — refuted, needs the guest-ACK design

   #797 is the candidate wedge fix, and its adjudication is honest about what it does not explain

   Established from artifacts, not argued: the benchmark passes no --kernel-profile, so it ran the default profile = Kata 6.12.47, whose embedded config on both arches reads # CONFIG_INET_DIAG is not set. So ss -K could destroy nothing, the documented conntrack -F fallback needs a binary the rootfs never installs, and its error arm is Err(_) => {}. Post-restore TCP cleanup on main has been a silent no-op. Independently, the old filter spared dst 10.0.2.0/24, and under pasta a host→guest published-port connection has peer 10.0.2.2 — exactly the socket class a new CDP client collides with. Both premises of the stale-socket theory hold on main and this PR fixes both.

   But it does not explain the ping predictor: a stale TCP socket cannot stop a guest answering ICMP echo. And the PR removes ip neigh flush all, which is a falsifiable risk — if the wedge is neighbour-state related this could make it worse. That prediction is written into the acceptance test.

   Acceptance test (needs Graviton3-class host with /dev/kvm): rebuild the default kernel, confirm CONFIG_INET_DIAG=y inside a booted guest, build a golden snapshot with a non-empty non-loopback socket set, check a restored clone's manifest for non-zero cookies before cleanup, then run reqbench.sh run at 809 requests for both backends. Report the ping-predictor table, not the overall rate. Failures to zero ⇒ this was the cause. Rate drops but ping-unanswered clones still fail disproportionately ⇒ stale sockets were contributory and the ICMP fault is separate. Unchanged ⇒ the PR is right on its own merits and is not this bug. Rate goes up ⇒ the ip neigh flush removal.

   #795 carries a live-bug fix and a defect sweep

   cleanup_vm moved the health-monitor JoinHandle into a tokio::select!, so the timeout arm dropped it — detaching rather than cancelling, leaving the monitor running while cleanup destroyed its netns. Witnessed red. All nine detached-JoinHandle sites in the tree were then examined; one more is worth a follow-up (fc-agent/src/proxy.rs:272,275, self-limiting but real), and the rest are correct or deliberate.

   #794 is remediation of a process failure

   #790 was merged without reading its review comments. Four P1s were waiting, including that make bench-quick could not run at all — criterion flags landed before Cargo's --. Running the fixes then exposed a fifth defect nobody had reported: criterion resolves a relative CARGO_TARGET_DIR against the bench binary's cwd, so every suite wrote to fuse-pipe/target/criterion while the ownership repair scanned target/ — the repair had never fired, and bench-protocol was silently persisting nothing. CRITERION_HOME now pins one absolute path.

   The metadata benchmarks now force real round trips via statx(AT_EMPTY_PATH|AT_STATX_FORCE_SYNC) and each asserts it reached the server before measuring. Measured: cache hits ~298 ns, round trips 83.1 µs and 120.3 µs — 293x and 396x, against the "~5% overhead" PERFORMANCE.md published. That conclusion is removed.

   Still open

   • Attribution of the 7.x wedge. Our FUSE patch is cleared by a verified control (7.1.8 minus that patch, still RED). Remaining candidates: upstream 7.x, or one of the four other arm64 patches. A fully unpatched 7.1.8 cannot be tested — without mmfr4-override there is no L2.
   • Why the readiness ping goes unanswered for ~0.8% of clones. Capture guest and host state when a CDP request fails #793 captures the guest's own view at failure time with a healthy control; Give every memory snapshot a proven TCP generation boundary #797 may or may not remove the failures entirely.
   • Rewiring bench/chromium's golden phase onto prepare now that Add fcvm podman prepare with a nameable, keepable snapshot #796 gives it a caller-chosen tag, a User snapshot type that survives prune, and --force.
   • Branch deletions for the twelve adjudicated WIPs, once Salvage the unlanded work from the archived WIP branches #795 lands and the salvage is confirmed on main.
   • fc-agent/src/proxy.rs detached handles; spawn_victim()'s orphaned sleep 600 wedging the next make for up to 10 minutes; tests/test_serve.rs unused import under --no-default-features.
5. ejc3 commented on Aug 14, 2026

   

   ejc3

   on Aug 14, 2026

   OwnerAuthor

   More actions

   Superseded by #819, the current comprehensive tracker. Everything in this 2026-08-10 snapshot has either merged (#808/#809/#792/#805/#815/#816/#817), been root-caused and published in the benchmark report, or is enumerated as an open item in #819.
6. ejc3

   closed this as completedon Aug 14, 2026