Repository navigation
[17.0-stable] evetpm, newlogd, ZFS trim, EVE-k PVC upload and unit-test fixes - #6993
Merged
eriknordmark merged 14 commits intoNov 1, 2026
Merged
eriknordmark merged 14 commits into
eriknordmark merged 14 commits into
Conversation
When the vault key is fetched, a TPM error while checking for the sealed disk key was read as "no key present", so EVE generated and sealed a fresh key over the existing one. Data encrypted with the old key, such as a reused etcd zvol after a kvm-to-k vault migration redo, then could not be opened by any key. The check now distinguishes an undefined NV index from a TPM error, retries the error twice with a log line per attempt, and fails the fetch rather than generating a key if it persists. An index that is defined but never written still counts as absent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit e3969c3)
Fetching the vault key must not take a TPM error for an absent key, or it seals a fresh key over the one the vault is encrypted with. The tests put a proxy between EVE and the TPM simulator that fails the disk-key presence check while every other command, sealing included, keeps working. A transient failure must yield the sealed key, and a persistent one an error with the sealed key left in place. Against the earlier presence check, the transient case replaces the key. The proxy answers TPM_RC_NV_UNAVAILABLE rather than TPM_RC_RETRY because go-tpm retries TPM_RC_RETRY itself and never returns it to the caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit aaaaab0)
Vector creates its input sockets only once its topology is up, about 20 s into an EVE-k boot, while the services and pillar log their startup. newlogd queued 1000 entries per socket and silently dropped the rest, so the first 1000 log entries of a boot were followed by a gap: 866 entries on one EVE-k boot, including everything nodeagent logged before it published its status. For the first 60 s after it starts writing to vector, newlogd now waits for room in the queue instead of dropping. That stops it reading from memlogd, whose ring buffer holds the backlog meanwhile. Once the window ends a full queue drops as before, so a vector that never comes up or stalls later cannot hold up logging for longer than that. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0353a11)
The dropped-message counter of the vector socket writer is incremented by the logging goroutine when the buffer is full, by the writer goroutine when a socket write fails, and read when deciding whether to report drops, with no synchronization. Make it an atomic.Int64 so the race detector stays quiet and the count stays exact. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0b6761e)
On EVE-k, zfsmanager ran an unthrottled zpool trim of the persist pool at every boot. On a ~1 TB SATA disk that pass takes many hours, and on a device that reboots often it re-trims all free space on every boot. Under the resulting load Longhorn replicas hit their 16 s R/W timeout and app volume creation stalled in CREATING_VOLUME. Drop the boot-time trim and run the scheduled trim with zpool trim -r at 1.5 MiB/s, a rate ZFS applies to each disk in the pool. Cap each TRIM command at 16 MiB through zfs_trim_extent_bytes_max, set at boot with the other ZFS module parameters and checked by zfsmanager. A trim interrupted by a reboot resumes on its own, so the cron start covers the whole pool over time. A resumed trim keeps the rate it was started with, so at startup zfsmanager suspends any active trim and restarts it at the limit; this throttles a trim started by an older EVE, keeps its progress, and resets the start time zpool status reports to the boot time. A cron tick that finds a trim running is logged as a notice and leaves the published trim start time unchanged. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7caec15)
At the fixed 1.5 MiB/s a trim pass over a 4 TB disk takes about a month. The new storage.zfs.pool.trim.rate.bytespersec config item sets the rate, in bytes/s per disk as zpool trim -r applies it, with 1.5 MiB/s as the default. A change takes effect on the next scheduled trim and, within a minute, on a trim already in progress, which is suspended and restarted at the new rate without losing its progress. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 73a22b3)
On EVE-k, a reboot while CDI is uploading a volume's image leaves the PVC holding part of the data. After the reboot volumemgr adopts PVCs it finds and marked this one created, so the app started on a volume with missing blocks and no error was reported. Decide from the PVC itself whether an upload was started and whether it finished: virtctl marks every PVC it uploads into with CDI's upload target annotation, and CDI records completion in the pod phase. A PVC with an unfinished upload is no longer adopted after a reboot; it goes through the create path, which re-drives the upload into it. The same check now applies to volumes without an image reference, such as one filled from a disk file, when deciding whether an existing PVC can be reused and whether the volume can be reported created. The re-drive is refused while a pod other than CDI's own uses the PVC, since an app activated by an older EVE may already be running on it and CDI does not stop an upload into a claim that is in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kgg2Z8uxUyj6gY12oHbqxT Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit c8c3ab4)
TestSingleEthInterface and TestSingleEthInterfaceWithIPv6 failed intermittently on loaded CI runners, with DNS servers still reported after the interface lost its addresses. One simulated interface update publishes address, DNS and route events, and the first of them wakes the test. The reconciler's watcher picks randomly between a queued event and the pause request from Reconcile, so Reconcile could run before the DNS change was processed and skip the DNS state. Wait until the watcher has received every published event before reconciling. The mock network monitor reports the number of events still queued for its subscribers for this purpose. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5c2a208)
TestManageStatFileSizeTargetSizesAvgThreshold asserted that rewriting a 6.8 MB stat file takes under 5 s. Under -race on a loaded CI runner it sometimes does not, failing the test although the trimmed file is correct. The test checks the result, which is what it is for; log the duration and drop the bound. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 30428f1)
TestWork required each sleeping job to finish within 100 ms of its nominal duration, which a loaded CI runner under -race does not guarantee. Run TestWork and TestLength under testing/synctest so the measured durations are exact. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5fe0822)
The test sleeps for twice the stats window and expects the slice to have filled at one sample per millisecond, which fails when real sleeps run long under -race on a loaded runner. Run it under testing/synctest, record every sample, and wait for the monitor to exit before the bubble ends. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 60471e4)
Probe latency is measured in real time around the mock's simulated RTT, so a slow scheduler only inflates it, and the fixed headroom the tests allowed was exceeded on loaded CI runners. A probe that outlasts UserProbeTimeout fails instead of reporting latency, so use that as the upper bound. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 50fd414)
The first global config adds the lastresort DPC, and the PNAC tests waited only for some verification to finish. When lastresort was verified first, the PNAC event was handled before the test's DPC was applied and the reacquire was canceled, so the retry never came. Wait for the test's DPC by key, and disable the periodic connectivity test, which blocks the main loop and delays the timed retries. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d431622)
The send and verify tests reached google.com and postman-echo.com, so a slow resolver or a CI runner without Internet access failed them; a 5 s DNS timeout did so on CI. Serve them from an httptest TLS server bound to the port's IPv4 address instead. The URL is an IP literal, so these tests no longer exercise DNS resolution. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b81f434)
github-actions
Bot
requested review from
OhmSpectator,
andrewd-zededa,
christoph-zededa,
europaul,
milan-zededa,
naiming-zededa,
rucoder,
shjala and
zedi-pramodh
October 30, 2026 18:51
This was referenced Oct 30, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 17.0-stable #6993 +/- ##
===============================================
+ Coverage 23.24% 23.67% +0.43%
===============================================
Files 516 526 +10
Lines 95468 97299 +1831
===============================================
+ Hits 22193 23039 +846
- Misses 71401 72198 +797
- Partials 1874 2062 +188 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
eriknordmark
requested review from
rene
and removed request for
naiming-zededa,
rucoder,
shjala and
zedi-pramodh
October 30, 2026 20:36
eriknordmark
marked this pull request as ready for review
October 30, 2026 20:36
eriknordmark
requested review from
christoph-zededa,
naiming-zededa,
rucoder and
shjala
as code owners
October 30, 2026 20:36
This was referenced Oct 30, 2026
rene
approved these changes
Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6691, #6839, #6841, #6760, #6953 to
17.0-stable, combined intoone PR to save CI runs. Every commit is cherry-picked with
-xfrom master, inmaster merge order:
at boot no longer makes EVE replace the vault's disk key with a new one.
first seconds of boot are no longer lost while the log pipeline starts.
trim no longer runs at every boot, and the scheduled trim is rate-limited.
volume whose CDI upload was cut short by a reboot is uploaded again instead
of being used with missing data.
Adaptations:
sealTestDiskKeydoes not callextendFirmwareAnchorPCRs, which comes with evetpm: refuse to seal the vault key against all-zero firmware PCRs (defense in depth) #6162 (refuse to seal againstall-zero firmware PCRs) and is not on this branch. Without evetpm: refuse to seal the vault key against all-zero firmware PCRs (defense in depth) #6162's guard the
call has nothing to satisfy. Production code is unchanged.
eve-dom0-ztoolsFROM hash inpkg/pillar/Dockerfileandpkg/vtpm/Dockerfileis this branch's own, recomputed after the000-mod-paramschange.fix's one-line change inside that code (passing
pvcExiststoRolloutDiskToPVC) and the conflicting migration test are left out.Everything else the fix adds is unchanged.
the test it fixes does not exist on this branch. The other seven commits
apply unchanged.
All other commits' diffs are identical to master's.
PR dependencies
None.
How to test and validate this PR
gofmt,go build ./...(with and without-tags k),go vetandgo testof every touched pillar package, and ofpkg/newlog/cmd, pass onthis branch.
make check-docker-hashes-consistencyexits 0.tests/tpm/prep-and-run.sh).TestDiskKeyPresentFailsOnTPMErrorfails on this branch whendiskKeyPresenttreats a TPM error as an absent key again.
TestCSIHandler_PopulateRejectsPartialUploadfails on this branch with thepartial-upload check in
Populateremoved.Changelog notes
new one.
pipeline starts up.
trim is rate-limited so it does not slow down application storage.
uploaded again instead of being used with missing data.
PR Backports
Checklist
included.
device testing is on the source PRs.
For backport PRs:
I've added a reference link to the original PR
PR's title follows the template
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
🤖 Generated with Claude Code