Skip to content

[17.0-stable] evetpm, newlogd, ZFS trim, EVE-k PVC upload and unit-test fixes - #6993

Merged
eriknordmark merged 14 commits into
lf-edge:17.0-stablefrom
eriknordmark:backports-2026-10-30-17.0-stable
Nov 1, 2026
Merged

eriknordmark merged 14 commits into
lf-edge:17.0-stablefrom
eriknordmark:backports-2026-10-30-17.0-stable

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

Backport of #6691, #6839, #6841, #6760, #6953 to 17.0-stable, combined into
one PR to save CI runs. Every commit is cherry-picked with -x from master, in
master merge order:

Adaptations:

All other commits' diffs are identical to master's.

PR dependencies

None.

How to test and validate this PR

  • gofmt, go build ./... (with and without -tags k), go vet and
    go test of every touched pillar package, and of pkg/newlog/cmd, pass on
    this branch. make check-docker-hashes-consistency exits 0.
  • The evetpm tests run against swtpm (tests/tpm/prep-and-run.sh).
    TestDiskKeyPresentFailsOnTPMError fails on this branch when diskKeyPresent
    treats a TPM error as an absent key again.
  • TestCSIHandler_PopulateRejectsPartialUpload fails on this branch with the
    partial-upload check in Populate removed.
  • Device-level validation is described in each source PR.

Changelog notes

  • A transient TPM error at boot no longer replaces the vault's disk key with a
    new one.
  • Device logs from the first seconds of boot are no longer lost while the log
    pipeline starts up.
  • On EVE-k nodes the ZFS pool trim no longer runs at every boot. The scheduled
    trim is rate-limited so it does not slow down application storage.
  • EVE-k: an app volume whose image upload was interrupted by a reboot is
    uploaded again instead of being used with missing data.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation — the docs changes of zfsmanager: rate-limit pool trim, drop boot trim #6841 are
    included.
  • I've tested my PR on amd64 device — unit tests on this branch only; the
    device testing is on the source PRs.
  • I've tested my PR on arm64 device — not tested.
  • I've written the test verification instructions
  • I've set the proper labels to this PR — none apply to a backport.

For backport PRs:

  • I've added a reference link to the original PR

  • PR's title follows the template

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

eriknordmark and others added 14 commits October 30, 2026 10:44
When the vault key is fetched, a TPM error while checking for the sealed
disk key was read as "no key present", so EVE generated and sealed a
fresh key over the existing one. Data encrypted with the old key, such as
a reused etcd zvol after a kvm-to-k vault migration redo, then could not
be opened by any key. The check now distinguishes an undefined NV index
from a TPM error, retries the error twice with a log line per attempt,
and fails the fetch rather than generating a key if it persists. An index
that is defined but never written still counts as absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
(cherry picked from commit e3969c3)
Fetching the vault key must not take a TPM error for an absent key, or
it seals a fresh key over the one the vault is encrypted with. The tests
put a proxy between EVE and the TPM simulator that fails the disk-key
presence check while every other command, sealing included, keeps
working. A transient failure must yield the sealed key, and a persistent
one an error with the sealed key left in place. Against the earlier
presence check, the transient case replaces the key.

The proxy answers TPM_RC_NV_UNAVAILABLE rather than TPM_RC_RETRY because
go-tpm retries TPM_RC_RETRY itself and never returns it to the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
(cherry picked from commit aaaaab0)
Vector creates its input sockets only once its topology is up, about
20 s into an EVE-k boot, while the services and pillar log their
startup. newlogd queued 1000 entries per socket and silently dropped
the rest, so the first 1000 log entries of a boot were followed by a
gap: 866 entries on one EVE-k boot, including everything nodeagent
logged before it published its status.

For the first 60 s after it starts writing to vector, newlogd now
waits for room in the queue instead of dropping. That stops it reading
from memlogd, whose ring buffer holds the backlog meanwhile. Once the
window ends a full queue drops as before, so a vector that never comes
up or stalls later cannot hold up logging for longer than that.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0353a11)
The dropped-message counter of the vector socket writer is incremented
by the logging goroutine when the buffer is full, by the writer
goroutine when a socket write fails, and read when deciding whether to
report drops, with no synchronization. Make it an atomic.Int64 so the
race detector stays quiet and the count stays exact.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0b6761e)
On EVE-k, zfsmanager ran an unthrottled zpool trim of the persist pool
at every boot. On a ~1 TB SATA disk that pass takes many hours, and on
a device that reboots often it re-trims all free space on every boot.
Under the resulting load Longhorn replicas hit their 16 s R/W timeout
and app volume creation stalled in CREATING_VOLUME.

Drop the boot-time trim and run the scheduled trim with zpool trim -r
at 1.5 MiB/s, a rate ZFS applies to each disk in the pool. Cap each
TRIM command at 16 MiB through zfs_trim_extent_bytes_max, set at boot
with the other ZFS module parameters and checked by zfsmanager. A trim
interrupted by a reboot resumes on its own, so the cron start covers
the whole pool over time. A resumed trim keeps the rate it was started
with, so at startup zfsmanager suspends any active trim and restarts it
at the limit; this throttles a trim started by an older EVE, keeps its
progress, and resets the start time zpool status reports to the boot
time. A cron tick that finds a trim running is logged as a notice and
leaves the published trim start time unchanged.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7caec15)
At the fixed 1.5 MiB/s a trim pass over a 4 TB disk takes about a
month. The new storage.zfs.pool.trim.rate.bytespersec config item sets
the rate, in bytes/s per disk as zpool trim -r applies it, with 1.5
MiB/s as the default. A change takes effect on the next scheduled trim
and, within a minute, on a trim already in progress, which is suspended
and restarted at the new rate without losing its progress.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 73a22b3)
On EVE-k, a reboot while CDI is uploading a volume's image leaves the
PVC holding part of the data. After the reboot volumemgr adopts PVCs it
finds and marked this one created, so the app started on a volume with
missing blocks and no error was reported.

Decide from the PVC itself whether an upload was started and whether it
finished: virtctl marks every PVC it uploads into with CDI's upload
target annotation, and CDI records completion in the pod phase. A PVC
with an unfinished upload is no longer adopted after a reboot; it goes
through the create path, which re-drives the upload into it. The same
check now applies to volumes without an image reference, such as one
filled from a disk file, when deciding whether an existing PVC can be
reused and whether the volume can be reported created.

The re-drive is refused while a pod other than CDI's own uses the PVC,
since an app activated by an older EVE may already be running on it and
CDI does not stop an upload into a claim that is in use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kgg2Z8uxUyj6gY12oHbqxT
Signed-off-by: eriknordmark <erik@zededa.com>
(cherry picked from commit c8c3ab4)
TestSingleEthInterface and TestSingleEthInterfaceWithIPv6 failed
intermittently on loaded CI runners, with DNS servers still reported
after the interface lost its addresses. One simulated interface update
publishes address, DNS and route events, and the first of them wakes
the test. The reconciler's watcher picks randomly between a queued
event and the pause request from Reconcile, so Reconcile could run
before the DNS change was processed and skip the DNS state.

Wait until the watcher has received every published event before
reconciling. The mock network monitor reports the number of events
still queued for its subscribers for this purpose.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5c2a208)
TestManageStatFileSizeTargetSizesAvgThreshold asserted that rewriting a
6.8 MB stat file takes under 5 s. Under -race on a loaded CI runner it
sometimes does not, failing the test although the trimmed file is
correct. The test checks the result, which is what it is for; log the
duration and drop the bound.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 30428f1)
TestWork required each sleeping job to finish within 100 ms of its
nominal duration, which a loaded CI runner under -race does not
guarantee. Run TestWork and TestLength under testing/synctest so the
measured durations are exact.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5fe0822)
The test sleeps for twice the stats window and expects the slice to
have filled at one sample per millisecond, which fails when real
sleeps run long under -race on a loaded runner. Run it under
testing/synctest, record every sample, and wait for the monitor to
exit before the bubble ends.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 60471e4)
Probe latency is measured in real time around the mock's simulated
RTT, so a slow scheduler only inflates it, and the fixed headroom the
tests allowed was exceeded on loaded CI runners. A probe that outlasts
UserProbeTimeout fails instead of reporting latency, so use that as
the upper bound.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 50fd414)
The first global config adds the lastresort DPC, and the PNAC tests
waited only for some verification to finish. When lastresort was
verified first, the PNAC event was handled before the test's DPC was
applied and the reacquire was canceled, so the retry never came.
Wait for the test's DPC by key, and disable the periodic connectivity
test, which blocks the main loop and delays the timed retries.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d431622)
The send and verify tests reached google.com and postman-echo.com, so
a slow resolver or a CI runner without Internet access failed them; a
5 s DNS timeout did so on CI. Serve them from an httptest TLS server
bound to the port's IPv4 address instead. The URL is an IP literal,
so these tests no longer exercise DNS resolution.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b81f434)
@codecov

codecov Bot commented Oct 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 42.33129% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 23.67%. Comparing base (1791e05) to head (0564940).

Files with missing lines Patch % Lines
pkg/pillar/cmd/zfsmanager/handlepooltrim.go 0.00% 47 Missing ⚠️
pkg/pillar/kubeapi/cdiupload.go 43.47% 26 Missing ⚠️
pkg/pillar/volumehandlers/csihandler.go 75.75% 6 Missing and 2 partials ⚠️
pkg/pillar/evetpm/tpm.go 76.66% 6 Missing and 1 partial ⚠️
pkg/pillar/netmonitor/mock.go 0.00% 6 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           17.0-stable    #6993      +/-   ##
===============================================
+ Coverage        23.24%   23.67%   +0.43%     
===============================================
  Files              516      526      +10     
  Lines            95468    97299    +1831     
===============================================
+ Hits             22193    23039     +846     
- Misses           71401    72198     +797     
- Partials          1874     2062     +188     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark
eriknordmark merged commit 8a4b36d into lf-edge:17.0-stable Nov 1, 2026
64 of 65 checks passed
@eriknordmark
eriknordmark deleted the backports-2026-10-30-17.0-stable branch November 6, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants