Skip to content

zfsmanager: rate-limit pool trim, drop boot trim - #6841

Merged
rene merged 2 commits into
lf-edge:masterfrom
eriknordmark:zfs-trim-rate-limit
Oct 27, 2026
Merged

rene merged 2 commits into
lf-edge:masterfrom
eriknordmark:zfs-trim-rate-limit

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

On EVE-k, zfsmanager runs an unthrottled zpool trim of the persist pool at
every boot. On a ~1 TB SATA disk that pass takes many hours, and on a device
that reboots often it restarts against all free space on every boot. Under that
load Longhorn replicas hit their 16 s R/W timeout and app volume creation
stalls in CREATING_VOLUME.

Even without frequent reboots of the device there is a performance impact when the periodic trim runs.

This PR drops the boot-time trim, runs the scheduled trim as
zpool trim -r 1572864 persist (1.5 MiB/s), and caps each TRIM command at
16 MiB through zfs_trim_extent_bytes_max. A trim interrupted by a reboot
resumes on its own, so the existing timer.zfs.pool.trim.cron start covers the
whole pool over time (about a week per TB of free space).

A resumed trim keeps the rate it was started with, so a trim started by an
older EVE would stay unthrottled. At startup zfsmanager suspends any active
trim and restarts it at the limit; progress is kept, but the start time
reported by zpool status -t becomes the boot time. A cron tick that finds a
trim running is logged as a notice instead of an error. docs/ZFS.md is
updated to match.

Partially addresses #6650.

How to test and validate this PR

On an EVE-k node with a ZFS persist pool:

  1. After boot, zpool status -t persist shows no new trim started at boot, and
    cat /sys/module/zfs/parameters/zfs_trim_extent_bytes_max prints 16777216.
  2. At the cron time (default Sat/Sun 03:00), a trim starts; discard throughput
    stays near 1.5 MiB/s (e.g. /proc/diskstats discard sectors).
  3. Reboot mid-trim: zfsmanager logs resumed trim of persist at 1572864 bytes/s, zpool status -t persist shows the trim continuing from its
    previous percentage with the boot time as start time, and discards resume
    at ~1.5 MiB/s.
  4. Upgrade from an image with the unthrottled boot trim while that trim is
    running: after boot the discard rate drops to ~1.5 MiB/s.

Latency results under the ztest soak:
#6841 (comment)

Changelog notes

On EVE-k nodes the ZFS pool trim no longer runs at every boot. The scheduled
trim is rate-limited so it does not slow down application storage.

PR Backports

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj

@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 2.08333% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.61%. Comparing base (ce3fd33) to head (a547fa4).
⚠️ Report is 13 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/cmd/zfsmanager/handlepooltrim.go 0.00% 47 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6841      +/-   ##
==========================================
+ Coverage   28.33%   28.61%   +0.28%     
==========================================
  Files         537      548      +11     
  Lines       98110    99897    +1787     
==========================================
+ Hits        27802    28589     +787     
- Misses      68054    68858     +804     
- Partials     2254     2450     +196     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark eriknordmark added the stable Should be backported to stable release(s) label Oct 19, 2026
@eriknordmark
eriknordmark marked this pull request as ready for review October 19, 2026 09:28
@eriknordmark
eriknordmark force-pushed the zfs-trim-rate-limit branch 2 times, most recently from 87712a2 to ea678cb Compare October 19, 2026 09:34
if !base.IsHVTypeKube() {
return
}
if err := os.WriteFile(trimExtentMaxParam,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upgrade path: a device that was mid-trim on the previous image resumes that trim at import with its stored rate (vdev ZAP trim_rate, see vdev_trim_restart), i.e. unthrottled. zpool trim -r on it returns EBUSY while the trim thread exists (spa_vdev_trim_impl). So the devices this PR targets keep the unthrottled trim for one more full pass, which on a frequently rebooting box may never finish. Needs a one-time fixup at startup.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in b55c78a: at startup zfsmanager runs zpool trim -s and, if a trim was active, restarts it with -r. The restarted thread resumes from the saved offset, so no progress is lost; the cost is that the start time zpool status -t shows becomes the boot time. zpool status doesn't expose the rate, so this runs on every boot rather than only after an upgrade.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eriknordmark Just noticed, the rest of the zfs parameters are set in pkg/dom0-ztools/rootfs/etc/init.d/000-mod-params. Moving this over there would lose this narrow error-case log line but I suppose this path could verify the value was set as expected?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved in adebb5b: 000-mod-params sets zfs_trim_extent_bytes_max to 16 MiB with the other ZFS parameters, and zfsmanager logs an error at startup if the value is not 16 MiB.

@@ -77,7 +93,8 @@ func runPoolTrimSchedule(ctx *zfsContext) {
func runZpoolTrim(ctx *zfsContext) {
ctx.setTrimStart(time.Now())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a new issue, but much more likely now that a pass takes days: a cron tick that lands on a trim still in progress gets "cannot trim '/dev/sda9': currently trimming", logged as an error, after setTrimStart has already overwritten PoolTrimStatus.LastStartTime to imply a fresh start. Suggest setting the start time only on success and logging the busy case as a notice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b55c78a: LastStartTime is set only when zpool trim succeeds, and "currently trimming" is logged as a notice.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

Latency results from a 1 TB SATA EVE-k device (image built from ea678cb) under the ztest soak, sampling disk
latency every ~134 s:

Condition Samples PSI io full % p50 / p95 write ms/op p50 / p95 read ms/op p50 / p95
Unthrottled boot trim (before) 560 19.0 / 46.2 15.6 / 20.2 17.5 / 21.3
No trim running 2136 0.19 / 3.4 0.45 / 3.0 0.29 / 3.0
Rate-limited trim (this PR) 255 0.23 / 3.8 0.51 / 3.2 0.31 / 3.4

const (
// poolTrimRate caps the average rate of a pool trim, in bytes/s. At this
// rate one pass over 1 TB of free space takes about a week.
poolTrimRate = 1536 * 1024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a 4 TB pool a pass takes about a month. The cron spec is already a config key, can the rate be as well?

@eriknordmark eriknordmark Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in a9c2199 as storage.zfs.pool.trim.rate.bytespersec, in bytes/s per disk as zpool trim -r applies it: default 1572864 (1.5 MiB/s), min 65536 (64 KiB/s). A change also restarts a trim in progress at the new rate, keeping its progress.

@rene

rene commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 2m 54s —— View job


Review

I reviewed the diff against origin/master. The change is well-scoped and the earlier review concerns (upgrade-path re-throttle, busy-tick logged as notice, configurable rate, moving zfs_trim_extent_bytes_max into 000-mod-params) are all addressed.

Correctness checks I verified:

  • trimRate can't be read as 0: the startup resumeTrimAtRate runs only after the GCInitialized gate (zfsmanager.go:134) which first invokes refreshTrimConfig, and GlobalValueInt falls back to the spec default (1572864) when the key is absent, so an -r 0 (unlimited) start can't be issued.
  • Rate type is consistent end-to-end (uint32 field ↔ GlobalValueInt ↔ AddIntItem); default/min/max (1572864 / 65536 / 0xFFFFFFFF) match the docs and CONFIG-PROPERTIES.md.
  • trimExtentMaxParam uses the established /hostfs/sys/module/zfs/parameters/... path (same as ZFSArcMaxSizeFile).
  • -s → -r suspend/restart is guarded so a rate change while no trim is active doesn't start an off-schedule trim (noActiveTrimMsg short-circuits).
  • LastStartTime is set only on a successful fresh zpool trim, and the busy case is a notice — consistent with the updated PoolTrimStatus doc.

One minor note (posted inline on resumeTrimAtRate): the re-throttle depends on ZFS having already auto-resumed the interrupted trim at import before zpool trim -s runs. Practically safe since the pool is imported well before zfsmanager starts; flagged only as an assumed ordering.

No blocking issues from me — looks good.
· branch zfs-trim-rate-limit

// rate to check, so every active trim is re-issued at startup. A suspend
// waits for the trim thread to exit and keeps the trim's progress, but resets
// the start time that `zpool status -t` reports.
func resumeTrimAtRate(rate uint32) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor / ordering note: the re-throttle relies on zpool trim -s finding the trim thread that ZFS auto-resumes at import (vdev_trim_restart). If this ran before import had spawned that thread, -s would hit noActiveTrimMsg, return early, and the old unthrottled trim would then auto-resume unthrottled for that boot. In practice the persist pool is imported long before zfsmanager starts, so the thread already exists — not a blocker, just worth being aware of that this is an assumed ordering rather than a guaranteed one.

@rene

rene commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@eriknordmark you have some conflicts to fix

eriknordmark and others added 2 commits October 26, 2026 07:45
On EVE-k, zfsmanager ran an unthrottled zpool trim of the persist pool
at every boot. On a ~1 TB SATA disk that pass takes many hours, and on
a device that reboots often it re-trims all free space on every boot.
Under the resulting load Longhorn replicas hit their 16 s R/W timeout
and app volume creation stalled in CREATING_VOLUME.

Drop the boot-time trim and run the scheduled trim with zpool trim -r
at 1.5 MiB/s, a rate ZFS applies to each disk in the pool. Cap each
TRIM command at 16 MiB through zfs_trim_extent_bytes_max, set at boot
with the other ZFS module parameters and checked by zfsmanager. A trim
interrupted by a reboot resumes on its own, so the cron start covers
the whole pool over time. A resumed trim keeps the rate it was started
with, so at startup zfsmanager suspends any active trim and restarts it
at the limit; this throttles a trim started by an older EVE, keeps its
progress, and resets the start time zpool status reports to the boot
time. A cron tick that finds a trim running is logged as a notice and
leaves the published trim start time unchanged.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At the fixed 1.5 MiB/s a trim pass over a 4 TB disk takes about a
month. The new storage.zfs.pool.trim.rate.bytespersec config item sets
the rate, in bytes/s per disk as zpool trim -r applies it, with 1.5
MiB/s as the default. A change takes effect on the next scheduled trim
and, within a minute, on a trim already in progress, which is suspended
and restarted at the new rate without losing its progress.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@andrewd-zededa andrewd-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, my concerns have been addressed.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

@eriknordmark you have some conflicts to fix

@rene fixed

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rene
rene merged commit 73a22b3 into lf-edge:master Oct 27, 2026
61 of 66 checks passed
@eriknordmark
eriknordmark deleted the zfs-trim-rate-limit branch October 27, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants