Repository navigation
zfsmanager: rate-limit pool trim, drop boot trim - #6841
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6841 +/- ##
==========================================
+ Coverage 28.33% 28.61% +0.28%
==========================================
Files 537 548 +11
Lines 98110 99897 +1787
==========================================
+ Hits 27802 28589 +787
- Misses 68054 68858 +804
- Partials 2254 2450 +196 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
87712a2 to
ea678cb
Compare
| if !base.IsHVTypeKube() { | ||
| return | ||
| } | ||
| if err := os.WriteFile(trimExtentMaxParam, |
There was a problem hiding this comment.
Upgrade path: a device that was mid-trim on the previous image resumes that trim at import with its stored rate (vdev ZAP trim_rate, see vdev_trim_restart), i.e. unthrottled. zpool trim -r on it returns EBUSY while the trim thread exists (spa_vdev_trim_impl). So the devices this PR targets keep the unthrottled trim for one more full pass, which on a frequently rebooting box may never finish. Needs a one-time fixup at startup.
There was a problem hiding this comment.
Good catch, thanks. Fixed in b55c78a: at startup zfsmanager runs zpool trim -s and, if a trim was active, restarts it with -r. The restarted thread resumes from the saved offset, so no progress is lost; the cost is that the start time zpool status -t shows becomes the boot time. zpool status doesn't expose the rate, so this runs on every boot rather than only after an upgrade.
There was a problem hiding this comment.
@eriknordmark Just noticed, the rest of the zfs parameters are set in pkg/dom0-ztools/rootfs/etc/init.d/000-mod-params. Moving this over there would lose this narrow error-case log line but I suppose this path could verify the value was set as expected?
There was a problem hiding this comment.
Moved in adebb5b: 000-mod-params sets zfs_trim_extent_bytes_max to 16 MiB with the other ZFS parameters, and zfsmanager logs an error at startup if the value is not 16 MiB.
| @@ -77,7 +93,8 @@ func runPoolTrimSchedule(ctx *zfsContext) { | |||
| func runZpoolTrim(ctx *zfsContext) { | |||
| ctx.setTrimStart(time.Now()) | |||
There was a problem hiding this comment.
This is not a new issue, but much more likely now that a pass takes days: a cron tick that lands on a trim still in progress gets "cannot trim '/dev/sda9': currently trimming", logged as an error, after setTrimStart has already overwritten PoolTrimStatus.LastStartTime to imply a fresh start. Suggest setting the start time only on success and logging the busy case as a notice.
There was a problem hiding this comment.
Fixed in b55c78a: LastStartTime is set only when zpool trim succeeds, and "currently trimming" is logged as a notice.
ea678cb to
b55c78a
Compare
|
Latency results from a 1 TB SATA EVE-k device (image built from ea678cb) under the ztest soak, sampling disk
|
| const ( | ||
| // poolTrimRate caps the average rate of a pool trim, in bytes/s. At this | ||
| // rate one pass over 1 TB of free space takes about a week. | ||
| poolTrimRate = 1536 * 1024 |
There was a problem hiding this comment.
On a 4 TB pool a pass takes about a month. The cron spec is already a config key, can the rate be as well?
There was a problem hiding this comment.
Added in a9c2199 as storage.zfs.pool.trim.rate.bytespersec, in bytes/s per disk as zpool trim -r applies it: default 1572864 (1.5 MiB/s), min 65536 (64 KiB/s). A change also restarts a trim in progress at the new rate, keeping its progress.
f992738 to
a9c2199
Compare
|
Claude finished @rene's task in 2m 54s —— View job ReviewI reviewed the diff against Correctness checks I verified:
One minor note (posted inline on No blocking issues from me — looks good. |
| // rate to check, so every active trim is re-issued at startup. A suspend | ||
| // waits for the trim thread to exit and keeps the trim's progress, but resets | ||
| // the start time that `zpool status -t` reports. | ||
| func resumeTrimAtRate(rate uint32) { |
There was a problem hiding this comment.
Minor / ordering note: the re-throttle relies on zpool trim -s finding the trim thread that ZFS auto-resumes at import (vdev_trim_restart). If this ran before import had spawned that thread, -s would hit noActiveTrimMsg, return early, and the old unthrottled trim would then auto-resume unthrottled for that boot. In practice the persist pool is imported long before zfsmanager starts, so the thread already exists — not a blocker, just worth being aware of that this is an assumed ordering rather than a guaranteed one.
|
@eriknordmark you have some conflicts to fix |
On EVE-k, zfsmanager ran an unthrottled zpool trim of the persist pool at every boot. On a ~1 TB SATA disk that pass takes many hours, and on a device that reboots often it re-trims all free space on every boot. Under the resulting load Longhorn replicas hit their 16 s R/W timeout and app volume creation stalled in CREATING_VOLUME. Drop the boot-time trim and run the scheduled trim with zpool trim -r at 1.5 MiB/s, a rate ZFS applies to each disk in the pool. Cap each TRIM command at 16 MiB through zfs_trim_extent_bytes_max, set at boot with the other ZFS module parameters and checked by zfsmanager. A trim interrupted by a reboot resumes on its own, so the cron start covers the whole pool over time. A resumed trim keeps the rate it was started with, so at startup zfsmanager suspends any active trim and restarts it at the limit; this throttles a trim started by an older EVE, keeps its progress, and resets the start time zpool status reports to the boot time. A cron tick that finds a trim running is logged as a notice and leaves the published trim start time unchanged. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At the fixed 1.5 MiB/s a trim pass over a 4 TB disk takes about a month. The new storage.zfs.pool.trim.rate.bytespersec config item sets the rate, in bytes/s per disk as zpool trim -r applies it, with 1.5 MiB/s as the default. A change takes effect on the next scheduled trim and, within a minute, on a trim already in progress, which is suspended and restarted at the new rate without losing its progress. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a9c2199 to
a547fa4
Compare
andrewd-zededa
left a comment
There was a problem hiding this comment.
LGTM, my concerns have been addressed.
@rene fixed |
Description
On EVE-k, zfsmanager runs an unthrottled
zpool trimof the persist pool atevery boot. On a ~1 TB SATA disk that pass takes many hours, and on a device
that reboots often it restarts against all free space on every boot. Under that
load Longhorn replicas hit their 16 s R/W timeout and app volume creation
stalls in
CREATING_VOLUME.Even without frequent reboots of the device there is a performance impact when the periodic trim runs.
This PR drops the boot-time trim, runs the scheduled trim as
zpool trim -r 1572864 persist(1.5 MiB/s), and caps each TRIM command at16 MiB through
zfs_trim_extent_bytes_max. A trim interrupted by a rebootresumes on its own, so the existing
timer.zfs.pool.trim.cronstart covers thewhole pool over time (about a week per TB of free space).
A resumed trim keeps the rate it was started with, so a trim started by an
older EVE would stay unthrottled. At startup zfsmanager suspends any active
trim and restarts it at the limit; progress is kept, but the start time
reported by
zpool status -tbecomes the boot time. A cron tick that finds atrim running is logged as a notice instead of an error.
docs/ZFS.mdisupdated to match.
Partially addresses #6650.
How to test and validate this PR
On an EVE-k node with a ZFS persist pool:
zpool status -t persistshows no new trim started at boot, andcat /sys/module/zfs/parameters/zfs_trim_extent_bytes_maxprints16777216.stays near 1.5 MiB/s (e.g.
/proc/diskstatsdiscard sectors).resumed trim of persist at 1572864 bytes/s,zpool status -t persistshows the trim continuing from itsprevious percentage with the boot time as start time, and discards resume
at ~1.5 MiB/s.
running: after boot the discard rate drops to ~1.5 MiB/s.
Latency results under the ztest soak:
#6841 (comment)
Changelog notes
On EVE-k nodes the ZFS pool trim no longer runs at every boot. The scheduled
trim is rate-limited so it does not slow down application storage.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
🤖 Generated with Claude Code
https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj