Repository navigation
kube: expire events after 1h, drop the 72h backlog - #6934
Conversation
On a single-node EVE-k device, k3s stores its state in sqlite through kine. kine keeps lease expiry timers in memory and, at every k3s start, sets each key's expiry to the start time plus its lease. With event-ttl at 72h, a device that reboots more often than every three days never expires an event. On sc-sm-e300-10, which reboots several times a day, 26 days of events accumulated: 295600 of 298249 rows and 236 MiB of the 644 MB datastore. kine lists scan the whole name index, so every list and every compaction gets slower as the table grows, and listing the events no longer finishes within two minutes. A 1h TTL is shorter than a typical uptime, so events expire within an hour of each k3s start, including those carried over from before a reboot. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
Lowering event-ttl only affects events written afterwards: kine stores each event's lease in its row and re-arms expiry from that lease at every k3s start. On a device that reboots more often than every three days, events written under the former 72h TTL therefore never expire, and an upgraded device keeps its whole backlog. Before starting k3s, kube-init now deletes the events that carry the 72h lease and vacuums the datastore if any were removed. Events from a user override with a different TTL are left alone. On sc-sm-e300-10 the same deletion, done by hand, removed 297835 events in 42 s and shrank the datastore from 644 MB to 81 MB. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
c4b164e to
a7f156c
Compare
evetest: skipped suites for this PRThis PR only touches Unrelated (different subsystem, or default to KVM and never touch the changed kube-init/k3s code):
Redundant with a kept EVE-K test (same incidental fresh-boot / upgrade-prune path, no distinct coverage for this change):
Kept for direct/representative coverage: An engineer can comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6934 +/- ##
==========================================
+ Coverage 28.48% 28.77% +0.28%
==========================================
Files 540 551 +11
Lines 98258 100005 +1747
==========================================
+ Hits 27990 28772 +782
- Misses 67998 68771 +773
- Partials 2270 2462 +192 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Description
On a single-node EVE-k device, k3s stores its state in sqlite through kine,
which re-arms each key's expiry at every k3s start from the lease stored in its
row. With
event-ttl=72h, a device that reboots more often than every threedays never expires an event. On a lab device that reboots several times a day,
events made up 297835 of 299035 rows and most of a 644 MB datastore; event
lists took minutes, and the slow kine queries came with apiserver timeouts and
lost CDI leader elections that failed app deployments.
event-ttl=1h, below a typical uptime, so events expire within an hourof each k3s start.
lease and vacuums the datastore. Without this an upgraded device keeps its
whole backlog, since lowering the TTL does not touch existing rows. Events
from a user override with a different TTL are left alone.
@andrewd-zededa, the 72h TTL came from b871d8d for debugging. Is 1h enough
for what you needed it for?
How to test and validate this PR
backlog: in the kube container,
sqlite3 -readonly /var/lib/rancher/k3s/server/db/state.db "select lease, count(*) from kine where name like '/registry/events/%' group by 1"shows a large count with lease 259260.
/persist/kubelog/k3s-install.logshowsprune legacy events: removed N events, and the query above shows onlylease 3660 rows.
ls -lon the datastore shows it shrunk.worth of events.
Unit tests cover the prune: no datastore, nothing to delete, and delete
followed by vacuum.
The 1h TTL was tested on sc-sm-e300-10 (EVE-k, amd64) in a soak image that
also carried #6841: 78 ztest runs over about 26 h, with
test_app_multidiskand
test_oneapp, which had failed on the bloated datastore, passing everyrun. The legacy-event deletion was done by hand there with the same statement
(297835 rows in 42 s, 644 MB to 81 MB, integrity check ok); the kube-init code
path itself has not run on a device yet.
Changelog notes
EVE-k: Kubernetes events now expire after one hour, and events left from the
previous 72-hour setting are removed on upgrade, which keeps the k3s datastore
small on devices that reboot often.
PR Backports
event-ttl=72hbut no kube-init, so thecleanup would need a cluster-init.sh variant.
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
No documentation change: the TTL is internal k3s configuration. Not tested on
arm64; nothing here is architecture-specific. The amd64 test covers the TTL and
the hand-run deletion, not the kube-init prune itself (see above).
🤖 Generated with Claude Code
https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj