Skip to content

kube: expire events after 1h, drop the 72h backlog - #6934

Merged
rene merged 2 commits into
lf-edge:masterfrom
eriknordmark:kube-event-ttl
Oct 30, 2026
Merged

rene merged 2 commits into
lf-edge:masterfrom
eriknordmark:kube-event-ttl

Conversation

@eriknordmark

Copy link
Copy Markdown
Contributor

Description

On a single-node EVE-k device, k3s stores its state in sqlite through kine,
which re-arms each key's expiry at every k3s start from the lease stored in its
row. With event-ttl=72h, a device that reboots more often than every three
days never expires an event. On a lab device that reboots several times a day,
events made up 297835 of 299035 rows and most of a 644 MB datastore; event
lists took minutes, and the slow kine queries came with apiserver timeouts and
lost CDI leader elections that failed app deployments.

  • Set event-ttl=1h, below a typical uptime, so events expire within an hour
    of each k3s start.
  • Before starting k3s, kube-init deletes the events that still carry the 72h
    lease and vacuums the datastore. Without this an upgraded device keeps its
    whole backlog, since lowering the TTL does not touch existing rows. Events
    from a user override with a different TTL are left alone.

@andrewd-zededa, the 72h TTL came from b871d8d for debugging. Is 1h enough
for what you needed it for?

How to test and validate this PR

  1. On an EVE-k device that has run the old image for a few days, check the
    backlog: in the kube container,
    sqlite3 -readonly /var/lib/rancher/k3s/server/db/state.db "select lease, count(*) from kine where name like '/registry/events/%' group by 1"
    shows a large count with lease 259260.
  2. Upgrade to this image. /persist/kubelog/k3s-install.log shows
    prune legacy events: removed N events, and the query above shows only
    lease 3660 rows. ls -l on the datastore shows it shrunk.
  3. Over the following hours the event count stays bounded, roughly an hour's
    worth of events.

Unit tests cover the prune: no datastore, nothing to delete, and delete
followed by vacuum.

The 1h TTL was tested on sc-sm-e300-10 (EVE-k, amd64) in a soak image that
also carried #6841: 78 ztest runs over about 26 h, with test_app_multidisk
and test_oneapp, which had failed on the bloated datastore, passing every
run. The legacy-event deletion was done by hand there with the same statement
(297835 rows in 42 s, 644 MB to 81 MB, integrity check ok); the kube-init code
path itself has not run on a device yet.

Changelog notes

EVE-k: Kubernetes events now expire after one hour, and events left from the
previous 72-hour setting are removed on upgrade, which keeps the k3s datastore
small on devices that reboot often.

PR Backports

  • 17.0-stable: To be decided. It has event-ttl=72h but no kube-init, so the
    cleanup would need a cluster-init.sh variant.
  • 16.0-stable: To be decided, as for 17.0-stable.
  • 14.5-stable: To be decided, as for 17.0-stable.
  • 13.4-stable: No, it does not set event-ttl.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

No documentation change: the TTL is internal k3s configuration. Not tested on
arm64; nothing here is architecture-specific. The amd64 test covers the TTL and
the hand-run deletion, not the kube-init prune itself (see above).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj

eriknordmark and others added 2 commits October 28, 2026 15:19
On a single-node EVE-k device, k3s stores its state in sqlite through
kine. kine keeps lease expiry timers in memory and, at every k3s
start, sets each key's expiry to the start time plus its lease. With
event-ttl at 72h, a device that reboots more often than every three
days never expires an event. On sc-sm-e300-10, which reboots several
times a day, 26 days of events accumulated: 295600 of 298249 rows and
236 MiB of the 644 MB datastore. kine lists scan the whole name index,
so every list and every compaction gets slower as the table grows, and
listing the events no longer finishes within two minutes.

A 1h TTL is shorter than a typical uptime, so events expire within an
hour of each k3s start, including those carried over from before a
reboot.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
Lowering event-ttl only affects events written afterwards: kine stores
each event's lease in its row and re-arms expiry from that lease at
every k3s start. On a device that reboots more often than every three
days, events written under the former 72h TTL therefore never expire,
and an upgraded device keeps its whole backlog.

Before starting k3s, kube-init now deletes the events that carry the
72h lease and vacuums the datastore if any were removed. Events from a
user override with a different TTL are left alone. On sc-sm-e300-10 the
same deletion, done by hand, removed 297835 events in 42 s and shrank
the datastore from 644 MB to 81 MB.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

evetest: skipped suites for this PR

This PR only touches pkg/kube (EVE-K's kube-init/k3s event-ttl config and legacy-event pruning on upgrade), so I'm skipping suites/tests that are either unrelated to EVE-K entirely or that redundantly re-exercise the same incidental "boot EVE-K with an empty kine datastore" path.

Unrelated (different subsystem, or default to KVM and never touch the changed kube-init/k3s code):

  • TestApplicationConnectivitySuite
  • TestAppsSuite
  • TestBootstrapSuite
  • TestControllerFaultsSuite
  • TestDatastoreSuite
  • TestDeviceConnectivitySuite
  • TestDiagSuite
  • TestHardwareSuite
  • TestKonvertSuite (storage repartition/geometry/volume-migration, not events/k3s config)
  • TestLPSSuite
  • TestPcibackErrorSuite
  • TestPortScanSuite (open-port scanning, unrelated to kine/event pruning)
  • TestSecuritySuite
  • TestStorageSuite
  • TestTelemetrySuite
  • TestEVEUpgradeKVMtoKVM, TestEVEUpgradeKVMtoKVMWithOCIDatastore (KVM-hypervisor upgrade variants)

Redundant with a kept EVE-K test (same incidental fresh-boot / upgrade-prune path, no distinct coverage for this change):

  • TestKubeRegistrationSuite, TestNodeClusterResilienceSuite — each subtest just forms a fresh EVE-K cluster, same no-op pruning path as TestSingleNodeCluster (kept)
  • TestNativeKubeAppNetworking, TestAppInstancePurge, TestThreeNodesCluster, TestThreeNodesPortScan, TestMgmtProxy, TestTieBreakerCluster, TestClusterToSingleConversion, TestClusterVMApp — the rest of TestNodeClusterSuite, redundant with TestSingleNodeCluster
  • TestThreeNodesUpgrade — redundant with TestEVEUpgradeKubevirtToKubevirt (kept), which is the most direct test of the new prune-on-upgrade behavior (upgrades from a pre-this-PR EVE-K image that still used the 72h event-ttl)

Kept for direct/representative coverage: TestSingleNodeCluster (fresh EVE-K boot sanity) and TestEVEUpgradeKubevirtToKubevirt (exercises the actual legacy-event prune-on-upgrade path).

An engineer can comment /evetest all to force the full suite to run instead, or /evetest <name> <name> ... to run only specific tests/suites.

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.77%. Comparing base (02ad7ce) to head (a7f156c).
⚠️ Report is 47 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6934      +/-   ##
==========================================
+ Coverage   28.48%   28.77%   +0.28%     
==========================================
  Files         540      551      +11     
  Lines       98258   100005    +1747     
==========================================
+ Hits        27990    28772     +782     
- Misses      67998    68771     +773     
- Partials     2270     2462     +192     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene
rene merged commit 6babf29 into lf-edge:master Oct 30, 2026
76 of 81 checks passed
@eriknordmark
eriknordmark deleted the kube-event-ttl branch November 6, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants