Skip to content

mecak8s: make Kubernetes Lease renewal capacity explicit and scalable #1515

Description

@ChrisJBurns

Problem

The Kubernetes client uses client-go's default 5 QPS / 10 burst limits. Each held session Lease renewal performs a GET followed by an UPDATE, normally every ten seconds. Roughly 25 concurrently held sessions consume the entire steady-state client budget before acquisitions, releases, scheduler election, retries, or other Kubernetes work.

Throttled renewals can miss the 30-second Lease TTL, cancel live work, and trigger avoidable lease-loss recovery.

Evidence

  • internal/app/build.go:newK8sClientset does not set rest.Config.QPS or Burst.
  • internal/adapter/k8slease/k8slease.go:Renew performs GET + UPDATE.
  • cmd/mecak8s/flags.go defaults the Lease TTL to 30 seconds and renewal interval to TTL / 3.

Proposed direction

Define and document a capacity model. Make QPS/Burst explicit and configurable or derive safe values from supported concurrency. Consider reducing renewal round trips or moving high-cardinality session fencing away from the Kubernetes API.

Acceptance criteria

  • Supported concurrent-session capacity is documented.
  • Client throttling settings are explicit and observable.
  • A load test exercises acquisitions and renewals at the supported limit.
  • Sustained supported load does not cause renewal loss or live-run cancellation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions