fix(server): clear lease-loss tombstones for awaiting/cancelled sessions - #1344
Open
jhrozek wants to merge 2 commits into
Open
fix(server): clear lease-loss tombstones for awaiting/cancelled sessions#1344jhrozek wants to merge 2 commits into
jhrozek wants to merge 2 commits into
Conversation
…ons (fixes #1334) onLeaseLost drives a session OUT of StateRunning while handling a declared lease loss (to awaiting via preserveAwaiting, or eventually cancelled), so the StateRunning-only stale-session sweep (SessionStale/SettleIfStale) could never rediscover it and the lostOwnership tombstone - a permanent fail-fast by design for every ordinary caller - stayed wedged short of CloseSession or a process restart. Add Service.LostOwnershipCandidates (an in-memory read of this process's own lease-loss tombstones, not a store-wide scan) and Service.ReconcileLeaseLossTombstone (a bounded trial-Acquire+immediate-release against the real backend, mirroring SessionStale's own refinement, that clears the tombstone plus any stale invalid heldLeases bookkeeping once the lease is proven genuinely free - never unconditionally). Wire both into the existing composition-level stale-session sweep so a stranded tombstone self-heals on the next pass instead of needing a manual CloseSession or restart. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 1ccb837, addressing a panel review of that commit: - Extract the ~30-line trial-Acquire -> switch -> Release block SessionStale and ReconcileLeaseLossTombstone each duplicated into one shared Service.leaseTrial helper. held is non-nil only for a genuine ErrLeaseHeld; each caller keeps its own distinct ownership judgement (SessionStale's self-held-lease correction; ReconcileLeaseLossTombstone's leave-the-tombstone-in-place). - Add internal/app/session_reconcile_test.go's TestSweepStaleSessionsClearsLeaseLossTombstone: drives a real lease loss through a live run, then proves sweepStaleSessions itself (not a direct ReconcileLeaseLossTombstone call) clears the tombstone via reconcileLeaseLossTombstones - closing the composition-wiring coverage gap a regression dropping that one call would have slipped through. - Add TestReconcileLeaseLossTombstoneFailSafeOnGenericError: a bare non-sentinel Acquire error must leave the tombstone in place, never clear it on ambiguity. - Log a WARN when the trial's own Release fails (a leaked trial otherwise silently pins the lease until TTL with no diagnostic trail). - Update docs/adr/0027-cloud-native.md row 27, internal/syscaller's RootStaleSessionReconcile doc comment, and classification.go's rationale for that root to describe BOTH tombstone-clearing exceptions (SettleIfStale for StateRunning, ReconcileLeaseLossTombstone for awaiting/cancelled) instead of the now-stale "one narrow exception" text. - Note the bounded CloseSession/ReconcileLeaseLossTombstone interleave in the latter's doc comment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
Service.onLeaseLostdeclares a cross-process session-lease loss, it setsa permanent
lostOwnership[id]tombstone. Every normal caller ofacquireLease/reaffirmLease(StartRunContent,resumeFromAwaiting/ApproveRun,RenameSession,DeleteSession, etc.) fails fast on thattombstone forever, by design — the only two things that cleared it were
CloseSession(not something a normal client resuming a session calls) andthe
StateRunning-only stale-session reconcile sweep (#1302). ButonLeaseLostitself drives the session out ofStateRunningwhilehandling the loss (to
awaitingor eventuallycancelled), so the sweepnever rediscovers it and the tombstone stayed permanent — regardless of
whether the original loss was a genuine takeover or a false positive (see the
companion fix in #1333).
Adds
Service.LostOwnershipCandidates(an in-memory read of this process'sown lease-loss tombstones — deliberately not a store-wide scan, to avoid an
unbounded fan-out of trial-
Acquirecalls against every ordinaryawaiting/cancelled session) and
Service.ReconcileLeaseLossTombstone(abounded trial-
Acquire+immediate-release against the real backend, mirroringSessionStale's own refinement, that clears the tombstone — and any staleinvalid
heldLeasesbookkeeping — only on proof the lease is genuinely free,never unconditionally). Both are wired into the existing composition-level
stale-session sweep ticker, so a stranded tombstone self-heals on the next
pass instead of needing a manual
CloseSessionor process restart.Reviewed by a go-architect + kubernetes-operator-expert panel before
implementation, and by a full four-axis panel review after — no blockers
found; four important findings (a stale ADR line, two test-coverage gaps, and
a ~20-line duplicated trial-Acquire block versus
SessionStale) are alladdressed in the second commit, which also extracts the shared
leaseTrialhelper both call sites now use.
Development stage
Contract linkage
Service-level additions consumed only by the existing composition-level sweep (no new goroutine, no new gRPC/HTTP surface — confirmed by the security review below); gated by the samestaleReconcileAuthorizedsystem-root authorization every sibling stale-reconcile method already uses.Interface conformance
Servicemethods (LostOwnershipCandidates,ReconcileLeaseLossTombstone), both gated by the existingstaleReconcileAuthorizedsystem-root check — not reachable from any gRPC/HTTP-authenticated caller (verified: no new handler registers them; the only caller is the composition-level sweep). Registered inclassification.goand the SDK-parity test alongside their siblings.Issue relationship
Fixes #1334
Type of change
Test plan
Baseline checks
golangci-lint run --config .golangci.yml ./internal/adapter/server/... ./internal/app/...)go test ./internal/adapter/server/... ./internal/app/... -race -count=1)internal/, outside the engine module/panel-review-equivalent four-axis panel run manually —PANEL: ship_blockers=0 important=4 advisory=4 reviewer_failures=0; all four important findings fixed in the second commitFull-repo
task lint/task testintentionally not run in this environment (disk space constrained during development); scoped checks above cover every touched package plus a fullgo vet ./...sanity pass on the root module. CI will run the full gates.Changes
internal/adapter/server/service.goLostOwnershipCandidates,ReconcileLeaseLossTombstone, extracted sharedleaseTrialhelperinternal/adapter/server/classification.gointernal/adapter/server/sdk_typescript_release_test.gointernal/adapter/server/lease_test.gointernal/app/session_reconcile.gointernal/app/session_reconcile_test.goServicemethod) clears a tombstonedocs/adr/0027-cloud-native.mdinternal/syscaller/syscaller.goUser-facing change
A session whose cross-process lease was lost while parked awaiting a
permission approval, or after being cancelled, is no longer permanently stuck
read-only. It self-heals on the next stale-session sweep pass once the
underlying lease is verifiably free — no manual intervention or process
restart required.
Special notes for reviewers
The safety-critical property here is that the tombstone is never
cleared unconditionally — only after a real trial-
Acquireproves the leasefree, with a fail-safe error path that leaves the tombstone in place. This is
covered by
TestReconcileLeaseLossTombstoneRefusesWhenGenuinelyHeldElsewhereand the new
TestReconcileLeaseLossTombstoneFailSafeOnGenericError. Asecure-code-reviewer pass specifically traced the TOCTOU/split-brain question
and confirmed no double-holder scenario is reachable. See #1333 for the
companion fix (preventing a false-positive loss in the first place on the
single-host flock backend).