Skip to content

fix(server): clear lease-loss tombstones for awaiting/cancelled sessions - #1344

Open
jhrozek wants to merge 2 commits into
mainfrom
worktree-agent-af54f2ebd711919a9
Open

fix(server): clear lease-loss tombstones for awaiting/cancelled sessions#1344
jhrozek wants to merge 2 commits into
mainfrom
worktree-agent-af54f2ebd711919a9

Conversation

@jhrozek

@jhrozek jhrozek commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

When Service.onLeaseLost declares a cross-process session-lease loss, it sets
a permanent lostOwnership[id] tombstone. Every normal caller of
acquireLease/reaffirmLease (StartRunContent, resumeFromAwaiting/
ApproveRun, RenameSession, DeleteSession, etc.) fails fast on that
tombstone forever, by design — the only two things that cleared it were
CloseSession (not something a normal client resuming a session calls) and
the StateRunning-only stale-session reconcile sweep (#1302). But
onLeaseLost itself drives the session out of StateRunning while
handling the loss (to awaiting or eventually cancelled), so the sweep
never rediscovers it and the tombstone stayed permanent — regardless of
whether the original loss was a genuine takeover or a false positive (see the
companion fix in #1333).

Adds Service.LostOwnershipCandidates (an in-memory read of this process's
own lease-loss tombstones — deliberately not a store-wide scan, to avoid an
unbounded fan-out of trial-Acquire calls against every ordinary
awaiting/cancelled session) and Service.ReconcileLeaseLossTombstone (a
bounded trial-Acquire+immediate-release against the real backend, mirroring
SessionStale's own refinement, that clears the tombstone — and any stale
invalid heldLeases bookkeeping — only on proof the lease is genuinely free,
never unconditionally). Both are wired into the existing composition-level
stale-session sweep ticker, so a stranded tombstone self-heals on the next
pass instead of needing a manual CloseSession or process restart.

Reviewed by a go-architect + kubernetes-operator-expert panel before
implementation, and by a full four-axis panel review after — no blockers
found; four important findings (a stale ADR line, two test-coverage gaps, and
a ~20-line duplicated trial-Acquire block versus SessionStale) are all
addressed in the second commit, which also extracts the shared leaseTrial
helper both call sites now use.

Development stage

  • Spike / Routine — acceptance-plan spine exempt; Spike evidence does not ship as-is

Contract linkage

  • Work classification: Routine — mechanical, reversible bug fix closing a self-heal gap in an existing mechanism (the stale-session sweep, fix(server): let the stale-session sweep re-acquire a lost lease #1302); introduces no new durable architecture decision, no public API/persistence/trust-boundary change.
  • Classification rationale: New methods are internal Service-level additions consumed only by the existing composition-level sweep (no new goroutine, no new gRPC/HTTP surface — confirmed by the security review below); gated by the same staleReconcileAuthorized system-root authorization every sibling stale-reconcile method already uses.
  • Decision record: None — bug fix, not a new architectural decision.
  • Human waiver of spine: No
  • Acceptance plan: N/A (Routine)
  • Human decisions resolved and recorded: N/A
  • Plan / Interface PR: N/A
  • Approved commit baseline: N/A
  • Combined/exemption rationale: N/A (Routine, not Combined)

Interface conformance

  • Two new exported Service methods (LostOwnershipCandidates, ReconcileLeaseLossTombstone), both gated by the existing staleReconcileAuthorized system-root check — not reachable from any gRPC/HTTP-authenticated caller (verified: no new handler registers them; the only caller is the composition-level sweep). Registered in classification.go and the SDK-parity test alongside their siblings.

Issue relationship

Fixes #1334

Type of change

  • Bug fix

Test plan

Baseline checks

  • Linting (scoped: golangci-lint run --config .golangci.yml ./internal/adapter/server/... ./internal/app/...)
  • Offline test suite (scoped: go test ./internal/adapter/server/... ./internal/app/... -race -count=1)
  • Guarded engine API affected: not applicable — change is entirely in internal/, outside the engine module
  • Final implementation review: /panel-review-equivalent four-axis panel run manually — PANEL: ship_blockers=0 important=4 advisory=4 reviewer_failures=0; all four important findings fixed in the second commit

Full-repo task lint/task test intentionally not run in this environment (disk space constrained during development); scoped checks above cover every touched package plus a full go vet ./... sanity pass on the root module. CI will run the full gates.

Changes

File Change
internal/adapter/server/service.go LostOwnershipCandidates, ReconcileLeaseLossTombstone, extracted shared leaseTrial helper
internal/adapter/server/classification.go Register the two new methods under the existing stale-reconcile root
internal/adapter/server/sdk_typescript_release_test.go SDK-parity table entries
internal/adapter/server/lease_test.go New tests: recovers-awaiting, recovers-cancelled, refuses-when-genuinely-held-elsewhere, fail-safe-on-generic-error
internal/app/session_reconcile.go Wire the new reconcile pass into the existing sweep ticker
internal/app/session_reconcile_test.go New test proving the sweep wiring itself (not just the Service method) clears a tombstone
docs/adr/0027-cloud-native.md Correct the now-stale "one narrow exception" tombstone-clearing text
internal/syscaller/syscaller.go Extend the stale-reconcile root's authorized-scope rationale

User-facing change

A session whose cross-process lease was lost while parked awaiting a
permission approval, or after being cancelled, is no longer permanently stuck
read-only. It self-heals on the next stale-session sweep pass once the
underlying lease is verifiably free — no manual intervention or process
restart required.

Special notes for reviewers

The safety-critical property here is that the tombstone is never
cleared unconditionally — only after a real trial-Acquire proves the lease
free, with a fail-safe error path that leaves the tombstone in place. This is
covered by TestReconcileLeaseLossTombstoneRefusesWhenGenuinelyHeldElsewhere
and the new TestReconcileLeaseLossTombstoneFailSafeOnGenericError. A
secure-code-reviewer pass specifically traced the TOCTOU/split-brain question
and confirmed no double-holder scenario is reachable. See #1333 for the
companion fix (preventing a false-positive loss in the first place on the
single-host flock backend).

jhrozek and others added 2 commits September 10, 2026 13:40
…ons (fixes #1334)

onLeaseLost drives a session OUT of StateRunning while handling a declared
lease loss (to awaiting via preserveAwaiting, or eventually cancelled), so the
StateRunning-only stale-session sweep (SessionStale/SettleIfStale) could never
rediscover it and the lostOwnership tombstone - a permanent fail-fast by
design for every ordinary caller - stayed wedged short of CloseSession or a
process restart.

Add Service.LostOwnershipCandidates (an in-memory read of this process's own
lease-loss tombstones, not a store-wide scan) and
Service.ReconcileLeaseLossTombstone (a bounded trial-Acquire+immediate-release
against the real backend, mirroring SessionStale's own refinement, that clears
the tombstone plus any stale invalid heldLeases bookkeeping once the lease is
proven genuinely free - never unconditionally). Wire both into the existing
composition-level stale-session sweep so a stranded tombstone self-heals on
the next pass instead of needing a manual CloseSession or restart.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 1ccb837, addressing a panel review of that commit:

- Extract the ~30-line trial-Acquire -> switch -> Release block SessionStale
  and ReconcileLeaseLossTombstone each duplicated into one shared
  Service.leaseTrial helper. held is non-nil only for a genuine
  ErrLeaseHeld; each caller keeps its own distinct ownership judgement
  (SessionStale's self-held-lease correction; ReconcileLeaseLossTombstone's
  leave-the-tombstone-in-place).
- Add internal/app/session_reconcile_test.go's
  TestSweepStaleSessionsClearsLeaseLossTombstone: drives a real lease loss
  through a live run, then proves sweepStaleSessions itself (not a direct
  ReconcileLeaseLossTombstone call) clears the tombstone via
  reconcileLeaseLossTombstones - closing the composition-wiring coverage
  gap a regression dropping that one call would have slipped through.
- Add TestReconcileLeaseLossTombstoneFailSafeOnGenericError: a bare
  non-sentinel Acquire error must leave the tombstone in place, never clear
  it on ambiguity.
- Log a WARN when the trial's own Release fails (a leaked trial otherwise
  silently pins the lease until TTL with no diagnostic trail).
- Update docs/adr/0027-cloud-native.md row 27, internal/syscaller's
  RootStaleSessionReconcile doc comment, and classification.go's rationale
  for that root to describe BOTH tombstone-clearing exceptions
  (SettleIfStale for StateRunning, ReconcileLeaseLossTombstone for
  awaiting/cancelled) instead of the now-stale "one narrow exception" text.
- Note the bounded CloseSession/ReconcileLeaseLossTombstone interleave in
  the latter's doc comment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lease-loss tombstone (lostOwnership) is permanent for awaiting/cancelled sessions; stale-sweep only covers StateRunning

1 participant