fix(server): clear lease-loss tombstones for awaiting/cancelled sessions

docs/adr/0027-cloud-native.md

@@ -826,7 +826,7 @@ durable artifact survives and is reloaded), or **lost** (gone, possibly leaking)
 | 24 | `modelRouterBreaker` (per-run model-router circuit breaker, ADR 0031; ADR 0034 reuses the SAME per-run breaker for team members + Parallel branches — NO new breaker) | `agent.Run` | run | dies with the run | lost (run-scoped by design, mirroring `askReviewBreaker` row 22) | `engine/agent/modelrouter.go` (`modelRouterBreaker`); armed in `engine/agent/loop.go` (`startRun`) |
 | 25 | `gitWorktreeLister` (osfs-backed worktree discovery, issue #102) | `app.Build` | process lifetime | none (value type, no goroutine, no Close needed) | reconstructible (rebuilt from `cfg.Shell` at next Build; no state) | `internal/app/build.go` (`buildWorktreeLister`) |
 | 26 | routed team-member / Parallel-branch child engine (ADR 0034) | `buildMemberEngine` (member) / `buildParallelEngineFactory` (branch), minted in composition | per-AddMember (member; reused across rounds, torn down on member teardown) / per-call (branch; torn down with the branch fork) | dies with the member/branch (the SAME lifecycle as the non-routed member/branch engine it replaces — routing changes only the model, not the lifetime) | reconstructible (a new team/Parallel call re-classifies + re-mints); decision = derive (nothing persisted; the routed model is List-2 row 18) | `internal/app/build.go` (`buildMemberEngine`, `buildParallelEngineFactory`) |
-| 27 | held session leases + per-session renewer goroutines (`Service.heldLeases`, Phase 4) + local retained generation-liveness flock fds + local mutation-capability/lost-owner maps (`SessionMutationCapability`, `Service.lostOwnership`, ADR 0294) | `server.Service` owns the lifecycle and shares the capability with its guarded SessionStore/EventLog/ToolCallRecorder projections; `flocklease.Lease` owns each local generation fd | session (one lease + renewer, retained generation fd, and capability state per leased session; one lightweight lost-owner entry until local teardown after definitive renewal loss) | acquire grants the capability. Close and joined graceful drain cancel renewal, invalidate locally, then release with a cancel-detached bounded context. Lease loss or drain timeout invalidates before cancelling the run; a definitive non-awaiting loss releases the exact generation, while awaiting loss retains the invalid hold without Release so takeover preserves the durable handoff point. Capability and held-lease tombstones are removed when stale lifecycle references settle, while `lostOwnership` prevents that stale Service from reacquiring until explicit `CloseSession` teardown — the one narrow exception being the stale-session reconcile sweep's `SettleIfStale`, whose caller is independently pre-verified (age-horizon + local liveness) as recovering a genuine crash orphan rather than a live handoff, and which clears `lostOwnership` itself on a successful re-Acquire (`acquireLeaseCore`'s `bypassTombstone` path, used only via `acquireMutationLeaseForStaleSettle`). Awaiting loss also retracts the local ask while preserving the durable snapshot. No mutation is admitted after invalidation starts, while an already-admitted backend call may still finish because this is local invalidation rather than token-bearing storage fencing. Local exact-token Release durably tombstones then closes/unlocks its generation fd; stale release cannot touch a successor, the stable per-session sentinel is operation-scoped, and failure paths close newly opened handles or retain failed-close references for retry. SIGKILL makes the kernel close every retained fd | **reconstructible/reset-by-design** (a restart starts with empty capability/lost-owner maps and re-acquires on the next admitted operation; record expiry preserves the generic TTL takeover contract while a crashed local holder's generation flock releases immediately so a survivor can take over before expiry and increment the token preserved in the durable record. No local validity/tombstone is persisted, and it must not survive process identity. The durable awaiting `PendingAsk` and session state remain in the snapshot; see List 2 row 42). Constructed when a lease backend is explicitly wired or store-provided, and automatically as the existing flock lease beneath every local JSONL StoreDir. Other no-lease shareable stores do not gain destructive-maintenance authority; automatic retention fails closed | `internal/adapter/server/service.go` (`heldLeases`, `lostOwnership`, `acquireLease`, `acquireLeaseCore`, `acquireMutationLeaseForStaleSettle`, `renewLoop`, `onLeaseLost`, `GracefulDrain`, `releaseLease`); `internal/adapter/server/mutation_capability.go` (`SessionMutationCapability`); `internal/adapter/flocklease/flocklease.go` (`Lease`, `heldLease`, `Acquire`, `Release`); wired at `internal/app/build.go` (`buildSessionLease`) |
+| 27 | held session leases + per-session renewer goroutines (`Service.heldLeases`, Phase 4) + local retained generation-liveness flock fds + local mutation-capability/lost-owner maps (`SessionMutationCapability`, `Service.lostOwnership`, ADR 0294) | `server.Service` owns the lifecycle and shares the capability with its guarded SessionStore/EventLog/ToolCallRecorder projections; `flocklease.Lease` owns each local generation fd | session (one lease + renewer, retained generation fd, and capability state per leased session; one lightweight lost-owner entry until local teardown after definitive renewal loss) | acquire grants the capability. Close and joined graceful drain cancel renewal, invalidate locally, then release with a cancel-detached bounded context. Lease loss or drain timeout invalidates before cancelling the run; a definitive non-awaiting loss releases the exact generation, while awaiting loss retains the invalid hold without Release so takeover preserves the durable handoff point. Capability and held-lease tombstones are removed when stale lifecycle references settle, while `lostOwnership` prevents that stale Service from reacquiring until explicit `CloseSession` teardown — with TWO independent, narrowly-scoped exceptions, both root-authorized (`stale-session-reconcile`) and both re-verifying under `s.mu` before writing: (1) the stale-session reconcile sweep's `SettleIfStale`, whose caller is independently pre-verified (age-horizon + local liveness) as recovering a genuine crash orphan rather than a live handoff, and which clears `lostOwnership` itself on a successful re-Acquire (`acquireLeaseCore`'s `bypassTombstone` path, used only via `acquireMutationLeaseForStaleSettle`) — this covers ONLY a `StateRunning` candidate; and (2) `ReconcileLeaseLossTombstone` (issue #1334), the awaiting/cancelled counterpart, for the sessions `onLeaseLost` itself drives OUT of `StateRunning` while handling the very loss that set the tombstone (to `awaiting` via the `preserveAwaiting` branch, or eventually `cancelled`) — a population `SettleIfStale`'s `StateRunning`-only candidacy can never rediscover. It performs a bounded TRIAL Acquire+immediate-Release (the shared `leaseTrial` helper, also used by `SessionStale`'s own refinement) against exactly the ids `Service.lostOwnership` already names (an in-memory read, never a store-wide scan — that population is the steady state for huge numbers of ordinary finished sessions), clears the tombstone plus any stale invalid `heldLeases` entry `onLeaseLost`'s `preserveAwaiting` branch left behind, and never holds a lease or repairs session state itself: the next genuine run-entry's existing `loadAndReopen`/`resumeFromAwaiting` still does that. Wired into the SAME composition-level sweep pass as `SettleIfStale` (`internal/app/session_reconcile.go`'s `reconcileLeaseLossTombstones`), no new goroutine. Awaiting loss also retracts the local ask while preserving the durable snapshot. No mutation is admitted after invalidation starts, while an already-admitted backend call may still finish because this is local invalidation rather than token-bearing storage fencing. Local exact-token Release durably tombstones then closes/unlocks its generation fd; stale release cannot touch a successor, the stable per-session sentinel is operation-scoped, and failure paths close newly opened handles or retain failed-close references for retry. SIGKILL makes the kernel close every retained fd | **reconstructible/reset-by-design** (a restart starts with empty capability/lost-owner maps and re-acquires on the next admitted operation; record expiry preserves the generic TTL takeover contract while a crashed local holder's generation flock releases immediately so a survivor can take over before expiry and increment the token preserved in the durable record. No local validity/tombstone is persisted, and it must not survive process identity. The durable awaiting `PendingAsk` and session state remain in the snapshot; see List 2 row 42). Constructed when a lease backend is explicitly wired or store-provided, and automatically as the existing flock lease beneath every local JSONL StoreDir. Other no-lease shareable stores do not gain destructive-maintenance authority; automatic retention fails closed | `internal/adapter/server/service.go` (`heldLeases`, `lostOwnership`, `acquireLease`, `acquireLeaseCore`, `acquireMutationLeaseForStaleSettle`, `leaseTrial`, `SessionStale`, `ReconcileLeaseLossTombstone`, `LostOwnershipCandidates`, `renewLoop`, `onLeaseLost`, `GracefulDrain`, `releaseLease`); `internal/app/session_reconcile.go` (`reconcileLeaseLossTombstones`); `internal/adapter/server/mutation_capability.go` (`SessionMutationCapability`); `internal/adapter/flocklease/flocklease.go` (`Lease`, `heldLease`, `Acquire`, `Release`); wired at `internal/app/build.go` (`buildSessionLease`) |
 | 28 | MCP standalone-SSE listener goroutine (`handleSSE`) per connected server | `mcp.Server` | per connected server (rides the SDK session, opened after `initialize` when `DisableStandaloneSSE: false`) | `Server.Close()` → `session.Close()` → `conn.Close()` cancels `connCtx` → `handleSSE` returns (async; the `mcp` package's `goleak` gate has a targeted ignore list for the SDK + stdlib goroutines that unwind asynchronously after close) | none (the SDK reconnects the stream itself on a transient drop; #177/ADR 0056 reconnects the whole session when the SSE reconnect exhausts → `ErrSessionMissing`) | `internal/adapter/mcp/mcp.go` (`dial`); ADR 0057 |
 | 29 | guardrail session waiver (`WaiverHolder`, ADR 0062) | `app.Build` constructs; the engine arms it via the `modelhook.Runner`'s `port.HookApprovalLearner` on a human `VerdictAllowAlways`, `modelhook.Runner.check` consults it | process | self-clearing; dies with the process (no `Close` — a nil `*WaiverHolder` is the byte-identical OFF posture) | **lost** (in-memory; a waiver never silently survives restart — fail-safe: the call re-blocks/re-asks until a human re-approves it, ADR 0062) | `internal/adapter/modelhook/waiver.go` (`WaiverHolder`); armed via `internal/adapter/modelhook/modelhook.go` (`LearnHookApproval`); constructed in `internal/app/build.go` |
 | 30 | scheduler tick goroutine (Phase 5, ADR 0059) | `internal/adapter/scheduler` (`Scheduler`), held by `server.Service.scheduler` | process | `Scheduler.Stop` cancels the tick loop + joins in-flight fires (with a grace) + releases the leader lease; `Service.Close` stops it FIRST so fires drain while the service is alive | **reconstructible** (a restarted process re-acquires the leader lease or ticks standalone, and re-polls `ScheduleStore.Due` — the store is ground truth, the lookahead is derived); only constructed when `--scheduler` is set (byte-identical default when unwired) | `internal/adapter/scheduler/scheduler.go` (`tickLoop`, `Start`, `Stop`); wired at `internal/app/build.go` (`startScheduler`) |

docs/design/IMPLEMENTATION-NOTES.md

@@ -881,8 +881,17 @@ signals the run. The Service retracts local ask delivery and prevents later rela
 persistence, but leaves the durable `PendingAsk` unresolved and byte-identical for TTL
 takeover. Settled stale run
 references remove heavyweight held-lease/capability tombstones; the lightweight
-`lostOwnership` denial remains until explicit local session teardown so that stale
-Service cannot reacquire.
+`lostOwnership` denial otherwise fails every ordinary caller fast so that a stale
+Service cannot reacquire. It is cleared by explicit local session teardown
+(`CloseSession`), or automatically by the composition-level stale-session sweep's
+`ReconcileLeaseLossTombstone` (issue #1334): a bounded trial Acquire+immediate-Release
+against the real backend proves the lease is genuinely free before the tombstone is
+dropped, letting the next real run-entry repair the session (Interrupt for cancelled,
+the awaiting-resume machinery for awaiting) without waiting for teardown or a process
+restart. That trial is serialized against `onLeaseLost`'s own Release for the same id
+via a dedicated per-id lock (`leaseLossMu`), since `engine/port.SessionLease`'s
+same-id calls are caller-serialized and a conforming backend need not make an
+overlapping Acquire/Release safe on its own.
 
 The gRPC in-stream approval path also enters a Service-owned live-run gate: holding the
 Service mutex orders the verdict against lease invalidation before it reaches the parent

internal/adapter/server/classification.go

@@ -410,6 +410,8 @@ var serviceAccessTable = map[string]ClassificationEntry{
 	"LeaseSweepDisabled":                 {KindExempt, "reads the process-wide sticky sweep-disabled flag SessionStale sets, consumed only by the composition-owned sweep"},
 	"StaleRunningCandidates":             {KindSharedInfrastructure, "root-authorized metadata-only enumeration of owned running, non-scheduled sessions; returns no transcript content"},
 	"SettleIfStale":                      {KindSharedInfrastructure, "root-authorized authoritative reload and settlement of a stale running candidate; ownerless records are rejected under ownership enforcement"},
+	"LostOwnershipCandidates":            {KindSharedInfrastructure, "root-authorized in-memory enumeration of this process's own lease-loss tombstone ids (issue #1334); no store scan, no transcript content"},
+	"ReconcileLeaseLossTombstone":        {KindSharedInfrastructure, "root-authorized trial-Acquire clearing this process's own lease-loss tombstone once the backend proves it free (issue #1334); no session-state mutation"},
 	"DeleteSessionForRetention":          {KindExempt, "legacy composition retention callback; revalidates durable taxonomy/state and acquires the session mutation lease before deletion"},
 	"DeleteSessionForRetentionCandidate": {KindExempt, "composition-owned retention callback over planner metadata; holds run-entry, lease, and backend family exclusions through conditional deletion"},
 }
@@ -472,7 +474,7 @@ var systemAccessTable = map[syscaller.Root]ClassificationEntry{
 	},
 	syscaller.RootStaleSessionReconcile: {
 		KindSharedInfrastructure,
-		"stale-session repair: may enumerate metadata and settle only owned, running, non-scheduled crash orphans through the root-authorized narrow server seam; cannot read transcripts or use caller memory",
+		"stale-session repair: may enumerate metadata and settle only owned, running, non-scheduled crash orphans through the root-authorized narrow server seam, AND (issue #1334) clear this process's own lease-loss tombstones for awaiting/cancelled sessions via a bounded trial-Acquire; cannot read transcripts or use caller memory",
 	},
 }