LOU-U7, LOU-U8, LOU-U9: durable execution gaps (approval mid-batch, resume with new input, checkpoint after each LLM turn)

README.md

@@ -250,6 +250,7 @@ console.log(session.id, session.messages.length);
 - [Configuration](docs/configuration.md) - agent spec fields, provider env vars, `AgentExecutor.execute()` options, CLI flags
 - [Deployment](docs/deployment.md) - `loushy build` targets: Node server, Docker, Cloudflare Workers
 - [API Overview](docs/api-overview.md) - the main exports; `npm run docs:build` generates the full TypeDoc reference
+- [Durable execution](docs/durable-execution.md) - checkpoints, crash resume, multi-turn sessions, approvals mid-batch, at-least-once tools
 - [Streaming](docs/streaming.md) - `agent.stream()`: the typed event schema, terminal and SSE examples
 - [Workspace tools](docs/workspace-tools.md) - file system and shell tools for coding agents, and their security model
 - [Tracing and observability](docs/observability.md) - OpenTelemetry GenAI spans, attribute table, content opt-in
@@ -398,21 +399,26 @@ const result = await resumeAfterApproval(
 
 ### Durable execution / checkpoints
 
-Pass a `sessionId` and a `checkpointStore`; `AgentExecutor` saves a
-checkpoint after every tool result and rehydrates from it on the next call
-with the same `sessionId` — so a crash mid-conversation resumes rather than
-restarts:
+Pass a `sessionId` and a `checkpointStore`; `AgentExecutor` checkpoints
+after every model response and every tool result, on any host. The next
+call with the same `sessionId` resumes an unfinished run (running only the
+tool calls that have no result yet, without asking the model again),
+continues a finished one as a multi-turn conversation, and refuses to
+bypass a pending approval:
 
 ```typescript
 import { AgentExecutor, LocalStorageCheckpointStore } from '@loushy/build-ai-agent';
 
 const checkpointStore = new LocalStorageCheckpointStore(storage);
 
-await AgentExecutor.execute({ agent, input, provider, sessionId: 'session-123', checkpointStore });
-// ...process restarts...
-await AgentExecutor.execute({ agent, input: 'continue', provider, sessionId: 'session-123', checkpointStore });
+await AgentExecutor.execute({ agent, input: 'Book a table for 2', provider, sessionId: 'session-123', checkpointStore });
+// ...process restarts, or the user simply replies...
+await AgentExecutor.execute({ agent, input: 'Make it 3 people', provider, sessionId: 'session-123', checkpointStore });
 ```
 
+See [Durable execution](docs/durable-execution.md) for the exact guarantees
+(including the at-least-once caveat for tools that were mid-flight).
+
 ### Sub-agents and delegation
 
 Give a lead agent named sub-agents; it delegates with one auto-registered

apps/agent-forge/server/__tests__/runRegistry.test.ts

@@ -99,24 +99,25 @@ describe('RunManager', () => {
     await runManager.run(agentId, 'please use current-date', SPEC);
     await waitForStatus(runManager, agentId, (s) => s.status === 'stopped');
 
-    // The checkpoint from the aborted run must survive - AgentExecutor only
-    // deletes it on a successful terminal completion, never on an aborted
-    // run (see abortableProvider.ts's doc comment).
+    // The checkpoint from the aborted run must survive, still unfinished
+    // (see abortableProvider.ts's doc comment).
     const checkpoint = await checkpointStore.load(agentId);
     expect(checkpoint).not.toBeNull();
     expect(checkpoint!.stepIndex).toBeGreaterThanOrEqual(1);
 
     // Running again picks the same checkpoint back up (AgentExecutor
-    // rehydrates from it and ignores the new `input`) rather than starting
-    // a brand new conversation from scratch.
-    await runManager.run(agentId, 'this input is ignored on resume', SPEC);
+    // rehydrates from it; LOU-U8 appends the new `input` after the resumed
+    // turn) rather than starting a brand new conversation from scratch.
+    await runManager.run(agentId, 'a follow-up appended on resume', SPEC);
     const resumed = await waitForStatus(runManager, agentId, (s) => s.status === 'stopped');
     expect(resumed.resultText).toBe('This is a mock response.');
 
-    // And the checkpoint is cleared once that resumed run completes
-    // successfully.
+    // And the checkpoint is kept, marked finished, once that resumed run
+    // completes successfully (LOU-U8: the next run continues the session).
     const afterResume = await checkpointStore.load(agentId);
-    expect(afterResume).toBeNull();
+    expect(afterResume?.status).toBe('finished');
+    expect(afterResume?.messages.map((m) => m.content)).toContain('please use current-date');
+    expect(afterResume?.messages.map((m) => m.content)).toContain('a follow-up appended on resume');
   });
 
   it('emits structured log entries derived from the ExecutionEvent stream (O1)', async () => {

apps/agent-forge/server/abortableProvider.ts

@@ -22,11 +22,10 @@
  * provider is itself abort-aware) throws, which propagates out of
  * AgentExecutor.execute()'s catch block (it only emits an 'error' event and
  * rethrows - it does not swallow) as a rejected promise. Crucially,
- * AgentExecutor only clears the CheckpointStore entry on a *successful*
- * terminal completion (see the bottom of runAgentLoop(), reached only after
- * the while-loop `break`s) - never from the catch block - so an abort
- * leaves the last-saved checkpoint (written after each completed tool
- * result) intact. The next `run()` call for the same agent reuses the same
+ * AgentExecutor only marks the CheckpointStore entry 'finished' on a
+ * *successful* terminal completion - never from the catch block - so an
+ * abort leaves the last-saved 'in-progress' checkpoint (written after each
+ * model response and each completed tool result) intact. The next `run()` call for the same agent reuses the same
  * sessionId, so AgentExecutor.execute() rehydrates from exactly that
  * checkpoint instead of starting over. This is what makes Stop-then-Run
  * "resume from the last checkpoint" rather than "restart from scratch".

apps/agent-forge/server/runRegistry.ts

@@ -355,19 +355,16 @@ export class RunManager extends EventEmitter {
    * what makes Stop-then-Run resume from the last checkpoint (N3) instead
    * of starting over: AgentExecutor.execute() (see its doc comments)
    * rehydrates from `checkpointStore.load(sessionId)` when a checkpoint
-   * exists under that id, and only clears it on a *successful* terminal
-   * completion - never when the run was aborted (see abortableProvider.ts).
-   * So after a stop(), the checkpoint from the last completed tool result
-   * is still there, and the very next run() call for this agentId picks it
+   * exists under that id, and only marks it 'finished' on a *successful*
+   * terminal completion - never when the run was aborted (see
+   * abortableProvider.ts). So after a stop(), the unfinished checkpoint is
+   * still there, and the very next run() call for this agentId picks it
    * back up automatically.
    *
-   * One consequence worth calling out: when a checkpoint exists,
-   * AgentExecutor.execute() ignores `input` entirely and continues the
-   * rehydrated conversation - this mirrors resumeAfterApproval()'s
-   * documented behavior for the approval-gate case and is intentional: a
-   * resumed run is a continuation of the same paused conversation, not a
-   * new question. `input` is only actually used to seed a genuinely fresh
-   * run (no prior checkpoint).
+   * LOU-U8: `input` is appended to the rehydrated conversation as a new
+   * user turn (after the interrupted turn's pending tool results), unless
+   * it re-sends the message the interrupted run started from; after a
+   * finished run it continues the conversation.
    */
   async run(agentId: string, input: string, spec?: AgentSpec): Promise<void> {
     const existing = this.entries.get(agentId);
@@ -403,15 +400,10 @@ export class RunManager extends EventEmitter {
    *   string instead, so AgentExecutor's normal buildMessages() path builds
    *   the system+user turn from scratch exactly like a Topbar "Run" click.
    *
-   * Known limitation (left for LOU-Q): if the agent was Stopped mid-flight
-   * (a checkpoint exists from an in-progress run, not a completed one),
-   * AgentExecutor.execute() ignores `input` entirely and resumes straight
-   * from that checkpoint (see run()'s original doc comment above) - so a
-   * chat message sent in that state is shown optimistically in the thread
-   * but is NOT actually incorporated into the resumed run's conversation.
-   * This mirrors existing Stop→Run checkpoint-resume behavior rather than
-   * inventing new semantics for it; a future epic may want a dedicated
-   * "discard checkpoint and continue with this new message instead" action.
+   * The checkpoint under this agent's sessionId already holds that same
+   * history; AgentExecutor.execute() (LOU-U8) recognises the re-sent prefix
+   * and appends only the new user turn - after a Stop mid-flight, once the
+   * interrupted turn's pending tool results are recorded.
    */
   async sendMessage(agentId: string, text: string, spec?: AgentSpec): Promise<void> {
     const existing = this.entries.get(agentId) ?? this.freshEntry(agentId);

docs/api-overview.md

@@ -17,6 +17,7 @@ npm run docs:build   # writes docs/api/index.html
 | `AgentExecutor.execute(opts)` | Static executor: runs an agent (LLM + tool-calling loop) and resolves to an `ExecutionResult`. |
 | `AgentType`                   | Agent type enum (e.g. `AgentType.SmartAssistant`).                          |
 | `resumeAfterApproval()`       | Resume an execution paused for human approval.                             |
+| `SessionAwaitingApprovalError` | Thrown by `execute()` when its `sessionId` is paused on an approval (see [Durable execution](./durable-execution.md)). |
 | `createDelegateTool()`        | Wrap a child agent as a tool for multi-agent delegation.                    |
 
 ### Sub-agents
@@ -36,6 +37,18 @@ Pass `skills: [defineSkill({ name, description, content }), ...(await loadSkills
 the system prompt and the model loads bodies through an auto-registered
 `load_skill` tool. See [Skills](./skills.md).
 
+### Durable execution
+
+`sessionId` + `checkpointStore` make a run crash-safe and a session
+multi-turn: the run is checkpointed after every model response, every tool
+result and every pause, and calling `execute()` again with the same
+`sessionId` resumes an unfinished run (without re-calling the model for a
+turn it already has), continues a finished conversation with the new
+input, or throws `SessionAwaitingApprovalError` while an approval is
+pending. Tools run at-least-once across a crash; `execute` receives the
+call's `toolCallId` to use as an idempotency key. See
+[Durable execution](./durable-execution.md) for the exact guarantees.
+
 ### Cancellation
 
 Pass an `AbortSignal` to stop a run: `agent.send(input, { signal })`,
@@ -75,9 +88,11 @@ How it behaves:
 - `onEvent` receives an `abort` event (its `abortReason` is the signal's
   `reason`), then `finish` with `finishReason: 'aborted'`.
 - With `sessionId` + `checkpointStore`, the state is checkpointed. Calling
-  `execute()` again with the same `sessionId` resumes where the run stopped.
-  Tool calls the run never reached get an `{ error }` result saying they were
-  cancelled, so the conversation stays valid for the provider.
+  `execute()` again with the same `sessionId` resumes where the run stopped;
+  new `input` is appended as the next user message (see
+  [Durable execution](./durable-execution.md)). Tool calls the run never
+  reached get an `{ error }` result saying they were cancelled, so the
+  conversation stays valid for the provider.
 - An already-aborted signal returns at once without calling the provider.
 
 ### Parallel tool calls
@@ -120,10 +135,10 @@ Guarantees, whatever the limit:
 - **Approvals.** The first call that needs approval stops the batch: the
   calls before it run (concurrently) and their results are recorded, then the
   run pauses on that call (`finishReason: 'awaiting-approval'`). Calls after
-  it never start in this run. **Known gap:** after `resumeAfterApproval()`
-  those later calls are still not run and get no result (tracked as LOU-U7).
-  Until that lands, use `toolConcurrency: 1` and keep approval tools out of
-  multi-call turns if this matters to you.
+  it never start in this run. `resumeAfterApproval()` records the paused
+  call's result (or rejection) and then runs those later calls the same way,
+  so every call of the turn gets exactly one result - see
+  [Durable execution](./durable-execution.md#approvals-in-the-middle-of-a-tool-batch).
 - **Failures are isolated.** A tool that throws gets its own error result;
   its siblings carry on. A propagating error (`PropagatingToolError`, such as
   the delegation depth guard, or a throwing hook) stops new calls from
@@ -133,9 +148,11 @@ Guarantees, whatever the limit:
   `finishReason: 'aborted'`. Calls that finished keep their results; the rest
   get a "cancelled" result. Running tools see the abort through their
   `abortSignal`.
-- **Checkpoints.** With `sessionId` + `checkpointStore`, a checkpoint is
-  written each time the in-order run of finished calls grows (with `1`, after
-  every call, as before). A resumed run never runs a recorded call again.
+- **Checkpoints.** With `sessionId` + `checkpointStore`, the model's turn is
+  checkpointed before any call starts, then again each time the in-order run
+  of finished calls grows (with `1`, after every call). A resumed run never
+  runs a recorded call again, and never asks the model again for a turn it
+  already checkpointed.
 
 ## Declarative specs