diff --git a/docs/approvals.md b/docs/approvals.md index 2d482908..c2472452 100644 --- a/docs/approvals.md +++ b/docs/approvals.md @@ -568,7 +568,7 @@ continued run as an `AgentRun` (see | `InMemoryApprovalStore` | This process only; the default for `createAgent()`. | | `StorageServiceApprovalStore(storage)` | JSON files through a `StorageService`. | | `fileStore(dir)` | One JSON file per pause, `/approvals/.json`, on one machine; an approval can be resolved by only one process (see [Choosing a store](./sessions.md#choosing-a-store)). | -| `SqliteStore.approvals` | One SQLite file, shared safely by several processes; an approval can be resolved by only one of them. | +| `SqliteStore.approvals` | One SQLite file, shared by several processes on one host ([details](./sessions.md#sharing-a-sqlite-file)); an approval can be resolved by only one of them. | An `ApprovalStore` needs `save()` and `resolve()`. `load(id)` (read a pause without claiming it) and `list()` (every pause not resolved yet, oldest first) diff --git a/docs/deployment.md b/docs/deployment.md index c879f09b..716b883e 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -216,7 +216,11 @@ directory's `auth.ts`, with `LOUSHO_API_TOKEN` appended to it: see | `memory` (default) | `memoryStore()`: gone when the process restarts, and not shared between instances. | | `sqlite:` | A [`SqliteStore`](./sessions.md) file (created with its directory): sessions survive restarts. Needs Node 22 (`node:sqlite`); the `docker` image uses `node:22-slim`. Mount the file's directory as a volume (the `docker` image owns `/app/data` for this). | -SQLite is one file on one disk, so run a single instance per database file. +Several instances on one host may share one `sqlite:` file on a local disk (for +example two replicas on one Docker volume); a busy session answers `409` with +`Retry-After`. A network filesystem (NFS, EFS, SMB) or several hosts are not +supported: use a store you implement. See +[Sharing a SQLite file](./sessions.md#sharing-a-sqlite-file). ### Crash recovery diff --git a/docs/durable-execution.md b/docs/durable-execution.md index 2e3dbc0d..5e160842 100644 --- a/docs/durable-execution.md +++ b/docs/durable-execution.md @@ -302,6 +302,11 @@ before any tool runs or any model is called (a resolved approval stays pending). Two replicas behind a load balancer therefore never run one turn's tool calls twice. +This holds for replicas on one host sharing one SQLite file on a local disk, or +for a store you implement over a shared database; SQLite on a network +filesystem or across hosts is not supported. See +[Sharing a SQLite file](./sessions.md#sharing-a-sqlite-file). + A lease whose holder died is taken over once it expires. Expiry is judged by the clock of the process that wants the lease against the expiry the holder wrote with its own clock, so the file and SQLite stores wait one more lease diff --git a/docs/pause-and-resume.md b/docs/pause-and-resume.md index 7ee1ab75..16063c5c 100644 --- a/docs/pause-and-resume.md +++ b/docs/pause-and-resume.md @@ -32,7 +32,7 @@ policies. Pauses and checkpoints live in the agent's `store`. The default is in memory, which a restart loses. Use a durable one: `SqliteStore` (one transactional file, -shared safely by several processes) or `fileStore(dir)` (plain JSON files). See +shared by several processes on one host) or `fileStore(dir)` (plain JSON files). See [Choosing a store](./sessions.md#choosing-a-store). ## 3. Send: the run pauses diff --git a/docs/sessions.md b/docs/sessions.md index 7a060ec6..6ada5c36 100644 --- a/docs/sessions.md +++ b/docs/sessions.md @@ -349,7 +349,7 @@ implementation by where the process runs: | --- | --- | --- | --- | --- | | In memory (`memoryStore()`) | `MemorySessionStore` | in memory | `InMemoryApprovalStore` | Tests, scripts, one process that never restarts | | Files (`fileStore(dir)`) | `/sessions/.json` | `/checkpoints/`, `/checkpoint-history/` | `/approvals/.json` | One machine, you want plain inspectable files | -| SQLite | `store.sessions` | `store.checkpoints` | `store.approvals` | A Node server: one durable, transactional file, shared safely by several processes | +| SQLite | `store.sessions` | `store.checkpoints` | `store.approvals` | A Node server: one durable, transactional file, shared by several processes on one host (see [Sharing a SQLite file](#sharing-a-sqlite-file)) | | Cloudflare KV (`KVStore` from `@lousho/build-ai-agent/kv`) | `store.sessions` | `store.checkpoints` | `store.approvals` | Workers deployments (see [Deployment](deployment.md)) | Each one is an `AgentStore` part: pass them together as @@ -433,8 +433,9 @@ With the executor directly, pass `store.checkpoints` as `checkpointStore` and - The directory is created if missing. The schema is versioned with `PRAGMA user_version` and migrated on open; a database written by a newer release is refused. -- WAL mode and a 5 s busy timeout let two processes share the file. An approval - can be resolved by only one of them. +- WAL mode and a short lock wait with retries (about 5 s in all, then + `LOUSHO_STORAGE_BUSY`) let several processes share the file. An approval can + be resolved by only one of them. See [Sharing a SQLite file](#sharing-a-sqlite-file). - Checkpoints and approval snapshots are stored as opaque JSON, so new fields round-trip unchanged. - `prune()` removes sessions and checkpoints not updated within `olderThanMs`, @@ -447,6 +448,27 @@ With the executor directly, pass `store.checkpoints` as `checkpointStore` and - A file that is not a SQLite database fails with an error naming the path; using the store after `close()` throws a clear error. +### Sharing a SQLite file + +This is the one statement the other pages link to. + +- **Supported:** several processes (for example two Docker replicas on one + named volume, or a server and a CLI) on **one host**, sharing one SQLite file + on a **local disk**. The file runs in WAL mode, and each session turn holds a + [run lease](./durable-execution.md#run-leases), so replicas can alternate + messages on one session. A write to a session another process is running + fails with `LOUSHO_SESSION_BUSY`; `POST /chat` and the channels answer that + with `409` and a `Retry-After` header, so retry. +- **Not supported:** a SQLite file on a network filesystem (NFS, EFS, SMB), + whose locking SQLite cannot rely on, and several hosts. For several hosts, + implement `SessionStore`, `CheckpointStore` and `ApprovalStore` over a shared + database such as Postgres or Redis. The SDK ships no network store. +- **Clocks and PIDs:** leases are judged by each process's own clock, so keep + replica clocks in sync (NTP). A crashed run is taken over early only when + its process is provably gone on the same host (boot id, PID namespace and + process start time are checked); otherwise it waits 30 to 60 seconds for the + lease to expire. See [Run leases](./durable-execution.md#run-leases). + ## Project instructions Not part of sessions, but often wanted together: see "Project instructions" in diff --git a/llms-full.txt b/llms-full.txt index 9725ff68..75ad6419 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3015,7 +3015,7 @@ continued run as an `AgentRun` (see | `InMemoryApprovalStore` | This process only; the default for `createAgent()`. | | `StorageServiceApprovalStore(storage)` | JSON files through a `StorageService`. | | `fileStore(dir)` | One JSON file per pause, `/approvals/.json`, on one machine; an approval can be resolved by only one process (see [Choosing a store](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md#choosing-a-store)). | -| `SqliteStore.approvals` | One SQLite file, shared safely by several processes; an approval can be resolved by only one of them. | +| `SqliteStore.approvals` | One SQLite file, shared by several processes on one host ([details](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md#sharing-a-sqlite-file)); an approval can be resolved by only one of them. | An `ApprovalStore` needs `save()` and `resolve()`. `load(id)` (read a pause without claiming it) and `list()` (every pause not resolved yet, oldest first) @@ -7274,7 +7274,11 @@ directory's `auth.ts`, with `LOUSHO_API_TOKEN` appended to it: see | `memory` (default) | `memoryStore()`: gone when the process restarts, and not shared between instances. | | `sqlite:` | A [`SqliteStore`](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md) file (created with its directory): sessions survive restarts. Needs Node 22 (`node:sqlite`); the `docker` image uses `node:22-slim`. Mount the file's directory as a volume (the `docker` image owns `/app/data` for this). | -SQLite is one file on one disk, so run a single instance per database file. +Several instances on one host may share one `sqlite:` file on a local disk (for +example two replicas on one Docker volume); a busy session answers `409` with +`Retry-After`. A network filesystem (NFS, EFS, SMB) or several hosts are not +supported: use a store you implement. See +[Sharing a SQLite file](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md#sharing-a-sqlite-file). ### Crash recovery @@ -7723,6 +7727,11 @@ before any tool runs or any model is called (a resolved approval stays pending). Two replicas behind a load balancer therefore never run one turn's tool calls twice. +This holds for replicas on one host sharing one SQLite file on a local disk, or +for a store you implement over a shared database; SQLite on a network +filesystem or across hosts is not supported. See +[Sharing a SQLite file](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md#sharing-a-sqlite-file). + A lease whose holder died is taken over once it expires. Expiry is judged by the clock of the process that wants the lease against the expiry the holder wrote with its own clock, so the file and SQLite stores wait one more lease @@ -14181,7 +14190,7 @@ policies. Pauses and checkpoints live in the agent's `store`. The default is in memory, which a restart loses. Use a durable one: `SqliteStore` (one transactional file, -shared safely by several processes) or `fileStore(dir)` (plain JSON files). See +shared by several processes on one host) or `fileStore(dir)` (plain JSON files). See [Choosing a store](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md#choosing-a-store). ## 3. Send: the run pauses @@ -16766,7 +16775,7 @@ implementation by where the process runs: | --- | --- | --- | --- | --- | | In memory (`memoryStore()`) | `MemorySessionStore` | in memory | `InMemoryApprovalStore` | Tests, scripts, one process that never restarts | | Files (`fileStore(dir)`) | `/sessions/.json` | `/checkpoints/`, `/checkpoint-history/` | `/approvals/.json` | One machine, you want plain inspectable files | -| SQLite | `store.sessions` | `store.checkpoints` | `store.approvals` | A Node server: one durable, transactional file, shared safely by several processes | +| SQLite | `store.sessions` | `store.checkpoints` | `store.approvals` | A Node server: one durable, transactional file, shared by several processes on one host (see [Sharing a SQLite file](#sharing-a-sqlite-file)) | | Cloudflare KV (`KVStore` from `@lousho/build-ai-agent/kv`) | `store.sessions` | `store.checkpoints` | `store.approvals` | Workers deployments (see [Deployment](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/deployment.md)) | Each one is an `AgentStore` part: pass them together as @@ -16850,8 +16859,9 @@ With the executor directly, pass `store.checkpoints` as `checkpointStore` and - The directory is created if missing. The schema is versioned with `PRAGMA user_version` and migrated on open; a database written by a newer release is refused. -- WAL mode and a 5 s busy timeout let two processes share the file. An approval - can be resolved by only one of them. +- WAL mode and a short lock wait with retries (about 5 s in all, then + `LOUSHO_STORAGE_BUSY`) let several processes share the file. An approval can + be resolved by only one of them. See [Sharing a SQLite file](#sharing-a-sqlite-file). - Checkpoints and approval snapshots are stored as opaque JSON, so new fields round-trip unchanged. - `prune()` removes sessions and checkpoints not updated within `olderThanMs`, @@ -16864,6 +16874,27 @@ With the executor directly, pass `store.checkpoints` as `checkpointStore` and - A file that is not a SQLite database fails with an error naming the path; using the store after `close()` throws a clear error. +### Sharing a SQLite file + +This is the one statement the other pages link to. + +- **Supported:** several processes (for example two Docker replicas on one + named volume, or a server and a CLI) on **one host**, sharing one SQLite file + on a **local disk**. The file runs in WAL mode, and each session turn holds a + [run lease](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/durable-execution.md#run-leases), so replicas can alternate + messages on one session. A write to a session another process is running + fails with `LOUSHO_SESSION_BUSY`; `POST /chat` and the channels answer that + with `409` and a `Retry-After` header, so retry. +- **Not supported:** a SQLite file on a network filesystem (NFS, EFS, SMB), + whose locking SQLite cannot rely on, and several hosts. For several hosts, + implement `SessionStore`, `CheckpointStore` and `ApprovalStore` over a shared + database such as Postgres or Redis. The SDK ships no network store. +- **Clocks and PIDs:** leases are judged by each process's own clock, so keep + replica clocks in sync (NTP). A crashed run is taken over early only when + its process is provably gone on the same host (boot id, PID namespace and + process start time are checked); otherwise it waits 30 to 60 seconds for the + lease to expire. See [Run leases](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/durable-execution.md#run-leases). + ## Project instructions Not part of sessions, but often wanted together: see "Project instructions" in