Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions apps/agent-forge/src/screens/threads/ThreadsScreen.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,12 @@ import {
type ThreadsUrlState,
} from './threadsUrl';

/** Session statuses the daemon writes: `active`, then `interrupted` when the agent's crash sweep closes it. */
const STATUSES = ['active', 'interrupted'];
/**
* Session statuses the daemon writes: `active`; `ended` once its idle sweep sees
* nothing run for the idle window; `interrupted` when the agent's crash sweep
* closes it. New activity makes an ended or interrupted session `active` again.
*/
const STATUSES = ['active', 'ended', 'interrupted'];

/** The session's span, start to its latest finished observation; `-` while nothing has finished. */
const span = (s: ObserveSessionRow): number | null => (s.ended_at === null ? null : s.ended_at - s.started_at);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,15 @@ describe('ThreadsScreen list', () => {
expect(custom?.getAttribute('aria-pressed')).toBe('true');
});

it('offers the ended status the idle sweep writes', async () => {
const container = await mountAt('#/threads?status=ended');
expect(listRequests().at(-1)!.get('status')).toBe('ended');
const statusBtn = [...container.querySelectorAll<HTMLButtonElement>('.facet-btn')].find((b) => b.textContent?.startsWith('Status'))!;
act(() => statusBtn.click());
const options = [...container.querySelectorAll('.facet-pop .facet-opt')].map((o) => o.textContent);
expect(options).toEqual(['active', 'ended', 'interrupted']);
});

it('opening a row navigates to #/threads/:sessionId', async () => {
const container = await mountAt('#/threads?status=active');
const row = container.querySelector<HTMLTableRowElement>('tbody tr[data-row-index="0"]')!;
Expand Down
11 changes: 10 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ lousho mcp <agent.yaml|json|agent-dir|agent.ts> [--http --port N --host H]
lousho doctor [agent.yaml|json|agent-dir|agent.ts] [--json] [--ping]
lousho eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]] [--url <base> [--token <bearer>]] [--config vitest.config.ts] [--timeout <ms>]
lousho traces [traceId|prefix] [--dir D] [--limit N] [--json] [--content]
lousho observe <start|status|stop|rebuild-rollups|prune|export|token> [--port N] [--db path] [--host H] [--require-token] [--project X] [--webhook URL] [--retention-days N] [--older-than <N>d] [--since <N>d|<N>h|<N>m] [--agent X] [--out file]
lousho observe <start|status|stop|rebuild-rollups|prune|export|token> [--port N] [--db path] [--host H] [--require-token] [--project X] [--webhook URL] [--retention-days N] [--session-idle-minutes N] [--older-than <N>d] [--since <N>d|<N>h|<N>m] [--agent X] [--out file]
lousho prompt <push <file|->|list|get <name[@version|@label]>|label <name> <label> <version>> [--url U] [--port N] [--token T] [--json] [--name N]
lousho --version
```
Expand Down Expand Up @@ -306,6 +306,15 @@ prune stays committed, the error goes to stderr and the exit code is 0. It is
safe to run while the daemon is up. A bad N is a usage
error (exit 2). See [Retention](./observability.md#retention).

`start --session-idle-minutes N` sets when a conversation counts as finished
(default 30; `0` turns it off, so a session stays `active` until its agent goes
down). The daemon's sweep marks a session `ended` once none of its runs is
still open (a run marked `interrupted` does not count) and nothing in it has started or finished for N minutes; its
`ended_at` is the time of that last activity. A new run in an `ended` (or
`interrupted`) session makes it `active` again. Ended sessions are then pruned
by `--retention-days` like any other finished session. A bad N is a usage
error (exit 2). See [Idle sessions](./observability.md#idle-sessions).

`export` writes traces as JSONL: one line per trace, `{ trace, observations,
scores }` (the same rows the trace API returns), oldest first, with no cap on
how many. `--since <N>d` (or `<N>h`, `<N>m`) keeps traces that started within
Expand Down
30 changes: 28 additions & 2 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -628,7 +628,8 @@ however old. With a trace go its scores and its prompt-use links; scores older
than the window whose trace has no observations left go too. A session is
deleted only when it is not `active`, ended (or, with no end, started) before
the window, and no remaining observation references it, so an old thread
that is still active is never pruned.
that is still active is never pruned. Idle sessions end on their own (see
[Idle sessions](#idle-sessions)), so finished conversations do age out.

Pruning never touches the Insights rollups (charts for pruned days stay),
the prompt registry, hub entries, the agent registry or tokens. The daily
Expand Down Expand Up @@ -693,12 +694,37 @@ session row is `{ session_id, agent_id, agent_name, status, started_at,
ended_at, traces, input_tokens, output_tokens, cost_usd, last_input_preview }`:
`traces` counts distinct trace ids, tokens and cost sum the session's
observations, and `last_input_preview` is the first 200 characters of the root
input of its latest trace.
input of its latest trace. `status` is `active`, `ended` (idle, see below) or
`interrupted` (its agent went down).

`GET /api/v1/sessions/:id` returns `{ session, traces }` with the traces
(same rows as the trace list) ordered by start ascending, capped at 500; an
unknown id is a 404 with code `LOUSHO_REQUEST_INVALID`.

### Idle sessions

The SDK sends no "session closed" event: a session row exists because its
runs carry its id, so the daemon decides when a conversation is over. Each
sweep (every 30 s) marks a session `ended` when none of its runs is still open
(a run in progress keeps it `active` however long it takes) and nothing in it
has started or finished for `--session-idle-minutes` (default 30; `0` turns
this off). A run marked `interrupted` (its process died while the agent kept
running) no longer counts as in progress; it counts as activity at the moment
it was interrupted. Its `ended_at` is the time of that last activity, not the time of
the sweep. A session no run references falls back to when the daemon last
updated it. Sessions in a db written by an older daemon close on the first
sweep that finds them idle.

A new run in an `ended` session makes it `active` again (`ended_at` follows
its runs again), and so does new activity in an `interrupted` one. A
re-sent or reordered event for activity at or before the session's
`ended_at` does not reopen it. The idle pass never touches `interrupted`
sessions. Once ended, a session is pruned by [retention](#retention) like
any other finished session, so Threads shows only live conversations as
`active`. The idle window is judged against the timestamps your agents
report, so an agent clock far behind the daemon's can end a session early;
its next run reopens it.

## The built server

The server `lousho build --target=node-server` or `--target=docker` generates
Expand Down
41 changes: 38 additions & 3 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4991,7 +4991,7 @@ lousho mcp <agent.yaml|json|agent-dir|agent.ts> [--http --port N --host H]
lousho doctor [agent.yaml|json|agent-dir|agent.ts] [--json] [--ping]
lousho eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]] [--url <base> [--token <bearer>]] [--config vitest.config.ts] [--timeout <ms>]
lousho traces [traceId|prefix] [--dir D] [--limit N] [--json] [--content]
lousho observe <start|status|stop|rebuild-rollups|prune|export|token> [--port N] [--db path] [--host H] [--require-token] [--project X] [--webhook URL] [--retention-days N] [--older-than <N>d] [--since <N>d|<N>h|<N>m] [--agent X] [--out file]
lousho observe <start|status|stop|rebuild-rollups|prune|export|token> [--port N] [--db path] [--host H] [--require-token] [--project X] [--webhook URL] [--retention-days N] [--session-idle-minutes N] [--older-than <N>d] [--since <N>d|<N>h|<N>m] [--agent X] [--out file]
lousho prompt <push <file|->|list|get <name[@version|@label]>|label <name> <label> <version>> [--url U] [--port N] [--token T] [--json] [--name N]
lousho --version
```
Expand Down Expand Up @@ -5261,6 +5261,15 @@ prune stays committed, the error goes to stderr and the exit code is 0. It is
safe to run while the daemon is up. A bad N is a usage
error (exit 2). See [Retention](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/observability.md#retention).

`start --session-idle-minutes N` sets when a conversation counts as finished
(default 30; `0` turns it off, so a session stays `active` until its agent goes
down). The daemon's sweep marks a session `ended` once none of its runs is
still open (a run marked `interrupted` does not count) and nothing in it has started or finished for N minutes; its
`ended_at` is the time of that last activity. A new run in an `ended` (or
`interrupted`) session makes it `active` again. Ended sessions are then pruned
by `--retention-days` like any other finished session. A bad N is a usage
error (exit 2). See [Idle sessions](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/observability.md#idle-sessions).

`export` writes traces as JSONL: one line per trace, `{ trace, observations,
scores }` (the same rows the trace API returns), oldest first, with no cap on
how many. `--since <N>d` (or `<N>h`, `<N>m`) keeps traces that started within
Expand Down Expand Up @@ -14494,7 +14503,8 @@ however old. With a trace go its scores and its prompt-use links; scores older
than the window whose trace has no observations left go too. A session is
deleted only when it is not `active`, ended (or, with no end, started) before
the window, and no remaining observation references it, so an old thread
that is still active is never pruned.
that is still active is never pruned. Idle sessions end on their own (see
[Idle sessions](#idle-sessions)), so finished conversations do age out.

Pruning never touches the Insights rollups (charts for pruned days stay),
the prompt registry, hub entries, the agent registry or tokens. The daily
Expand Down Expand Up @@ -14559,12 +14569,37 @@ session row is `{ session_id, agent_id, agent_name, status, started_at,
ended_at, traces, input_tokens, output_tokens, cost_usd, last_input_preview }`:
`traces` counts distinct trace ids, tokens and cost sum the session's
observations, and `last_input_preview` is the first 200 characters of the root
input of its latest trace.
input of its latest trace. `status` is `active`, `ended` (idle, see below) or
`interrupted` (its agent went down).

`GET /api/v1/sessions/:id` returns `{ session, traces }` with the traces
(same rows as the trace list) ordered by start ascending, capped at 500; an
unknown id is a 404 with code `LOUSHO_REQUEST_INVALID`.

### Idle sessions

The SDK sends no "session closed" event: a session row exists because its
runs carry its id, so the daemon decides when a conversation is over. Each
sweep (every 30 s) marks a session `ended` when none of its runs is still open
(a run in progress keeps it `active` however long it takes) and nothing in it
has started or finished for `--session-idle-minutes` (default 30; `0` turns
this off). A run marked `interrupted` (its process died while the agent kept
running) no longer counts as in progress; it counts as activity at the moment
it was interrupted. Its `ended_at` is the time of that last activity, not the time of
the sweep. A session no run references falls back to when the daemon last
updated it. Sessions in a db written by an older daemon close on the first
sweep that finds them idle.

A new run in an `ended` session makes it `active` again (`ended_at` follows
its runs again), and so does new activity in an `interrupted` one. A
re-sent or reordered event for activity at or before the session's
`ended_at` does not reopen it. The idle pass never touches `interrupted`
sessions. Once ended, a session is pruned by [retention](#retention) like
any other finished session, so Threads shows only live conversations as
`active`. The idle window is judged against the timestamps your agents
report, so an agent clock far behind the daemon's can end a session early;
its next run reopens it.

## The built server

The server `lousho build --target=node-server` or `--target=docker` generates
Expand Down
4 changes: 4 additions & 0 deletions src/cli/observe.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,10 @@ describe('lousho observe', () => {
expect(await runObserve(['start', `--retention-days=${bad}`, '--db', file, '--port', '0'])).toBe(2);
}
expect(errors.at(-1)).toContain('--retention-days');
for (const bad of ['-1', '1.5', 'x', '', '30m']) {
expect(await runObserve(['start', `--session-idle-minutes=${bad}`, '--db', file, '--port', '0'])).toBe(2);
}
expect(errors.at(-1)).toContain('--session-idle-minutes');
});

describe('export', () => {
Expand Down
37 changes: 28 additions & 9 deletions src/cli/observe.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* `lousho observe` - the collector daemon command. It owns the single writer
* to `~/.lousho/observe.db` and serves the ingest + Studio read API.
*
* lousho observe start [--port N] [--db path] [--host H] [--require-token] [--retention-days N]
* lousho observe start [--port N] [--db path] [--host H] [--require-token] [--retention-days N] [--session-idle-minutes N]
* lousho observe status [--port N | --url U]
* lousho observe stop [--db path]
* lousho observe rebuild-rollups [--db path]
Expand All @@ -25,6 +25,7 @@ import {
defaultPidPath,
isObserveDaemon,
OBSERVE_RETENTION_DAYS_DEFAULT,
OBSERVE_SESSION_IDLE_MINUTES_DEFAULT,
sameDbPath,
probeHealth,
startObserveDaemon,
Expand All @@ -38,7 +39,7 @@ import { VERSION } from '../version';
import { exitCodeFor, parseCommand, portValue, stringValue, usageError, type CommandSpec } from './args';

const USAGE = `Usage: lousho observe <command> [options]
observe start [--port N] [--db path] [--host H] [--require-token] [--webhook URL] [--retention-days N]
observe start [--port N] [--db path] [--host H] [--require-token] [--webhook URL] [--retention-days N] [--session-idle-minutes N]
observe status [--port N] [--url U]
observe stop [--db path]
observe rebuild-rollups [--db path]
Expand All @@ -63,6 +64,10 @@ Options:
'start': once a day, delete finished traces older than N days (default
${OBSERVE_RETENTION_DAYS_DEFAULT}; 0 keeps everything). Open runs, active sessions,
rollups, prompts and hub entries are never pruned.
--session-idle-minutes N
'start': end a session once nothing in it has run for N minutes (default
${OBSERVE_SESSION_IDLE_MINUTES_DEFAULT}; 0 keeps sessions active until their agent goes down).
New activity reopens it; retention then prunes ended sessions.
--older-than <N>d 'prune': delete finished traces older than N days now, then VACUUM the db
--since <N>d 'export': only traces that started within the last N days (or <N>h / <N>m)
--agent X 'export': only this agent (its agent_id, or its name when that names one agent)
Expand All @@ -76,10 +81,11 @@ type ObserveSubcommand = 'start' | 'status' | 'stop' | 'rebuild-rollups' | 'prun
const SUB_SPECS: Record<ObserveSubcommand, CommandSpec> = {
start: {
command: 'observe start',
usage: 'Usage: lousho observe start [--port N] [--db path] [--host H] [--require-token] [--webhook URL] [--retention-days N]',
usage: 'Usage: lousho observe start [--port N] [--db path] [--host H] [--require-token] [--webhook URL] [--retention-days N] [--session-idle-minutes N]',
options: {
port: { type: 'string' },
'retention-days': { type: 'string' },
'session-idle-minutes': { type: 'string' },
db: { type: 'string' },
host: { type: 'string' },
'require-token': { type: 'boolean' },
Expand Down Expand Up @@ -135,11 +141,17 @@ function stringListValue(value: string | boolean | (string | boolean)[] | undefi

const DAY_MS = 86_400_000;

/** `--retention-days N`: a non-negative integer (0 disables the daily prune), or the default when absent. */
function retentionDaysValue(spec: CommandSpec, value: string | boolean | (string | boolean)[] | undefined): number {
/** A non-negative integer flag (`--retention-days N`, `--session-idle-minutes N`; 0 disables), or `fallback` when absent. */
function wholeNumberValue(
spec: CommandSpec,
flag: string,
meaning: string,
value: string | boolean | (string | boolean)[] | undefined,
fallback: number
): number {
const given = stringValue(value);
if (given === undefined) return OBSERVE_RETENTION_DAYS_DEFAULT;
if (!/^\d+$/.test(given)) throw usageError(spec, `--retention-days must be a non-negative integer (days; 0 disables pruning), got '${given}'.`);
if (given === undefined) return fallback;
if (!/^\d+$/.test(given)) throw usageError(spec, `${flag} must be a non-negative integer (${meaning}), got '${given}'.`);
return Number(given);
}

Expand Down Expand Up @@ -210,12 +222,19 @@ async function runStart(args: string[]): Promise<number> {
const host = stringValue(values.host) ?? '127.0.0.1';
const requireToken = values['require-token'] === true;
const webhooks = stringListValue(values.webhook);
const retentionDays = retentionDaysValue(spec, values['retention-days']);
const retentionDays = wholeNumberValue(spec, '--retention-days', 'days; 0 disables pruning', values['retention-days'], OBSERVE_RETENTION_DAYS_DEFAULT);
const sessionIdleMinutes = wholeNumberValue(
spec,
'--session-idle-minutes',
'minutes; 0 never ends idle sessions',
values['session-idle-minutes'],
OBSERVE_SESSION_IDLE_MINUTES_DEFAULT
);
const pidPath = defaultPidPath(dbPath);

let daemon;
try {
daemon = await startObserveDaemon({ dbPath, port, host, requireToken, pidPath, webhooks, retentionDays });
daemon = await startObserveDaemon({ dbPath, port, host, requireToken, pidPath, webhooks, retentionDays, sessionIdleMinutes });
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'EADDRINUSE') return reportPortInUse(port, host, dbPath);
throw error;
Expand Down
17 changes: 16 additions & 1 deletion src/observe/daemon.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,24 @@ export interface ObserveDaemonOptions {
* it; `0`, a negative value or `NaN` disables pruning.
*/
retentionDays?: number;
/**
* Idle window in minutes after which the sweep ends an `active` session
* with no open observation (default {@link OBSERVE_SESSION_IDLE_MINUTES_DEFAULT}).
* Only a finite number `> 0` enables it; `0` keeps sessions active until
* their agent goes down.
*/
sessionIdleMinutes?: number;
}

/** `lousho observe start`'s default retention window: finished traces older than 30 days are pruned daily. */
export const OBSERVE_RETENTION_DAYS_DEFAULT = 30;

/** `lousho observe start`'s default idle window: a session quiet for 30 minutes with nothing running is ended. */
export const OBSERVE_SESSION_IDLE_MINUTES_DEFAULT = 30;

/** The sweep's idle window in ms from {@link ObserveDaemonOptions.sessionIdleMinutes} (absent = the default). */
const minutesToMs = (minutes: number = OBSERVE_SESSION_IDLE_MINUTES_DEFAULT): number => minutes * 60_000;

export interface ObserveDaemon {
port: number;
host: string;
Expand Down Expand Up @@ -236,14 +249,16 @@ export async function startObserveDaemon(options: ObserveDaemonOptions = {}): Pr
// The monitoring sweep: rewrites agents.status from heartbeat freshness,
// interrupts a crashed agent's open work, and posts alerts to webhooks
// (fire-and-forget; delivery lives in alerts.ts). It also runs the daily
// retention prune (sweep.ts) - never a VACUUM, which would stall ingest.
// retention prune (sweep.ts) - never a VACUUM, which would stall ingest -
// and ends idle sessions.
const alerter = createAlerter(alertConfig, options.fetchFn ?? fetch);
const sweep = startStatusSweep(db, {
intervalMs: options.sweepIntervalMs,
now: options.nowFn,
errorRateThreshold: alertConfig.errorRateThreshold,
onAlerts: (alerts) => alerter.post(alerts),
retentionDays: options.retentionDays ?? OBSERVE_RETENTION_DAYS_DEFAULT,
sessionIdleMs: minutesToMs(options.sessionIdleMinutes),
});

return {
Expand Down
Loading
Loading