docs(changelog): observe and Studio round 2

CHANGELOG.md

@@ -14,19 +14,58 @@ opt-out.
 ### Added
 
 - The `lousho observe` daemon: a local collector (loopback, `http://127.0.0.1:7777`, SQLite at `~/.lousho/observe.db`) that gathers every agent's traces, heartbeats and eval scores when `LOUSHO_OBSERVE` is set — no per-agent code changes. `lousho observe start|status|stop|rebuild-rollups|token create`; a second `start` on a busy port attaches when it is already the observe daemon. See [Observability](docs/observability.md#the-observe-daemon).
-- Agent Forge gains Observe tabs — Traces, Insights, Monitoring, Scores — reading the daemon through a token-gated `/observe/**` proxy on the studio server.
+- Agent Forge is rebuilt around a left navigation grouped Build (Agents, Playground), Observe (Traces, Threads, Monitoring), Evaluate (Evaluations: Scores, Datasets, Evaluators, Annotation queues) and Assets (Prompts, Context Hub), plus Settings. It replaces the Build/Observe toggle and the old Observe tabs, and the Build toolbar shows only on Agents. See [Agent Forge](docs/agent-forge.md#navigation).
+- Every Studio screen has a hash route with its filters in the query string (`#/traces?agent=…&status=error`, `#/traces/<id>?obs=<step>`), so a reload, a bookmark, a pasted link or back/forward lands on the same view. See [Routes and deep links](docs/agent-forge.md#routes-and-deep-links).
+- Studio keyboard: `Ctrl+K` / `Cmd+K` opens a command palette (go to a screen, open an agent, open a pasted trace id, toggle the light/dark theme), `?` lists the shortcuts, and `j`/`k`/`Enter` move through and open table rows. See [Keyboard](docs/agent-forge.md#keyboard).
+- The last Studio nav item shows whether the collector answers. While the daemon is down, each daemon screen shows the command to start it and fills in once it is back. Daemon screens read it through the token-gated `/observe/**` proxy on the studio server, which forwards reads plus only the writes the screens make. See [The collector](docs/agent-forge.md#the-collector-observe-daemon).
+- Studio Traces and Trace screens: the list filters by search (name, input, output), agent, status, model, project, prompt and time range (15m to 30d or custom, default 24 h), sorts by duration, tokens or cost, and shows input/output previews, the error message and the model. A trace opens in one viewer (tree or waterfall, the failed step selected by default) whose inspector shows the error, input and output as chat bubbles with tool-call chips (Rendered, Raw or JSON), thumbs up/down feedback, usage and attributes; the header links the agent and thread and has **Copy link**, **Add to dataset** and **Add to queue**. Model-call steps carry **Open in Playground**. See [Traces](docs/agent-forge.md#traces).
+- Studio Threads (`#/threads`, one row per session, a session's turns oldest first), Monitoring (`#/monitoring`: the fleet table with status, last heartbeat, runs today, error rate, p95 latency and dropped events, plus Runs, Latency p95, Error rate, Cost and Tokens charts over 1h to 30d whose buckets open the matching traces) and Scores (`#/evals/scores`: one row per eval with pass rate, trend, per-check pass rates and latest failures, filtered by eval, agent, source and range). See [Monitoring](docs/agent-forge.md#monitoring).
+- Studio Playground (`#/playground`): system, user and assistant messages with `{name}` variables, a provider and model (`mock` with no key), tools borrowed from a saved agent, a streamed run with **Stop**, **Open trace** when it finishes, and **Save as new prompt version**. See [Playground](docs/agent-forge.md#playground).
+- Playground deep links: `#/playground?trace=<id>&fromObs=<obs>` loads a model call from a trace, and `#/playground?prompt=<name>[@<ref>]` loads a registry prompt.
+- `POST /playground/run` on the studio server runs a playground request and streams SSE `delta`, `tool_call`, `done` and `error` events; runs report to the daemon as agent `playground`. See [Playground run endpoint](docs/agent-forge.md#playground-run-endpoint).
+- Studio Prompts (`#/prompts`: versions, labels, runs that used each version, a side-by-side diff, **Move `<label>` here**, **Open in Playground**) and Context Hub (`#/hub`: namespaces viewed as one agent, `private` or `fleet` scope, and editing a fleet entry). See [Prompts](docs/agent-forge.md#prompts).
+- The Agents builder: node cards show the model, tool count and status; an empty canvas offers templates; Inspector sections collapse and the bottom drawer resizes (both remembered); and a run's status pill links **Open trace**. Builder runs report to the observe daemon under their agent id, with captured content kept out of the project's trace files. See [Agents](docs/agent-forge.md#agents).
+- Datasets: the daemon stores named datasets of `{ input, expected? }` examples (`GET`/`POST /api/v1/datasets`, `GET /api/v1/datasets/:name`, `POST /api/v1/datasets/:name/examples` with an example or `{ from_trace_id }`, which copies a trace's root input and output, and `DELETE …/examples/:id`). `defineEval({ dataset: 'name' })` in place of `cases` runs one test per example (`c` is `{ input, expected, label, example_id, source_trace_id }`) and fails with `LOUSHO_DATASET_UNAVAILABLE` when the daemon cannot serve it. Studio's `#/evals/datasets` lists, creates and browses datasets and deletes examples, and a trace's **Add to dataset** copies it into one. See [Datasets](docs/observability.md#datasets) and [Evals](docs/evals.md#from-the-daemon).
+- Online evaluator rules: `/api/v1/evaluators` holds rules with a `filter` on agent, status and name, a `sample_rate` and an `enabled` switch. Each enabled rule scores new traces as their root ends, as `source: online` scores named after the rule. See [Online evaluator rules](docs/observability.md#online-evaluator-rules).
+- Code evaluators: a `code` rule is a function `(trace) => number | { value, comment }` the daemon runs in the isolate sandbox. It needs `isolated-vm`, plus `NODE_OPTIONS=--no-node-snapshot` on Node 20+; without them code rules are skipped with one warning. `POST /api/v1/evaluators/:id/test` dry-runs a rule on one trace without writing a score. See [Online evaluators](docs/observability.md#online-evaluators).
+- Judge evaluators: a `judge` rule is a model and a rubric. The daemon queues a job per matching trace, and Agent Forge runs the judge with its provider keys (a key saved in Settings, or the provider's env var; the saved key wins), so judges run only while Studio is running. `LOUSHO_STUDIO_JUDGE=0` turns Studio's judge loop off. Studio's `/observe/api/v1/evaluators/:id/test` dry-runs judge rules too. See [Judge rules](docs/observability.md#judge-rules-through-agent-forge).
+- Studio Evaluators screen (`#/evals/evaluators`): lists the rules with their last run and runs in the last 24 hours, and creates, edits, enables, disables and deletes them. **Test on trace** dry-runs a rule on one trace id. The judge kind stays disabled until Studio has a provider key, saved in Settings or set in the provider's env var. See [Evaluators](docs/agent-forge.md#evaluators).
+- Annotation queues: `/api/v1/queues` holds named lists of traces with an optional rubric (`name`, `description`, `min`, `max` per score); `GET /api/v1/queues/:id/next` returns the oldest pending item and `POST …/items/:item/complete` writes its scores as human scores (or skips it). Studio's `#/evals/queues` creates queues and reviews them one trace at a time beside the rubric (Enter submits, `S` skips), and a trace's **Add to queue** adds it. See [Annotation queues](docs/observability.md#annotation-queues).
+- Human feedback scores: `POST /api/v1/scores` `{ trace_id, obs_id?, name, value, comment? }` adds a `source: human` score (`201 { score_id }`; unknown trace or step `404`). Each eval case also records one `<evalName>` score: 1 when no gate check failed and the case did not throw, else 0 with the failed gates in the comment. See [The observe daemon](docs/observability.md#the-observe-daemon).
+- Prompt labels: a label (`prod`, `staging`) points at one version; `lousho prompt label <name> <label> <version>` (or `PUT /api/v1/prompts/:name/labels/:label`) moves it, `push --labels` sets it on the pushed version, and `createAgent({ prompt: 'name@prod' })`, `lousho prompt get name@prod` and `GET /api/v1/prompts/:name/:ref` resolve it. A push whose text and variables equal the latest version's mints nothing (`unchanged: true`). See [`lousho prompt`](docs/cli.md#lousho-prompt).
+- Trace list API: `GET /api/v1/traces` takes `from`/`to` (half-open ms), `q` (substring of name, input or output), `min_duration_ms`, `model`, `prompt` (`<name>[@<version>]`), `sort` (`start`, `duration`, `tokens`, `cost`) and `order`, and rows add `agent_name`, `model`, `error_message` and input/output previews. See [Trace list API](docs/observability.md#trace-list-api).
+- Sessions API: `GET /api/v1/sessions` (filter by agent, status and start window, paged) and `GET /api/v1/sessions/:id` (the session and its traces, oldest first) with per-session trace counts, tokens, cost and last input. See [Sessions API](docs/observability.md#sessions-threads-api).
+- Retention: once a day the daemon deletes finished traces whose last observation ended more than `lousho observe start --retention-days N` ago (default 30, `0` keeps everything), with their scores and prompt-use links, plus older scores whose trace is gone and sessions that are not `active` and no run references. A trace with a run still open and an `active` session are kept. Rollups, prompts, hub entries, the agent registry, tokens, datasets, evaluator rules and annotation queues are never pruned. `lousho observe prune --older-than <N>d` prunes now and runs `VACUUM`. See [Retention](docs/observability.md#retention).
+- `lousho observe export [--since 7d] [--agent X] [--out file]` writes traces as JSONL (`{ trace, observations, scores }` per line) straight from the db, without a running daemon. See [Export](docs/observability.md#export).
+- Insights series carry `p95_latency_ms` and `error_rate` per agent and bucket. A db written by an older daemon lacks them for past buckets: run `lousho observe rebuild-rollups` once to backfill. See [The observe daemon](docs/observability.md#the-observe-daemon).
+- Heartbeats carry the reporter's dropped-event count, and each `/api/v1/agents` row shows it as `dropped` (`null` for older SDKs); Studio's Monitoring fleet table shows it. See [The observe daemon](docs/observability.md#the-observe-daemon) and [Monitoring](docs/agent-forge.md#monitoring).
 - Prompt registry: `createAgent({ prompt: 'name@version' })` or `LOUSHO_PROMPT` resolves versioned instructions from the daemon at build; every run records the version used. `lousho prompt push|list|get` manages it. When the daemon is unreachable, embedded `instructions` are used; with no `instructions` the agent fails closed: every run rejects with `LOUSHO_PROMPT_UNRESOLVED` instead of running without a system prompt. A failed fetch is not cached for good: runs in the next 30 seconds resolve at once, then the next run fetches again (waiting up to 2.5 seconds).
 - Context hub: `hub()` / `hubTools()` publish and query namespaced shared memory scoped `private` (the writing agent) or `fleet` (every agent on the daemon).
 - Fleet monitoring: heartbeat-derived agent status, `agent.deregister` on graceful exit, a crashed agent's open sessions and unfinished observations flip to `interrupted`, and webhook alerts (`--webhook`, `alerts.json`) on down transitions and error-rate thresholds.
 - Eval scores report to the daemon: every `AssertionResult` lands as a score row on the exercised run's trace (`source: eval` or `judge`); `lousho eval --observe` forces emission. Each case's `eval` span and score rows are flushed when the case finishes, each wait capped at 1 second; the scores share the reporter's queue, so they go out after the span and are retried while the daemon is down. A wait that runs out skips the waits of the next 30 seconds of cases.
 - Reporting to the daemon: spans that finished without an error report status `ok`. A down daemon's batches are requeued and retried with exponential backoff (1 s up to 30 s); batches are capped at 1000 events and 4 MB, and a batch the daemon refuses (`400`/`413`) is dropped and counted in `dropped`. The queue holds at most 5000 events or 32 MB, oldest evicted first. A process dying of an uncaught exception or unhandled rejection sends the newest 16 MB it holds, blocking for at most 1 second; a graceful exit makes one final attempt capped at about 2 seconds. A run's root span is reported when the run starts, so a run whose process is killed is still listed and turns `interrupted`.
 - `LOUSHO_OBSERVE` accepts `1`/`true`/`on`/`yes`, an `http(s)://` URL, or `0`/`false`/`off`/`no`/`none`/`disabled`/empty (any case). Any other value warns once naming `LOUSHO_OBSERVE` and disables reporting, instead of being posted to as a URL.
-- With `LOUSHO_OBSERVE` pointing at a loopback daemon, `createAgent` captures message and tool content for the daemon by default (`captureContent: false` turns it off). Only the daemon gets it: the agent's own `exporter` (trace files, OTLP or vendor exporters, the built server's OTel export) still records no content unless the agent sets `captureContent: true`. `redactContent` does not scrub this `gen_ai.*` content; it only governs the legacy `input`/`prompt`/`args`/`result` attributes. See [Observability](docs/observability.md#message-and-argument-content-is-opt-in).
+- With `LOUSHO_OBSERVE` pointing at a loopback daemon, `createAgent` captures message and tool content for the daemon by default (`captureContent: false` turns it off). Only the daemon gets it: the agent's own `exporter` (trace files, OTLP or vendor exporters, the built server's OTel export) still records no content unless the agent sets `captureContent: true`. Captured content goes through `redactContent` first (see the scrubber entry under Changed). See [Observability](docs/observability.md#message-and-argument-content-is-opt-in).
 - Each observe daemon writes its own pidfile beside its db (`foo.db` gives `foo.pid`), so two dbs in one folder never share one. `lousho observe start` exits 1 when the port already serves another db, and `stop` refuses to signal when the daemon on the pidfile's port reports another db or answers as a different service. `/health` names the db it serves only to callers on the same machine. A daemon started with `--db foo.db` by an earlier build of this feature wrote the shared pidfile and is not found by the new `stop`: stop it before updating.
 - `lousho observe start` and Studio print the `LOUSHO_OBSERVE` value for the port and host the daemon really uses (`LOUSHO_OBSERVE=http://127.0.0.1:7788` on a non-default port), and the Studio proxy answers `/observe/__target` with the daemon URL reporters should use.
-- Agent Forge navigates with hash routes (`#/traces/<id>`: a trace has a link of its own, and back and forward work), has a left navigation instead of the Build/Observe toggle, and shows the Build controls only on Agents.
 - `sandbox({ driver: 'isolate' })` runs tool code in an `isolated-vm` V8 isolate (optional peer `isolated-vm`); `defineTool({ sandbox })` opts a tool in, timeouts and OOM map to `sandbox.error` on the tool span. `'docker'` and `'e2b'` driver names are reserved for later drivers.
 
+### Changed
+
+- **Behavior change:** content captured for traces is scrubbed for secrets by default. `redactContent` is `boolean | ((text) => string)`: unset or `true` runs a built-in scrubber over message text, system instructions, tool-call arguments and tool results that replaces API-key shapes (`sk-…`, `lsv2_…`, GitHub tokens, AWS access key ids, the token after `Bearer`) with `[redacted]`; a function replaces the scrubber (text it throws on is recorded as `[redacted]`); `false` records content as is. It catches common key formats, not every secret, so set `captureContent: false` if content must never reach the db. See [Secrets in captured content](docs/observability.md#secrets-in-captured-content).
+- **Behavior change:** sessions end. The daemon marks a session `ended` once none of its runs is open and nothing in it has started or finished for `lousho observe start --session-idle-minutes N` (default 30, `0` turns it off); `ended_at` is the last activity, and a new run makes it `active` again. Sessions in an existing db close on the first sweep that finds them idle, and ended sessions then age out with retention. See [Idle sessions](docs/observability.md#idle-sessions).
+- **Behavior change:** the daemon keeps 30 days of runs by default. On its first sweep after start, and daily after that, it deletes traces whose observations all finished more than `--retention-days` ago, with their scores; a trace with a run still open, and a session still `active`, is kept. Start it with `--retention-days 0` to keep an existing db's history. See [Retention](docs/observability.md#retention).
+
+### Fixed
+
+- Insights runs/day averages over the days an agent was active in the window, so an agent first seen today with 12 runs reports 12/day instead of 12/7.
+- An open run's root start is kept when the reporter's buffer trims or a crash flush cuts to its newest 16 MB, so a run whose child spans overflow the buffer is still listed.
+- The run inspector shows `error`, not `running`, for a failed step that has no end time.
+
+### Documentation
+
+- [Agent Forge](docs/agent-forge.md) documents the left navigation, every screen, the deep-link routes and the keyboard shortcuts, with screenshots in `docs/images`; [Observability](docs/observability.md), [CLI](docs/cli.md) and [Evals](docs/evals.md) link to the screens. The screenshots come from an opt-in Playwright spec: `LOUSHO_DOC_SHOTS=1 npx playwright test -c apps/agent-forge/playwright.config.ts docShots`. See [Keyboard](docs/agent-forge.md#keyboard) and [Routes and deep links](docs/agent-forge.md#routes-and-deep-links).
+
 ## [1.0.0-alpha.29] - 2026-10-21
 
 ### Changed