diff --git a/README.md b/README.md index 22aa449..cbbe586 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,8 @@ npm run sync -- /path/to/agent-sdk ``` The script (`scripts/sync-sdk-docs.mjs`) adds front matter, rewrites links -between pages, points links to source files at GitHub, and escapes what MDX +between pages, points links to source files at GitHub, copies the SDK's +`docs/images/` to `images/` (and points image links there), and escapes what MDX would otherwise read as JSX. It also drops internal tracker ids (such as `LOU-...` tickets) from the prose, since they mean nothing to a reader of the site, and prints any it could not remove. It reports any SDK doc that has no diff --git a/agent-forge.mdx b/agent-forge.mdx index d73f06d..c1bae76 100644 --- a/agent-forge.mdx +++ b/agent-forge.mdx @@ -3,16 +3,23 @@ title: "Agent Forge" --- {/* Generated from docs/agent-forge.md in LinuxDevil/agent-sdk by scripts/sync-sdk-docs.mjs. Do not edit here. */} -Agent Forge (`apps/agent-forge`) is this SDK's companion visual dashboard: -build an agent's graph on a canvas, run it, watch it execute in a live -debug console, chat with it, and author sandboxed pre/post hooks - all -reading and writing the exact same `AgentSpec` YAML that `lousho dev` and -`lousho build` use. It's launched with one command, `lousho studio`, and -ships as part of this SDK's npm package. - -This doc covers: installation, the `lousho studio` quickstart, a -first-agent walkthrough, hook authoring, time travel (replaying a run from -a past step), and how settings/secrets/deploy wiring works. +Agent Forge (`apps/agent-forge`, also called the Studio) is this SDK's +companion dashboard: one app with a left navigation to build and run agents, +try prompts, and read what your agents did from the observe daemon. Build an +agent's graph on a canvas, run it, chat with it and author sandboxed +pre/post hooks (all reading and writing the same `AgentSpec` YAML that +`lousho dev` and `lousho build` use), try prompts in a playground, and read +the [observe daemon](/observability#the-observe-daemon)'s traces, +threads, fleet monitoring, eval scores, datasets, annotation queues, prompt +registry and context hub. It's launched with one command, +`lousho studio`, and ships as part of this SDK's npm package. + +This doc covers: installation, the `lousho studio` quickstart, the +[navigation and screens](#navigation), a first-agent walkthrough, hook +authoring, time travel (replaying a run from a past step), the playground +endpoint, and how settings/secrets/deploy wiring works. + +![The Agents screen: nav, canvas, inspector and the bottom drawer](/images/studio-agents.png) ## Installation @@ -37,12 +44,19 @@ npx lousho studio ``` This starts one local server and prints its URL (default -`http://127.0.0.1:4750/?token=`). Open that exact URL in a browser - you'll see the canvas, -the left rail (your saved agents + a node/hook palette), the Inspector -(right), and a bottom drawer with Chat/Logs/Trace/Output/Settings tabs. -Below 860px wide the rail and the Inspector become slide-over panels you -open from the top bar, and Debug, Import, Export and Save move into its -**More** (`⋯`) menu. +`http://127.0.0.1:4750/?token=`). Open that exact URL in a browser. +It lands on the Agents screen (`#/agents`): the left navigation, the build +toolbar (Run, Stop, Debug, Import, Export, Save and the provider pill), the +agent rail (your saved agents plus a node/hook palette), the canvas, the +Inspector (right), and a bottom drawer with Chat, Logs, Trace, Output, +History and Settings tabs. The other screens are one click away in the +navigation; see [Navigation](#navigation). + +Below 860px wide the navigation becomes a drawer opened from the +**Navigation** (menu) button at the top; picking a screen, the scrim or Esc +closes it. On the Agents screen the agent rail and the Inspector become +slide-over panels you open from the top bar, and Debug, Import, Export and +Save move into its **More** (`⋯`) menu. Useful flags: @@ -89,7 +103,7 @@ ran `lousho studio` from (agent YAML files under `.lousho/agents/`, checkpoints and approvals alongside them) - the same `.lousho/` layout `lousho dev`/`lousho build` use. -The left rail lists exactly the agents in that `.lousho/agents/`: a YAML +The Agents screen's agent rail lists exactly the agents in that `.lousho/agents/`: a YAML file you add there shows up, and an agent you create in the studio is written there at once (autosave and **Save** both write the file), so it can be chatted with straight away. Another project, opened in its own @@ -117,10 +131,353 @@ longer parses is skipped with a warning in the server log. `lousho studio` auto-detects based on whether `apps/agent-forge/dist-server` exists. +## Navigation + +The left navigation lists every screen, grouped: + +| Group | Item | Screen | +| --- | --- | --- | +| Build | **Agents** | The builder: canvas, Inspector, bottom drawer. The build toolbar (Run, Stop, Debug, Import, Export, Save, provider pill) shows only here. | +| Build | **Playground** | Try messages against a model without editing an agent. | +| Observe | **Traces** | Every run the observe daemon holds, filtered and sorted; a row opens the trace. | +| Observe | **Threads** | Sessions (conversations): one row per session, one turn per run. | +| Observe | **Monitoring** | The fleet's health and charts over a time window. | +| Evaluate | **Evaluations** | Tabs for Scores, Datasets, Evaluators and Annotation queues. | +| Assets | **Prompts** | The prompt registry: versions, labels, diffs. | +| Assets | **Context Hub** | Shared key-values agents publish to the daemon. | +| | **Settings** | Provider keys, settings profiles and deploy. | + +Off the Agents screen the top bar names the screen and keeps the live run +status pill, nothing else. The last nav item is the collector status (see +[The collector](#the-collector-observe-daemon)). + +### Routes and deep links + +Every screen has a hash route, and a screen's filters live in the URL's +query string, so a reload, a bookmark or a pasted link opens the same screen +with the same filters. Defaults are never written (a bare `#/traces` is the +default 24 h list). Filter, sort and page edits replace the current history +entry, and opening a row pushes one, so the browser's Back returns from a +trace to the list instead of walking through filter edits. An empty or +unknown hash goes to `#/agents` (the `?token=` stays). + +| Route | Screen | Query keys | +| --- | --- | --- | +| `#/agents`, `#/agents/` | Agents, with that agent open | | +| `#/playground` | Playground | `trace` + `fromObs`, or `prompt` (see [Playground](#playground)) | +| `#/traces` | Traces list | `agent`, `status`, `model`, `project`, `q`, `prompt` (`` or `@`), `from` + `to` (ms epoch, half-open), `sort` (`duration`, `tokens`, `cost`) + `order` (`asc`, `desc`), `offset` | +| `#/traces/` | One trace | `obs` (the selected step) | +| `#/threads` | Threads list | `agent`, `status`, `from` + `to`, `offset` | +| `#/threads/` | One thread | | +| `#/monitoring` | Monitoring | `window` (`1h`, `24h`, `7d`, `30d`; default `24h`), `agent` | +| `#/evals/scores` | Scores | `eval` (comma list), `agent`, `source` (`eval`, `judge`, `human`, `online`), `range` (default `30d`), `open` (the expanded eval) | +| `#/evals/datasets`, `#/evals/datasets/` | Datasets, one dataset | `offset` | +| `#/evals/evaluators`, `#/evals/evaluators/` | Evaluators, one rule's edit view | `kind` (`code`, `judge`), `enabled` (`true`, `false`), `offset`; on a rule, `trace` (prefills Test on trace) | +| `#/evals/queues`, `#/evals/queues/` | Annotation queues, one queue's review | `offset` (the list) | +| `#/prompts` | Prompt registry | | +| `#/prompts/`, `#/prompts/@` | One prompt (latest version by default) | `diff=,` | +| `#/hub`, `#/hub/` | Context Hub | `agent` (the "Viewing as" agent), `scope` (`private`, `fleet`) | +| `#/settings` | Settings | | + +A time range picked from a preset other than the default is written as +absolute `from`/`to` bounds, so a reload shows exactly the same rows. The +full route list is `apps/agent-forge/src/screens/routeTable.ts`. + +### Keyboard + +| Keys | Does | +| --- | --- | +| `Ctrl+K` / `Cmd+K` | Open (or close) the command palette, anywhere, even while typing. | +| `?` | Show the keyboard shortcuts (not while typing in a field). | +| `j` / `k` | Next / previous row in a focused table. | +| `Enter` | Open the selected row. | +| `Esc` | Close the palette, the shortcut sheet, a header dialog or the narrow nav drawer. | + +The command palette lists **Go to** every nav screen, **Open agent** +for each saved agent, and **Toggle theme** (light/dark, remembered in the +browser); type to filter. Paste a trace id (8 or more hex digits and +dashes) and the first entry is **Open trace** for it. Arrow keys move, +Enter runs. + +![The command palette over the Traces screen](/images/studio-command-palette.png) + +In an annotation queue's review, Enter submits the scores and `S` skips +the item (see [Annotation queues](#annotation-queues)). + +### Status words + +Every status in the Studio is the same pill: + +- Runs: `ok`, `error`, `running`, `interrupted` (the agent's process died + mid-run; see [The observe daemon](/observability#the-observe-daemon)). +- Agents: `healthy`, `stale` (missed heartbeats), `down`, `stopped`. +- Threads: `active`, `ended` (idle, see + [Idle sessions](/observability#idle-sessions)), `interrupted`. +- Scores: `pass`, `fail`. +- Builder runs: `idle`, `running`, `done`, `stopped`, `error`, + `awaiting approval`. + +### The collector (observe daemon) + +Everything under Observe, Evaluate and Assets reads the observe daemon +(`lousho observe start`) through the Studio server's `/observe` proxy. The +proxy reads only `LOUSHO_OBSERVE_URL` (default `http://127.0.0.1:7777`); +`LOUSHO_OBSERVE` does not change where the screens read (it only picks where +builder runs report, see the [walkthrough](#first-agent-walkthrough)). When +`LOUSHO_OBSERVE_TOKEN` is set, the proxy sends it to the daemon as the bearer +token on every read and write (as run reporting and the judge evaluators do), +so the screens also work against a daemon started with `--require-token`. +Without it, such a daemon answers `401` to every `/api/` read and the screens +show that error instead of data (`/health` still answers, so the collector +item says running). The browser's own credentials (the Studio token, cookies, +`Authorization`) are never passed on. The proxy only forwards reads plus the +writes the screens make (feedback scores, new prompt versions from the +Playground, prompt label moves, hub entry edits, dataset and example writes, +annotation queue writes, evaluator rule writes); every other write, ingest +above all, is refused with `405`. It always calls the daemon at +`LOUSHO_OBSERVE_URL`, whatever host the request line names. + +The last nav item says whether the daemon answers: **Collector: checking…** +until the first probe, **Collector: running at ``**, or **Collector: +not reachable**. The probe keeps polling. While the daemon is down, each +daemon screen shows the command to start it instead of an error, keeps its +URL, and swaps the data back in by itself once the daemon answers. The +Agents screen and Playground runs do not need the daemon. + +## Screens + +### Agents + +The builder: see the [First-agent walkthrough](#first-agent-walkthrough). +`#/agents/` opens that agent; leaving the screen (or switching agents) +with unsaved changes asks first. Runs started here report to the observe +daemon, and the Run status pill's **Open trace** link opens the run on the +Trace screen. + +### Playground + +`#/playground` (Build > Playground) tries messages against a model +without editing an agent: + +![The Playground, loaded from a trace's model call](/images/studio-playground.png) + +- **Messages**: system, user and assistant rows. Every `{name}` placeholder + in any row gets a **Variables** input, in the order the names first appear. +- **Model**: a provider (the ones with a key under Settings > Provider keys, + plus `ollama` and `mock`; `mock` when nothing is configured) and a model + name, sent as `provider/model`. +- **Tools**: pick a saved agent to borrow its tools, then switch each one on + or off. +- **Run** streams the reply as it arrives; **Stop** aborts it. When the run + finishes, **Open trace** goes to its trace. +- **Save as new prompt version** pushes the system message (and its + variables) to the prompt registry, under the loaded prompt's name or one + you type. It is disabled while there is no system message. + +Two links pre-fill it. **Open in Playground** on a model call (generation) +in a trace opens `#/playground?trace=&fromObs=`, which loads +that generation's messages and model. `#/playground?prompt=[@