Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,16 @@ jobs:

- run: pip install -r requirements-docs.txt

# Diagrams are hand-placed figures in docs/figures/, rendered to PNG
# (docs/brand.md says why). mkdocs.yml no longer has a mermaid fence, so
# one would build as a plain code block rather than fail; this makes it fail.
- name: No mermaid
run: |
if grep -rn --include='*.md' '^ *```mermaid' docs/; then
echo "::error::mermaid diagram under docs/: draw it as a figure in docs/figures/ instead (docs/brand.md)"
exit 1
fi

# The cookbook includes collector/examples/jobs/*.json verbatim via the
# snippets extension, so a missing or renamed example fails the build
# here as well as failing the collector's own suite.
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,20 @@ a CI job fails when they disagree.

## [Unreleased]

### Changed

- **The explore docs' three diagrams are drawn, not generated.** The night,
one mission's sequence and the code map were mermaid blocks; they are now
figures in `docs/figures/` rendered to PNG like the rest
(`explore-night`, `explore-step`, `explore-code`). The docs workflow fails on
a mermaid fence under `docs/`, and `mkdocs.yml` no longer configures one.
- **The workloads figure shows all 34 shipped workloads.** It had 28; `enrol`,
`explore`, `upgrade-test`, `size-report`, `tvloop` and `llm-eval` are added.
- **`devices.png` is the new device drawings, and there is a `findings.png`.**
Both are captured read-only from a running brain by the new
`npm run shoot:live`, filtered to what was online and to the `bug-garden`
fixture app; docs/brand.md says why a throwaway collector cannot make them.

### Fixed

- **`fleet skill` failed on an installed 0.8.0.** The 0.8.0 archives carry
Expand Down
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,10 @@ something a runner can do without the collector shipping a release.

## What it can run

![Twenty-eight workloads in three columns by who claims them: on the device, on a host, and on a machine](docs/img/workloads.png)
![Thirty-four workloads in three columns by who claims them: on the device, on a host, and on a machine](docs/img/workloads.png)

Thirty-four workloads (the figure predates six of them, `explore` and `tvloop`
among them), and the column a workload sits in is the answer to "who can
physically do this". A benchmark runs inside the app on the phone. An
Thirty-four workloads, and the column a workload sits in is the answer to
"who can physically do this". A benchmark runs inside the app on the phone. An
install needs a cable and a Mac. A build needs a checkout and a toolchain.
Each has [its own page](https://addisdev.github.io/fleet-runner/workloads/),
saying what it measures and what it refuses to guess.
Expand Down
1 change: 1 addition & 0 deletions collector/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"chaos": "tsx scripts/chaos.ts",
"assets": "tsx scripts/render-assets.ts",
"shoot:dash": "tsx scripts/shoot-dash.ts",
"shoot:live": "tsx scripts/shoot-live.ts",
"shoot:motion": "tsx scripts/shoot-motion.ts"
},
"dependencies": {
Expand Down
45 changes: 45 additions & 0 deletions collector/scripts/shoot-live.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// The two dashboard screenshots a throwaway collector cannot honestly produce,
// captured read-only from a running brain.
//
// npm run shoot:live -- --url http://127.0.0.1:18788
//
// shoot-dash.ts starts a fresh collector with one machine agent, so its Devices
// screen is a single laptop and its Findings screen is empty: the device
// drawings need a shelf of real devices, and findings need a real explore night
// with a real model. Seeding either would be inventing data. This script only
// opens two pages on a brain that already has both, and writes nothing to it.
//
// Both views are filtered, and the filter is visible in the picture:
// devices.png status=online, so the shelf shows what is plugged in now
// findings.png app=bug-garden, the fixture app whose defects are planted on
// purpose; findings against a real product stay out of the docs
import path from "node:path";
import { chromium } from "playwright";

const at = process.argv.indexOf("--url");
const BASE = (at === -1 ? process.env.FLEET_URL : process.argv[at + 1])?.replace(/\/$/, "");
if (!BASE) {
console.error("usage: npm run shoot:live -- --url http://brain:8788");
process.exit(2);
}
const OUT = path.resolve(import.meta.dirname, "../../docs/img");

const SHOTS: [route: string, name: string][] = [
["/dash/devices?status=online", "devices"],
["/dash/findings?app=bug-garden", "findings"],
];

const browser = await chromium.launch();
try {
const page = await browser.newPage({ viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 });
await page.emulateMedia({ colorScheme: "dark" });
for (const [route, name] of SHOTS) {
await page.goto(`${BASE}${route}`, { waitUntil: "networkidle" });
// The dashboard renders from its own fetches, not from the document.
await page.waitForTimeout(2000);
await page.screenshot({ path: path.join(OUT, `${name}.png`) });
console.log(`wrote ${name}.png from ${BASE}${route}`);
}
} finally {
await browser.close();
}
25 changes: 20 additions & 5 deletions docs/brand.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,16 @@ dependency, so nothing new is installed.
| `img/architecture.png` | 2560×1280 | `figures/architecture.html` |
| `img/lifecycle.png` | 2560×1360 | `figures/lifecycle.html` |
| `img/protocol.png` | 2560×1120 | `figures/protocol.html` |
| `img/workloads.png` | 2560×1200 | `figures/workloads.html` |
| `img/workloads.png` | 2560×1360 | `figures/workloads.html` |
| `img/platforms.png` | 2560×1400 | `figures/platforms.html` |
| `img/explore-night.png` | 2560×1400 | `figures/explore-night.html` |
| `img/explore-step.png` | 2560×2040 | `figures/explore-step.html` |
| `img/explore-code.png` | 2560×1384 | `figures/explore-code.html` |
| `img/first-result.png`, `alerts.png`, `visual.png` | 1440 wide | `npm run shoot:dash` |
| `img/fanout.gif` | 1280 wide, 18 s | `npm run shoot:motion` |
| `img/shelf-banner.jpg`, `shelf-social.jpg` | 2560×1280, JPEG | `figures/shelf-banner.html`, `figures/shelf-social.html` — **skipped until [the photograph](#the-shelf-photograph) exists** |
| `img/overview.png`, `devices.png`, `jobs.png`, `results.png` | | Dashboard pages, captured with Playwright against a running collector |
| `img/overview.png`, `jobs.png`, `results.png` | | Dashboard pages, captured with Playwright against a running collector |
| `img/devices.png`, `findings.png` | 1440×900 | `npm run shoot:live -- --url <brain>`, read-only against the lab's running brain |
| `img/runner-web.png` | | `npm run shoot:runner-web`, against a throwaway collector |
| `runner-ios/.../AppIcon.appiconset/icon-1024.png` | 1024×1024, no alpha | `img/icon-source.svg` |
| `runner-android/.../mipmap*/ic_launcher.xml` | vector | Drawn as Android vector drawables from the same geometry |
Expand All @@ -99,7 +103,8 @@ The diagrams are placed by hand rather than generated by mermaid because a
layout engine is for people who do not want to place boxes, and a diagram
somebody will look at for ten seconds is placed boxes. Mermaid also renders in
GitHub's own theme, which is the one thing on the page that cannot be made to
match the rest.
match the rest. No page uses it any more: the docs workflow fails on a
mermaid fence anywhere under `docs/`, and the site no longer loads it.

### The live captures

Expand Down Expand Up @@ -194,9 +199,9 @@ holds.
gets believed and turns out to be invented costs more than a missing
picture.

!!! note "Two of these were captured against different fleets, on purpose"
!!! note "These were captured against different fleets, on purpose"

`overview.png`, `devices.png` and `jobs.png` show a **live** two-device
`overview.png` and `jobs.png` show a **live** two-device
fleet — a laptop and an iPhone simulator, both running the real agents,
with a thermal run in progress and a `self-check` correctly failing because
the agent was started by hand rather than by launchd. They were captured
Expand All @@ -207,6 +212,16 @@ holds.
captured while the laptop was on a different network from the shelf. That
set was honest and read as abandoned.

**`devices.png` and `findings.png` are from the lab's running brain**, on
15 October 2026, because a throwaway collector cannot have either: the
device drawings need a shelf of real hardware, and a finding needs a real
`explore` night with a real model. `npm run shoot:live` only opens the two
pages; it writes nothing. Devices is filtered to what was online (five of
twenty registered), and Findings to `bug-garden`, the fixture app whose
defects are planted on purpose, so no real product's open bugs are
published. Both filters are visible in the picture. The four open alerts in
the banner are real too: shelf devices that had not checked in.

**`results.png` is deliberately not from that fleet.** It shows real
llama.cpp measurements from an SM-X930 — 125.0 prefill and 47.4 decode
tok/s on a Dimensity 9400 — which is stored history a fresh database cannot
Expand Down
2 changes: 2 additions & 0 deletions docs/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ SQLite file. One thing to keep alive, one URL, no CORS.
| **Jobs** — the queue, with a composer and cancel/retry | **Results** — benchmark trends, eval accuracy, UI-test matrices, drain curves |
| ![Visual](img/visual.png) | ![Alerts](img/alerts.png) |
| **Visual** — every page across every profile, diffed against its accepted baseline | **Alerts** — one row per condition for as long as it holds, with acknowledge and snooze |
| ![Findings](img/findings.png) | |
| **Findings** — what an [`explore`](explore/index.md) night reproduced, with a verdict for each | |

These screenshots are from 2026-09-05. They predate the device drawings
described below, and there is none of the Findings screen.
Expand Down
2 changes: 2 additions & 0 deletions docs/explore/findings.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ What a night leaves behind, and how your judgement of it steers the next one.

**Findings** in the dashboard's top bar, or <kbd>g</kbd> <kbd>f</kbd>.

![The Findings page filtered to the bug-garden fixture app: 59 open findings, none triaged yet, each row with its severity, check, title and screen, how often and on how many builds it was seen, "reproduced 2/2", and when it was first and last seen](../img/findings.png)

The **list** can be filtered by app, check and status (open, triaged, all).
Each row shows severity, check, title, screen, how many times and on how many
builds it has been seen, how it replayed ("reproduced 2/2", "flaky 1/2",
Expand Down
27 changes: 1 addition & 26 deletions docs/explore/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,7 @@ One `explore` job is one night on one or more devices. For each device it runs
mission cards in order until the night's budget is spent, then confirms what it
found and files what holds up.

```mermaid
sequenceDiagram
participant E as Executor (explore)
participant D as Device (actuator)
participant M as Model (driver)
participant J as Judge
participant C as Collector
E->>D: reset (install, clear data, launch, sign in)
loop each step, until answer, budget, time or stuck
E->>D: observe (screenshot, tree, foreground, focus)
E->>E: screen map, oracle checks
E-->>J: new screen? judge it (once per screen)
E->>M: screenshot + harness notes
M-->>E: tool calls
E->>E: snap and leash each action
E->>D: act, settle
E->>D: crashes since last step
E-->>C: beacon (renews the lease)
end
E->>J: goal met? (cards with a success line)
loop each candidate, worst first
E->>D: reset, replay the steps, ask the check again (x2)
end
E->>C: POST /findings (reproduced, or a crash)
E->>C: result row with metrics
```
![One mission as a sequence between the executor, the device, the model, the judge and the collector: reset the device; then each step observe, place the screen and run the checks, have the judge look at a new screen, send the model the screenshot and get tool calls back, snap and leash the actions, act, read the crashes and beacon the collector; after the mission ask the judge whether the goal was met, replay each candidate twice, and post the findings and the result row](../img/explore-step.png)

## Before the first step

Expand Down
13 changes: 1 addition & 12 deletions docs/explore/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,7 @@ in Spanish. A model reads each screenshot and picks the next action; the
harness checks that action, performs it, and looks for trouble after every
step. Nothing it sees is believed until it happens again on a clean install.

```mermaid
flowchart TD
A[Tonight's build] --> B[Mission cards]
B --> C{{"Look, decide, act<br/>(the model)"}}
C --> D[Checks after every step]
D -->|candidate| E[Replay twice on a clean install]
E -->|reproduced| F[Finding]
E -->|not reproduced| X[Dropped]
F --> G[Findings page, digest, optional issue]
G -->|your verdict| H[Precision per check]
H -->|under 30%| D
```
![One night of exploratory QA: tonight's build and the mission cards go to the model, which looks, decides and acts while checks look for trouble after every step; a candidate is replayed twice on a clean install and becomes a finding only if it reproduces at least once or was a crash with a log, otherwise it is dropped; findings go to the Findings page, the digest and optionally an issue, and your verdicts give each check a precision that switches it off for that app under 30% once ten are judged](../img/explore-night.png)

## What you get in the morning

Expand Down
17 changes: 1 addition & 16 deletions docs/explore/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,22 +47,7 @@ The model speaks 0–1000; `convertCall` maps to pixels; each actuator maps
pixels to its own input units (device pixels on Android, points on Apple
devices).

```mermaid
flowchart TB
subgraph Executor
IDX[index.ts<br/>per device, per mission] --> LOOP[loop.ts<br/>runMission / confirm]
LOOP --> MODEL[model.ts<br/>Conversation]
LOOP --> JUDGE[judge.ts]
LOOP --> LEASH[leash.ts]
LOOP --> MAP[screenmap.ts]
LOOP --> ACT{{actuators/}}
IDX --> REP[report.ts, replay.ts]
end
ACT --> ADB[adb] & FD[FleetDriver over HTTP] & TV[tvloop adapter]
MODEL & JUDGE --> GW[(gateway<br/>pilot / vision)]
IDX -->|POST /findings, /results, /artifacts| COL[(Collector)]
COL --> DASH[Findings page]
```
![The explorer's code map: index.ts calls loop.ts and report.ts and replay.ts; loop.ts uses model.ts, judge.ts, leash.ts and screenmap.ts each step and actuators/ to reach the device over adb, FleetDriver or tvloop; model.ts and judge.ts call the gateway's pilot and vision models; index.ts posts findings, results and artifacts to the collector and its Findings page](../img/explore-code.png)

## Tests

Expand Down
88 changes: 88 additions & 0 deletions docs/figures/explore-code.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
<!doctype html>
<meta charset="utf-8">
<title>explore-code</title>
<link rel="stylesheet" href="brand.css">
<!--
The explorer's code map: which file calls which, and the three things outside
the executor it talks to. Every box is a file or directory that exists under
collector/src/workloads/explore/ or a process it reaches; the arrows are
imports and requests, not data flow in general.

Rendered by `npm run assets` in collector/ to docs/img/explore-code.png.
-->
<div class="figure" style="width:1280px;height:692px">
<svg viewBox="0 0 1280 692" width="1280" height="692" role="img"
aria-label="Inside the executor: index.ts runs each device and mission, calls loop.ts, and uses report.ts and replay.ts. loop.ts runs a mission and its replay, using model.ts, judge.ts, leash.ts and screenmap.ts on each step, and the actuators directory to reach the device: adb for Android and Fire TV, FleetDriver over HTTP for Apple devices, tvloop for a Roku. model.ts and judge.ts call the gateway's pilot and vision models. index.ts posts findings, results and artifacts to the collector, whose Findings page is where the verdicts are given. types.ts is the contract they all agree on.">
<g transform="translate(0 -48)">

<!-- ─── the executor ──────────────────────────────────────────────── -->
<rect class="l-structure" x="48" y="96" width="812" height="604" rx="12"/>

<rect x="72" y="152" width="360" height="76" rx="10" fill="var(--panel)" stroke="var(--amber)" stroke-width="1.5"/>
<text class="t-title" x="90" y="186">index.ts</text>
<text class="t-mono" x="90" y="210">per device, per mission; confirm, file</text>

<rect class="l-panel" x="492" y="152" width="344" height="76" rx="10"/>
<text class="t-title" x="510" y="186">report.ts · replay.ts</text>
<text class="t-mono" x="510" y="210">steps in words, the replay flow</text>
<path class="l-protocol" d="M432 190H484" marker-end="url(#arrow-amber)"/>

<rect class="l-panel" x="72" y="292" width="360" height="76" rx="10"/>
<text class="t-title" x="90" y="326">loop.ts</text>
<text class="t-mono" x="90" y="350">runMission · confirm</text>
<path class="l-protocol" d="M252 228V284" marker-end="url(#arrow-amber)"/>

<rect class="l-panel" x="492" y="268" width="344" height="176" rx="10"/>
<text class="t-mono" x="510" y="296">each step</text>
<g>
<rect class="l-structure" x="508" y="312" width="148" height="44" rx="7"/>
<text class="t-mono-light" x="582" y="339" text-anchor="middle" style="font-size:15px">model.ts</text>
<rect class="l-structure" x="672" y="312" width="148" height="44" rx="7"/>
<text class="t-mono-light" x="746" y="339" text-anchor="middle" style="font-size:15px">judge.ts</text>
<rect class="l-structure" x="508" y="372" width="148" height="44" rx="7"/>
<text class="t-mono-light" x="582" y="399" text-anchor="middle" style="font-size:15px">leash.ts</text>
<rect class="l-structure" x="672" y="372" width="148" height="44" rx="7"/>
<text class="t-mono-light" x="746" y="399" text-anchor="middle" style="font-size:15px">screenmap.ts</text>
</g>
<path class="l-protocol" d="M432 330H484" marker-end="url(#arrow-amber)"/>

<rect class="l-panel" x="72" y="504" width="360" height="76" rx="10"/>
<text class="t-title" x="90" y="538">actuators/</text>
<text class="t-mono" x="90" y="562">android.ts · apple.ts · roku.ts</text>
<path class="l-protocol" d="M252 368V496" marker-end="url(#arrow-amber)"/>

<text class="t-mono-light" x="72" y="628">types.ts is the contract every file here agrees on.</text>
<text class="t-mono" x="72" y="652">Outside model.ts, every coordinate is in screenshot pixels.</text>
<text class="t-mono-amber" x="72" y="684">the executor · collector/src/workloads/explore/</text>

<!-- ─── what it reaches ───────────────────────────────────────────── -->
<path class="l-protocol" d="M300 152V116H1200V160" marker-end="url(#arrow-amber)"/>
<text class="t-mono" x="900" y="148">findings, results, artifacts</text>

<rect x="900" y="168" width="332" height="76" rx="10" fill="var(--panel)" stroke="var(--amber)" stroke-width="1.5"/>
<text class="t-title" x="918" y="202">collector</text>
<text class="t-mono" x="918" y="226">the findings store</text>

<path class="l-structure" d="M1066 244V276" marker-end="url(#arrow-muted)"/>
<rect class="l-panel" x="900" y="284" width="332" height="76" rx="10"/>
<text class="t-title" x="918" y="318">Findings page</text>
<text class="t-mono" x="918" y="342">where your verdicts go</text>

<path class="l-protocol" d="M836 424H892" marker-end="url(#arrow-amber)"/>
<rect class="l-panel" x="900" y="386" width="332" height="76" rx="10"/>
<text class="t-title" x="918" y="420">gateway</text>
<text class="t-mono" x="918" y="444">pilot drives · vision judges</text>

<path class="l-cable" d="M432 542H892" marker-end="url(#arrow-muted)"/>
<rect class="l-panel" x="900" y="516" width="332" height="148" rx="10"/>
<text class="t-title" x="918" y="550">the device</text>
<text class="t-mono-light" x="918" y="582">adb</text>
<text class="t-mono" x="1074" y="582">Android, Fire TV</text>
<text class="t-mono-light" x="918" y="610">FleetDriver</text>
<text class="t-mono" x="1074" y="610">Apple, over HTTP</text>
<text class="t-mono-light" x="918" y="638">tvloop</text>
<text class="t-mono" x="1074" y="638">Roku</text>
</g>
</svg>
</div>
<script src="symbols.js"></script>
Loading
Loading