diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index dd68603..59aa5a3 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -41,6 +41,16 @@ jobs: - run: pip install -r requirements-docs.txt + # Diagrams are hand-placed figures in docs/figures/, rendered to PNG + # (docs/brand.md says why). mkdocs.yml no longer has a mermaid fence, so + # one would build as a plain code block rather than fail; this makes it fail. + - name: No mermaid + run: | + if grep -rn --include='*.md' '^ *```mermaid' docs/; then + echo "::error::mermaid diagram under docs/: draw it as a figure in docs/figures/ instead (docs/brand.md)" + exit 1 + fi + # The cookbook includes collector/examples/jobs/*.json verbatim via the # snippets extension, so a missing or renamed example fails the build # here as well as failing the collector's own suite. diff --git a/CHANGELOG.md b/CHANGELOG.md index b1b6bbf..fbeaeb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,20 @@ a CI job fails when they disagree. ## [Unreleased] +### Changed + +- **The explore docs' three diagrams are drawn, not generated.** The night, + one mission's sequence and the code map were mermaid blocks; they are now + figures in `docs/figures/` rendered to PNG like the rest + (`explore-night`, `explore-step`, `explore-code`). The docs workflow fails on + a mermaid fence under `docs/`, and `mkdocs.yml` no longer configures one. +- **The workloads figure shows all 34 shipped workloads.** It had 28; `enrol`, + `explore`, `upgrade-test`, `size-report`, `tvloop` and `llm-eval` are added. +- **`devices.png` is the new device drawings, and there is a `findings.png`.** + Both are captured read-only from a running brain by the new + `npm run shoot:live`, filtered to what was online and to the `bug-garden` + fixture app; docs/brand.md says why a throwaway collector cannot make them. + ### Fixed - **`fleet skill` failed on an installed 0.8.0.** The 0.8.0 archives carry diff --git a/README.md b/README.md index c46f42b..76957e9 100644 --- a/README.md +++ b/README.md @@ -138,11 +138,10 @@ something a runner can do without the collector shipping a release. ## What it can run -![Twenty-eight workloads in three columns by who claims them: on the device, on a host, and on a machine](docs/img/workloads.png) +![Thirty-four workloads in three columns by who claims them: on the device, on a host, and on a machine](docs/img/workloads.png) -Thirty-four workloads (the figure predates six of them, `explore` and `tvloop` -among them), and the column a workload sits in is the answer to "who can -physically do this". A benchmark runs inside the app on the phone. An +Thirty-four workloads, and the column a workload sits in is the answer to +"who can physically do this". A benchmark runs inside the app on the phone. An install needs a cable and a Mac. A build needs a checkout and a toolchain. Each has [its own page](https://addisdev.github.io/fleet-runner/workloads/), saying what it measures and what it refuses to guess. diff --git a/collector/package.json b/collector/package.json index 57ba697..acb267d 100644 --- a/collector/package.json +++ b/collector/package.json @@ -26,6 +26,7 @@ "chaos": "tsx scripts/chaos.ts", "assets": "tsx scripts/render-assets.ts", "shoot:dash": "tsx scripts/shoot-dash.ts", + "shoot:live": "tsx scripts/shoot-live.ts", "shoot:motion": "tsx scripts/shoot-motion.ts" }, "dependencies": { diff --git a/collector/scripts/shoot-live.ts b/collector/scripts/shoot-live.ts new file mode 100644 index 0000000..3fc9e94 --- /dev/null +++ b/collector/scripts/shoot-live.ts @@ -0,0 +1,45 @@ +// The two dashboard screenshots a throwaway collector cannot honestly produce, +// captured read-only from a running brain. +// +// npm run shoot:live -- --url http://127.0.0.1:18788 +// +// shoot-dash.ts starts a fresh collector with one machine agent, so its Devices +// screen is a single laptop and its Findings screen is empty: the device +// drawings need a shelf of real devices, and findings need a real explore night +// with a real model. Seeding either would be inventing data. This script only +// opens two pages on a brain that already has both, and writes nothing to it. +// +// Both views are filtered, and the filter is visible in the picture: +// devices.png status=online, so the shelf shows what is plugged in now +// findings.png app=bug-garden, the fixture app whose defects are planted on +// purpose; findings against a real product stay out of the docs +import path from "node:path"; +import { chromium } from "playwright"; + +const at = process.argv.indexOf("--url"); +const BASE = (at === -1 ? process.env.FLEET_URL : process.argv[at + 1])?.replace(/\/$/, ""); +if (!BASE) { + console.error("usage: npm run shoot:live -- --url http://brain:8788"); + process.exit(2); +} +const OUT = path.resolve(import.meta.dirname, "../../docs/img"); + +const SHOTS: [route: string, name: string][] = [ + ["/dash/devices?status=online", "devices"], + ["/dash/findings?app=bug-garden", "findings"], +]; + +const browser = await chromium.launch(); +try { + const page = await browser.newPage({ viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 }); + await page.emulateMedia({ colorScheme: "dark" }); + for (const [route, name] of SHOTS) { + await page.goto(`${BASE}${route}`, { waitUntil: "networkidle" }); + // The dashboard renders from its own fetches, not from the document. + await page.waitForTimeout(2000); + await page.screenshot({ path: path.join(OUT, `${name}.png`) }); + console.log(`wrote ${name}.png from ${BASE}${route}`); + } +} finally { + await browser.close(); +} diff --git a/docs/brand.md b/docs/brand.md index 9b8e7dd..4abce47 100644 --- a/docs/brand.md +++ b/docs/brand.md @@ -72,12 +72,16 @@ dependency, so nothing new is installed. | `img/architecture.png` | 2560×1280 | `figures/architecture.html` | | `img/lifecycle.png` | 2560×1360 | `figures/lifecycle.html` | | `img/protocol.png` | 2560×1120 | `figures/protocol.html` | -| `img/workloads.png` | 2560×1200 | `figures/workloads.html` | +| `img/workloads.png` | 2560×1360 | `figures/workloads.html` | | `img/platforms.png` | 2560×1400 | `figures/platforms.html` | +| `img/explore-night.png` | 2560×1400 | `figures/explore-night.html` | +| `img/explore-step.png` | 2560×2040 | `figures/explore-step.html` | +| `img/explore-code.png` | 2560×1384 | `figures/explore-code.html` | | `img/first-result.png`, `alerts.png`, `visual.png` | 1440 wide | `npm run shoot:dash` | | `img/fanout.gif` | 1280 wide, 18 s | `npm run shoot:motion` | | `img/shelf-banner.jpg`, `shelf-social.jpg` | 2560×1280, JPEG | `figures/shelf-banner.html`, `figures/shelf-social.html` — **skipped until [the photograph](#the-shelf-photograph) exists** | -| `img/overview.png`, `devices.png`, `jobs.png`, `results.png` | | Dashboard pages, captured with Playwright against a running collector | +| `img/overview.png`, `jobs.png`, `results.png` | | Dashboard pages, captured with Playwright against a running collector | +| `img/devices.png`, `findings.png` | 1440×900 | `npm run shoot:live -- --url `, read-only against the lab's running brain | | `img/runner-web.png` | | `npm run shoot:runner-web`, against a throwaway collector | | `runner-ios/.../AppIcon.appiconset/icon-1024.png` | 1024×1024, no alpha | `img/icon-source.svg` | | `runner-android/.../mipmap*/ic_launcher.xml` | vector | Drawn as Android vector drawables from the same geometry | @@ -99,7 +103,8 @@ The diagrams are placed by hand rather than generated by mermaid because a layout engine is for people who do not want to place boxes, and a diagram somebody will look at for ten seconds is placed boxes. Mermaid also renders in GitHub's own theme, which is the one thing on the page that cannot be made to -match the rest. +match the rest. No page uses it any more: the docs workflow fails on a +mermaid fence anywhere under `docs/`, and the site no longer loads it. ### The live captures @@ -194,9 +199,9 @@ holds. gets believed and turns out to be invented costs more than a missing picture. -!!! note "Two of these were captured against different fleets, on purpose" +!!! note "These were captured against different fleets, on purpose" - `overview.png`, `devices.png` and `jobs.png` show a **live** two-device + `overview.png` and `jobs.png` show a **live** two-device fleet — a laptop and an iPhone simulator, both running the real agents, with a thermal run in progress and a `self-check` correctly failing because the agent was started by hand rather than by launchd. They were captured @@ -207,6 +212,16 @@ holds. captured while the laptop was on a different network from the shelf. That set was honest and read as abandoned. + **`devices.png` and `findings.png` are from the lab's running brain**, on + 15 October 2026, because a throwaway collector cannot have either: the + device drawings need a shelf of real hardware, and a finding needs a real + `explore` night with a real model. `npm run shoot:live` only opens the two + pages; it writes nothing. Devices is filtered to what was online (five of + twenty registered), and Findings to `bug-garden`, the fixture app whose + defects are planted on purpose, so no real product's open bugs are + published. Both filters are visible in the picture. The four open alerts in + the banner are real too: shelf devices that had not checked in. + **`results.png` is deliberately not from that fleet.** It shows real llama.cpp measurements from an SM-X930 — 125.0 prefill and 47.4 decode tok/s on a Dimensity 9400 — which is stored history a fresh database cannot diff --git a/docs/dashboard.md b/docs/dashboard.md index eeb6c64..65d56af 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -11,6 +11,8 @@ SQLite file. One thing to keep alive, one URL, no CORS. | **Jobs** — the queue, with a composer and cancel/retry | **Results** — benchmark trends, eval accuracy, UI-test matrices, drain curves | | ![Visual](img/visual.png) | ![Alerts](img/alerts.png) | | **Visual** — every page across every profile, diffed against its accepted baseline | **Alerts** — one row per condition for as long as it holds, with acknowledge and snooze | +| ![Findings](img/findings.png) | | +| **Findings** — what an [`explore`](explore/index.md) night reproduced, with a verdict for each | | These screenshots are from 2026-09-05. They predate the device drawings described below, and there is none of the Findings screen. diff --git a/docs/explore/findings.md b/docs/explore/findings.md index 52dc2d7..56ca534 100644 --- a/docs/explore/findings.md +++ b/docs/explore/findings.md @@ -6,6 +6,8 @@ What a night leaves behind, and how your judgement of it steers the next one. **Findings** in the dashboard's top bar, or g f. +![The Findings page filtered to the bug-garden fixture app: 59 open findings, none triaged yet, each row with its severity, check, title and screen, how often and on how many builds it was seen, "reproduced 2/2", and when it was first and last seen](../img/findings.png) + The **list** can be filtered by app, check and status (open, triaged, all). Each row shows severity, check, title, screen, how many times and on how many builds it has been seen, how it replayed ("reproduced 2/2", "flaky 1/2", diff --git a/docs/explore/how-it-works.md b/docs/explore/how-it-works.md index 6302881..a7fdf54 100644 --- a/docs/explore/how-it-works.md +++ b/docs/explore/how-it-works.md @@ -4,32 +4,7 @@ One `explore` job is one night on one or more devices. For each device it runs mission cards in order until the night's budget is spent, then confirms what it found and files what holds up. -```mermaid -sequenceDiagram - participant E as Executor (explore) - participant D as Device (actuator) - participant M as Model (driver) - participant J as Judge - participant C as Collector - E->>D: reset (install, clear data, launch, sign in) - loop each step, until answer, budget, time or stuck - E->>D: observe (screenshot, tree, foreground, focus) - E->>E: screen map, oracle checks - E-->>J: new screen? judge it (once per screen) - E->>M: screenshot + harness notes - M-->>E: tool calls - E->>E: snap and leash each action - E->>D: act, settle - E->>D: crashes since last step - E-->>C: beacon (renews the lease) - end - E->>J: goal met? (cards with a success line) - loop each candidate, worst first - E->>D: reset, replay the steps, ask the check again (x2) - end - E->>C: POST /findings (reproduced, or a crash) - E->>C: result row with metrics -``` +![One mission as a sequence between the executor, the device, the model, the judge and the collector: reset the device; then each step observe, place the screen and run the checks, have the judge look at a new screen, send the model the screenshot and get tool calls back, snap and leash the actions, act, read the crashes and beacon the collector; after the mission ask the judge whether the goal was met, replay each candidate twice, and post the findings and the result row](../img/explore-step.png) ## Before the first step diff --git a/docs/explore/index.md b/docs/explore/index.md index 378041f..871414e 100644 --- a/docs/explore/index.md +++ b/docs/explore/index.md @@ -10,18 +10,7 @@ in Spanish. A model reads each screenshot and picks the next action; the harness checks that action, performs it, and looks for trouble after every step. Nothing it sees is believed until it happens again on a clean install. -```mermaid -flowchart TD - A[Tonight's build] --> B[Mission cards] - B --> C{{"Look, decide, act
(the model)"}} - C --> D[Checks after every step] - D -->|candidate| E[Replay twice on a clean install] - E -->|reproduced| F[Finding] - E -->|not reproduced| X[Dropped] - F --> G[Findings page, digest, optional issue] - G -->|your verdict| H[Precision per check] - H -->|under 30%| D -``` +![One night of exploratory QA: tonight's build and the mission cards go to the model, which looks, decides and acts while checks look for trouble after every step; a candidate is replayed twice on a clean install and becomes a finding only if it reproduces at least once or was a crash with a log, otherwise it is dropped; findings go to the Findings page, the digest and optionally an issue, and your verdicts give each check a precision that switches it off for that app under 30% once ten are judged](../img/explore-night.png) ## What you get in the morning diff --git a/docs/explore/internals.md b/docs/explore/internals.md index 85e1561..eb08788 100644 --- a/docs/explore/internals.md +++ b/docs/explore/internals.md @@ -47,22 +47,7 @@ The model speaks 0–1000; `convertCall` maps to pixels; each actuator maps pixels to its own input units (device pixels on Android, points on Apple devices). -```mermaid -flowchart TB - subgraph Executor - IDX[index.ts
per device, per mission] --> LOOP[loop.ts
runMission / confirm] - LOOP --> MODEL[model.ts
Conversation] - LOOP --> JUDGE[judge.ts] - LOOP --> LEASH[leash.ts] - LOOP --> MAP[screenmap.ts] - LOOP --> ACT{{actuators/}} - IDX --> REP[report.ts, replay.ts] - end - ACT --> ADB[adb] & FD[FleetDriver over HTTP] & TV[tvloop adapter] - MODEL & JUDGE --> GW[(gateway
pilot / vision)] - IDX -->|POST /findings, /results, /artifacts| COL[(Collector)] - COL --> DASH[Findings page] -``` +![The explorer's code map: index.ts calls loop.ts and report.ts and replay.ts; loop.ts uses model.ts, judge.ts, leash.ts and screenmap.ts each step and actuators/ to reach the device over adb, FleetDriver or tvloop; model.ts and judge.ts call the gateway's pilot and vision models; index.ts posts findings, results and artifacts to the collector and its Findings page](../img/explore-code.png) ## Tests diff --git a/docs/figures/explore-code.html b/docs/figures/explore-code.html new file mode 100644 index 0000000..a254fd0 --- /dev/null +++ b/docs/figures/explore-code.html @@ -0,0 +1,88 @@ + + +explore-code + + +
+ + + + + + + + index.ts + per device, per mission; confirm, file + + + report.ts · replay.ts + steps in words, the replay flow + + + + loop.ts + runMission · confirm + + + + each step + + + model.ts + + judge.ts + + leash.ts + + screenmap.ts + + + + + actuators/ + android.ts · apple.ts · roku.ts + + + types.ts is the contract every file here agrees on. + Outside model.ts, every coordinate is in screenshot pixels. + the executor · collector/src/workloads/explore/ + + + + findings, results, artifacts + + + collector + the findings store + + + + Findings page + where your verdicts go + + + + gateway + pilot drives · vision judges + + + + the device + adb + Android, Fire TV + FleetDriver + Apple, over HTTP + tvloop + Roku + + +
+ diff --git a/docs/figures/explore-night.html b/docs/figures/explore-night.html new file mode 100644 index 0000000..4290455 --- /dev/null +++ b/docs/figures/explore-night.html @@ -0,0 +1,90 @@ + + +explore-night + + +
+ + + One night, per device. Nothing the model or a check sees is believed until it happens again on a clean install. + + + + tonight's build + installed, data cleared + + + mission cards + today's changes first + + + look · decide · act + the model reads the screen + + + checks after every step + crash · dead control · the judge + + + + + + + leashed action + + next screenshot + + + + candidate + + + replay twice + clean install, the same steps + + + at least 1 of 2, + or a crash log + + + finding + screenshot, steps, a flow + + + + + Findings page · digest · issue + a repeat raises a count, not a second report + + + + not reproduced + + dropped + counted in explore_not_reproduced + + + + your verdict + + + precision per check + Real · Duplicate · Not a bug · Agent's mistake + + + + under 30% once ten are judged: switched off for that app from the next night + crashes and ANRs are never switched off; their evidence is the platform's own log + +
+ diff --git a/docs/figures/explore-step.html b/docs/figures/explore-step.html new file mode 100644 index 0000000..cb1d107 --- /dev/null +++ b/docs/figures/explore-step.html @@ -0,0 +1,86 @@ + + +explore-step + + +
+ + + One mission. Only the executor talks; the device, both models and the collector never talk to each other. + + + + + + + + + executor + explore + + + device + actuator + + model + pilot + + judge + vision + + collector + the brain + + + reset · install, clear, sign in + + each step, until answer, budget, time or stuck + + observe · screenshot, tree, focus + + screen map, then the checks + + a new screen? judge it, once per screen + + the screenshot and the harness notes + + tool calls + + snap and leash each action + + act, then let the screen settle + + crashes since the last step + + beacon · renews the lease + + goal met? cards with a success line + + each candidate, worst first · twice + + reset, replay, ask the check again + + POST /findings · reproduced, or a crash + + the result row, with its metrics + + + + + the executor asks + + the model answers + gateway defaults: pilot drives, vision judges + +
+ diff --git a/docs/figures/workloads.html b/docs/figures/workloads.html index c8d5318..7e52e10 100644 --- a/docs/figures/workloads.html +++ b/docs/figures/workloads.html @@ -6,18 +6,20 @@ Everything the fleet knows how to run, grouped by who claims it. The columns are the three executors; the routing is on declared capabilities, so this is a picture of what the agents in this repository say they can do rather than - a list the collector hardcodes. + a list the collector hardcodes. The source of truth is the shipped entries in + collector/src/workloads/catalog.ts (runs_on); the three reserved names are + left out because nothing here runs them. Rendered by `npm run assets` in collector/ to docs/img/workloads.png. --> -
- +
+ - 28 workloads. Which agent claims one is decided by the capabilities it declared, not by anything hardcoded. + 34 workloads. Which agent claims one is decided by the capabilities it declared, not by anything hardcoded. - - + + on the device @@ -48,34 +50,44 @@ an executor on a Mac with a cable - install + enrol - ui-test + install - cold-start + upgrade-test - app-soak + ui-test - a11y-audit + cold-start - locale-shots + app-soak - drain + a11y-audit + + locale-shots + + explore + + drain - web-test + size-report - web-shots + tvloop - web-audit + web-test - web-unfurl + web-shots - soak + web-audit - archive + web-unfurl - digest + soak + + archive + + digest @@ -93,13 +105,15 @@ shell - self-check - + llm-eval + + self-check + - - also claimable elsewhere — benchmark and vantage run on a machine too, and self-check on a host + + also claimable elsewhere — benchmark and vantage run on a machine too, and self-check on a host
diff --git a/docs/img/devices.png b/docs/img/devices.png index 301228d..e701e2b 100644 Binary files a/docs/img/devices.png and b/docs/img/devices.png differ diff --git a/docs/img/explore-code.png b/docs/img/explore-code.png new file mode 100644 index 0000000..327117a Binary files /dev/null and b/docs/img/explore-code.png differ diff --git a/docs/img/explore-night.png b/docs/img/explore-night.png new file mode 100644 index 0000000..6732cad Binary files /dev/null and b/docs/img/explore-night.png differ diff --git a/docs/img/explore-step.png b/docs/img/explore-step.png new file mode 100644 index 0000000..eba5629 Binary files /dev/null and b/docs/img/explore-step.png differ diff --git a/docs/img/findings.png b/docs/img/findings.png new file mode 100644 index 0000000..5117309 Binary files /dev/null and b/docs/img/findings.png differ diff --git a/docs/img/workloads.png b/docs/img/workloads.png index 96d49b2..c89f939 100644 Binary files a/docs/img/workloads.png and b/docs/img/workloads.png differ diff --git a/docs/workloads/index.md b/docs/workloads/index.md index 0f9084c..dbdb9b0 100644 --- a/docs/workloads/index.md +++ b/docs/workloads/index.md @@ -4,7 +4,7 @@ What the fleet can be asked to do. A workload is the `workload` field of a job spec, and which agent claims it is decided by [capabilities](../concepts.md#capabilities) rather than by anything hardcoded. -![The workloads in three columns by who claims them: on the device, on a host, and on a machine](../img/workloads.png) +![Thirty-four workloads in three columns by who claims them: on the device, on a host, and on a machine](../img/workloads.png) The collector ships 34 workloads. `GET /api/v1/workloads` (or `fleetr workloads`) lists the same set with each one's params and an example diff --git a/mkdocs.yml b/mkdocs.yml index 3c8a399..4da58a6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -75,11 +75,7 @@ markdown_extensions: - pymdownx.highlight: anchor_linenums: true - pymdownx.inlinehilite - - pymdownx.superfences: - custom_fences: - - name: mermaid - class: mermaid - format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.superfences - pymdownx.tabbed: alternate_style: true # The cookbook includes the example job specs verbatim from collector/, which