diff --git a/AGENTS.md b/AGENTS.md index a1c68e6..35d170b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,11 +9,14 @@ from an agent, see [`skills/fleet-runner/SKILL.md`](skills/fleet-runner/SKILL.md | Path | What | |---|---| | `collector/` | The brain: Fastify + `node:sqlite`, the queue, the dashboard (`dash/`), the host executor | +| `collector/src/workloads/explore/` | Exploratory QA: a vision model drives an app from mission cards (`collector/examples/missions/`); see `docs/explore/internals.md` | | `collector/src/api/v1/` | The control API. `routes.ts` is the single route table: it registers the routes, generates `openapi.json` and the typed client | | `collector/src/client/` | The generated client (`api.gen.ts`) and its transport. The CLI and the MCP server use nothing else | | `fleet/` | The `fleet` / `fleetr` command: runs a brain or a runner, plus the work commands, `fleet mcp` and the remote gateway | -| `runner-machine/`, `runner-android/`, `runner-ios/`, `runner-roku/`, `desktop/` | Runners. They speak the runner protocol (`/devices/register`, `/devices/:id/next-job`, `/results`), not `/api/v1` | +| `desktop/` | A Tauri menu-bar app around `fleet`. A front door, not a runner; compiled in CI and never launched | +| `runner-machine/`, `runner-android/`, `runner-ios/`, `runner-roku/`, `collector/runner-web/` | Runners. They speak the runner protocol (`/devices/register`, `/devices/:id/next-job`, `/results`), not `/api/v1` | | `skills/fleet-runner/` | The agent skill. Its tables are generated | +| `bug-garden/` | An Android fixture app with twenty planted defects, for scoring exploratory QA | | `docs/` | The mkdocs site. Built with `--strict`, so a broken link fails CI | ## Suites diff --git a/CHANGELOG.md b/CHANGELOG.md index 4506e4e..b1b6bbf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,11 @@ a CI job fails when they disagree. the bundle gains a directory the installers do not copy. The installers are fetched from `main`, so re-running the install command fixes an existing 0.8.0 install; no new release is needed. +- **`bug-garden/tools/check-bench.mjs` passed `bench-delete-plant` without + checking the plant was gone.** The card's `check` gained `absent_text`, which + the harness enforces and the standalone checker ignored; it also matched + case-sensitively where the harness does not. It now reads `absent_text` and + matches case-insensitively, as the bug garden's README says. ## [0.8.0] — 2026-10-15 @@ -126,6 +131,63 @@ pull request that touches them: unversioned routes with hand-written types ([#83](https://github.com/addisdev/fleet-runner/issues/83)). +### Also in this release + +These merged into `develop` before 0.8.0 was cut and shipped in it, but were +left out of the notes above on release day. + +- **Exploratory QA, the `explore` workload.** A vision model works through + mission cards on tonight's build, driving a device the way a person would, + while the harness checks every step for crashes, ANRs, frozen and blank + screens, dead controls, accessibility and visual defects. Each candidate is + replayed twice on a clean install and only what reproduces is filed; a crash + stands on its log. See [Exploratory QA](docs/explore/index.md). + ([#70](https://github.com/addisdev/fleet-runner/pull/70)) +- **Findings, verdicts and a morning digest.** `POST /findings` merges repeats + by fingerprint; the dashboard's Findings page has Real, Duplicate, Not a bug + and Agent's mistake. A check under 30% precision after ten verdicts is + switched off for that app (crashes and ANRs never are). GitHub issues stay a + dry run unless `FLEET_GITHUB_ISSUES=1` and a token are set, and the collector + only ever creates them. ([#70](https://github.com/addisdev/fleet-runner/pull/70)) +- **What it drives:** Android phones, tablets, Android TV and Fire TV over adb; + iOS and tvOS through FleetDriver, a new XCUITest command server in + `runner-ios`; a Roku through tvloop. The leash refuses delete, purchase, + invite and sign-out controls unless a card allows them and never types into + a password field. ([#70](https://github.com/addisdev/fleet-runner/pull/70)) +- **The bug garden**, an Android fixture app with twenty planted defects, a + scoring rule and scorer, and twenty bench cards for scoring a model without + a judge. Its screenshot walks capture each screen in the base, dark, Spanish + and large-text settings. ([#70](https://github.com/addisdev/fleet-runner/pull/70), + [#74](https://github.com/addisdev/fleet-runner/pull/74)) +- **Nights on ultra.** `scripts/explore-night.ts` is the night-queue command: + it boots the devices, opens its own tunnel to the brain, runs one job and + exits with its result. `nightly-builds.sh` builds GreenFolio iOS and + Dozehound tvOS from `origin/develop`; the tvOS and iOS nights run after the + products' own night jobs. Holo4 35B-A3B measured 100 of 100 on pointing and + a median step of 6.8 s. Fire TV is parked and the TV question is answered on + tvOS. ([#73](https://github.com/addisdev/fleet-runner/pull/73), + [#77](https://github.com/addisdev/fleet-runner/pull/77), + [#78](https://github.com/addisdev/fleet-runner/pull/78)) +- **The dashboard draws each device as the object it is**: 22 drawings + (phones, tablets, TV boxes and sticks, computers, watches, a headset, a + board, a browser) instead of seven silhouettes, chosen from the model string + first because older agents report a Roku as `android` and an Apple TV or a + watch as `ios`. ([#69](https://github.com/addisdev/fleet-runner/pull/69)) +- **The Android runner starts on Android 7.0 and 7.1.** `RunnerService` called + `startForegroundService`, which exists from API 26, so API 24 and 25 threw + `NoSuchMethodError`; it uses `startService` there now. + ([#75](https://github.com/addisdev/fleet-runner/pull/75)) +- **The Android runner is linted in CI.** The `android.hardware.type.watch` + feature line that failed `lintDebug` is gone; the runner still detects a + watch at run time. ([#76](https://github.com/addisdev/fleet-runner/pull/76)) + +Not verified, from those pull requests: a physical iPhone or Apple TV through +FleetDriver; a real Roku (its developer password is not on the host yet); any +model-driven night on an Apple surface (the iOS and tvOS simulators were driven +without a model); the GreenFolio Android cards; the Android 7 fix on a real +API 24 or 25 device (Robolectric only); and any Wear device, before or after +the lint change. + ## [0.7.2] — 2026-09-30 ### Fixed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f5e7d56..835d0a2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,7 +11,7 @@ that is not on this shelf is data this repository does not have. ## Running the suites -Four components, four suites. Only run the one you touched. +Five components, five suites. Only run the one you touched. === "collector" @@ -80,7 +80,7 @@ Four components, four suites. Only run the one you touched. npm install && npm test ``` - 122 tests, about a second. This is the easiest component to contribute to + About a second. This is the easiest component to contribute to and the reference implementation of the protocol. ## Things this project learned the hard way @@ -162,7 +162,9 @@ the others. One version covers every component, so a release is one number everywhere. 1. On a branch from `develop`, run `node scripts/version.mjs set X.Y.Z`. It - writes `VERSION` and every component's own version. Then run + writes `VERSION` and every component's own version except the desktop + app's (`desktop/src-tauri/Cargo.toml` and `tauri.conf.json` are not among + the places it writes, and still say 0.5.0). Then run `npm --prefix collector run gen:api`, because `openapi.json` carries the version and CI fails when it is stale. Android's `versionCode` is not derived from it; bump that by hand if the Play Store build needs it. diff --git a/README.md b/README.md index 9ab2b35..c46f42b 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,10 @@ One queued job installs a build on every attached device, runs a UI suite across them, benchmarks llama.cpp on real silicon, classifies a few hundred images through Core ML and LiteRT, screenshots a website on two real phone screens and diffs it against a baseline, or drains a battery on purpose and -plots the curve. Everything lands in one results database with one dashboard -in front of it. +plots the curve. Overnight, a local vision model can use tonight's build the +way a person would and leave a short list of bugs that reproduced on a clean +install. Everything lands in one results database with one dashboard in front +of it. Built to answer a question I could not otherwise answer: **would on-device machine learning actually be good enough to ship in my apps, and on which @@ -27,7 +29,7 @@ hardware?** It turned out to be yes, and the fleet is how I know. | **Problem** | Mobile tests and ML benchmarks are only useful when they run on the hardware, under conditions, and with the exact artifacts a product will ship. | | **System** | One collector owns the registry, queue, leases, artifacts, results and dashboard; independently implemented agents claim work through one tested JSON protocol. | | **Reach** | Five runner implementations across TypeScript, Swift, Kotlin, JavaScript and BrightScript, plus host-driven iOS and Android workloads. | -| **Workloads** | 28 workload types spanning UI tests, visual regression, llama.cpp, Core ML, LiteRT, battery experiments, builds and deployment. | +| **Workloads** | 34 workload types spanning UI tests, visual regression, exploratory QA by a vision model, llama.cpp, Core ML, LiteRT, battery experiments, builds and deployment. | | **Proof** | The first production question found an int8 CPU model faster to load and run than its fp32 GPU alternative while preserving useful accuracy. | **What this demonstrates:** distributed systems, mobile platforms, protocol @@ -40,8 +42,8 @@ trade-offs. ![The Overview while one fan-out benchmark is claimed by three agents at once: the queue fills, three rows appear under running now, and the done count climbs as each reports back](docs/img/fanout.gif) -One `POST /jobs` with `"fanout": true`, and every agent the expression matches -claims its own child job. Above: a laptop and two browser runners on a collector +One job with `"fanout": true`, and every agent the expression matches claims +its own child job. Above: a laptop and two browser runners on a collector that did not exist a minute earlier. Nothing there is staged — it is the shipping agents against a real collector, recorded a frame every 250 ms. @@ -63,8 +65,9 @@ curl -fsSL https://raw.githubusercontent.com/addisdev/fleet-runner/main/install. > The installer verifies the download against the release's `SHASUMS256.txt` and > refuses to install if it does not match. It needs Node 22.13 or newer and > carries no runtime of its own. **`fleet up` itself has been watched to work on -> macOS/arm64 only** -- CI runs the suites on Windows and Linux, which is not the -> same as somebody having run a fleet on one. +> macOS only**, on Apple silicon and on a 2016 Intel Mac that runs this lab's +> brain -- CI runs the suites on Windows and Linux, and starts the container and +> the systemd unit, which is not the same as somebody having run a fleet on one. > **[docs/install/](https://addisdev.github.io/fleet-runner/install/)** is honest > per platform. @@ -137,15 +140,16 @@ something a runner can do without the collector shipping a release. ![Twenty-eight workloads in three columns by who claims them: on the device, on a host, and on a machine](docs/img/workloads.png) -Twenty-eight workloads, and the column a workload sits in is the answer to -"who can physically do this". A benchmark runs inside the app on the phone. An +Thirty-four workloads (the figure predates six of them, `explore` and `tvloop` +among them), and the column a workload sits in is the answer to "who can +physically do this". A benchmark runs inside the app on the phone. An install needs a cable and a Mac. A build needs a checkout and a toolchain. Each has [its own page](https://addisdev.github.io/fleet-runner/workloads/), saying what it measures and what it refuses to guess. ## What is in here -Seven projects in one repository, in three groups: **one brain**, **five +Eight projects in one repository, in three groups: **one brain**, **five runners** that speak its protocol and share no code with each other or with it, and **two front doors** onto the whole thing. They ship independently but they are versioned together, because the protocol is the thing that breaks and a @@ -155,7 +159,7 @@ change to it touches three of them at once. | | What it is | |---|---| -| **[collector/](collector)** | Device registry, job queue with leases, artifact store, results database, scheduler, alert engine, and the dashboard above. Node + Fastify + SQLite, no broker, no cloud. It also holds the host executor, which drives phones from outside over adb, simctl and devicectl. | +| **[collector/](collector)** | Device registry, job queue with leases, artifact store, results database, scheduler, alert engine, and the dashboard above. Node + Fastify + SQLite, no broker, no cloud. It also holds the host executor, which drives phones from outside over adb, simctl and devicectl, and the [exploratory QA](https://addisdev.github.io/fleet-runner/explore/) workload a vision model drives. | **The runners** -- five hand-written implementations of one protocol, in five languages, sharing not one line @@ -175,6 +179,13 @@ languages, sharing not one line | **[fleet/](fleet)** | The CLI. `fleet up` runs this machine's components under a supervisor that backs off and gives up loudly rather than restarting a broken component forever; `fleet doctor` says what the machine can run and why not; `fleet service install` keeps it up. It wraps the three programs above and reimplements none of them. | | **[desktop/](desktop)** | A Tauri menu-bar app around that CLI: a tray menu, role switches, the dashboard in a window, and a notification when a component gives up. It exists to test one hypothesis about macOS local-network permissions, and **it has never been launched**: CI compiles its Rust and that is all -- read its README before believing anything in it. | +**Alongside them** + +| | What it is | +|---|---| +| **[skills/fleet-runner/](skills/fleet-runner)** | The agent skill `fleet skill install` copies into a coding agent: which commands to run in what order. Its tool and exit-code tables are generated from the code. | +| **[bug-garden/](bug-garden)** | An Android app with twenty planted defects, so exploratory QA has something with a known answer to be scored against. | + Each directory has its own README, its own tests and its own CI job, filtered by path so a change to a phone runner does not build the dashboard. @@ -272,8 +283,9 @@ that is not on this shelf is data this repository does not have. the handful of things this project learned the hard way. Behaviour is covered by the [Contributor Covenant](CODE_OF_CONDUCT.md). -**There is no authentication, by design** — the collector is built for a home -LAN or a tailnet. [`SECURITY.md`](SECURITY.md) says what that does and does not +**By default there is no authentication** — the collector is built for a home +LAN or a tailnet, and [scoped tokens](https://addisdev.github.io/fleet-runner/deploy/auth/) +are opt-in. [`SECURITY.md`](SECURITY.md) says what that does and does not cover before you report it. ## License diff --git a/bug-garden/README.md b/bug-garden/README.md index a7712d7..a3ef1ae 100644 --- a/bug-garden/README.md +++ b/bug-garden/README.md @@ -21,8 +21,10 @@ mission and before a replay) puts it back exactly where it started. | `android/` | The app. Kotlin, Jetpack Compose, minSdk 26, AndroidX only. | | `defects.json` | The twenty defects: class, which check should catch it, screen, how to reach it, what it looks like, and a match pattern. | | `tools/check-bench.mjs` | Checks a bench mission's `check` block against a `uiautomator dump`, with no model. | +| `tools/verify-on-device.sh` | Walks to every defect on a device and checks each one still fires; see [Proving it on a device](#proving-it-on-a-device). | | `../collector/examples/missions/bug-garden/` | Seven exploration missions and twenty bench missions (`bench-*`). | -| `../collector/examples/flows/bug-garden/` | `sign-in.yaml` (the missions' setup flow), `smoke.yaml`, and `bench/`, a scripted answer to each bench mission. | +| `../collector/examples/flows/bug-garden/` | `sign-in.yaml` (the missions' setup flow), `smoke.yaml`, `bench/` (a scripted answer to each bench mission), `probes/` (a walk to each defect), and the `t1-*.yaml` walks that take one screenshot of each screen in the base, dark, Spanish and large-text settings. | +| `../collector/scripts/explore-bench/garden-score.ts` | Scores a night's findings against `defects.json` by the rule under [Scoring](#scoring). | ## Build and install @@ -89,7 +91,7 @@ in any of the three languages. ## The defects -Twenty, in eight classes. `defects.json` has the steps for each; in brief: +Twenty, in nine classes. `defects.json` has the steps for each; in brief: | Id | Class | Check | Screen | Defect | |---|---|---|---|---| @@ -202,9 +204,15 @@ with quantity 3", "Change the language to Spanish"), defects off (`launch_args: ["garden_defects=false"]`), and a `check` block a harness evaluates against the final `uiautomator dump`: -- `text`: every string is a case-sensitive substring of some node's `text` or - `content-desc`. (Content-desc counts: the Settings icon on Home is labelled - "Ajustes" in Spanish, so the Spanish check passes on Home or on Settings.) +- `text`: every string is a case-insensitive substring of some node's `text` + or `content-desc`. (Content-desc counts: the Settings icon on Home is + labelled "Ajustes" in Spanish, so the Spanish check passes on Home or on + Settings.) +- `absent_text`: no node's `text` or `content-desc` contains any of these + strings. `bench-delete-plant` uses it to check that Aloe Vera is gone, not + just that the list is one plant shorter. +- `focused_label`: the focused node's label contains the string. The TV cards + use it, since a D-pad screen's state is where focus sits. - `screen`: some node has `resource-id="screen_"`. The checks only use text that the seed data cannot produce by itself. No seeded @@ -213,11 +221,10 @@ the Office, so "Qty 3" can only come from the plant the mission added. Where a check names no screen, it holds on both the plant list and the plant's detail page, which is where a model doing the task naturally ends. -One limit: the check block has no way to say text must be absent, so -`bench-delete-plant` checks for "8 plants" (one fewer than the seed) rather -than for Aloe Vera's absence. - -`tools/check-bench.mjs` implements exactly these rules: +`tools/check-bench.mjs` implements these rules. The one difference from the +harness is `screen`: the harness matches the screen map's name for the final +screen, while the tool, which has only a dump, looks for the bug garden's +`screen_` tag. To check a dump by hand: ```sh adb exec-out uiautomator dump /dev/tty > /tmp/dump.xml diff --git a/bug-garden/tools/check-bench.mjs b/bug-garden/tools/check-bench.mjs index 1afe7b1..73a7ff4 100644 --- a/bug-garden/tools/check-bench.mjs +++ b/bug-garden/tools/check-bench.mjs @@ -6,17 +6,19 @@ * adb exec-out uiautomator dump /dev/tty | node check-bench.mjs - * * Exit 0 and "PASS" when every part of the mission's `check` holds, exit 1 and - * "FAIL" with what was missing otherwise. + * "FAIL" with what was wrong otherwise. * * The rules, which are the ones README.md states for the harness: - * text every string must be a substring (case-sensitive) of some + * text every string must be a substring (case-insensitive) of some * node's text OR content-desc. Content-desc counts because an * icon button's label is its content-desc (the Settings icon on * Home is "Ajustes" in Spanish). + * absent_text no node's text or content-desc may contain any of these + * strings (case-insensitive): "the plant is gone". * screen some node has resource-id "screen_". The bug garden * tags every screen's root that way. * focused_label some node with focused="true" has the string in its text or - * content-desc. + * content-desc (case-insensitive). * * Standalone on purpose: no imports from the collector, so it can be pointed * at a dump from any device without building anything. @@ -40,7 +42,7 @@ const nodes = []; for (const m of xml.matchAll(/]*?)\/?>/g)) { const a = {}; for (const kv of m[1].matchAll(/([\w:-]+)="([^"]*)"/g)) a[kv[1]] = unescape(kv[2]); - nodes.push({ text: a.text ?? "", desc: a["content-desc"] ?? "", id: a["resource-id"] ?? "", focused: a.focused === "true" }); + nodes.push({ text: (a.text ?? "").toLowerCase(), desc: (a["content-desc"] ?? "").toLowerCase(), id: a["resource-id"] ?? "", focused: a.focused === "true" }); } if (nodes.length === 0) { console.log(`FAIL ${mission.id}: the dump has no nodes`); @@ -48,20 +50,24 @@ if (nodes.length === 0) { } const check = mission.check ?? {}; +const shows = (n, t) => n.text.includes(t.toLowerCase()) || n.desc.includes(t.toLowerCase()); const missing = []; for (const t of check.text ?? []) { - if (!nodes.some((n) => n.text.includes(t) || n.desc.includes(t))) missing.push(`text ${JSON.stringify(t)}`); + if (!nodes.some((n) => shows(n, t))) missing.push(`missing text ${JSON.stringify(t)}`); +} +for (const t of check.absent_text ?? []) { + if (nodes.some((n) => shows(n, t))) missing.push(`still shows ${JSON.stringify(t)}`); } if (check.screen && !nodes.some((n) => n.id === `screen_${check.screen}`)) { const on = nodes.find((n) => n.id.startsWith("screen_"))?.id ?? "no tagged screen"; - missing.push(`screen ${check.screen} (on ${on})`); + missing.push(`missing screen ${check.screen} (on ${on})`); } -if (check.focused_label && !nodes.some((n) => n.focused && (n.text.includes(check.focused_label) || n.desc.includes(check.focused_label)))) { - missing.push(`focused ${JSON.stringify(check.focused_label)}`); +if (check.focused_label && !nodes.some((n) => n.focused && shows(n, check.focused_label))) { + missing.push(`missing focus on ${JSON.stringify(check.focused_label)}`); } if (missing.length) { - console.log(`FAIL ${mission.id}: missing ${missing.join(", ")}`); + console.log(`FAIL ${mission.id}: ${missing.join(", ")}`); process.exit(1); } console.log(`PASS ${mission.id}`); diff --git a/collector/README.md b/collector/README.md index 6f40b81..f44d9fd 100644 --- a/collector/README.md +++ b/collector/README.md @@ -1,8 +1,9 @@ # collector *Part of [Fleet Runner](../README.md), alongside the -[Android](../runner-android), [iOS](../runner-ios) and -[desktop](../runner-machine) runners.* +[Android](../runner-android), [iOS](../runner-ios), +[desktop](../runner-machine), [Roku](../runner-roku) and +[web](runner-web) runners.* The brain of **Fleet Runner**: a personal device lab that turns a shelf of old phones into something you can send work to. It holds the device registry, the @@ -12,10 +13,12 @@ One queued job can install a build on every attached device, run a Maestro or XCUITest suite across them, benchmark llama.cpp on real silicon, classify a few hundred images through Core ML and LiteRT, screenshot a website on two real phones and diff it against a baseline, or drain a battery on purpose and -plot the curve. The runners are separate apps — -[iOS](../runner-ios) and -[Android](../runner-android) — that speak a -shared JSON protocol, not shared code. +plot the curve. The runners are separate programs — +[iOS](../runner-ios) (with a tvOS target), +[Android](../runner-android), the [machine](../runner-machine) agent for +laptops and desktops, a [web](runner-web) runner this collector serves at +`/runner`, and a [Roku](../runner-roku) channel — that speak a shared JSON +protocol, not shared code. Node + Fastify + SQLite in WAL mode. No build step for the server, no broker, no cloud. @@ -33,7 +36,7 @@ APK or tapping through a UI test is not something an app can do to itself. ## Run it -Needs Node 22 or newer. Nothing else. +Needs Node 22.13 or newer. Nothing else. ```bash npm install @@ -45,7 +48,7 @@ build a runner from one of the app repos, point it at this address, and enqueue a job: ```bash -curl -X POST http://127.0.0.1:8788/jobs -H 'content-type: application/json' -d '{ +curl -X POST http://127.0.0.1:8788/api/v1/jobs -H 'content-type: application/json' -d '{ "schema": 1, "job_id": "bench-1", "workload": "benchmark", "executor": "device", "backend": "synthetic", "params": { "prompt_tokens": 256, "gen_tokens": 64, "measure_iters": 3 }, @@ -53,6 +56,10 @@ curl -X POST http://127.0.0.1:8788/jobs -H 'content-type: application/json' -d ' }' ``` +Or, with the `fleet` CLI installed, `fleetr run benchmark --backend synthetic +--pool ml-capable --param prompt_tokens=256 ...`, which validates, queues, +waits, and exits with the verdict. + ```bash npm test # typecheck, build the dashboard, run the suite npm run executor # host executor: claims host jobs, drives devices @@ -86,6 +93,15 @@ so it never touches a real fleet's history. | `llm-eval` | machine | Scores what a device generated against an eval set — exact, regex, JSON-shape and model-judged rules — so a quantisation is measured for whether it is still RIGHT, not only how fast it is wrong | | `self-check` | host, machine | The fleet inspects its own hosts: disk, tool versions, clock drift, agents actually loaded | | `archive` / `digest` | host | Pull store reviews and Search Console data; then have the shelf summarize its own reviews using its own models | +| `enrol` | host | Point every attached device's runner at a collector, without anybody typing | +| `explore` | host | A vision model explores a build and files reproduced crashes, hangs, dead controls and accessibility defects as findings | +| `tvloop` | host | tvloop's hardware suite on a Roku the fleet has locked: doctor, sideload, contract tests, a JUnit replay of its flows | +| `model-convert` / `dataset-prep` | machine | A checkpoint into GGUF, Core ML or TFLite; an eval set that carries its licence and source | +| `serve` / `shell` | machine | Host a model for the length of a lease; run a script whose hash the machine's owner pinned by hand | + +That is all 34 shipped workloads. `GET /api/v1/workloads` (or `fleetr +workloads`) lists them with their params and an example spec, plus anything a +registered agent declares that the collector does not know. Full reference, including every parameter and the failure modes worth knowing: **[the documentation](https://addisdev.github.io/fleet-runner/)**. @@ -93,6 +109,16 @@ Start at [Concepts](https://addisdev.github.io/fleet-runner/concepts/) for how the queue thinks, or [Workloads](https://addisdev.github.io/fleet-runner/workloads/) for what it can be asked to do. +## Control API + +Scripts, CI, the `fleet` CLI and AI agents use the versioned API under +`/api/v1`: devices, workloads, jobs (enqueue, validate, wait, cancel, retry), +results and artifacts. `GET /api/v1/openapi.json` describes it, and +`GET /metrics` serves the fleet in Prometheus text format. The older +unversioned routes still work and answer with a `Deprecation` header. Runners +speak the separate runner protocol. See +[the HTTP API](https://addisdev.github.io/fleet-runner/api/). + ## Status and threat model Built and running: every phase of the original plan, a dashboard with live @@ -101,13 +127,20 @@ an [on-device plant-ID evaluation](evals/greenfolio-plant-id.md) across both platforms — 77% top-1 at 7 ms per image on a phone, which answered a product question that had been guesswork. -**There is no authentication, by design.** The collector is reachable on a home -LAN and is meant to stay that way; anyone who can reach it can enqueue a job. -`FLEET_DASH_TOKEN` guards the dashboard's mutations, but that is a speed bump -against a stray browser tab, not an access control — `POST /jobs` stays open so -`curl` and CI keep working. Do not put this on the internet. If you want to, -the honest starting point is that every endpoint would need to be re-thought, -not that a token would need to be added. +**There is no authentication by default.** The collector is reachable on a +home LAN and is meant to stay that way; anyone who can reach it can enqueue a +job. `FLEET_DASH_TOKEN` guards the dashboard's mutations, but that is a speed +bump against a stray browser tab, not an access control — `POST /jobs` stays +open so `curl` and CI keep working. + +Scoped bearer tokens are opt-in: `FLEET_AUTH=tokens` requires one on the +control API, and `FLEET_AUTH_PROTOCOL=token` also on the runner protocol, for +fleets whose runners can send one (the phone, TV and browser runners cannot). +See [Authentication](https://addisdev.github.io/fleet-runner/deploy/auth/). +With tokens on, still serve the collector over TLS or keep it on a private +network: tokens are bearer secrets with no proof of possession, and unless +`FLEET_AUTH_PROTOCOL=token` is set the runner protocol answers anyone who can +reach it. That posture depends on every agent being on the same LAN, which stops being true the moment one of them is a laptop that leaves the house. `FLEET_BIND` diff --git a/collector/scripts/seed-schedules.ts b/collector/scripts/seed-schedules.ts index d176226..b93e8ac 100644 --- a/collector/scripts/seed-schedules.ts +++ b/collector/scripts/seed-schedules.ts @@ -258,8 +258,8 @@ const SCHEDULES: Record = { // Overnight exploratory QA (src/workloads/explore). The night slot the plan // proposes is 22:00-00:30, before Dozehound's video queue takes the Studio's // memory; the window constraint keeps a late claim from running into it. - // All three arrive switched off like every schedule here, and the TV one - // should stay off until the plan's phase-3 test says a model can drive a TV. + // All four arrive switched off like every schedule here. The TV one targets + // Fire TV, which is parked (WB-297): the TV question is answered on tvOS. "nightly-explore-greenfolio": { "cron": "0 22 * * *", "template": { diff --git a/docs/agents.md b/docs/agents.md index 17b0836..5c703e8 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -21,22 +21,55 @@ fleetr devices # what is online fleetr workloads benchmark # params and an example spec fleetr run benchmark --pool phones --validate # check without queueing fleetr run benchmark --pool phones # queue, wait, print the verdict -fleetr artifact --out ./artifacts +mkdir -p artifacts && fleetr artifact --out ./artifacts ``` +`--out` is a directory only if it already exists (and has no extension in its +name); then the file lands there under the artifact's own name. Otherwise it is +the file name to write. `--out -` writes to stdout. + - `--json` on every command prints the API's answer and nothing else; progress goes to stderr. - The exit code is the verdict: **0** passed, **1** failed (or mixed, cancelled), **2** usage error or invalid spec, **3** still running when `--timeout` ran out, **4** skipped (nothing ran), **5** refused or unreachable. So `fleetr run ... && deploy` means what it says. -- `fleetr run` takes a workload and flags (`--device`, `--pool`, `--match`, - `--app`, `--param key=value`, `--fanout`) or a whole spec (`--spec FILE`, - `-` for stdin), and validates before it queues. +- `fleetr run` takes a workload and flags, or a whole spec (`--spec FILE`, + `-` for stdin) that the flags then add to, and validates before it queues. + Validation is the control API's, so an unknown top-level field in the spec + is an error. + +| `fleetr run` flag | | +|---|---| +| `--device ID` · `--pool P` · `--match EXPR` | `targets.device_id`, `.pool`, `.match` | +| `--target k=v` | Any other `targets` field; repeatable, dotted keys nest | +| `--exclusive` | `targets.exclusive: true` | +| `--app NAME` | `app.name`, with `--build` and `--sha256` (both default `latest`) and `--platform` | +| `--param k=v` | A workload param; repeatable, dotted keys nest, values parsed as JSON where they parse | +| `--fanout` | One child job per matching device | +| `--backend B` · `--priority N` · `--id JOB_ID` | The spec fields of the same names; `--id` is an idempotency key | +| `--validate` | Validate, print where it would land, and stop | +| `--no-validate` | Skip the dry run and let the enqueue refuse it instead | +| `--no-wait` | Print the job id and exit 0 | +| `--timeout S` | How long to wait for the verdict; default 1800. Exit 3 if it runs out | + +| Other commands | | +|---|---| +| `fleetr devices [--all] [--platform P] [--pool P]` | Online devices, or the whole shelf | +| `fleetr workloads [NAME] [--available]` | What the fleet can run; NAME prints the params schema and an example spec | +| `fleetr job ID` · `fleetr wait ID [--timeout S]` | The verdict now, or when it finishes (default 1800 s); exits with the verdict | +| `fleetr cancel ID [--reason R]` | Cancel; cascades over a fan-out and fails whatever waited on it | +| `fleetr retry ID [--device ID] [--pool P] [--wait]` | Run again under a new id; `--wait` waits and exits with the verdict | +| `fleetr results [JOB] [--device] [--workload] [--job] [--failures] [--limit N]` | Result rows, final rows only unless a job is named; default 20 | +| `fleetr flaky [--device ID] [--app NAME]` | Builds that both passed and failed on one device | +| `fleetr whoami` | Which collector, its auth mode, who you are to it and your scopes | +| `fleetr login [--url U] [--token T]` · `fleetr logout [--url U]` | Save a token for a collector (from `--token` or stdin), or forget it | Which collector: `--url`, then `FLEET_RUNNER_URL`, then `FLEET_URL`, then `~/.fleet/config.json`. Which token: `--token`, then `FLEET_RUNNER_TOKEN`, then -`FLEET_TOKEN`, then what `fleetr login` saved for that URL. +`FLEET_TOKEN`, then what `fleetr login` saved for that URL. `FLEET_SOURCE` +overrides the `source` recorded on jobs (`cli` by default), for example +`FLEET_SOURCE=ci` in a workflow. ## The skill @@ -63,7 +96,9 @@ suite, so they cannot fall behind the binary. } ``` -Add `"--url", "http://host:8788"` to the args to point at another collector. +Add `"--url", "http://host:8788"` to the args to point at another collector, +and `"--token", "fr_..."` if it has tokens on (or set `FLEET_RUNNER_TOKEN` in +the server's `env`; a saved `fleetr login` for that URL also works). Tools: `fleet_devices`, `fleet_workloads`, `fleet_validate`, `fleet_run`, `fleet_job`, `fleet_cancel`, `fleet_retry`, `fleet_results`, `fleet_flaky`, `fleet_artifact`, `fleet_whoami`. Each carries read-only and destructive @@ -71,8 +106,11 @@ annotations and, where it returns a summary, an output schema. Screenshots come back as images. Jobs, workloads, devices and artifacts are also readable as resources (`fleet://job/{id}` and so on). -`fleet_run` waits for the verdict by default (up to 15 minutes locally); pass -`wait: false` and follow with `fleet_job` for anything longer. +`fleet_run` waits for the verdict by default: `timeout_s` defaults to 900 +(15 minutes) locally and is capped at 3600; through the remote gateway it +defaults to 45 s and is capped at 55. Pass `wait: false` and follow with +`fleet_job`, whose `wait_s` holds the call up to 120 s (55 remotely; default 0, +answer now). ## The remote gateway diff --git a/docs/api.md b/docs/api.md index b6b4083..38bf3db 100644 --- a/docs/api.md +++ b/docs/api.md @@ -20,8 +20,8 @@ disagree: the test suite fails if they do. | `GET /api/v1/devices` · `/devices/{id}` | read | The shelf, with status (`online`, `stale`, `offline`) and current job | | `GET /api/v1/workloads` · `/workloads/{name}` | read | What the fleet can run: executor, params schema, a valid example spec, how many live agents can take it | | `GET /api/v1/templates` | read | Saved job templates | -| `GET /api/v1/jobs` | read | Jobs; filter by `status`, `workload`, `device`, `pool`, `source`, `requested_by`, `mine=true`, `from`/`to` | -| `POST /api/v1/jobs` | run | Enqueue. 201 with the job id, its status, and links to its summary and wait URLs | +| `GET /api/v1/jobs` | read | Jobs; filter by `status`, `workload`, `executor` (`device` or `host`), `device`, `pool`, `source`, `requested_by`, `mine=true`, `q` (substring of the job id), `from`/`to` | +| `POST /api/v1/jobs` | run | Enqueue. 201 with the job id, its status, and links to its summary and wait URLs. 422 `no_capable_agent` when no live agent declares the workload (or backend); 400 `no_matching_devices` for a fan-out that matches nothing, and 400 `no_build` when `latest` names no published build | | `POST /api/v1/jobs/validate` | run | Dry run: schema errors by path, the devices it would land on, warnings. Always 200 | | `GET /api/v1/jobs/{id}` | read | One job, with its spec | | `GET /api/v1/jobs/{id}/summary` | read | The verdict across every device, per-device outcomes and artifact refs. Works for a fan-out parent | @@ -31,6 +31,7 @@ disagree: the test suite fails if they do. | `GET /api/v1/artifacts/{sha256}` · `/content` | read | What an artifact is, and its bytes. Active types (HTML, SVG) are served as attachments with a sandbox CSP | | `GET /api/v1/schedules` · `/alerts` | read | Recurring jobs and open alerts | | `GET /api/v1/audit` | admin | Who did what, when tokens are on | +| `GET /metrics` | read | The fleet in Prometheus text format: devices by status, jobs by status, oldest queued job, per-device success ratio over 24 h, open alerts, artifact counts. Computed from the database on each scrape. Not under `/api/v1` and not in the OpenAPI document | **Lists** are `{items, page, per_page, total, pages}` with `page` and `per_page` (≤ 500) query parameters. @@ -45,7 +46,8 @@ disagree: the test suite fails if they do. **Job specs are validated strictly** against `job.schema.json`: an unknown top-level field is refused rather than ignored, so a typo is an error and not a setting that silently did nothing. A workload's params are checked against -its manifest. +its manifest. This is `/api/v1/jobs` (and so `fleetr run`) only: the legacy +`POST /jobs` and `POST /api/jobs` do not refuse unknown fields. **Verdicts** in a summary: `passed`, `failed`, `mixed`, `skipped` (nothing ran, never folded into passed), `cancelled`, and while unfinished `queued`, @@ -57,10 +59,13 @@ unverified; with tokens on, `requested_by` is the token's name, or the person a `delegate` token names in `X-Fleet-Actor`. See [Authentication](deploy/auth.md). **The older routes are deprecated, not removed.** `GET /api/devices`, -`/api/jobs`, `/api/results`, `/devices`, `/schedules`, `GET /jobs/{id}` and +`GET /api/devices/{id}`, `GET /api/jobs`, `GET /api/jobs/{id}`, +`GET /api/results`, `GET /devices`, `GET /schedules`, `GET /jobs/{id}` and `POST /jobs` · `/api/jobs` answer with `Deprecation` (RFC 9745) and a -`Link: <...>; rel="successor-version"` header. There is no sunset date. The -runner protocol is not deprecated. +`Link: <...>; rel="successor-version"` header. Only those exact routes carry +the headers: `/api/devices/{id}/beacons`, `/api/results/bench` and the rest of +the dashboard's read API do not. There is no sunset date. The runner protocol is +not deprecated. Timestamps are ISO-8601 **UTC with a `Z`**. SQLite stores `YYYY-MM-DD HH:MM:SS` with no zone marker, which JavaScript parses as local @@ -69,8 +74,12 @@ Every list is bounded. ## Agent API -Never guarded, whatever `FLEET_DASH_TOKEN` is set to — the fleet has to keep -running whether or not anyone has a token in a browser. +Not guarded by `FLEET_DASH_TOKEN`, with the one exception marked below — the fleet has to keep running whether or +not anyone has a token in a browser. Open by default even with +`FLEET_AUTH=tokens`; with `FLEET_AUTH_PROTOCOL=token` as well, every call here +needs a token with the `agent` scope (a `read` token may still download +artifacts). See [the protocol](protocol.md#authentication-optional) and +[Authentication](deploy/auth.md). | Method & path | Purpose | |---|---| @@ -81,9 +90,15 @@ running whether or not anyone has a token in a browser. | `POST /artifacts` | Upload bytes; returns `sha256` | | `GET /artifacts/:sha256` | Download, supports Range | | `POST /findings` | An `explore` finding from an executor; merged on its fingerprint. See [Findings and verdicts](explore/findings.md#the-api) | +| `POST /api/jobs/:id/mirror` | One raw JPEG frame from an executor driving a device, for the live view. Kept in memory only (the last three frames), never stored; 2 MB cap per frame. The one row here that `FLEET_DASH_TOKEN` does guard; the `explore` workload sends it | ## Queue +With `FLEET_AUTH=tokens`: `POST /jobs` needs `run`; `/jobs/:id/release`, +`/locks/*` and `/events/*` are runner protocol, so they follow +`FLEET_AUTH_PROTOCOL` like the table above; any other GET needs `read` and +anything else `admin`. With auth off, all of it is open. + | Method & path | Purpose | |---|---| | `POST /jobs` | Enqueue. 409 on duplicate `job_id`; the reply's `status` says whether it queued or is `waiting` on `depends_on` | @@ -97,6 +112,8 @@ running whether or not anyone has a token in a browser. | `PATCH /schedules/:id` · `DELETE /schedules/:id` | Enable, disable or remove | | `POST /schedules/tick` | Force an evaluation; fires due schedules at most once a minute | | `POST /power/:pool/:state` | Fire a pool's smart-plug webhook | +| `GET /devices` | The raw registry: descriptor, pools, capabilities, last beacon. Deprecated; `GET /api/v1/devices` replaces it | +| `GET /status-reports` | The last 100 commit-status rows, raw. `GET /api/status-reports` is the dashboard's version | ## Read API @@ -129,6 +146,9 @@ Every endpoint is `GET` and side-effect free. | `GET /api/peers` | Other brains this one knows about, with liveness and device counts | | `GET /api/peers/:id/*` | One peer's read API, fetched server to server. GET only, and only the endpoints on an allow-list | | `GET /api/status-reports` | The commit-status audit trail, posted or not | +| `GET /api/templates` | Saved job templates, with their specs | +| `GET /api/events` · `/api/events/:topic?limit=` | Pipeline-event topics with counts, and one topic's newest events (≤ 500) | +| `GET /api/jobs/:id/mirror` · `/mirror/status` | The live view of a host job as an MJPEG stream (404 when nothing is streaming), and whether a stream exists without opening it | | `GET /api/stream` | SSE | ### `/api/stream` carries a nudge, not a payload @@ -157,15 +177,27 @@ From the [`explore`](explore/index.md) workload. Full shapes and replies are in ## Mutations +With `FLEET_AUTH=tokens`, `POST /api/jobs`, `/api/jobs/preview-targets` and +`/api/jobs/:id/cancel` · `/retry` need `run`, and unlike their `/api/v1` +equivalents they do not check that the job is yours (see +[Authentication](deploy/auth.md#tokens-on-the-collector)). The GET here needs +`read`, `/api/session` needs no token up front, and everything else needs +`admin`. + | Endpoint | Purpose | |---|---| | `POST /api/jobs` | Enqueue; forwards to `POST /jobs` so validation and fan-out have one implementation | -| `POST /api/jobs/:id/cancel` · `/retry` | Cancel, or clone the spec under `-r2` | +| `POST /api/jobs/:id/cancel` · `/retry` | Cancel, or clone the spec under `-r2` (body may set `pool`, `device_id`, `priority`) | | `PATCH /api/jobs/:id` | Set `priority` | | `POST /api/jobs/preview-targets` | "N devices match", using the matcher fan-out uses | | `PATCH /api/devices/:id` | Name, notes, pool override (`pools: null` clears it) | | `DELETE /api/devices/:id` | Forget a device; refuses while it is running a job | | `POST /api/devices/:id/release-lock` | Drop a stuck lock | +| `POST /api/templates` · `DELETE /api/templates/:id` | Save a job template (`id`, `name`, `spec` without a `job_id`), or delete one | +| `POST /api/schedules` · `PATCH` · `DELETE /api/schedules/:id` | Forwarded to `/schedules` | +| `POST /api/schedules/:id/run` | Fire one schedule now, as `-manual-`, without touching its own next firing | +| `POST /api/power/:pool/:state` | Forwarded to `POST /power/:pool/:state` | +| `POST /api/session` · `DELETE /api/session` | Exchange a token for the dashboard's read-only cookie, or clear it. See [Authentication](deploy/auth.md#tokens-on-the-collector) | | `POST /api/visual/baselines/accept` | Set the baseline for (suite, page, profile) | | `GET /api/artifacts/gc-candidates?days=` | Artifacts nothing references | | `DELETE /api/artifacts/:sha256` | Delete one; refuses while referenced or pinned | @@ -177,8 +209,11 @@ From the [`explore`](explore/index.md) workload. Full shapes and replies are in `GET /api/peers/:id/*` proxies rather than letting the browser call the other brain directly, and that is a security decision rather than a convenience. -There is no authentication: opening the read API cross-origin would let any -website in the operator's browser read their whole fleet from any tab. The +There is no authentication by default (see [Authentication](deploy/auth.md)): +opening the read API cross-origin would let any website in the operator's +browser read their whole fleet from any tab. With `FLEET_AUTH=tokens` the proxy +needs `read` like any other GET. It sends no token to the peer, so a peer that +has tokens on answers 401 through it. The proxy keeps one origin, one token, and one place that decides what may be asked for — an **allow-list** of endpoints, because the failure of a deny-list is that the next endpoint somebody adds is exposed by default. @@ -211,10 +246,13 @@ report; both stay visible in `GET /api/devices`. ## The guard `FLEET_DASH_TOKEN` guards every mutation above: set it and the dashboard must -send `X-Fleet-Token`. +send `X-Fleet-Token`. With `FLEET_AUTH=tokens` it steps aside: the scoped token +has already been checked, and every mutation needs `run` or `admin` (see +[Authentication](deploy/auth.md)). **It is deliberately unset on the reference deployment.** The collector is LAN -and tailnet only, `POST /jobs` is open anyway so `curl` and CI keep working, and +and tailnet only with auth off, `POST /jobs` is open then anyway so `curl` and +CI keep working, and the token buys protection against a stray browser tab at the cost of a paste in every browser — not a trade worth making on a home network. It is a speed bump, not an access control. [The network is the access diff --git a/docs/concepts.md b/docs/concepts.md index 44fe2e7..d2718ea 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -66,6 +66,12 @@ registered agent declares.** Anything else is a 422 naming the missing capability, at enqueue time, rather than a job that sits queued forever with nothing to explain why. +Through [`/api/v1/jobs`](api.md#control-api-apiv1) (and so `fleetr run`) the +spec is also validated strictly: an unknown top-level field is refused as +`invalid_spec` rather than ignored, so a typo is an error instead of a setting +that silently did nothing. The older `POST /jobs` does not check for unknown +fields. + That is what lets a new runner add a workload the collector has never heard of without a release in the collector. [Writing a runner](writing-a-runner.md) is the other half of it. @@ -336,6 +342,7 @@ a function in the middle of the executor. | `adb` | Android phones, tablets, TV sticks, headsets and watches — one driver, every Android shape | | `simctl` | booted Apple simulators: iOS, tvOS, watchOS, visionOS | | `devicectl` | cabled or paired Apple hardware: iPhone, iPad, Apple TV, Watch, Vision Pro | +| `roku` | Roku players and Roku TVs on the LAN, found over SSDP and spoken to over ECP. It discovers and launches; it does not install | Every target carries both `platform` (what it is) and `driver` (what reaches it), because the two do not line up: one adb drives four shapes that all call diff --git a/docs/dashboard.md b/docs/dashboard.md index 1c7c1a5..eeb6c64 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -6,12 +6,15 @@ SQLite file. One thing to keep alive, one URL, no CORS. | | | |---|---| | ![Overview](img/overview.png) | ![Devices](img/devices.png) | -| **Overview** — the fleet and the queue at a glance, and what is running now | **Devices** — every phone on the shelf with battery, thermal state and what it is doing | +| **Overview** — the fleet and the queue at a glance, and what is running now | **Devices** — every device on the shelf with battery, thermal state and what it is doing | | ![Jobs](img/jobs.png) | ![Results](img/results.png) | | **Jobs** — the queue, with a composer and cancel/retry | **Results** — benchmark trends, eval accuracy, UI-test matrices, drain curves | | ![Visual](img/visual.png) | ![Alerts](img/alerts.png) | | **Visual** — every page across every profile, diffed against its accepted baseline | **Alerts** — one row per condition for as long as it holds, with acknowledge and snooze | +These screenshots are from 2026-09-05. They predate the device drawings +described below, and there is none of the Findings screen. + ## Building it The dashboard is a Preact app with its own `package.json`, built by Vite, and @@ -36,6 +39,13 @@ button for each one. Device detail has a 24 h battery and thermal chart; job detail has per-device results, beacons and artifacts. +**The Devices screen draws each device as the object it is**: one of 22 +drawings, from phones and tablets through Roku boxes and sticks, an Apple TV, +laptops, watches and a headset to a board and a browser. The drawing is chosen +from the model string first, because agents that predate the `platform` field +are reported as `ios` or `android`, so a Roku arrives as `android` and an +Apple TV or Apple Watch as `ios`. + **Filters live in the URL**, so a filtered view is a link you can send. Jobs can be composed, enqueued, cancelled, retried and reprioritised; devices can be renamed, annotated and re-pooled; schedules enabled, fired now and deleted; @@ -43,7 +53,21 @@ artifacts uploaded and garbage-collected. Alerts appear as a banner on every screen. The layout works on a phone, and `?` lists the keyboard shortcuts — `g j` jobs, `g d` devices, `g f` findings, `g n` new job, -`/` search. +`/` search, and on a finding `r`, `d`, `n` or `a` for its verdict (real, +duplicate, not a bug, the agent's mistake). + +## When the collector wants a token + +If the collector runs with `FLEET_AUTH=tokens`, or has `FLEET_DASH_TOKEN` set, +and this browser holds no token, every screen shows a banner with a field to +paste one. With `FLEET_AUTH=tokens` nothing can be seen or changed without it; +`fleet token create NAME --scope read` on the collector's host makes one, and +`run` or `admin` lets you change things too. With only `FLEET_DASH_TOKEN`, +reads work and the banner asks for that value before anything can be changed. +The token is kept in the browser's local storage and sent as `X-Fleet-Token`. +With tokens on it is also exchanged for an HttpOnly session cookie that the +collector honours for reads only, so images and the live stream work. See +[Authentication](deploy/auth.md). ## Naming devices diff --git a/docs/deploy/alerts.md b/docs/deploy/alerts.md index 85d9249..0119178 100644 --- a/docs/deploy/alerts.md +++ b/docs/deploy/alerts.md @@ -1,6 +1,6 @@ # Alerts -Evaluated every 60 seconds. **Alerts are state, not events**: one row per (rule, +Evaluated every 60 seconds (`FLEET_ALERT_TICK_MS`). **Alerts are state, not events**: one row per (rule, subject) for as long as the condition holds, resolved when it stops. A device offline for six hours is one row with a rising `seen_count`, not 360 notifications — and nothing is notified twice, ever. @@ -16,12 +16,16 @@ notifications — and nothing is notified twice, ever. | `thermal-critical` | A device still reporting is thermally critical | | `low-battery` | Below `FLEET_ALERT_LOW_BATTERY_PCT` (default 15) and not charging | | `job-failed` | A job failed in the last 24 h | -| `job-stuck` | Claimed, lease still being renewed, but no result rows after 2× the lease TTL | -| `benchmark-regressed` | Decode throughput fell more than `FLEET_ALERT_REGRESSION_PCT` (10%) against its own 7-day median | +| `job-stuck` | Claimed, lease still being renewed, but no result rows after `FLEET_ALERT_STUCK_LEASE_MULTIPLE` (default 2) × the lease TTL | +| `benchmark-regressed` | Decode throughput fell more than `FLEET_ALERT_REGRESSION_PCT` (10%) against its own `FLEET_ALERT_REGRESSION_DAYS` (7) day median | | `cold-start-regressed` | Launch time rose past the same threshold against its own median | | `batch-regressed` | Top-1 accuracy fell past the same threshold | -| `schedule-missed` | An enabled schedule that has run before missed its firing by 5 min | -| `db-size` / `log-size` | Past `FLEET_ALERT_DB_BYTES` / `FLEET_ALERT_LOG_BYTES` | +| `schedule-missed` | An enabled schedule that has run before missed its firing by `FLEET_ALERT_SCHEDULE_LATE_S` (default 5 min) | +| `db-size` / `log-size` | Past `FLEET_ALERT_DB_BYTES` (default 2 GiB) / `FLEET_ALERT_LOG_BYTES` (200 MiB) | + +Every threshold is read once, at collector start; a value that is not a +positive number falls back to the default. `job-failed`'s 24-hour window is +fixed. `job-stuck` is the case the lease sweep cannot see: beacons keep renewing the claim, so the job never lapses, and without this rule a runner that is alive but @@ -37,8 +41,9 @@ The comparison key matters more than the threshold. Two runs are only comparable on the same device with the same model, quant and backend, and **comparing across any of those is how a "regression" turns out to be a different model**. -The baseline is the median of the trailing seven days, needs at least four prior -runs, and the alert's subject is keyed on the same tuple the comparison was — so +The baseline is the median of the trailing seven days +(`FLEET_ALERT_REGRESSION_DAYS`), needs at least four prior runs +(`FLEET_ALERT_REGRESSION_MIN_SAMPLES`), and the alert's subject is keyed on the same tuple the comparison was — so a device that has genuinely got slower is one alert with a rising `seen_count`, not a new one every night. @@ -62,6 +67,12 @@ host. Point `FLEET_ALERT_WEBHOOK` at the collector's own loopback port and let [the tunnel's](networking.md) reverse forward deliver it — loopback at both ends, so the alert never crosses the LAN and never leaves the house. +## Reading them + +The Alerts screen, `GET /api/v1/alerts` (`?state=open,acked,snoozed,resolved`; +all but resolved by default, `read` scope when tokens are on), and the +`fleet_alerts_open` gauge on [`GET /metrics`](../api.md#control-api-apiv1). + ## Quieting one **Acknowledge** keeps an alert listed but stops it nagging. **Snooze** quiets it diff --git a/docs/deploy/auth.md b/docs/deploy/auth.md index c496f55..b644a06 100644 --- a/docs/deploy/auth.md +++ b/docs/deploy/auth.md @@ -24,11 +24,15 @@ environment or in the `env` object of `~/.fleet/config.json` (`"env": {"FLEET_AUTH": "tokens"}`). The token file is re-read when it changes, so creating and revoking need no restart. +The file is `$FLEET_DATA_DIR/tokens.json` (`~/.fleet/data/tokens.json` under +`fleet up`). `FLEET_TOKENS_FILE` moves it for the collector, and +`fleet token ... --tokens-file PATH` points the command at the same place. + | Scope | Allows | |---|---| | `read` | Every GET: devices, jobs, results, artifacts, the dashboard's data | -| `run` | Enqueue and validate jobs; cancel or retry the jobs this token started | -| `admin` | Everything else: other people's jobs, devices, schedules, the audit log | +| `run` | Enqueue and validate jobs; cancel or retry the jobs this token started (checked on `/api/v1` only, see below) | +| `admin` | Everything else: other people's jobs, devices, schedules, the audit log, and the `shell` workload | | `agent` | The runner protocol, when `FLEET_AUTH_PROTOCOL=token` | | `delegate` | Name the person a request is for (`X-Fleet-Actor`). For a gateway only | @@ -36,6 +40,17 @@ changes, so creating and revoking need no restart. `delegate`, because a token that can put a name on a job can put *anyone's* name on it. +**Two checks are made on `/api/v1` only, and not yet on the older routes.** +`POST /api/v1/jobs/{id}/cancel` and `/retry` refuse a `run` token acting on a +job somebody else started; the dashboard's `POST /api/jobs/:id/cancel` and +`/retry` need `run` and check nothing more, so any `run` token can cancel or +retry any job there. Likewise `POST /api/v1/jobs` refuses the `shell` workload +to a token without `admin`, but `POST /jobs` and `POST /api/jobs` accept it +from any `run` token. Both are gaps in the code, not intended behaviour. Until +they close, treat `run` as able to stop anyone's job. For `shell`, what bounds +it is each machine's own allowlist of script hashes: a machine with no +allowlist file never declares `shell`, so nothing can be sent to it. + **Runners stay open unless you close them.** Phones, TVs and the browser runner do not send tokens, so with `FLEET_AUTH=tokens` the runner protocol still answers without one. `FLEET_AUTH_PROTOCOL=token` requires the `agent` @@ -43,10 +58,12 @@ scope; set it only when every runner can send `FLEET_TOKEN` (runner-machine and the host executor can). **People** run `fleetr login` once per collector; the token is saved to -`~/.fleet/credentials.json` (mode 0600), keyed by URL. **The dashboard** asks -for a token in a banner and exchanges it for an HttpOnly, SameSite=Strict -cookie that is honoured for reads only, so a link on another site cannot -make the browser change anything. +`~/.fleet/credentials.json` (mode 0600), keyed by URL. `fleetr whoami` shows +what the collector makes of it and `fleetr logout` forgets it; which token wins +when several are set is in [AI agents and automation](../agents.md#the-cli). +**The dashboard** asks for a token in a banner and exchanges it for an +HttpOnly, SameSite=Strict cookie that is honoured for reads only, so a link on +another site cannot make the browser change anything. **Every change is audited**: who, through which front door (`source`), what, and the result, readable at `GET /api/v1/audit` with `admin`. @@ -95,6 +112,10 @@ stateless, so any number of replicas can sit behind one. | Flag | Environment | Default | | |---|---|---|---| +| `--url` | `FLEET_RUNNER_URL`, `FLEET_URL` | the config | The collector, resolved the way the CLI resolves it | +| `--token` | `FLEET_RUNNER_TOKEN`, `FLEET_TOKEN` | saved login | The gateway's own collector token (`run,delegate`) | +| `--host` | `FLEET_MCP_HOST` | `127.0.0.1` | Address to listen on. `0.0.0.0` behind a proxy | +| `--port` | `FLEET_MCP_PORT` | 8790 | | | `--issuer` | `FLEET_MCP_ISSUER` | | The IdP's issuer, exactly as in the token's `iss` | | `--audience` | `FLEET_MCP_AUDIENCE` | | This resource, as in `aud`. Required with `--issuer` | | `--jwks` | `FLEET_MCP_JWKS` | discovered | Signing keys URL, if not at the issuer's OpenID configuration | @@ -107,7 +128,8 @@ stateless, so any number of replicas can sit behind one. | `--no-auth` | | | Development only; refused off loopback | Endpoints: `POST /mcp`, `GET /.well-known/oauth-protected-resource/mcp` -(RFC 9728: tells a client which IdP to sign in with), `GET /healthz`. A +(RFC 9728: tells a client which IdP to sign in with; also answered without the +`/mcp` suffix), `GET /healthz`. A request without a valid token gets a 401 whose `WWW-Authenticate` header points at that metadata, which is how an MCP client discovers where to log in. diff --git a/docs/deploy/explore-on-ultra.md b/docs/deploy/explore-on-ultra.md index 34d0ad6..fdd7323 100644 --- a/docs/deploy/explore-on-ultra.md +++ b/docs/deploy/explore-on-ultra.md @@ -106,8 +106,10 @@ Serve it two ways and keep the faster one behind the gateway name `pilot`: ``` Merge `deploy/explore/litellm-pilot.yaml` into the gateway's `model_list`. -`pilot` has no cloud fallback on purpose. Then put the gateway key in the -Keychain, where the workload reads it: +`pilot` has no cloud fallback on purpose. On ultra the gateway key is the file +`~/.config/harness/keys/fleet.key` (see the table at the top: nothing started +over SSH or by launchd can write the login Keychain there). On a laptop, the +Keychain works: ```bash security add-generic-password -s fleet-explore-gateway -a gateway -w @@ -180,10 +182,12 @@ replayed real-build findings marked real. Under 30% real: switch the judge off ## Phase 3: the TV question The fake Roku first (`npx @tvloop/fakeroku --agent`), then the real Roku once -its developer password is in fleet-host's Keychain, then Fire TV through adb -D-pad keys. The TV cards are in `examples/missions/fakeroku/` and -`examples/missions/dozehound-tv/`; each bench card's end state is a focused -element the harness can read. +its developer password is in fleet-host's Keychain (WB-101). Fire TV was the +next step and is parked (WB-297, above), so the question is answered on a tvOS +simulator with the Dozehound stub catalog: the `fleet-explore-tv` night. The TV +cards are in `examples/missions/fakeroku/`, `examples/missions/dozehound-tv/` +and `examples/missions/dozehound-tvos/`; each bench card's end state is a +focused element the harness can read. **Exit:** the better of Holo4 and plain Qwen completes at least half of 20 navigation missions. If neither does, TVs keep scripted tvloop replays with the @@ -213,7 +217,8 @@ FLEET_URL=http://127.0.0.1:18788 npm run seed:schedules ultra is on the Mini's side of the home network and cannot route to fleet-host's 192.168.50.x address, so the tunnel goes over the tailnet (`fleet-ts`). Turn `nightly-explore-greenfolio` on from the Schedules page after -phase 2 passes; leave `nightly-explore-dozehound-tv` off until phase 3 does. +phase 2 passes. Leave `nightly-explore-dozehound-tv` off: it is the Fire TV +schedule, parked with Fire TV itself. On the brain, set `FLEET_FINDINGS_DIGEST_AT=07:30` for the morning digest through the alert webhook. GitHub issues stay dry-run until both @@ -222,7 +227,8 @@ through the alert webhook. GitHub issues stay dry-run until both `/api/findings/issues` shows what would have been filed. **Exit:** seven nights in a row without help, morning triage under ten minutes, -and no night over five new findings per app. +and no night over five new findings per app (the bug garden's night allows 25 +on purpose and does not count). ## Guards diff --git a/docs/deploy/headless.md b/docs/deploy/headless.md index 8b6a06a..d149b17 100644 --- a/docs/deploy/headless.md +++ b/docs/deploy/headless.md @@ -14,8 +14,8 @@ a Mac collector reachable at all. `collector/deploy/install-agent.sh` and `runner-machine/deploy/install-agent.sh` are superseded by [`fleet service install`](index.md#keeping-it-up-fleet-service), which writes - one unit running `fleet up` instead of one unit per component. They ship one - more release and are then removed. + one unit running `fleet up` instead of one unit per component. Deprecated + since 0.5.0, they still ship in 0.8.0 and no removal date is set. They are documented here rather than deleted because they have more mileage: `fleet service install` now runs a real fleet's brain under launchd diff --git a/docs/deploy/index.md b/docs/deploy/index.md index e9feb26..8760078 100644 --- a/docs/deploy/index.md +++ b/docs/deploy/index.md @@ -65,9 +65,9 @@ depends on the machine: Also run, on macOS, are the older per-component plists in `collector/deploy/` and `runner-machine/deploy/`. Those are what the reference deployment actually uses today, and they are documented in - [headless hosts](headless.md). They are deprecated -- they ship one more - release and are then removed in favour of `fleet service` -- but they are - the ones with mileage on them. + [headless hosts](headless.md). They have been deprecated in favour of + `fleet service` since 0.5.0 and still ship in 0.8.0, with no removal date + set, because they are the ones with mileage on them. **One unit, not five.** The old deployment had a plist each for the collector, the executor, the iOS executor, the tunnel and the agent. One unit is the @@ -100,10 +100,11 @@ supervisor, which is the right division of labour. ## Binding and exposure -**There is no authentication, by design.** Anyone who can reach the collector +**There is no authentication by default.** Anyone who can reach the collector can enqueue a job. `FLEET_DASH_TOKEN` guards the dashboard's mutations but -`POST /jobs` stays open so `curl` and CI keep working. **Do not put this on the -internet.** +`POST /jobs` stays open so `curl` and CI keep working. Scoped tokens are opt-in +(`FLEET_AUTH=tokens`, and `FLEET_AUTH_PROTOCOL=token` for the runners): see +[Authentication](auth.md). With auth off, **do not put this on the internet.** The network is the access control, and `FLEET_BIND` is what decides which network that is. It takes a comma-separated list of addresses to answer on and @@ -121,6 +122,35 @@ wifi, no guest network. That matters more than it looks, because the LAN-only assumption stops being true the moment one of your agents is a laptop that leaves the house. +### The other collector settings + +`fleet up` sets the paths below from `~/.fleet`; started by hand, the +collector resolves the relative defaults against its working directory. The +port, bind list, tailnet allowlist, data and artifact directories and +`FLEET_NAME` are covered above and in [Install](../install/index.md). + +| Variable | Default | | +|---|---|---| +| `FLEET_DISCOVERY` | off (`fleet up` passes `collector.discovery`) | `1` advertises the collector over mDNS, below | +| `FLEET_PEERS` | none | Comma-separated base URLs of other collectors, for [more than one brain](#more-than-one-brain) | +| `FLEET_POWER_CONFIG` | `power.json` | The pool-to-smart-plug map, see [Energy](#energy) | +| `FLEET_SWEEP_MS` | 15000 | How often expired leases are swept | +| `FLEET_SCHEDULER_TICK_MS` | 20000 | How often schedules are evaluated | +| `FLEET_DASH_DIST` | `dash/dist` | The built dashboard. Missing, the collector serves a placeholder rather than failing | +| `FLEET_LOG` | `info` | Log level | +| `FLEET_LOG_FILE` | `~/Library/Logs/fleet-collector.log`; `~/.fleet/logs/collector.log` under `fleet up` | Where the service manager sends the collector's output. The collector does not write it; it reports its size and alerts on it (`log-size`) | +| `FLEET_TOKENS_FILE` | `$FLEET_DATA_DIR/tokens.json` | See [Authentication](auth.md) | + +Alert thresholds are `FLEET_ALERT_*`, listed in [Alerts](alerts.md). + +Three more belong to the machine agent, not the collector: + +| Variable | Default | | +|---|---|---| +| `FLEET_DEVICE_TTL_S` | unset: a permanent shelf device | Registers with `ttl_s`, for an agent whose absence is normal. See [the protocol](../protocol.md#ttl_s-if-you-are-temporary). `fleet up` passes `agent.ttlS` | +| `FLEET_SHELL_ALLOWLIST` | `$FLEET_CACHE_DIR/shell-allowlist.txt` (`~/.fleet/cache/` under `fleet up`) | The sha256 list of scripts the `shell` workload may run. No file, no `shell` capability | +| `FLEET_SERVE_BIND` | `loopback` | Where the `serve` workload's model server listens when the job does not say: `loopback`, `tailnet`, or this machine's tailnet address. Never a wildcard or a LAN address | + ### mDNS advertising is off by default `fleet config set collector.discovery true` makes a collector announce itself as @@ -189,9 +219,9 @@ brain A is A's job and lands in A's database, whatever device ran it. `GET /api/peers` lists other brains and `/api/peers/:id/*` proxies one peer's read API server to server, which the dashboard's brain switcher uses. The proxy -rather than CORS is a security decision: there is no authentication, so opening -the read API cross-origin would let any website in the operator's browser read -the whole fleet from any tab. Peers are addressed by stable id rather than URL, +rather than CORS is a security decision: there is no authentication by default +(see [Authentication](auth.md)), so opening the read API cross-origin would let +any website in the operator's browser read the whole fleet from any tab. Peers are addressed by stable id rather than URL, so a link cannot be turned into a request to an arbitrary host by editing the address bar; only an allow-list of endpoints is proxied; and nothing that mutates is proxied at all. @@ -294,5 +324,8 @@ Three things are deliberate and worth knowing before you trust a figure: - **[Networking](networking.md)** -- the SSH tunnel, why it exists, and network shaping. - **[The iOS executor](ios-executor.md)** -- standing one up on a Mac with Xcode. +- **[Exploratory QA on ultra](explore-on-ultra.md)** -- the executor, emulator, + simulators and model gateway behind the overnight explore jobs. Listed under + Exploratory QA in the navigation. - **[Alerts](alerts.md)** -- the rules, and getting a notification somewhere you will see it. diff --git a/docs/explore/findings.md b/docs/explore/findings.md index 8f0267e..52dc2d7 100644 --- a/docs/explore/findings.md +++ b/docs/explore/findings.md @@ -98,8 +98,10 @@ fact created. ### `POST /findings` -From executors. Open, like `POST /results`: the collector's access control for -executors is the network, not the dashboard token. The body is an +From executors. A protocol route, like `POST /results`: by default the +collector's access control for executors is the network, not the dashboard +token, and with `FLEET_AUTH_PROTOCOL=token` it needs a token with the `agent` +scope ([Authentication](../deploy/auth.md)). The body is an `ExploreFinding`: ```json diff --git a/docs/explore/index.md b/docs/explore/index.md index 30b7689..378041f 100644 --- a/docs/explore/index.md +++ b/docs/explore/index.md @@ -138,9 +138,15 @@ a second report. Your verdicts decide which checks keep running. ## Honest status -Built and tested on 2 October 2026, before the Mac Studio it is meant to run on -had arrived. Every real-device run so far used **Qwen3.5-2B** as a stand-in -model on a laptop, so those runs say the harness works and say nothing about how -good the intended model is. The intended driver is **Holo4 35B-A3B** (Apache -2.0); every Holo4 number in these pages is the vendor's own until -[the measurements](measuring.md) are taken on ultra. +Built and tested on 2 October 2026 on a laptop, with **Qwen3.5-2B** standing in +for the real model. Those runs say the harness works and nothing about how good +the intended model is. + +On 8 October the intended driver, **Holo4 35B-A3B** (Apache 2.0), was +[measured on ultra](../deploy/explore-on-ultra.md#measured-on-8-october-2026): +100 of 100 pointing targets, a median whole step of 6.8 s, and a tool call on +all 40 steps of a mission once `tool_choice` was `required`. Its first +bug-garden mission filed BG-01 and BG-03, and a night through the brain filed +BG-16, each reproduced 2/2. That is one model on one fixture app. No +model-driven night has run on an Apple surface yet, and the phase exits in the +[ultra runbook](../deploy/explore-on-ultra.md) are still the bar. diff --git a/docs/explore/internals.md b/docs/explore/internals.md index c1ad7f0..85e1561 100644 --- a/docs/explore/internals.md +++ b/docs/explore/internals.md @@ -30,8 +30,12 @@ collector/scripts/explore-run.ts missions on a device, no collec collector/scripts/explore-bench/ pointing, steptime, image-size, bakeoff, garden-score collector/scripts/explore-apple-check.ts FleetDriver end to end on a simulator collector/scripts/explore-tv-check.ts the TV actuators end to end +collector/scripts/explore-night.ts ultra's night-queue command: devices, tunnel, one job collector/examples/missions/ mission cards -collector/deploy/explore/ gateway names, Holo4 fetch, night device script +collector/examples/jobs/explore-*-ultra.json the specs ultra's nights run +collector/deploy/explore/ gateway names, Holo4 fetch, night-devices.sh +collector/deploy/explore/nightly-builds.sh tonight's iOS and tvOS builds from origin/develop +collector/deploy/explore/night/ the four night-queue manifests and their priorities ``` ## The contract diff --git a/docs/explore/job.md b/docs/explore/job.md index 4766967..916b819 100644 --- a/docs/explore/job.md +++ b/docs/explore/job.md @@ -25,14 +25,14 @@ And the weekly mission bench: | `bench` | `false` | Run only `bench-` cards, with no judge; file nothing; report end states reached | | `model` | see [below](#models) | The driver | | `judge` | see below | The second model, or `false` for none | -| `minutes` | 150 | The night's budget, per device | +| `minutes` | 150 | The night's budget, per device (1 to 600) | | `steps` | | Cap every card at this many steps, for a short trial night | -| `findings_per_night` | 5 | New findings filed per app per night | +| `findings_per_night` | 5 | New findings filed per app per night (0 to 50) | | `replay_attempts` | 2 | Clean-install replays per candidate (1 to 5) | | `confirm_per_mission` | 6 | Candidates replayed per mission, worst first | | `conditions` | `["baseline"]` | Rotated across the night's missions; see [below](#conditions) | -| `changed` | | `{repo, since}` reads `git log`; `{files: [...]}` takes a list. Puts matching cards first | -| `previous_app` | | `{sha256}` of the previous build; each reproduced finding is replayed on it too | +| `changed` | | `{repo, since}` reads `git log` (`since` is a sha or a git date, default `24 hours ago`); `{files: [...]}` takes a list. Puts matching cards first | +| `previous_app` | | `{sha256, build?}` of the previous build; each reproduced finding is replayed on it too | | `surface` | from the device | `touch` or `dpad`. Android TV, Fire TV, Roku and tvOS default to `dpad` | | `credentials` | | `{account, email_var, password_var}` for setup flows; the password comes from the host's Keychain | | `mirror` | `true` | Stream each step's screenshot to the job's live view on the dashboard | @@ -54,8 +54,11 @@ And the weekly mission bench: | `extra_body` | | Merged into every request, for server-specific switches | The API key is never in the spec (the collector refuses specs carrying -`api_key` or `token`). It comes from the executor's environment -(`FLEET_EXPLORE_API_KEY`) or its Keychain: +`api_key` or `token`). The executor looks for it in this order: +`FLEET_EXPLORE_API_KEY`, then `HARNESS_KEY` (what ultra's night queue hands +its jobs), then the file `FLEET_EXPLORE_API_KEY_FILE` (default +`~/.config/harness/keys/fleet.key`; a process started over SSH or by launchd +cannot write the login Keychain), then the Keychain: ```bash security add-generic-password -s fleet-explore-gateway -a gateway -w @@ -88,11 +91,14 @@ the judge which language to expect, so English left in a Spanish build is a | Variable | Default | | |---|---|---| -| `FLEET_EXPLORE_BASE_URL` | `http://ultra.local:4000` | The gateway | +| `FLEET_EXPLORE_BASE_URL` | `HARNESS_GATEWAY`, then `http://ultra.local:4000` | The gateway | | `FLEET_EXPLORE_MODEL` | `pilot` | The driver's name on it | | `FLEET_EXPLORE_JUDGE_MODEL` | `vision` | The judge's name on it | -| `FLEET_EXPLORE_API_KEY` | | Otherwise the Keychain item `fleet-explore-gateway` | -| `FLEET_MISSIONS_DIR` | `examples/missions` | Mission cards | +| `FLEET_EXPLORE_API_KEY` | | Otherwise `HARNESS_KEY`, the key file, then the Keychain item `fleet-explore-gateway` | +| `FLEET_EXPLORE_API_KEY_FILE` | `~/.config/harness/keys/fleet.key` | The key as a file | +| `FLEET_KEYCHAIN_SERVICE` | `fleet-ui-test` | The Keychain service holding the sign-in passwords `credentials` names | +| `FLEET_DASH_TOKEN` | | Sent with live-view frames, when the brain guards the dashboard with one | +| `FLEET_MISSIONS_DIR` | `examples/missions` in the working directory, then beside the bundle | Mission cards | | `FLEET_FLOWS_DIR` | `examples/flows` | Setup flows | | `FLEET_STATE_DIR` | `~/.fleet` | Screen maps and last-run times live under `explore//` | | `FLEET_TVLOOP_DIR` | `~/tvloop` | A built tvloop checkout, for Rokus | @@ -159,6 +165,21 @@ recommended: a night that died halfway should not start again at 3 a.m. | Schedule | When | What | |---|---|---| | `nightly-explore-greenfolio` | 22:00 daily, window 22–01 | GreenFolio Android on ultra, 100 minutes, six conditions, aimed at the day's commits | -| `nightly-explore-dozehound-tv` | 23:45 daily | Dozehound on Fire TV, D-pad, 40 minutes. Leave off until a model passes the TV test | +| `nightly-explore-dozehound-tv` | 23:45 daily | Dozehound on Fire TV, D-pad, 40 minutes. Parked: the Fire TV app needs the production backend, which is the owner's call (WB-297); the TV question is being answered on tvOS instead | | `weekly-explore-shelf` | Wednesdays 23:00 | GreenFolio on the Galaxy S8+ (Android 9) through fleet-host's executor | | `weekly-explore-bench` | Sundays 22:00 | The bug garden's bench cards, for a weekly model score | + +The nights that actually run today do not come from these schedules. On ultra +the harness's night queue runs `scripts/explore-night.ts` from the manifests in +`collector/deploy/explore/night/`, each with a job spec from +`examples/jobs/`: + +| Night job | Priority | Spec | +|---|---|---| +| `fleet-explore-builds` | 19 | none: `deploy/explore/nightly-builds.sh` builds GreenFolio iOS and Dozehound tvOS from `origin/develop` and uploads them | +| `fleet-explore` | 20 | `explore-garden-ultra.json`, the bug garden on the Android emulator | +| `fleet-explore-tv` | 85 | `explore-dozehound-tvos-ultra.json` and the two tvOS bench specs, on a tvOS simulator | +| `fleet-explore-apple` | 86 | `explore-greenfolio-ios-ultra.json`, on an iOS simulator | + +A lower number runs earlier; the products' own night jobs sit between 20 and +85. [Running it on ultra](../deploy/explore-on-ultra.md) has the setup. diff --git a/docs/explore/missions.md b/docs/explore/missions.md index 3948da1..781ffa1 100644 --- a/docs/explore/missions.md +++ b/docs/explore/missions.md @@ -119,8 +119,8 @@ evaluate without a model: | `focused_label` | The focused element's label contains it (TV cards) | | `screen` | The final screen's name contains it | -Bench cards run only in bench mode (`params.bench: true`) and exploration cards -only outside it. Bench missions run with no judge and file nothing; the number +Bench cards run only in bench mode (`params.bench: true`), unless `missions` +names them, and exploration cards only outside it. Bench missions run with no judge and file nothing; the number that comes out is how many end states the model reached. See [Measuring a model](measuring.md#the-mission-bench-b4). @@ -132,5 +132,5 @@ that comes out is how many end states the model reached. See | `greenfolio/` | 3 Android: the sign-in gate, forgot password, a read-only signed-in tour | | `greenfolio-ios/` | 2 iOS, offline: the sign-in gate, creating an account with no network | | `dozehound-tv/` | Android TV / Fire TV: 3 exploration, 10 bench | -| `dozehound-tvos/` | tvOS: 1 exploration, 1 bench (its end-state text is unverified) | +| `dozehound-tvos/` | tvOS: 1 exploration, 10 bench, end states taken from a hand walk on a tvOS 26.5 simulator | | `fakeroku/` | 10 bench, for tvloop's fake Roku | diff --git a/docs/explore/models.md b/docs/explore/models.md index 0313138..aab84b8 100644 --- a/docs/explore/models.md +++ b/docs/explore/models.md @@ -16,6 +16,11 @@ to agree with itself. as `{...}`, or as a bare JSON object with `name` and `arguments`, are accepted too, so a model is not marked down for where its chat template puts them. +- **`tool_choice: "required"`.** The loop sends it on every turn. Holo4 under + `auto` sometimes answered its second turn in prose; a server that ignores the + field gets the same failure. +- **For the judge, `response_format` of type `json_schema`.** The judge asks + for a schema, not `json_object`, because LM Studio refuses `json_object`. - **About 16k tokens of context** by the middle of a mission, more with `keep_shots` raised. @@ -156,7 +161,7 @@ On a laptop that cannot hold Holo4, these were used: | Model | Size (Q4 + projector) | Licence | Notes | |---|---|---|---| -| Qwen3.5-2B | 1.3 GB + 0.7 GB | Apache-2.0 | Drove every real-device run so far. Needs `tree_hints` | +| Qwen3.5-2B | 1.3 GB + 0.7 GB | Apache-2.0 | Drove the laptop runs. Needs `tree_hints` | | Holo2-4B | 2.7 GB + 0.5 GB | Apache-2.0 | H Company's earlier GUI model; a reasoning model, so slow | | GUI-Owl-1.5-2B | 1.1 GB + 0.4 GB | MIT | A GUI-specialised small model | diff --git a/docs/explore/safety.md b/docs/explore/safety.md index b300afc..e7849f9 100644 --- a/docs/explore/safety.md +++ b/docs/explore/safety.md @@ -94,8 +94,9 @@ file an issue, or change a check; those are your verdicts. - **Debug builds and test accounts only.** A card whose build talks to production blocks anything that writes; see [production data](missions.md#production-data). -- **Budgets.** Steps and minutes per card, minutes per night, and at most five - new findings per app per night. +- **Budgets.** Steps and minutes per card, minutes per night, and a cap on new + findings per app per night: five by default, raised to 25 for the bug garden, + whose whole point is to be found. - **It files and never closes.** Even with GitHub issues armed, the collector only ever creates issues. - **No cloud fallback for the driver.** The `pilot` gateway name has none, so diff --git a/docs/explore/surfaces.md b/docs/explore/surfaces.md index 6e5d14b..2e55535 100644 --- a/docs/explore/surfaces.md +++ b/docs/explore/surfaces.md @@ -37,7 +37,8 @@ What it uses, all over adb: | Launch | The launcher activity, or the leanback launcher's on a TV; launch arguments become string extras | | Crashes | `logcat -b crash` for the package, as a running total; `logcat -b events` for `am_anr` | -**A TV** is recognised by the `android.software.leanback` feature. It gets +**A TV** is recognised by the `android.software.leanback` or +`android.hardware.type.television` feature. It gets `surface: "dpad"`, no touch tools, and a **focus line**: the focused node, named by its own label or, on Compose, the text inside it, with the labels of the containers around it. diff --git a/docs/getting-started.md b/docs/getting-started.md index 5e961c1..13bf3aa 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -127,7 +127,7 @@ than one brain needs. `~/.fleet/logs/agent.log`: ``` -[machine-macbookpro] fleet-runner-machine 0.5.0-dev on darwin/arm64, collector http://127.0.0.1:8788 +[machine-macbookpro] fleet-runner-machine 0.8.0 on darwin/arm64, collector http://127.0.0.1:8788 [machine-macbookpro] registered with http://127.0.0.1:8788; capabilities: benchmark, build, build:gradle, build:xcode, build:npm, self-check, llm-eval ``` @@ -175,6 +175,23 @@ curl -s http://127.0.0.1:8788/jobs/hello-fleet `queued` → `claimed` → `done`, in a few seconds. +!!! tip "The same with the CLI" + + `curl` is here because it shows exactly what goes over the wire, and these + unversioned routes keep working. Day to day, `fleetr` (installed beside + `fleet`, the same program) does steps 4 and 5 in one command, validating + the spec against [`/api/v1`](api.md#control-api-apiv1) first and waiting for + the verdict: + + ```bash + fleetr run benchmark --backend synthetic --pool machines \ + --param prompt_tokens=256 --param gen_tokens=64 \ + --param warmup_iters=1 --param measure_iters=3 + ``` + + Its exit code is the verdict: 0 passed, 1 failed, 4 skipped because nothing + ran. [AI agents and automation](agents.md) has the rest of the commands. + !!! tip "`targets.pool` has to match" The machine agent registers into `machines`. Ask for `ml-capable` -- the pool @@ -349,7 +366,7 @@ FLEET_URL=http://127.0.0.1:8788 npm start ``` ``` -[machine-your-hostname] fleet-runner-machine 0.5.0-dev on darwin/arm64, collector http://127.0.0.1:8788 +[machine-your-hostname] fleet-runner-machine 0.8.0 on darwin/arm64, collector http://127.0.0.1:8788 [machine-your-hostname] registered with http://127.0.0.1:8788; capabilities: benchmark, build, build:npm, self-check, llm-eval ``` @@ -365,7 +382,11 @@ it needs on `PATH`. - **[Concepts](concepts.md)** -- leases, capabilities and targeting, which is what you need before writing a job spec that does something interesting. -- **[Workloads](workloads/index.md)** -- the other twenty-odd things it can run. +- **[Workloads](workloads/index.md)** -- the other thirty-odd things it can run. +- **[Exploratory QA](explore/index.md)** -- a vision model using tonight's build + overnight, and the bugs it reproduces. +- **[AI agents and automation](agents.md)** -- `fleetr`, the skill and the MCP + tools, for scripts, CI and coding agents. - **[Adding devices](install/devices.md)** -- phones, televisions, Rokus, browsers. - **[Wire in your app](integration/index.md)** -- publish builds on merge and run @@ -375,8 +396,8 @@ it needs on `PATH`. !!! warning "Before you put it on a network" - There is no authentication, by design -- anyone who can reach the collector - can enqueue a job. `fleet up` binds every interface and warns that it did. + By default there is no authentication -- anyone who can reach the collector + can enqueue a job. Scoped tokens are opt-in ([Authentication](deploy/auth.md)). `fleet up` binds every interface and warns that it did. `FLEET_BIND` (or `fleet config set collector.bind`) decides which networks it answers on, and loopback plus your tailnet address is the configuration most people want. [Deploy](deploy/index.md#binding-and-exposure) has the detail. diff --git a/docs/index.md b/docs/index.md index 713bb7e..109ac8e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -49,7 +49,14 @@ fleet on. The answer is macOS; the suites run on all three. --- - All 28 of them, what each measures, and what each refuses to guess. + All 34 of them, what each measures, and what each refuses to guess. + +- __[Exploratory QA](explore/index.md)__ + + --- + + A vision model uses tonight's build the way a person would, and files + only the bugs that reproduce on a clean install. - __[The protocol](protocol.md)__ @@ -96,21 +103,23 @@ of phones. ## A warning worth reading before you deploy anything -**There is no authentication, by design.** The collector is meant for a home -LAN or a tailnet, and anyone who can reach it can enqueue a job. This is stated +**By default there is no authentication.** The collector is meant for a home +LAN or a tailnet, and anyone who can reach it can enqueue a job. Scoped tokens +and an audit log are opt-in with `FLEET_AUTH=tokens` +([Authentication](deploy/auth.md)); the default does not change. This is stated as a posture rather than apologised for, and [the security policy](https://github.com/addisdev/fleet-runner/blob/main/SECURITY.md) describes what that does and does not cover. Do not put it on the internet. ## Where things live -The code is one repository with seven components that ship independently and +The code is one repository with eight components that ship independently and share no code, only the protocol. One brain -- [`collector/`](https://github.com/addisdev/fleet-runner/tree/main/collector), -which also holds the host executor and the browser runner it serves at -`/runner`. +which also holds the host executor, the [exploratory QA](explore/index.md) +workload, and the browser runner it serves at `/runner`. Five runners, in five languages, sharing not one line: [`runner-android/`](https://github.com/addisdev/fleet-runner/tree/main/runner-android) diff --git a/docs/install/docker.md b/docs/install/docker.md index eb6d5cf..7f07282 100644 --- a/docs/install/docker.md +++ b/docs/install/docker.md @@ -91,7 +91,10 @@ objection to QEMU does not apply here. and a loopback bind would make the published port answer nothing at all. Everything in [binding and exposure](../deploy/index.md#binding-and-exposure) still applies to the *host* side of that mapping. **There is no - authentication.** + authentication by default**. [Authentication](../deploy/auth.md) turns it + on: `-e FLEET_AUTH=tokens`, and the first token created inside the container, + `docker exec node /app/bin/fleet.mjs token create ops --scope admin`, + so it lands on the volume. Nobody has done that in a container yet. - **A `HEALTHCHECK` that asks `/api/health`** rather than checking the process is alive. A collector whose port is bound but whose database will not open is exactly the failure that looks healthy from the outside. diff --git a/docs/install/index.md b/docs/install/index.md index e1cf6a1..de6d590 100644 --- a/docs/install/index.md +++ b/docs/install/index.md @@ -50,7 +50,7 @@ written into both files: `SHASUMS256.txt` cannot be fetched at all.** There is no `--force` past this. 4. Unpack into `~/.fleet` (`FLEET_INSTALL_DIR` moves it), replacing only the entries a release owns -- `bin`, `runner-web`, `dash`, `examples`, - `schemas`. It never removes the directory itself, because + `schemas`, `skills`. It never removes the directory itself, because `~/.fleet` is also where `config.json`, `data/`, `artifacts/` and `logs/` live, and an upgrade that deletes your results is worse than one that fails. 5. Run `fleet version` from the thing it just installed. If the bundle is @@ -69,7 +69,7 @@ something is wrong; stop and read it. ``` ~/.fleet/ config.json what fleet up reads; `fleet config set` writes it - bin/ fleet.mjs, and a `fleet` symlink (a fleet.cmd shim on Windows) + bin/ fleet.mjs, with `fleet` and `fleetr` symlinks (fleet.cmd and fleetr.cmd on Windows) data/ fleet.db, and collector.json -- this brain's name and id artifacts/ build products, screenshots, everything a job published logs/ one per supervised component @@ -107,3 +107,5 @@ to. browser to a fleet that is already running. - **[Deploy](../deploy/index.md)** -- keeping it up, and the network gotcha that costs an evening. +- **[AI agents and automation](../agents.md)** -- `fleet skill install` and + `fleet mcp`, so a coding agent can run jobs on the fleet. diff --git a/docs/integration/add-a-workload.md b/docs/integration/add-a-workload.md index 58662f3..a835ffa 100644 --- a/docs/integration/add-a-workload.md +++ b/docs/integration/add-a-workload.md @@ -5,10 +5,17 @@ collector. ## Why this works at all -`POST /jobs` accepts the workload names the collector and host executor ship -with, **plus any workload some registered agent declares in its capabilities**. -Anything else is a 422 naming the missing capability, at enqueue time, rather -than a job that sits queued forever with nothing to explain why. +`POST /api/v1/jobs` (and the older `POST /jobs`) accepts the workload names +the collector and host executor ship with, **plus any workload some registered +agent declares in its capabilities**. Anything else is a 422, +`no_capable_agent`, naming the missing capability, at enqueue time, rather than +a job that sits queued forever with nothing to explain why. + +`fleetr workloads` (`GET /api/v1/workloads`) lists an agent-declared workload +alongside the shipped ones, with the summary "Declared by N live agents; the +collector publishes no description or params for it." Its `params` are +therefore not validated by the collector: the agent is the only thing that +checks them, so it should refuse bad ones with a result that says why. So the entire path is: implement it in an agent, add it to that agent's route table, restart the agent. The job becomes enqueueable the moment it registers. diff --git a/docs/integration/add-your-app.md b/docs/integration/add-your-app.md index 70e6c1a..35f8b17 100644 --- a/docs/integration/add-your-app.md +++ b/docs/integration/add-your-app.md @@ -8,11 +8,15 @@ the code. | | Directory | Override | |---|---|---| -| Maestro flows | `collector/flows//` | `FLEET_FLOWS_DIR` | -| Playwright specs | `collector/web-specs//` | `FLEET_WEB_SPECS_DIR` | -| Visual capture manifests | `collector/web-specs//shots.json` | | -| Audit config | `collector/web-specs//audit.json` | | - +| Maestro flows | `collector/examples/flows//` | `FLEET_FLOWS_DIR` | +| Playwright specs | `collector/examples/web-specs//` | `FLEET_WEB_SPECS_DIR` | +| Visual capture manifests | `collector/examples/web-specs//shots.json` | | +| Audit config | `collector/examples/web-specs//audit.json` | | + +The defaults are `examples/flows` and `examples/web-specs`, resolved from the +directory the collector and executor run in, which is `collector/` for +`npm start`. Set the overrides to keep your own flows and specs outside the +checkout. [`examples/`](https://github.com/addisdev/fleet-runner/tree/main/collector/examples) has working versions of each, including a neutral `fleetrunner/smoke.yaml` you can copy as a starting point. @@ -36,7 +40,7 @@ Then: ```json { "workload": "ui-test", "executor": "host", - "app": { "name": "your-app-android", "sha256": "latest" }, + "app": { "name": "your-app-android", "build": "latest", "sha256": "latest" }, "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" }, "targets": { "exclusive": true } } ``` @@ -60,7 +64,7 @@ not a test. ```json { "workload": "ui-test", "executor": "host", - "app": { "name": "your-app-ios", "sha256": "latest" }, + "app": { "name": "your-app-ios", "build": "latest", "sha256": "latest" }, "suite": { "kind": "xcuitest", "bundle": "YourAppUITests-Runner.app" }, "targets": { "executor": "mac-xcode" } } ``` @@ -116,7 +120,18 @@ accurate and a match is a statement about what the job needs: { "targets": { "match": "os ~ 'android' && ram_mb >= 3000" } } ``` -Check what a match would select before you schedule it: +Check what a match would select before you schedule it. `--validate` runs +every check an enqueue makes and lists the devices the job would land on, +without queueing it: + +```bash +fleetr run --spec job.json --validate +fleetr run benchmark --match "os ~ 'android' && ram_mb >= 3000" --validate +``` + +That is `POST /api/v1/jobs/validate` underneath, which takes a whole spec. The +older `POST /api/jobs/preview-targets` still answers for a bare `targets` +object: ```bash curl -s -X POST http://fleet-host.local:8788/api/jobs/preview-targets \ @@ -132,6 +147,13 @@ into your app repository. It builds, then calls `ci-enqueue`, which exits non-zero on a red suite — so the check fails whether or not [commit statuses](index.md#commit-statuses) are armed. +`ci-enqueue` also exits 0 on a job that was skipped because nothing ran, and +sends no token, so it fails against a collector with `FLEET_AUTH=tokens` +([details](first-job.md#or-in-one-step-with-ci-enqueue)). For a check that +should block a merge, replace that step with `ci-upload` followed by +`fleetr run --spec job.json`, which exits 1 on a red suite and 4 on a skip, +and fails the step on either. + Give the job a unique `job_id` built from the pull request number, the run attempt and the sha. A duplicate is a 409, which is the correct answer but not the one you want on a re-run. diff --git a/docs/integration/cookbook.md b/docs/integration/cookbook.md index a2cc755..6000240 100644 --- a/docs/integration/cookbook.md +++ b/docs/integration/cookbook.md @@ -5,13 +5,27 @@ verbatim from [`collector/examples/jobs/`](https://github.com/addisdev/fleet-runner/tree/main/collector/examples/jobs) and validated against [`job.schema.json`](https://github.com/addisdev/fleet-runner/blob/main/collector/schemas/job.schema.json) -by `npm test`.** If one stops validating, CI fails — so what you copy from here -is what the collector accepts. +by `npm test`** (`scripts/check-examples.ts`). If one stops validating, CI +fails. That check is the schema file only. `/api/v1/jobs`, and so `fleetr run`, +is stricter: it refuses unknown top-level fields and checks `params` against +the workload's manifest where there is one. Every example passes both today, +but only the first is enforced. -Post any of them to `POST /jobs`: +Run any of them with `fleetr`: ```bash -curl -X POST http://fleet-host.local:8788/jobs \ +fleetr run --spec collector/examples/jobs/benchmark.json --validate # where it would land +fleetr run --spec collector/examples/jobs/benchmark.json --id bench-$(date +%s) +``` + +The examples carry fixed `job_id`s, so the second run of the same file is +refused with `duplicate_job` unless `--id` gives it a new one. `--spec` takes +one job; a chain is several files, enqueued in dependency order (see +[the chain below](#a-chain-build-install-test)). Without `fleetr`, post the same +file to `POST /api/v1/jobs`: + +```bash +curl -X POST http://fleet-host.local:8788/api/v1/jobs \ -H 'content-type: application/json' \ --data-binary @collector/examples/jobs/benchmark.json ``` @@ -182,8 +196,11 @@ shrinking the matrix. ``` `account` names a Keychain item on the executor host. The secret itself is never -in the spec — `POST /jobs` is unauthenticated, so a spec is not a place a -credential could safely live. +in the spec: a spec is stored, served by the API and shown on the dashboard to +anyone who can read jobs, and the collector has no authentication by default +(see [Authentication](../deploy/auth.md)). Enqueue enforces it, refusing with +`secret_in_spec` any spec in which `password`, `passwd`, `secret`, `token`, +`api_key` or `apikey` appears as a key (or as a whole string value). ### `digest` @@ -208,9 +225,14 @@ idle five minutes, rather than starting an Xcode build while somebody is typing. --8<-- "collector/examples/jobs/chain-build-install-test.json" ``` -Post `build.json`, `install.json` and this, and the last two arrive as -`waiting`. `${jobs.build-903.artifact}` is substituted at promotion with the -real hash the build uploaded. A failed build fails both waiters, naming the +This is the last link of three. Post `build.json`, then an `install` job with +`"job_id": "install-903"` and `"depends_on": ["build-903"]`, then this, in that +order, because a dependency must exist before the job that names it. The +`install.json` example above is a standalone nightly install +(`install-nightly`), not that middle link; [Publish on +merge](publish-on-merge.md#chaining-a-build-to-its-own-test) has all three +specs. The last two arrive as `waiting`. `${jobs.build-903.artifact}` is +substituted at promotion with the real hash the build uploaded. A failed build fails both waiters, naming the cause, rather than leaving them parked forever. ### `model-convert` diff --git a/docs/integration/first-job.md b/docs/integration/first-job.md index 0cfee0f..ab14c7d 100644 --- a/docs/integration/first-job.md +++ b/docs/integration/first-job.md @@ -6,42 +6,100 @@ the result. This assumes a collector running — [Get started](../getting-started.md) if not — and at least one agent registered. -## 1. Enqueue with curl +## 1. Enqueue with fleetr + +`fleetr` is installed beside `fleet` and is the same program. Point it at the +collector with `--url`, or set `FLEET_RUNNER_URL` once: + +```bash +export FLEET_RUNNER_URL=http://fleet-host.local:8788 + +fleetr devices # what is online; a job can only land on these +fleetr workloads benchmark # its params and a valid example spec +fleetr run benchmark --backend synthetic --match "os ~ 'android'" \ + --param prompt_tokens=256 --param gen_tokens=64 --param measure_iters=3 +``` + +`fleetr run` validates the spec against [`/api/v1`](../api.md#control-api-apiv1) +before it queues anything, waits for the verdict, prints it per device, and +**exits with it**: 0 passed, 1 failed, 2 usage error or invalid spec, 3 still +running when `--timeout` ran out, 4 skipped (nothing ran), 5 refused or +unreachable. Add `--validate` to see where a job would land without queueing +it, and `--no-wait` to print the job id and return; `fleetr wait ` +picks it up later. + +**A `job_id` you set must be unique.** Leave it out and the collector +generates one. Set it (`--id`, or `job_id` in a spec) and it is an idempotency +key: a second enqueue with the same id is refused with `duplicate_job` (a 409), +deliberately, so a retried CI step cannot silently overwrite an earlier +verdict. Building it from the commit sha and the run attempt is the pattern the +templates use. + +If the job sits `waiting` rather than `queued`, it has a `depends_on` that has +not closed yet. If nothing ever claims it, the usual cause is a `targets` +clause no registered device satisfies, which `--validate` shows before you +queue. + +### The same with curl + +Where `fleetr` is not installed, the control API takes the same spec: ```bash -curl -X POST http://fleet-host.local:8788/jobs \ +curl -s -X POST http://fleet-host.local:8788/api/v1/jobs \ -H 'content-type: application/json' \ -d '{ - "schema": 1, - "job_id": "smoke-'"$(date +%s)"'", "workload": "benchmark", "executor": "device", "backend": "synthetic", "params": { "prompt_tokens": 256, "gen_tokens": 64, "measure_iters": 3 }, "targets": { "match": "os ~ '\''android'\''" } }' -``` -```json -{ "ok": true, "job_id": "smoke-1788635539", "status": "queued" } +curl -s 'http://fleet-host.local:8788/api/v1/jobs//wait?timeout_s=120' +curl -s 'http://fleet-host.local:8788/api/v1/devices' ``` -**`job_id` must be unique** — a duplicate is a 409, deliberately, so a retried -CI step cannot silently overwrite an earlier verdict. Building it from the -commit sha and the run attempt is the pattern the templates use. +The first answers 201 with `job_id`, `status` and `links`; the second holds +for up to 120 seconds and returns the summary, whose `verdict` is `passed`, +`failed`, `mixed`, `skipped` or `cancelled` once `finished` is true. A collector +running with tokens on (see [Authentication](../deploy/auth.md)) also wants +`-H "authorization: Bearer $FLEET_TOKEN"`. The older `POST /jobs` and +`GET /api/devices` still answer, with a `Deprecation` header. -If the reply says `waiting` rather than `queued`, the job has a `depends_on` -that has not closed yet. If nothing ever claims it, the usual cause is a -`targets` clause no registered device satisfies: +## 2. Upload a build and test it + +Publish the build, then run the suite against it. From the collector's +checkout: ```bash -curl -s 'http://fleet-host.local:8788/api/devices' | grep -o '"device_id":"[^"]*"' +npx tsx scripts/ci-upload.ts \ + --collector http://fleet-host.local:8788 \ + --app your-app-android --build local --platform android \ + --artifact app/build/outputs/apk/debug/app-debug.apk + +cat > job.json <<'JSON' +{ + "workload": "ui-test", + "executor": "host", + "app": { "name": "your-app-android", "build": "latest", "sha256": "latest" }, + "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" }, + "targets": { "exclusive": true }, + "lease": { "ttl_s": 1200 } +} +JSON + +fleetr run --spec job.json ``` -## 2. Upload a build and test it +`"latest"` resolves at enqueue to the newest build published under that app +name, which from your own terminal is the one you just uploaded. Where two +uploads can race (two pull requests building at once), put the hash +`ci-upload` printed in `app.sha256` instead. + +### Or in one step with `ci-enqueue` -`ci-enqueue` is the script CI runs, and it works perfectly well from your own -terminal, which is where you should try it first. +`collector/scripts/ci-enqueue.ts` uploads, enqueues and polls in one command, +and the example workflow uses it: ```bash cat > job.json <<'JSON' @@ -64,9 +122,23 @@ npx tsx scripts/ci-enqueue.ts \ --timeout 1200 ``` -It uploads the artifact, rewrites `app.sha256` with the returned hash, enqueues, -polls to a verdict, and **exits 0 or 1**. That exit code is the whole contract — -everything else is detail. +It uploads the artifact, writes the returned hash into `app.sha256`, posts the +spec to the deprecated `POST /jobs`, polls `GET /jobs/` every ten seconds, +and exits 0 when the job is `done`, 1 when it failed or timed out, and 2 on a +usage error. `"replaced-by-upload"` is only a placeholder for the script to +overwrite: it is not a hash, so `fleetr run --spec` and `/api/v1/jobs` refuse +it. Know three things before you let it block a pull request: + +- **A skipped job exits 0.** A job where nothing ran (its `last_error` starts + `skipped:`) is still `done`, so the check goes green. `fleetr run` exits 4 + for the same job. +- **It sends no token**, so against a collector with `FLEET_AUTH=tokens` the + enqueue is refused and the step fails. +- **A duplicate `job_id` is not an error to it.** It prints "already queued" + and reports the earlier job's verdict. + +For a check that blocks a merge, prefer `ci-upload` followed by +`fleetr run --spec`. ## 3. Watch it on the dashboard diff --git a/docs/integration/index.md b/docs/integration/index.md index 4409e46..111e5c3 100644 --- a/docs/integration/index.md +++ b/docs/integration/index.md @@ -9,7 +9,10 @@ a verdict that can block a pull request. `ci-enqueue` and `ci-upload` are **verified standalone**: they upload, they enqueue, they poll to a verdict and they exit 0 or 1, and the artifact and `"latest"` resolution behind them are covered end to end by the collector's - test suite. + test suite. `ci-enqueue` predates `/api/v1`: it counts a skipped job as a + pass and sends no token ([details](first-job.md#or-in-one-step-with-ci-enqueue)). + `fleetr run`, which validates against `/api/v1` and exits 4 on a skip, is + the supported way to block on a verdict. **The GitHub Actions path has never run on a real GitHub runner.** The workflow templates below document a contract that is implemented on the @@ -21,7 +24,7 @@ a verdict that can block a pull request. | | | |---|---| -| **[Your first job](first-job.md)** | `curl`, then the CI script standalone, then the dashboard. No CI involved. | +| **[Your first job](first-job.md)** | `fleetr` (or `curl`), then a build and a suite, then the dashboard. No CI involved. | | **[Publish on merge](publish-on-merge.md)** | Every merge publishes a build the nightly can ask for by name. | | **[Add your app](add-your-app.md)** | Flows, suites, visual baselines, and targeting the right devices. | | **[Add a workload](add-a-workload.md)** | Teach the fleet something new without a collector release. | @@ -64,13 +67,18 @@ Arming it makes a red nightly block a pull request. That is the point, and it is also a decision with consequences for everyone who pushes, so it stays off until somebody chooses it deliberately. -Note that `ci-enqueue` already exits non-zero on a red suite, so the check fails -whether or not statuses are armed. Arming adds the status on the *nightly's* +Note that `ci-enqueue` and `fleetr run` already exit non-zero on a red suite, +so the check fails whether or not statuses are armed. Arming adds the status on the *nightly's* commit, which is the part a scheduled run cannot get any other way. ## What the runner needs to reach -The collector is LAN or tailnet only and has no authentication, so a +The collector is LAN or tailnet only and has no authentication by default +(tokens are opt-in; see [Authentication](../deploy/auth.md)), so a GitHub-hosted runner cannot reach it and should not be able to. The workflows below assume a **self-hosted runner on the same tailnet**, with the collector's address in a `FLEET_COLLECTOR_URL` secret. + +If the collector runs with `FLEET_AUTH=tokens`, give CI a token with the `run` +scope (`fleet token create ci --scope run`) in another secret and pass it to +`fleetr` as `FLEET_RUNNER_TOKEN`. `ci-enqueue` cannot send one. diff --git a/docs/integration/publish-on-merge.md b/docs/integration/publish-on-merge.md index c049d62..a604127 100644 --- a/docs/integration/publish-on-merge.md +++ b/docs/integration/publish-on-merge.md @@ -33,6 +33,16 @@ curl -X POST http://fleet-host.local:8788/artifacts \ Those headers are what stamp the publish ordering that `latest` reads. There is no `/artifacts/:sha/publish` endpoint and none is needed. +`POST /artifacts` is a runner-protocol route, so it stays open even with +`FLEET_AUTH=tokens` unless the collector also sets `FLEET_AUTH_PROTOCOL=token` +(see [Authentication](../deploy/auth.md)). Then it needs a token with the +`agent` scope (or `admin`; `run` is not enough): send +`-H "authorization: Bearer "` with curl. `ci-upload` has no bearer +option; it sends `FLEET_DASH_TOKEN` from its environment as `x-fleet-token`, +which the collector also accepts as a token, so put the `agent` token in that +variable. That route has been read from the code, not run against a +protocol-gated collector. + !!! note "An app with no published build is skipped, not failed" If nothing has ever been published for an app name, its nightly is @@ -89,16 +99,20 @@ names. ```json { "schema": 1, - "job_id": "nightly-ui", "workload": "ui-test", "executor": "host", - "app": { "name": "your-app-android", "sha256": "latest" }, + "app": { "name": "your-app-android", "build": "latest", "sha256": "latest" }, "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" }, "targets": { "match": "os ~ 'android'", "exclusive": true }, "lease": { "ttl_s": 1800 } } ``` +This is the schedule's template. It needs no `job_id`: each firing is enqueued +as `-`, and one in the template would be replaced. +`"build": "latest"` is there because the schema requires `app.build`; the +collector records the build string the published artifact carries. + Put it in [`scripts/seed-schedules.ts`](https://github.com/addisdev/fleet-runner/blob/main/collector/scripts/seed-schedules.ts) rather than only in the database, and `npm run seed:schedules`. That is @@ -109,27 +123,41 @@ watched it run once by hand. ## Chaining a build to its own test -If a machine agent does your building, the whole thing can be one enqueue — -[job chains](../concepts.md#job-chains) do the waiting: +If a machine agent does your building, [job chains](../concepts.md#job-chains) +do the waiting. Nothing accepts a list of jobs in one request, so enqueue them +one at a time, in dependency order, and let `depends_on` hold the later ones in +`waiting`: -```json -[ - { "job_id": "build-903", "workload": "build", "executor": "device", - "params": { "repo": "git@github.com:you/app.git", "ref": "main", - "kind": "gradle", "task": "assembleDebug", "app": "your-app-android" } }, - - { "job_id": "install-903", "workload": "install", "executor": "host", - "depends_on": ["build-903"], - "app": { "name": "your-app-android", "build": "903", - "sha256": "${jobs.build-903.artifact}" } }, - - { "job_id": "ui-903", "workload": "ui-test", "executor": "host", - "depends_on": ["install-903"], - "app": { "name": "your-app-android", "build": "903", - "sha256": "${jobs.build-903.artifact}" }, - "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" } } -] +```json title="build-903.json" +{ "job_id": "build-903", "workload": "build", "executor": "device", + "params": { "repo": "git@github.com:you/app.git", "ref": "main", + "kind": "gradle", "task": "assembleDebug", "app": "your-app-android" } } +``` + +```json title="install-903.json" +{ "job_id": "install-903", "workload": "install", "executor": "host", + "depends_on": ["build-903"], + "app": { "name": "your-app-android", "build": "903", + "sha256": "${jobs.build-903.artifact}" } } +``` + +```json title="ui-903.json" +{ "job_id": "ui-903", "workload": "ui-test", "executor": "host", + "depends_on": ["build-903", "install-903"], + "app": { "name": "your-app-android", "build": "903", + "sha256": "${jobs.build-903.artifact}" }, + "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" } } +``` + +```bash +fleetr run --spec build-903.json --no-wait +fleetr run --spec install-903.json --no-wait +fleetr run --spec ui-903.json --no-wait +fleetr wait ui-903 # exits with the UI test's verdict ``` -A broken build fails the install and the UI test with it, naming the cause, -rather than leaving them in `waiting` until somebody notices next week. +The ids are fixed here because the later specs name the earlier ones. A +dependency has to exist when its dependent is enqueued, which is why the order +matters, and a `${jobs..artifact}` reference may only name a job listed in +`depends_on`, which is why `ui-903` lists the build as well as the install. A broken build fails the install and the UI test with it, naming the +cause, rather than leaving them in `waiting` until somebody notices next week. diff --git a/docs/lab.md b/docs/lab.md index 161257b..bd0d54c 100644 --- a/docs/lab.md +++ b/docs/lab.md @@ -18,6 +18,7 @@ and nothing said which of them held the iPhone. | **runner-host** | MacBook Pro (Mac14,10), macOS 26.7, Xcode 27 | **Nothing of Fleet Runner's, since 2026-09-22.** It runs the GitHub Actions runner farm, which is its own project | `ssh runner-host` | | **the dev MacBook** | M1 Pro, 16 GB, macOS 27, Xcode 27 | A machine agent and the `mac-dev` executor in pool `personal`, plus the SSH tunnel and the alert receiver. Its executor refuses heavy work above load 50 or under 2 GB of free swap (`FLEET_MAX_LOAD`, `FLEET_MIN_FREE_SWAP_MB`) | it is the machine you are on | | **the mini** | Mac mini M4, macOS 15.6.1, up 224 days | Nothing fleet-related | `ssh mini`, over the tailnet | +| **ultra** | Mac Studio, M5 Ultra, 96 GB, macOS 27 | The exploratory QA nights: the `ultra` executor, its Android emulator and Apple simulators, and the model gateway they call. Its night queue runs them, not the brain's schedules; [Running it on ultra](deploy/explore-on-ultra.md) is its runbook | `ssh ultra`; it reaches the brain through its own tunnel | **Why the brain is the oldest machine.** It is on the shelf's subnet, and it is macOS 12, which has no local-network gate. Every other Mac here is macOS 26 or @@ -149,6 +150,12 @@ Three things have to be true for a status to appear: MacBook it reaches the brain through the tunnel, the same as the agent does. A result whose error starts with `skipped:` means nothing ran. +From 0.8.0 there is also `fleetr`, whose exit code is the verdict, and a skill +(`fleet skill install`) that tells an agent which commands to run in what +order. [AI agents and automation](agents.md) has both. The brain here is still +on 0.7.2, so neither `/api/v1` nor the new commands work against it until it is +upgraded. + ## The suites are not in git `~/fleet-collector/flows/` and `~/fleet-collector/web-specs/` on fleet-host hold @@ -185,7 +192,7 @@ Five enabled. All times local to fleet-host. |---|---|---|---| | `nightly-synthetic-shelf` | 02:00 | fanout, pool `machines` | A machine's throughput moved, or an agent is not claiming. This is the comparison table | | `nightly-self-check` | 02:15 | fanout, pool `machines` | Disk, clock drift, a tool that vanished, or the agent is not supervised | -| `nightly-fleet-ui-smoke` | 02:30 | `fleet-host`, an Android device | **Red since 2026-08-21.** Since 09-16 it is `no targets attached` — the Galaxy S8+ is unplugged | +| `nightly-fleet-ui-smoke` | 02:30 | `fleet-host`, an Android device | **Red since 2026-08-21.** From 09-16 to early October it was `no targets attached`. The Galaxy S8+ is plugged in again and the suite now runs on it, and fails, every night; why is not diagnosed yet | | `nightly-aliquant-shots` | 04:15 | `fleet-host`, `chromium` and `mobile-chrome` | Green. The baseline was re-accepted on 2026-09-22 from fleet-host's own captures, after Aliquant's sign-in page gained a "Forgot password?" link; the old one had diverged 2.7% on desktop and 6.1% on mobile since 09-10. The `webkit` and `mobile-safari` baselines are still the August ones, because nothing runs those projects now | | `nightly-tvloop-hardware` | 05:00 | `fleet-host`, the Roku, exclusive | tvloop's doctor, install, contract tests and flow replay. `skipped:` until the Roku password is in fleet-host's Keychain. Red from doctor means the Roku is asleep, off the network, or out of developer mode | @@ -216,15 +223,14 @@ that pool here. ```bash ssh fleet-host -fleet service status # says "running" even when the brain has given up — see below +fleet service status # names a component that gave up, and exits non-zero tail -40 ~/.fleet/logs/fleet.log curl -s localhost:8788/api/health ``` -`fleet service status` reports on the launchd job, not on what the supervisor is -doing, so it says `running` while the supervisor has permanently given up on the -brain. Read `fleet.log` for `GAVE UP`. Tracked in -[#45](https://github.com/addisdev/fleet-runner/issues/45). +Since 0.6.1 `fleet service status` reads the supervisor's own state, so a +component it has given up on shows as `GAVE UP at … after N restarts` rather +than `running`. `fleet.log` has the restarts that led there. ### Restarting it @@ -269,8 +275,9 @@ from, so a restore does not have to guess. ## What is not done, and who it needs -- **The shelf is unplugged.** No phone, tablet, TV or stick has been on the - fleet since 2026-08-18. That needs a powered USB hub and cables. +- **Most of the shelf is unplugged.** The Galaxy S8+ is back on fleet-host; + no other phone, tablet, TV or stick has been on the fleet since 2026-08-18. + That needs a powered USB hub and cables. - **The Roku's developer password is not in fleet-host's Keychain**, so the tvloop nightly skips. Developer mode is on. - **An Apple TV 4K** is paired to the dev MacBook and has never run the tvOS diff --git a/docs/platforms.md b/docs/platforms.md index 8d20453..094a2a9 100644 --- a/docs/platforms.md +++ b/docs/platforms.md @@ -14,7 +14,7 @@ that says which ones somebody has watched register is worth something. | Android phone, tablet | `runner-android` | `adb install`, or sideload | **yes** — the original shelf | | iPhone, iPad | `runner-ios` | Xcode, TestFlight internal | **yes** | | iOS simulator | `runner-ios` | `simctl install` | **yes** | -| macOS, Linux, Windows | `runner-machine` | `fleet up`, or `npm start`, or a LaunchAgent / systemd unit | **macOS yes, on both architectures** — an Apple M1 Pro and a 2016 Intel i7 both registered, passed `npm run conformance`, and produce comparable synthetic numbers with matching digests; both run under `fleet service`. Linux and Windows run their suite in CI on x64 and arm64. On Linux x64 the agent also registers, inside the Docker image in CI, against the collector beside it; Windows has never registered against a real collector — see [what each one reports](#what-a-cloud-runner-actually-reports) | +| macOS, Linux, Windows | `runner-machine` | `fleet up`, or `npm start`, or a LaunchAgent / systemd unit | **macOS yes, on both architectures** — an Apple M1 Pro and a 2016 Intel i7 both registered, passed `npm run conformance`, and produce comparable synthetic numbers with matching digests; both run under `fleet service`. Linux and Windows run their suite in CI on x64 and arm64, and CI installs `fleet service` as a systemd user unit on Linux, waits for the collector it starts, and uninstalls it; the launchd unit is only proven by this lab's own Macs and the Windows scheduled task has never been installed. On Linux x64 the agent also registers, inside the Docker image in CI, against the collector beside it; Windows has never registered against a real collector — see [what each one reports](#what-a-cloud-runner-actually-reports) | | Android emulator | `runner-android` | `adb install` to the emulator | **yes** | ## What Wave 5 added @@ -22,10 +22,10 @@ that says which ones somebody has watched register is worth something. | Platform | Runner | How it joins | Verified | |---|---|---|---| | **Any browser** | `collector/runner-web/` | open `http:///runner` | **yes** — driven end to end by a real Chromium against a real collector, both backends, digest checked | -| **Android TV, Google TV, Fire TV** | `runner-android` | `adb connect`, then `adb install` | **partly** — the built APK declares a leanback launcher and a banner, checked with `aapt2 dump badging`. No TV has run it | +| **Android TV, Google TV, Fire TV** | `runner-android` | `adb connect`, then `adb install` | **partly** — the built APK declares a leanback launcher and a banner, checked with `aapt2 dump badging`. No TV has run it. (Exploratory QA has driven an Android TV emulator from outside, over adb; that is the executor, not this runner — see [what it can drive](explore/index.md#what-it-can-drive)) | | **Meta Quest** | `runner-android` | sideload via `adb` | **no** — Quest is Android 12 and reports `UI_MODE_TYPE_NORMAL`, so the runner identifies it by the VR feature flag. Never run on one | | **Wear OS** | `runner-android` | `adb install` over Wi-Fi debugging | **no** — the watch feature flag is read at run time; the manifest deliberately does not declare it (that would make the APK a Wear-only app in a store). Never run on one | -| **Apple TV (tvOS)** | `runner-ios`, `FleetRunnerTV` target | `devicectl` or `simctl` | **builds** — `xcodebuild -scheme FleetRunnerTV` succeeds. No Apple TV has run it | +| **Apple TV (tvOS)** | `runner-ios`, `FleetRunnerTV` target | `devicectl` or `simctl` | **builds** — `xcodebuild -scheme FleetRunnerTV` succeeds. No Apple TV has run it. (Exploratory QA drives tvOS 26.5 simulators through FleetDriver, a separate UI-test bundle) | | **Vision Pro (visionOS)** | `runner-ios`, `FleetRunnerVision` target | `devicectl` or `simctl` | **no** — the target is written and has never been compiled; the visionOS platform is not installed on the machine it was written on | | **Raspberry Pi, Jetson, Steam Deck** | `runner-machine` | `install-agent.sh`, or the Docker image | **no** — but arm64 Linux in CI reports a nearly complete descriptor, so a board should register cleanly. No board has | | **Anything with Docker** | `fleet/Dockerfile` | `docker run ghcr.io/addisdev/fleet` | **in CI** — every release publishes linux/amd64 and linux/arm64. CI starts the amd64 image on every pull request that touches it and waits for its own machine agent to register; the arm64 image has not been started | @@ -37,6 +37,10 @@ that says which ones somebody has watched register is worth something. | **Roku player, Roku TV** | `runner-roku` | `build.sh --install `, then an ECP launch | **no** -- and less verified than anything above it. No Roku was available and no Roku emulator exists, so the channel has never run. CI compiles it with brighterscript's `bsc` on every pull request that touches it; the firmware's own compiler has never seen it. See [Roku: the one that has never run](#roku-the-one-that-has-never-run) | | **Roku, from the executor** | `collector/src/drivers/roku.ts` | SSDP discovery on the LAN | **partly** -- the SSDP and device-info parsers are checked against recorded response shapes in `drivers.test.ts`; the UDP socket path is exercised by nothing, and no Roku has answered it | +The desktop app is not a runner, but it is the other thing here that has never +been run: `desktop / cargo check` compiles its Rust on macOS on every pull +request that touches it, and nobody has launched it. + ## Android: one APK, every shape The Android agent is a foreground service and a text field. Nothing in it needs @@ -224,7 +228,7 @@ It deliberately does **not** install. That needs HTTP digest auth, which Node's something that looks finished and fails the first time somebody has a Roku. `runner-roku/build.sh` installs with `curl --digest` instead, reading the developer password from the executor host's Keychain -- never from a job spec, -which is unauthenticated and rendered on the dashboard. +which anyone who can enqueue can read, and which the dashboard renders. ## What a cloud runner actually reports diff --git a/docs/protocol.md b/docs/protocol.md index 7a884f6..f96a45c 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -294,6 +294,30 @@ Uploading with `x-artifact-app`, `x-artifact-build` and `x-artifact-platform` stamps the upload as a published build, which is what makes `"sha256": "latest"` resolvable. There is no separate publish call. +## Authentication (optional) + +Off by default, and off for the runner protocol even when the control API has +tokens on (`FLEET_AUTH=tokens`): phones, TVs, Rokus and the browser runner send +no token. With `FLEET_AUTH_PROTOCOL=token` as well, every call on this page +needs + +``` +Authorization: Bearer fr_... +``` + +from a token with the `agent` scope (or `admin`, which implies it). No token +is a 401; a token without the scope is a 403. `GET /artifacts/{sha256}` also +accepts a `read` token, so a person can open a screenshot. runner-machine and +the host executor send `FLEET_TOKEN` from their environment when it is set; +the Android, iOS, Roku and browser runners do not, so do not turn this on while +any of them is on the shelf. See [Authentication](deploy/auth.md). + +The same rule covers the protocol's smaller routes, which a minimal agent never +calls: `POST /jobs/{id}/release`, `POST /locks/acquire` · `/release`, +`POST /events/{topic}` and `GET /events/{topic}/poll`, `POST /findings` (an +`explore` finding from an executor) and `POST /api/jobs/{id}/mirror` (a live +JPEG frame). + ## Writing your own agent The contract above is the whole thing. [Writing a runner](writing-a-runner.md) @@ -313,6 +337,9 @@ Worth stating, because each of these is a design decision rather than a gap: which makes the next beacon return `lease_renewed: false`. Work in flight finishes; nothing new starts. - **It carries no secrets in job specs.** A job names a credential; the agent - reads it from its own environment or Keychain. This is enforced by the design - rather than by validation — `POST /jobs` is unauthenticated, so a spec is not - a place a secret could safely live. + reads it from its own environment or Keychain. Enqueue refuses, with 400 + `secret_in_spec`, any spec containing a key (or string value) named + `password`, `passwd`, `secret`, `token`, `api_key` or `apikey`, in any case. + The check is a backstop, not the reason: a spec is stored, served by the API + and shown on the dashboard, and `POST /jobs` is unauthenticated by default, + so a spec is not a place a secret could safely live. diff --git a/docs/workloads/host.md b/docs/workloads/host.md index 984fd33..b61db8b 100644 --- a/docs/workloads/host.md +++ b/docs/workloads/host.md @@ -179,6 +179,79 @@ Native libraries group **per ABI** rather than as one `lib/`. `lib/` being huge tells you nothing you did not know; `lib/arm64-v8a` being 18 MB of it tells you what to split. +## `tvloop` + +Runs [tvloop](../explore/surfaces.md#roku-through-tvloop)'s hardware suite +against a Roku on the executor host's network. tvloop is the Roku development +CLI and already has a hardware nightly, aimed at a self-hosted Actions runner +nobody carries; this workload runs the same steps on the fleet instead. +**tvloop drives, the fleet lends**: tvloop knows how to talk to a Roku, and the +fleet's part is the device lock, the address, the password and filing what came +back. + +```json +{ "schema": 1, "job_id": "tvloop-1", "workload": "tvloop", "executor": "host", + "targets": { "match": "os ~ 'roku'", "exclusive": true }, + "params": { "steps": ["doctor", "install", "hardware", "replay"] } } +``` + +| Step | What runs | | +|---|---|---| +| `doctor` | `tvloop doctor --json` | A failure stops the run | +| `install` | `tvloop install --force` | Sideloads the checkout's channel into the Roku's one dev slot, **replacing whatever is there**. A failure stops the run | +| `spike` | `tools/m0-spike.mjs --full` | Opt-in, not in the default: ten sideloads, slow | +| `hardware` | tvloop's vitest suite with `--testNamePattern hardware` | | +| `replay` | `tvloop replay --reporter junit` | The JUnit counts become the row's test counts | + +The default is doctor, install, hardware, replay. `params.flows` picks which +flows to replay (default: all of the checkout's `tests/flows`), and +`params.dir` the checkout, which must be built: `FLEET_TVLOOP_DIR`, then +`~/tvloop`. A host with no built checkout **fails** the job and says how to +build one, because that is misconfiguration. + +**`exclusive` matters more here than anywhere.** A Roku holds exactly one dev +channel, so two jobs sideloading onto it at once is a corrupted run, and the +lock is held around the whole suite. + +**The developer password is a Keychain item on the executor host**: service +`fleet-roku-dev` (`FLEET_ROKU_KEYCHAIN_SERVICE` to change), account `rokudev`. + +```bash +security add-generic-password -s fleet-roku-dev -a rokudev -w +``` + +Without it a job of only Rokus is **skipped, not failed**: its row reads +"skipped: no Roku developer password on this host" with that command, because +it is a setup step nobody has done yet rather than a broken build. A Keychain +that refuses access fails the job. The password is passed to tvloop in its +environment and scrubbed from every log and report before upload, since the +artifact store is readable by anyone who can open the dashboard. + +Only one executor should report the Rokus as its own devices: the host started +with `FLEET_ROKU=1`. + +### Fire TV and Android TV + +With `params.androidtv: { "package": "com.example.app.debug" }`, adb devices +that declare leanback run alongside the Rokus, each through a generated +`tvloop.toml`. Only `doctor` and `replay` run on them (install, spike and +hardware are the Roku adapter's), and they need no Roku password. +`params.androidtv.flows` replays a different set on the TVs. + +A green TV row says less than a Roku row. tvloop's Android TV adapter reads no +logcat, so `assert noErrors` cannot fail there; its launch is `monkey` with +one random event, so a replay can start with a stray key; and a flow that +asserts focus or visibility always fails, because it has no Android UI tree. + +### What is not proven + +The fleet's Roku has never run this suite: its developer password is not in +the host's Keychain, so no run from the fleet has got past the skip. The step +sequencing, the skip, the scrubbing and the Android TV path are covered by +unit tests that fake each step's exit code; the Android TV adapter's caveats +above were seen driving an Android TV emulator through tvloop, not from this +workload on a real TV. + ## `ui-test` Maestro flows or an XCUITest bundle, per device, with the JUnit report parsed @@ -192,7 +265,8 @@ back into results and uploaded as an artifact. "lease": { "ttl_s": 1200 } } ``` -Flows resolve relative to `flows/` (`FLEET_FLOWS_DIR` to change). `exclusive` +Flows resolve relative to `examples/flows/` in the executor's working +directory (`FLEET_FLOWS_DIR` to change). `exclusive` takes a device lock for the run, because two suites tapping the same phone is not a test. @@ -342,8 +416,11 @@ is kept forever. review pulls must run daily.** A lazy cadence loses data permanently. Credentials are Keychain items on the executor host. The job spec names the -account only, never the secret — `POST /jobs` is unauthenticated, so a spec is -not a place a credential could safely live. Until the Keychain item exists the +account only, never the secret: a spec is stored and shown to anyone who can +read jobs, and the collector has no authentication by default (see +[Authentication](../deploy/auth.md)), so a spec is not a place a credential +could safely live. Enqueue refuses a spec carrying a `password`, `token` or +similar key as `secret_in_spec`. Until the Keychain item exists the job fails with instructions rather than silently producing nothing. ## `digest` diff --git a/docs/workloads/index.md b/docs/workloads/index.md index 226b38f..0f9084c 100644 --- a/docs/workloads/index.md +++ b/docs/workloads/index.md @@ -4,7 +4,13 @@ What the fleet can be asked to do. A workload is the `workload` field of a job spec, and which agent claims it is decided by [capabilities](../concepts.md#capabilities) rather than by anything hardcoded. -![Twenty-eight workloads in three columns by who claims them: on the device, on a host, and on a machine](../img/workloads.png) +![The workloads in three columns by who claims them: on the device, on a host, and on a machine](../img/workloads.png) + +The collector ships 34 workloads. `GET /api/v1/workloads` (or +`fleetr workloads`) lists the same set with each one's params and an example +spec, and marks three more names as reserved: `push-latency`, `camera-eval` and +`desktop-ui-test` are accepted, but no runner in this repository implements +them yet. | Workload | Runs on | What it measures | |---|---|---| @@ -24,6 +30,9 @@ spec, and which agent claims it is decided by | [`a11y-audit`](host.md#a11y-audit) | host | The accessibility tree diffed against a baseline at the largest dynamic type | | [`locale-shots`](host.md#locale-shots) | host | A screenshot flow under every locale, including RTL, as a contact sheet | | [`explore`](host.md#explore) | host | A vision model explores tonight's build; reproduced crashes, hangs, dead controls, accessibility and visual defects, filed as findings | +| [`upgrade-test`](host.md#upgrade-test) | host | Install the version users have, seed it, upgrade in place, and check the data survived | +| [`size-report`](host.md#size-report) | host | Download, installed and archive size of a published build, with native libraries per ABI | +| [`tvloop`](host.md#tvloop) | host | tvloop's hardware suite on a Roku the fleet has locked; Fire TV and Android TV replay its flows too when asked | | [`web-test`](host.md#web-test) | host | Playwright suites, including on real phone screens | | [`web-shots`](host.md#web-shots) | host | Visual regression captures diffed against an accepted baseline | | [`web-audit`](host.md#web-audit) | host | Crawl-and-audit with a real browser | @@ -37,6 +46,7 @@ spec, and which agent claims it is decided by | [`dataset-prep`](machine.md#dataset-prep) | machine | An eval set, carrying its licence and source on the row | | [`serve`](machine.md#serve) | machine | Host a model for the length of a lease and announce its endpoint | | [`shell`](machine.md#shell) | machine | A pinned script, behind an allowlist the owner controls | +| [`llm-eval`](machine.md#llm-eval) | machine | Scores what a device generated against an eval set, so a fast quantised model is also shown to be right | | [`self-check`](machine.md#self-check) | host, machine | The fleet inspecting its own hosts: disk, tool versions, clock drift | ## How to read these pages diff --git a/docs/workloads/machine.md b/docs/workloads/machine.md index 188c35b..52d1468 100644 --- a/docs/workloads/machine.md +++ b/docs/workloads/machine.md @@ -78,7 +78,9 @@ to the one anybody asked for. ## `shell` Runs a pinned script. This is the one workload that cannot take the collector's -word for anything, because `POST /jobs` is unauthenticated by design. +word for anything, because by default `POST /jobs` is unauthenticated, and even +with [tokens](../deploy/auth.md) on, only `/api/v1` asks for `admin` before it +queues a `shell` job. The gating, in order: diff --git a/fleet/README.md b/fleet/README.md index 1afbaa0..4f55b71 100644 --- a/fleet/README.md +++ b/fleet/README.md @@ -45,10 +45,12 @@ once and then cannot start the fleet. |---|---| | `fleet up [--role brain,agent,executor] [--join ] [--port N]` | This machine's components, supervised. Roles are a list because the first machine this exists for is a brain and a runner at once | | `fleet collector` / `fleet agent` / `fleet executor` | Just the one component, in this process | -| `fleet join ` | Run a runner pointed at a brain elsewhere, and write that down | +| `fleet join ` / `fleet join --discover` | Run a runner pointed at a brain elsewhere, and write that down. `--discover` looks for a brain advertising over mDNS instead (off on the brain by default: `collector.discovery`) | | `fleet status` | What this fleet looks like right now: devices online, jobs by state, per collector | | `fleet doctor` | What this machine can and cannot run, **and why not** — adb, full Xcode against the Command Line Tools, Playwright, Maestro, and the capability list the agent would actually declare | | `fleet dash` | Open the dashboard | +| `fleet mcp [--url U] [--token T]` | The lab as MCP tools over stdio, for a coding agent on this machine. See below | +| `fleet mcp --http --issuer --audience ` | The same tools as a remote MCP server behind your identity provider's OAuth | | `fleet service install\|uninstall\|start\|stop\|status\|logs` | Start at login and stay up: launchd, systemd or a scheduled task, whichever this machine has | | `fleet config` / `config get ` / `config set ` | Read and write `~/.fleet/config.json` | | `fleet version` | | @@ -60,17 +62,19 @@ somebody put it in. ### Work commands Running jobs rather than running the fleet. Each calls `/api/v1` through the generated -client in `collector/src/client/`, takes `--url`, `--token` and `--json`, and exits with the -verdict: **0** passed, **1** failed, **2** usage or invalid spec, **3** still running, **4** -skipped (nothing ran), **5** refused or unreachable. +client in `collector/src/client/`, takes `--url`, `--token` and `--json` (except that +`token` reads the tokens file, `--tokens-file`, rather than calling a collector, and +`skill` and `logout` ignore `--token`), and exits with the verdict: **0** passed, **1** +failed, **2** usage or invalid spec, **3** still running, **4** skipped (nothing ran), +**5** refused or unreachable. | | | |---|---| | `fleet devices` / `fleet workloads [NAME]` | What is on the shelf; what it can run, with an example spec | -| `fleet run WORKLOAD [--device\|--pool\|--match] [--param k=v] [--spec FILE]` | Validate, queue, wait, print the verdict. `--validate` checks only; `--no-wait` prints the id | +| `fleet run WORKLOAD [--device\|--pool\|--match] [--app NAME] [--param k=v] [--target k=v] [--fanout] [--spec FILE\|-] [--timeout S]` | Validate, queue, wait (default 1800 s), print the verdict. `--validate` checks only; `--no-wait` prints the id. The full flag list is in [AI agents and automation](../docs/agents.md#the-cli) | | `fleet job ID` / `fleet wait ID` | A verdict now, or when it finishes | | `fleet cancel ID` / `fleet retry ID` | | -| `fleet results` / `fleet flaky` / `fleet artifact SHA [--out FILE]` | What ran, what flips, and the files it left | +| `fleet results` / `fleet flaky` / `fleet artifact SHA [--out FILE\|DIR\|-]` | What ran, what flips, and the files it left. `--out` is a directory only if it already exists | | `fleet login` / `logout` / `whoami` | Save a token for a collector (`~/.fleet/credentials.json`, 0600) | | `fleet token create\|list\|revoke` | On the brain's host: the collector's tokens, for `FLEET_AUTH=tokens` | | `fleet skill show\|install` | The agent skill, into Cursor, Claude Code and Codex | @@ -104,8 +108,11 @@ fleet skill install ``` It finds the brain the way the agent does -- `--url`, then `FLEET_RUNNER_URL` or -`FLEET_URL`, then the config -- and adds nothing the HTTP API does not already do: one -scheduler, one set of device locks, and nothing an agent can do that curl could not. +`FLEET_URL`, then the config -- and its token the way every work command does: `--token`, +then `FLEET_RUNNER_TOKEN`, then `FLEET_TOKEN`, then what `fleet login` saved for that URL. +`FLEET_SOURCE` overrides the `source` label jobs are recorded with. It adds nothing the +HTTP API does not already do: one scheduler, one set of device locks, and nothing an agent +can do that curl could not. `fleet mcp --http` is the same tools as a remote MCP server behind your identity provider's OAuth, for chat clients and people without `fleet` installed. See @@ -153,7 +160,7 @@ is almost always that. ### The settings this file has no key for -The three programs read about twenty `FLEET_*` variables and the config names six. The rest +The three programs read many more `FLEET_*` variables than the config has keys for. The rest are per-deployment facts with no good home — where this host's Maestro flows live, where its Playwright specs live, which Xcode project the generic iOS bundle builds from, whether web workloads are enabled here, where alerts go. `env` carries them: @@ -191,7 +198,9 @@ and which will otherwise restart a broken collector every ten seconds forever. ## Status -What has actually been watched to work, on **macOS/arm64 and nowhere else**: +What has actually been watched to work on **macOS/arm64**. The one other machine with +mileage is an Intel Mac on macOS 12 running a real brain under `fleet service` (below); +Windows and Linux have CI coverage only: - `fleet up` starting a supervised brain and agent, and a synthetic benchmark running end to end through them — from a checkout, and from the bundled release. diff --git a/fleet/src/skill.ts b/fleet/src/skill.ts index ae604c6..a08443e 100644 --- a/fleet/src/skill.ts +++ b/fleet/src/skill.ts @@ -15,7 +15,8 @@ const EXIT_MEANING: Record = { failed: "failed, mixed or cancelled: read `failing_step` and `error` before doing anything else", usage: "usage error or invalid spec: fix the flags or the spec; `--validate` names each problem", pending: "still running when `--timeout` ran out: `fleetr wait ID` picks it up again", - skipped: "skipped: nothing ran (no build for the device, no matching device). Never report this as a pass", + skipped: + "skipped: the job was claimed but reported `skipped:` instead of running (the app is not installed, the device was locked by another job, no attached device matched a host job, the host was under memory pressure, no Roku password on the host). Never report this as a pass. A spec nothing can run is refused before it runs instead: exit 2 from validation, or 5 when the collector refuses it at enqueue", error: "refused or unreachable: no permission, unknown workload or job, or no collector. Report it; do not work around it", }; diff --git a/mkdocs.yml b/mkdocs.yml index 2b67042..3c8a399 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -139,7 +139,6 @@ nav: - Headless hosts: deploy/headless.md - Networking: deploy/networking.md - The iOS executor: deploy/ios-executor.md - - Exploratory QA on ultra: deploy/explore-on-ultra.md - Alerts: deploy/alerts.md - Authentication and MCP gateway: deploy/auth.md - This lab: lab.md diff --git a/runner-android/README.md b/runner-android/README.md index 510a76a..48a1533 100644 --- a/runner-android/README.md +++ b/runner-android/README.md @@ -10,6 +10,12 @@ to: llama.cpp and LiteRT benchmarks, batch inference, and image-classification evals, with a telemetry beacon reporting battery, charging and thermal state every 60 seconds. +The same APK installs on Android TV, Google TV and Fire TV, on a Meta Quest +and on Wear OS, and works out at run time which of those it is on. That is +less proven than it sounds: the APK declares a leanback launcher and a banner +(checked with `aapt2 dump badging`) but no TV has run it, and no Quest or Wear +device ever has. [Platforms](../docs/platforms.md) has the state of each. + Old hardware is the point. A phone too slow for an LLM still runs the synthetic backend and still produces a comparable number, which is what makes a drawer of retired handsets useful as a test matrix instead of e-waste. @@ -41,7 +47,7 @@ Open the app, keep the default `http://127.0.0.1:8788`, tap **Start agent**, then enqueue a job: ``` -curl -X POST http://127.0.0.1:8788/jobs -H 'content-type: application/json' -d '{ +curl -X POST http://127.0.0.1:8788/api/v1/jobs -H 'content-type: application/json' -d '{ "schema": 1, "job_id": "bench-synthetic-1", "workload": "benchmark", "executor": "device", "backend": "synthetic", "params": { "prompt_tokens": 256, "gen_tokens": 64, "warmup_iters": 1, "measure_iters": 3 }, @@ -62,6 +68,11 @@ SM-X930, not an emulator. `benchmark`, `batch`, `batch:litert`, `pipeline`, dispatched from the same table, so the collector is never offered work this runner would bounce back. +**Android 7.0 and 7.1 (API 24-25) start the runner service with +`startService`**, because `startForegroundService` only exists from API 26 and +threw `NoSuchMethodError` there. That fix is covered by a Robolectric test and +has not run on a real API 24 or 25 device. + **There is no offline result queue.** Results retry through the agent loop's backoff and nothing else, so a device that loses the collector for longer than that loses those rows. This is the one gap worth knowing about before you trust @@ -77,6 +88,12 @@ the suite with no NDK installed: ./gradlew :app:testDebugUnitTest -Pfleet.skipNative ``` +CI also runs `./gradlew :app:lintDebug -Pfleet.skipNative`. Passing it meant +dropping the manifest's `android.hardware.type.watch` feature line, which lint +rejects as not-required and which a store reads as Wear-only; the runner still +detects a watch at run time. No Wear device has run the APK before or after +that change. + ## License MIT — see [LICENSE](LICENSE). llama.cpp, LiteRT, and the terms they carry are diff --git a/runner-machine/README.md b/runner-machine/README.md index a742a33..bea0461 100644 --- a/runner-machine/README.md +++ b/runner-machine/README.md @@ -29,11 +29,19 @@ deploy/install-agent.sh com.addisdev.fleet-runner-machine.plist # or the .serv | `benchmark` | `llama.cpp` | Model fetched by content hash and verified, then `llama-bench`: prefill/decode tok/s, load time, peak memory with the method named | | `build` | `gradle`, `xcode`, `npm` | Checks out `params.ref`, builds it, and publishes the product as the app's latest build. Reports `build_s` and `artifact_bytes` | | `self-check` | — | Disk, tool versions, NTP offset and whether the installed agent is the one running. Reports `disk_free_gb`, `clock_offset_ms`, `checks_failed` plus a per-check breakdown | +| `llm-eval` | — | Scores what a device generated against an eval set: exact, regex and JSON-shape rules, plus a model judge at an endpoint the job names | +| `model-convert` | `gguf`, `coreml`, `tflite` | A checkpoint into GGUF, Core ML or TFLite, declared per format whose toolchain resolves here; each output is uploaded for a dependent job | +| `dataset-prep` | — | An eval set, carrying its licence and source on the row. Declared only when a converter resolves | +| `serve` | — | `llama-server` for the length of a lease, bound to loopback or the tailnet, with its endpoint announced | +| `shell` | — | A script whose sha256 the owner pinned in `FLEET_SHELL_ALLOWLIST` (default `shell-allowlist.txt` in the data directory). Not declared without a non-empty allowlist | + +Each is described in [the machine workloads page](../docs/workloads/machine.md). Capabilities are declared only when honest: `benchmark:llama.cpp` needs a resolvable `llama-bench`, `benchmark:mlx` needs `import mlx_lm` to exit 0, `build:gradle`/`build:xcode`/`build:npm` each need -their binary (`gradle`, `xcodebuild`, `node`) to resolve, and a machine with none of it still -declares `benchmark` and `self-check`, both of which need nothing installed. +their binary (`gradle`, `xcodebuild`, `node`) to resolve, `model-convert:` and +`dataset-prep` need a converter toolchain, `serve` needs `llama-server`, and a machine with none +of it still declares `benchmark`, `self-check` and `llm-eval`, which need nothing installed. Bare `build` rides along with any kind — the collector matches a claim on the workload name and a build's kind lives in `params` where `capabilityMatches` cannot see it, so a machine declaring only `build:gradle` would never claim a build at all. @@ -76,6 +84,10 @@ against a throwaway collector, as are `build` (success, failure, log upload, pub against `kind: "npm"` only — the gradle and xcode command lines are unit-tested but have never run a real compiler, and the `.app` zip path and the `xcodegen` step with them. +`model-convert`, `dataset-prep` and `serve` have never run against a real toolchain: none of the +converters resolves on the development machine, the probes correctly decline to declare them, and +the command lines are unit-tested only. + The llama.cpp backend is unit-tested against llama-bench's documented JSON shape but never run against a real binary; the Windows probes are untested on Windows; `benchmark:mlx` has no backend yet; and off Linux `mem_method` says `max_rss`, a value the diff --git a/skills/fleet-runner/SKILL.md b/skills/fleet-runner/SKILL.md index b020ab1..a43684c 100644 --- a/skills/fleet-runner/SKILL.md +++ b/skills/fleet-runner/SKILL.md @@ -66,7 +66,7 @@ job) or `running`. `fleetr run`, `job`, `wait` and `retry --wait` exit with: - **1** failed, mixed or cancelled: read `failing_step` and `error` before doing anything else. - **2** usage error or invalid spec: fix the flags or the spec; `--validate` names each problem. - **3** still running when `--timeout` ran out: `fleetr wait ID` picks it up again. -- **4** skipped: nothing ran (no build for the device, no matching device). Never report this as a pass. +- **4** skipped: the job was claimed but reported `skipped:` instead of running (the app is not installed, the device was locked by another job, no attached device matched a host job, the host was under memory pressure, no Roku password on the host). Never report this as a pass. A spec nothing can run is refused before it runs instead: exit 2 from validation, or 5 when the collector refuses it at enqueue. - **5** refused or unreachable: no permission, unknown workload or job, or no collector. Report it; do not work around it.