Repository navigation
Four repositories become one - #2
Merged
Merged
Conversation
Job/result protocol (schema 1) with the device/host executor split, Fastify + SQLite collector with long-poll claiming, pool-aware dispatch, idempotent results, sha256 artifact store with Range support, and a server-rendered dashboard. End-to-end smoke script simulates both executor types (19 checks, all passing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Foreground service registers with the collector, long-polls for device-executor jobs, and reports results plus a 60 s telemetry beacon (battery/charging/thermal). ModelBackend abstraction with a synthetic SHA-256 throughput backend (real, device-comparable CPU numbers) and an honest fail-fast llama.cpp stub for Phase 1b. Benchmark engine: timed load, mandatory warmups, per-iteration rows, final summary closes the job. Verified end-to-end on an SM-X930 against a live collector. Unit tests cover protocol wire format and backend scaling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
llama.cpp pinned as a shallow submodule; thin JNI bench wrapper mirroring llama-bench (prefill batch + token-by-token decode, raw timings out). LlamaCppBackend fetches GGUF models through the collector's content-addressed artifact store with streaming download + sha256 verification. arm64-v8a only; big-core thread default (min(cores, 6), overridable via n_threads param). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CMAKE_BUILD_TYPE=Release for the native library even in debug app builds: unoptimized llama.cpp kernels understate benchmark numbers severely and would poison cross-device comparisons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A job claimed by a device that dies mid-run stayed 'claimed' forever — an emulator's low-memory killer took out the runner mid-benchmark and bench-qwen05b-emu-2 had to be marked failed by hand in sqlite. Claiming a job now takes a lease instead of holding it indefinitely: - Claims set lease_deadline = now + lease.ttl_s and bump attempts. - Beacons carrying a job_id renew the lease. The response's lease_renewed tells a runner whether it still holds the claim, so it can stop working a job that was swept out from under it. - A sweep (every FLEET_SWEEP_MS, default 15 s, on startup, or forced via POST /jobs/sweep) requeues lapsed claims and fails them once lease.max_attempts is spent, recording why in last_error. Defaults are 600 s and 3 attempts; drain and soak default to 4 h since they run for hours between beacons. The dashboard shows attempt counts, live deadlines, and the reason a job was requeued or given up on. Existing databases are migrated with ALTER TABLE, and jobs already sitting in 'claimed' get one lease window to report in rather than being swept immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The collector was started by hand, so nothing brought it back when it died — which is a poor foundation for a lease sweep whose whole job is surviving unexpected death. It also meant the service could be gone for hours with the only symptom being devices that never get work. KeepAlive rather than KeepAlive/SuccessfulExit: for a service that should always be listening, the exit status does not change what we want to happen. Verified against the real failure mode. tsx runs the server in a child process, so a low-memory kill lands on the child, not on the process launchd supervises; tsx propagates the exit, launchd notices, and the collector is serving again in about six seconds. Killing the supervised process directly recovers the same way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
src/executor.ts claims executor:"host" jobs and drives attached Android devices via adb + Maestro: install fans an APK artifact to every device; ui-test runs a Maestro flow per device, parses the JUnit report, uploads it back as an artifact, and reports per-device pass/fail. Devices without the flow's appId are reported as skipped, not failed. Flows live under flows/ (greenfolio smoke + fleet-runner dogfood smoke). Verified end-to-end: one queued job installed fleet-runner 0.2.0 on two emulators, a second ran the Maestro suite on both (1 passed / 0 failed each, JUnit artifacts stored). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # README.md
SwiftUI app mirroring the Android runner's protocol: async agent loop long-polling the collector, 60 s telemetry beacon (battery, thermal, phys_footprint — labeled, never compared to Android PSS), and a synthetic SHA-256 backend token-for-token identical to the Kotlin implementation so tok/s compares across the fleet. Headless start via simctl launch -autostart 1. Verified: one benchmark spec fanned across an iPhone 16 simulator, an Android ATD emulator, and the SM-X930 into one results table. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e flow Platform-aware targets (adb serials + booted sim UDIDs via simctl), install workload accepts iOS .app-bundle zips (ditto + simctl install), appId presence via get_app_container so one Maestro flow spans Android and iOS. Maestro's iOS driver cannot see simulators under Xcode 26.6, so the iOS ui-test path stays XCUITest (suite.kind already reserved); Android ui-test verified through the queue. Smoke flow now dismisses the first-launch notification permission dialog on full Android images. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LlamaCppBackend mirrors the Android JNI bench over the C API (prefill batch + token-by-token decode, Metal off in the simulator); ArtifactCache streams GGUF downloads from the collector with sha256 verification. The llama.xcframework is built from the Android repo's pinned llama.cpp checkout (build-xcframework.sh needs a space-free path) and gitignored; canImport(llama) compiles the backend out when absent, and such builds report an honest failure to llama jobs. Compiled and linked; on-simulator run pending a host without CoreSimulator contention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Zero-dep 5-field cron matcher drives a schedules table (upsert/list/toggle, 20 s tick, once-per-minute dedup, enqueue via POST /jobs so validation and fanout apply). POST /jobs fanout:true creates one child per registered pool device, pinned via targets.device_id — a whole-shelf benchmark is one request, and children for offline devices run when the device returns. device_locks: device-executor claims lock implicitly (released on final result and by the lease sweep); host executor acquires explicitly for targets.exclusive jobs so agents idle during UI tests. /power/:pool/:state fires smart-plug webhooks from power.json (see power.example.json). Example nightly schedules ship disabled. Smoke: fanout pinning, lock starvation and release, tick fire and minute dedup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Jobs may carry report_to.github_status (owner/repo@sha). On close the collector records a status_reports row; it only posts a real GitHub commit status when armed with FLEET_GITHUB_STATUS=1 + FLEET_GITHUB_TOKEN — unarmed (the default) the row is a dry run and GET /status-reports is the audit trail. scripts/ci-enqueue.ts is the CI step (upload artifact, enqueue, poll, exit 0/1); ci/example-workflow.yml documents the app-repo workflow and is installed nowhere. Verified end-to-end with a simulated device: pass exits 0, fail exits 1, both statuses recorded posted=0. Smoke covers the dry-run path; nothing in any app repo or CI system references the fleet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
soak (host+beacon): deterministic app launch via resolve-activity, periodic process-alive + battery checks that renew the job lease, per-check result rows; verified across a real tablet and two emulators. ui-test grows suite.kind xcuitest: the generic FleetRunnerUITests bundle asserts over any app via TEST_RUNNER_ env, per-sim pass/fail with the xcodebuild log as artifact — green on iOS 18.4; the iOS 27 beta runtime crashes UI-test runners at bootstrap, so fleet iOS UI tests target stable runtimes. Artifact uploads now stream to disk with incremental hashing (multi-GB GGUFs never buffer in memory). /dash/bench renders the cross-device comparison: latest passing run per device per model/quant/backend/token-count group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A benchmark longer than the lease TTL was previously swept mid-run because beacons never referenced the claim they were proof-of-life for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Beacons stamp the running job_id so long benchmarks aren't swept (fixed alongside Android). The llama artifact download uses the URL the agent was started with — @AppStorage defaults are never persisted, so a fresh install had none. FleetRunnerUITests: a generic smoke that launches any app by FLEET_APP_ID and asserts FLEET_ASSERTS via label/placeholder/title matching (a text field's placeholder is not a staticText); green through the fleet queue on iOS 18.4. Verified live: Qwen2.5-0.5B ran on-simulator through the full artifact-download + xcframework path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
events table + POST /events/:topic and long-poll GET /events/:topic/poll with a cursor — cheap devices publish triggers, capable devices process and publish to <topic>.out, no MQTT broker needed. Result schema gains an artifacts field for runner-uploaded outputs. Verified live: a tablet pipeline node consumed published prompts and returned real Qwen generations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…onesty nativeGenerate: tokenize -> prefill -> greedy sample -> detokenize, exposed through ModelBackend.generate (synthetic backend returns labeled digests). BatchEngine pulls an items artifact, processes each item, and uploads the outputs as a new artifact; PipelineEngine subscribes to a collector topic and publishes generations to <topic>.out. Verified on the SM-X930 with real Qwen completions over Wi-Fi. Device-state honesty from what that run taught: a partial wakelock while the agent runs, a doze-exemption prompt (guarded — ATD images lack the Settings activity), and enforcement of constraints.require_charging / min_battery_pct before a job runs, because on-battery screen-off throttling slowed decode ~100x and the lease sweep had to rescue the job. Scaling unit test given wider contrast to stop flaking under host load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LiteRtBackend: image classifier with job-fixed preprocessing (NCHW/NHWC, ImageNet/unit/raw normalization, CPU threads, opt-in GPU delegate that falls back to CPU and labels the fallback in the report rather than silently mislabeling numbers). VisionEvalEngine (batch job, backend litert): pulls a zipped eval set + manifest, classifies every image, reports top-1/top-5 accuracy with p50/p95 latency and per-image predictions as an artifact. This is what lets pre-LLM devices earn a place on the fleet. First run: the PlantNet-300K ResNet18 hits 77.5% top-1 / 90% top-5 on 120 held-out PlantNet test images at 54 ms p50 (emulator, CPU). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
evals/greenfolio-plant-id.md documents the on-device species-ID evaluation (PlantNet-300K ResNet18 + a houseplants-47 latency comparator over 120 PlantNet test images), the first numbers, and what they mean for replacing the cloud Plant.id call: viable, top-5 is the product surface, int8 is the next candidate. Eval manifest + species labels saved under evals/. A weekly fan-out schedule is seeded disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…curacy Post-training int8 quantization of the PlantNet-300K ResNet18 via the ai_edge_litert calibration wrapper (100 validation-split images, disjoint from the test-split eval set; float I/O preserved so app preprocessing is unchanged). Recipe in evals/plant-id-assets/quantize_int8.py. On the fleet: 12 MB / 11 ms p50 / 76.7% top-1 / 88.3% top-5 vs fp32's 47 MB / 54 ms / 77.5% / 90.0%. Host fp32 accuracy matches device fp32 exactly (bit-faithful preprocessing). int8 becomes the shipping recommendation: in-app asset size, viewfinder-capable on CPU, no GPU delegate dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, port 8788 targets.match: a small safe expression language over device descriptors (ram_mb >= 4000 && os ~ 'android') evaluated at claim time and in fan-out; validated at enqueue; smoke covers claim gating and fan-out filtering. Executor: physical iOS devices via devicectl (list/install/launch/apps/ battery) alongside sims; drain workload — launch app, replay a GPX route (simctl location on sims, adb emu geo on emulators, app-side replay noted for physical devices), sample battery + process-alive with lease-renewing beacons, report the drain curve and %/hour, refuse to run on a charging device unless allow_charging. Collector moves to :8788 (8787 was taken by another project's dev dashboard; a KeepAlive collector that can't bind respawns forever while every client talks to the wrong service). Plant-ID eval gains the iOS Core ML rows (fp16 + int8-weight, both ~76% top-1 / 90% top-5 on identical images) plus the conversion recipe and a Swift check harness; documents the Simulator's silent all-zero GPU/ANE logits failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CoreMLBackend classifies via compiled .mlmodelc/.mlpackage artifacts with ImageNet normalization folded into the model; Workloads.swift mirrors the Android VisionEval/Batch/Pipeline engines (llama.cpp generate added for batch/pipeline); MiniZip extracts fleet artifacts (stored+deflate) since iOS has no public unzip. The agent now enforces constraints.require_charging / min_battery_pct like Android. Hard-won: the iOS Simulator's emulated GPU/ANE returned an all-zero logits tensor with no error — the backend forces .cpuOnly on simulators and labels it (real devices honor compute_units). Verified: PlantNet-300K ResNet18 Core ML fp16 76.7%/90.0% and int8-weight 75.8%/90.8% top-1/top-5 on the shared 120-image eval set, matching Android. Default collector URL moves to :8788. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
params.sustained_minutes keeps iterating for N minutes instead of a fixed count — the per-iteration rows are the thermal curve, and tok/s over wall time is the honest number for real workloads. Renews the lease every five iterations since sustained runs outlive the TTL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…numbers The collector no longer lives on the workstation, where it collided with another project's dashboard on 8787 and died whenever the laptop slept. It now runs on fleet-host, a 2016 MacBook Pro doing nothing else: a user-local Node tarball, a LaunchAgent, and better-sqlite3 from a prebuild, so the whole stack installs over SSH with no sudo and no Xcode tools. adopt-fleet-host.sh provisions it and reports the facts that decide fitness (RAM, disk, battery condition). Migration moved 68 jobs, 213 results, 41 artifacts and 602 MB of models with counts verified on both sides. Executors stay next to their devices and reach the collector over the LAN. The move also drained the tablet's backlog the moment it rejoined, which finally answers the min-spec question with real hardware: on a Dimensity 9400, int8-on-CPU runs the plant classifier at 7 ms p50 — faster than fp32 with the GPU delegate (11 ms) and loading 18x quicker — so the shipping config needs no GPU delegate at all. Accuracy matched the emulator to the decimal, confirming the fixed preprocessing is bit-faithful across devices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The collector moved off the workstation to fleet-host (192.168.50.27:8788), so a device pointed at 127.0.0.1 now finds nothing. Loopback only ever worked via `adb reverse` on a tethered device anyway; the LAN address works for tethered, wireless and emulated devices alike. The host wants a DHCP reservation — a lease change would strand every device at once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The collector now runs on fleet-host (192.168.50.27:8788); simulators and real devices both reach it over the LAN. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registry was only visible through the HTML dashboard, which a script can't reasonably parse. Enrolling a shelf of devices needs to answer "did it actually check in?" without a human squinting at a page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Builds, installs, clears stale prefs, points the device at the collector via a base_url intent extra (so moving the shelf to a new collector needs no rebuild), starts the agent, and verifies the check-in. Names the two failure modes that actually happen instead of failing opaquely: a device sitting on its unanswered "Allow USB debugging?" prompt, and an API level below minSdk. 32-bit devices enroll too — they run the synthetic backend, which is what makes an old handset a useful baseline rather than dead weight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Without a license file nobody can legally use any of this, which is a strange thing to say about code you are publishing on purpose. MIT matches llama.cpp, which is the most consequential thing the fleet links. The notices file covers what a license scanner cannot see: the pinned llama.cpp commit, the Apache-2.0 vision runtimes, and — on the collector — the CC-BY-SA dataset behind the plant-ID eval. None of those are committed here. Models and eval images are fetched by content hash into the artifact store, which is why the share-alike images are distributed inside the fleet rather than checked in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without a license file nobody can legally use any of this, which is a strange thing to say about code you are publishing on purpose. MIT matches llama.cpp, which is the most consequential thing the fleet links. The notices file covers what a license scanner cannot see: the pinned llama.cpp commit, the Apache-2.0 vision runtimes, and — on the collector — the CC-BY-SA dataset behind the plant-ID eval. None of those are committed here. Models and eval images are fetched by content hash into the artifact store, which is why the share-alike images are distributed inside the fleet rather than checked in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README was 560 lines and opened with "Phase 0 of the Fleet Runner plan". That is the right document for the person operating this and the wrong one for everybody else: it answered "how do I configure web-shots profiles" before it answered "what is this". So the README is now the short version -- what the fleet does, a diagram of how the pieces talk, one screenshot of real benchmark numbers, and enough to get a collector running -- and every word of the reference moved intact to docs/operations.md, with its relative links repointed and a screenshot gallery of the dashboard added where it describes one. The status section says plainly that there is no authentication and that FLEET_DASH_TOKEN is a speed bump rather than an access control, because a reader deciding whether to trust this should not have to infer that from a sentence in the middle of a manual. npm test still ALL PASS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Opened with "Phase 3" and a protocol description, which assumes you already know what a fleet is. It now says what the app is for before it says how it works, and carries a screenshot of the agent registered with a collector and polling -- taken against a throwaway collector on a spare port, so nothing was added to a real fleet's registry to get it. The screenshot is also the honest picture: this is a headless agent with three controls, and pretending otherwise would misrepresent it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Said "Phase 1 of the plan" before it said what the app does. It now leads with the purpose, names the two things that make this worth reading -- that a phone too old for an LLM is still a useful fleet member via the synthetic backend, and the actual tok/s measured on real silicon -- and links the license. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dashboard had no favicon and a text-only brand. It now carries the Fleet Runner glyph — the pulse from the Android launcher mark, on its ink square — as the tab icon and beside the wordmark in the topbar. The SVG lives in dash/public so Vite copies it into dist and the collector serves it at /dash/favicon.svg; the legacy pages link the same file. A set of sixteen stroke icons on a 16 grid, all currentColor, lands in dash/src/icons.tsx. They sit beside their words rather than replacing them: online/stale/offline pills, the charging bolt (which replaces the emoji the devices table used), the thermometer before the thermal state, a dashed phone for simulators, and one icon per workload in every job table. Archive and digest get no icon on purpose — a wrong one would be worse than none. Verified: npm test (typecheck, dashboard build, smoke) passes, and the live collector renders the rebuilt bundle with no console errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The status pill ran straight into the simulator icon: the old marker carried its leading space inside the text, and JSX drops the newline between the two new children. An explicit space, as the line above it already has. From the same review: a low battery never rendered red because the bare .bad/.warn classes only exist scoped to stat tiles — the word classes are .text-bad/.text-warn, so Battery now uses those; IconName was widened to string by the Record annotation, so `satisfies` keeps the union; the comment on icon-less workloads names pipeline too; and the topbar glyph is decorative, since the wordmark beside it already says the name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dashboard assets: mark, favicon, and an icon set
Pipeline, archive and digest shipped without icons on the grounds that a wrong one is worse than none. Drawing them carefully turned out to be the answer rather than the risk: pipeline is three stages joined left to right, archive a lidded box, digest a document with its lines. Same 16 grid, same 1.5 stroke, same currentColor as the rest of the set, and they sit beside their words like the others. WORKLOAD_ICON now covers every workload WORKLOADS accepts, so no job table has a row whose workload is the odd one out. The topbar glyph's tile was ink on every theme. On the dark shell #1c2025 is within a hair of --panel, so the square disappeared and left the pulse floating with nothing under it. The tile takes its fill from the stylesheet now and lifts to #2a2e35 on dark, the value the README banner already used. It stays a literal hex rather than a token because it is brand ink, not part of the theme, and the favicon and docs mark are deliberately left alone: they sit on browser chrome and on paper, not on the dark shell. Verified: npm test (typecheck, dashboard build, smoke) passes, and the live collector renders a real pipeline job with its icon, the tile computing #1c2025 on light and #2a2e35 on dark. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three findings from the review of #29. The digest icon closed its outline with `z`, which runs a straight line from the bottom-left corner back to the start point rather than up the left side. The document was a trapezoid leaning a full grid unit over its height, with three rounded corners and one sharp slanted edge. Invisible at 14px, plain at the 32px the asset sheet draws it at. It closes up the left edge now, with the corner the other three already had. The glyph's tile keeps its ink as a presentation attribute again. Moving it wholly into the stylesheet meant the rect fell back to black whenever the CSS had not applied — a cached index.html asking for a hashed stylesheet that emptyOutDir has since deleted is the way that happens here. Presentation attributes lose to any selector, so the dark rule still wins and the light rule is gone. WORKLOAD_ICON claimed to cover every workload the collector accepts and nothing enforced it; the same comment had already drifted once. `npm test` now compares the two lists as text, since the dashboard deliberately does not import collector types. Verified it fails on a workload with no icon, on an icon for a workload that no longer exists, and on either literal being renamed out from under the regex — a check that cannot fail is not a check. Verified: npm test (typecheck, dashboard build, icon coverage, smoke) passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Icons for the last three workloads, and a glyph tile that survives dark
The collector answers a beacon with `lease_renewed`, and false means the claim is gone -- cancelled from the dashboard, or swept. We have claimed to honour that for a while without ever reading it: `post` threw the response body away, and the beacon task threw the result away with `try?` on top of that. Cancelling a sustained benchmark from the dashboard freed the queue slot and left the phone hashing for another twenty minutes. So the beacon now reads the flag, and a job named by a beacon that comes back explicitly not renewed lands in a small locked registry the workloads check at their iteration boundaries -- between measure iterations, between eval images, between batch prompts, between pipeline events. The workload stops there and posts its final row as ok: false, error: "cancelled", which is what the dashboard already knows how to show. Only an explicit false in a 2xx body counts. A timeout, a refused connection or a 502 leaves the flag alone and stays swallowed by the same `try?` as before -- a collector we cannot reach is not a collector telling us to stop, and a runner that quit on a flaky network would be worse than one that finishes work nobody wants. Registration also declares what this runner can actually run, which the collector is starting to route on. The list is the one the dispatch switch handles, in one constant next to it, so the two cannot drift and we don't get queued work we would only bounce back as unsupported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A beacon carrying a job_id already renews that job's lease, and the collector answers `lease_renewed: false` once the claim is gone — the dashboard cancelled the job, or the sweeper took it back. The runner threw that answer away: postResult discarded the body and the beacon loop swallowed everything, so a cancelled benchmark kept burning battery until it finished on its own. postResult now returns the ack, the beacon loop turns an explicit false into JobCancellation.cancel(jobId), and each engine checks the flag at its own iteration boundary — between iterations, between batch items, between pipeline polls — then unloads the model and closes the job with ok=false, error="cancelled". Only an explicit false in a 2xx body counts: a timeout, a dead collector or an HTML error page still reads as "lease held", because a flaky network must never look like a cancellation. Registration also declares capabilities now, for the collector's capability routing. The list lives beside the dispatch `when` it mirrors so the two cannot drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The queue routed on pools, which are labels a person has to keep accurate, and on a closed set of thirteen workload names, which meant a new runner could not add work without a release here. Both are now backed by a capability: a list an agent sends with every registration, describing its own code and toolchain. The queue never offers an agent a workload it did not declare. That check sits above targets.match on purpose — an expression narrows the set of eligible agents, it must not grant one a workload its code does not contain. Fan-out and the composer's preview use the same predicate, so a preview cannot promise devices the claim would then refuse. Two rules keep this from idling a running shelf on upgrade. No capabilities key means "no opinion", not "nothing", so every runner that predates the field is offered everything exactly as before; and a re-registration that omits the key keeps what was declared last, so rolling back to an older build cannot silently widen it back to all. POST /jobs still accepts the thirteen shipped workloads unconditionally — they live in the collector and the host executor, which do not register as devices. Beyond those it accepts anything some agent declares, and refuses the rest with a 422 naming the missing capability, at enqueue time rather than as a job that sits queued forever with nothing to explain why. Two things travel with it. FLEET_BIND, because the "no auth, LAN only" threat model stops being true the moment an agent is a laptop that leaves the house. It is a list of addresses to answer on, defaulting to every interface as before; set to loopback plus the host's tailnet address, the collector is reachable from your own devices anywhere and from nothing else. Verified both ways: two addresses both answer, and a loopback-only bind refuses the LAN. And a metric-name check, because the schema is a contract three codebases write and one reads, and nothing has ever enforced it. The last drift is why the plant-ID eval's accuracy rode in decode_tok_s and why no query can reproduce that write-up's numbers. It fails npm test on a metric the collector reads but the schema never declared, and regenerates schemas/metrics.json so the two runner repos have one list to check themselves against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
benchmark measures a cold device. Nobody knows whether the plant-ID eval's 7 ms per image survives five minutes in a warm pocket, and that question decides whether the feature ships behind a "device is warming up" banner. thermal answers it: measured iterations back to back for a fixed duration, one result row each, so the output is a curve. params.duration_s (default 900) and params.interval_s are the shape. The interval paces iteration starts rather than the gaps between them, so the cadence cannot drift as iterations slow down, which is the one thing this workload expects them to do. There is no second inference path -- the measure step is ModelBackend.runIteration, the same call BenchmarkEngine makes, against the same backend. The loop arithmetic lives in ThermalRun, apart from Context, the backend and the clock, because a quarter-hour on a warm phone is the worst possible place to find out the duration was off by an iteration or the lease went un-renewed. Beacons are due on elapsed time, never on iteration count, for the same reason. Cancellation is checked at the top of each iteration as the other engines check it. Protocol.kt mirrors elapsed_s and thermal_state from the collector's schemas/metrics.json. Per-iteration rows carry both -- the curve's two axes, measured by the runner rather than inferred from when a row happened to reach the collector -- alongside the thermal array, which stays because a one-element array is what every existing reader of these rows already understands and the scalar and the sequence answer different questions. The summary row carries the turn: elapsed_s at the first iteration whose state differed from the one the run started in, null when the device never got warm, plus decode_tok_s from the last iteration rather than the mean, since averaging a curve that falls by design describes no moment of the run. battery_pct is a beacon field and not a metric name, so per-iteration battery goes out as battery_end_pct, as it does in the benchmark's rows. benchmark's sustained_minutes param is gone with this. It was the thermal curve done by hand -- iteration count swapped for a wall-clock deadline, no elapsed axis on the rows, no summary of where the device turned, and a beacon every fifth iteration on the assumption that iterations keep their length, which is the one assumption a warming device breaks. Leaving it would be a second way to ask for a curve and get a worse one. Nothing outside BenchmarkEngine referenced it: no job spec, no seed, no collector code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plant-ID eval's 7 ms per image was measured on a phone out of a drawer. Whether it survives five minutes in a warm pocket is what decides whether that feature ships with a "device is warming up" banner, and until now there was no way to ask. `thermal` asks. It runs measured benchmark iterations back to back for params.duration_s (default 900) and posts one row per iteration carrying elapsed_s, decode_tok_s, thermal_state and the battery, so the output is a curve rather than a number. The collector already reads exactly those rows and derives the first and last tok/s, the drop between them, and the second at which the OS admitted the device was warming up. The iterations are the same iterations. BenchSession is the load, warmup and measure the benchmark already did, lifted out so both workloads drive it -- a warm curve is only readable against a cold number if identical code produced them. runBenchmark and runLlamaBenchmark now go through it too, which also gives the llama ms-to-tok/s conversion one home instead of two. Two things the metric names would not carry, left uncarried rather than smuggled: the cold tok/s has no second name to sit under on the final row, so it stays iteration 1's own row, and the per-sample battery rides in battery_end_pct because battery_pct is a beacon field. Inventing names here is how the vision numbers ended up unqueryable. sustained_minutes stays declared and unread on iOS. Android's benchmark loops on it; this runner never has, and `thermal` is that idea done properly, so there is no second way to do it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shelf has three kinds of agent and all of them are phones or the Mac that drives phones. A machine is neither: it registers and claims device work like a runner, but it is also the thing somebody is typing on, which is why the collector already grew require_ac, require_idle_s and max_load and had nothing sending the fields they read. This agent sends them. Same stance as the two phone runners: the protocol is reimplemented in TypeScript rather than importing the collector's fleet-client, because an agent that is a device on the fleet should depend on the wire format and not on the collector's source tree. The synthetic backend is ported from the Android and iOS ones token for token -- same 1000 rounds over the same 4 KiB block with the digest folded back in -- after reading both and confirming they agree with each other. A test pins the block state after one token's rounds against an independently computed chain, since a refactor that changed the work per token would silently invalidate every stored row on all three platforms and break nothing visibly. The descriptor carries model, soc, ram_mb, os and app_ver so expressions written for the phones keep working, plus kind, arch, gpu, vram_mb and cpu_cores, which are the facts about a computer no phone descriptor had a place for. Every probe degrades to a null field: seven tools across three operating systems, and any one of them missing must cost a field rather than registration. That rule found its own bug -- Number(null) is 0, so an unanswered probe was reporting a machine with no RAM at zero percent battery until finite() started rejecting null ahead of the conversion. Cancellation reads lease_renewed the way both phone runners now do, and only an explicit JSON false in a 2xx body stops work. An absent field, an empty body, a proxy's HTML and a string "false" all read as renewed, because a collector we cannot reach is not a collector telling us to stop. Verified against a throwaway collector on a spare port: the agent registers with its capabilities, claims a synthetic benchmark and posts its rows; cancelling from the dashboard lands a final ok=false error="cancelled" row at the next iteration boundary; killing the agent mid-job leaves the sweep to requeue rather than strand it. require_ac claimed and max_load 0.01 stayed queued, which is the beacon's new fields being read rather than merely stored. Capabilities are probed rather than declared. llama-bench has to resolve to an executable file and mlx_lm has to import; a machine with neither still declares benchmark, which is what the synthetic backend is for and what makes CI pass on a runner with no GPU. Two things are honest gaps rather than finished work: the llama.cpp backend is written against llama-bench's documented JSON and has never been run against a real binary, and off Linux mem_method says max_rss -- a value the collector's result schema does not list. Calling an RSS reading pss or phys_footprint would be exactly the laundering that schema's comment forbids, so the collector needs the third value instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wave 1. The collector half of making a laptop a first-class fleet member, plus
the two workloads that answer questions a single cold number cannot.
Constraints now have two enforcement points, and the split is the point.
require_charging and min_battery_pct stay with the runner, checked against live
state, refusing the job with a failed result — right for a contract about the
device, because a benchmark on a throttling phone produces a number that lies.
require_ac, require_idle_s, max_load and window are checked here, before the
claim, against the last beacon: an unsatisfied one leaves the job queued and
offers it again later. A laptop that is on battery, or busy, or awake at the
wrong hour has not failed at anything, and burning max_attempts on that would
exhaust a machine that was working perfectly. Stale beacons fail closed.
Fan-out grew two shapes. { distinct: "os" } enqueues one child per distinct
value of a descriptor field — a smoke pass on every OS on the shelf without
paying for every phone — and host jobs can fan out at all, pinned exactly as
device children are, which the executor's target selection already honoured.
thermal and cold-start arrive with their metric names, their result views and
their dashboard pages. thermal gets a four-hour lease: its whole value is an
unbroken curve, and a swept lease mid-run would requeue the one thing that
cannot be resumed. cold-start splits p50 and p95 by launch state, because a
percentile mixing cold and hot launches describes nothing that happens to a
user. On the host side both are real for Android and honestly refused for iOS,
where simctl returns on process spawn rather than first frame and filing that
into a field documented as time-to-first-frame would be a lie with a number
attached.
params.network can shape any host job, and refuses far more than it accepts:
no in-device 3g on an unrooted phone, no offline on an adb-tcp serial that
would cut the executor's own control channel, and never simctl's status-bar
override, which redraws an icon and moves no packet. Intent is journalled
before the device is touched and every attached device is restored at startup,
because a phone left offline by a crashed executor is a phone that looks dead
forever. The host dnctl path is opt-in behind three verified preconditions and
has not been run; that is said out loud in the docs rather than implied by its
presence.
Three alert rules watch a metric against its own seven-day median rather than a
fixed threshold. The comparison key carries more weight than the threshold: same
device, model, quant and backend, or a "regression" turns out to be a different
model. Percent rather than absolute, because one number in tok/s would be noise
on current silicon and silence on a 2016 phone.
Artifacts can be pinned, and accepted baselines pin themselves. The reference
scan reads specs, results, schedules and templates; a baseline is referenced by
a row and by nothing textual, so the scan would have offered every one of them
for collection while its entire purpose is to still exist months later to diff
against.
Two fixes found on the way. The regression query orders by rowid as well as
created_at, because SQLite stores whole seconds and "which run is the latest"
was undefined exactly when it mattered. And the lease test's two-second budget
against that same second-granular clock left under a second of margin, so on a
loaded machine it decided the sweep had eaten a lease it had not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wave 0: capability routing, bind control, and a metric-name check
Honour a lapsed lease, and say what this runner can run
Honour a lapsed lease, and say what this runner can run
Wave 1: laptop constraints, thermal and cold-start, shaping, regressions, pins
Does the cold number survive the device getting warm?
Does the cold number survive the device getting warm?
The split was right while these were four independent things. It stopped being right when the protocol started changing. A metric name is declared in collector/schemas/result.schema.json and mirrored by hand in three runners. A capability list is declared by an agent and enforced by the queue. A new workload touches a runner, the schema, a results endpoint and a dashboard view. Every one of those is one change that has to land in several places at once, and across repositories it landed as several pull requests that could each merge alone. That is not hypothetical drift: an eval's accuracy once rode in a field named decode_tok_s because vision had no field of its own, and no query can reproduce that report's numbers today. The four projects keep their own READMEs, tests, licences and notices, and still share no code — only the JSON protocol. What they no longer have is four places for a protocol change to half-land. collector/ the brain: queue, registry, artifacts, results, dashboard runner-android/ the Android agent runner-ios/ the iOS agent runner-machine/ the desktop agent History is preserved: each project came across with git subtree, so every commit and every author survives and `git log --follow` still works through the move. 105 commits. CI is four path-filtered workflows rather than four repositories' worth. A change to a phone runner no longer builds the dashboard. The filtering has one trap, written down in collector.yml where someone will meet it: a filtered workflow does not run on a PR that touches nothing it watches, and a required check that never ran reads as pending forever, so an iOS-only PR would block on the collector's suite. These stay advisory until there is a merge-gate job that always runs. Two path references actually broke and are fixed rather than left to fail at runtime: the executor's default path to the iOS project, and the same path in the collector's LaunchAgent installer. Both still honour FLEET_IOS_PROJECT, which is what the workstation executor uses. Verified in place, all four: the collector's npm test passes end to end against its own throwaway collector; the machine agent's 55 tests pass; the Android unit tests build and pass under -Pfleet.skipNative; and the iOS app builds for the simulator from a clean xcodegen run off project.yml, which is the spec a stranger's checkout uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The split was right while these were four independent things. It stopped being right when the protocol started changing.
A metric name is declared in
collector/schemas/result.schema.jsonand mirrored by hand in three runners. A capability list is declared by an agent and enforced by the queue. A new workload touches a runner, the schema, a results endpoint and a dashboard view. Each of those is one change that has to land in several places at once, and across repositories it landed as several pull requests that could each merge alone.That drift is not hypothetical. An eval's accuracy once rode in a field named
decode_tok_sbecause vision had no field of its own, and no query can reproduce that report's numbers today.Layout
Each keeps its own README, tests, licence and third-party notices, and they still share no code — only the JSON protocol. What they no longer have is four places for a protocol change to half-land.
History is preserved
Each project came across with
git subtree, so every commit and every author survives andgit log --followworks through the move. 105 commits.CI
Four path-filtered workflows replace four repositories' worth. A change to a phone runner no longer builds the dashboard.
collector.ymlwhere someone will meet it. A filtered workflow does not run on a PR that touches nothing it watches, and GitHub reports a required check that never ran as pending forever — so an iOS-only PR would block on the collector's suite. These stay advisory until there is a merge-gate job that always runs. Worth deciding before you re-add required checks to branch protection.Two references actually broke
Fixed rather than left to fail at runtime:
collector/src/executor.ts— the default path to the iOS projectcollector/deploy/install-agent.sh— the same path in the LaunchAgent installerBoth still honour
FLEET_IOS_PROJECT, which is what the workstation executor uses when the tree lives somewhere else.Verified in place, all four
npm testpasses end to end against its own throwaway collector-Pfleet.skipNativeproject.yml, the spec a stranger's checkout usesThe Android build needs
ANDROID_HOME;local.propertiesis gitignored and did not come across, which is the documented fresh-clone path anyway.Still to do after this merges
fleet-collector,fleet-runner-iosandfleet-runner-androidwith a pointer README. Their history lives here now, but their URLs are in the portfolio copy and in old PR descriptions, so they should redirect rather than disappear.fleet-runner-machinewas never pushed as its own repository, so there is nothing to archive for it.🤖 Generated with Claude Code