Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,12 @@ a CI job fails when they disagree.

### Changed

- **Explore loads its model at 45% free memory, not 65%.** ultra's resident
models alone hold it near 56% free, so the shelf's default bar meant no
explore night could load Holo4 (~25 GB); 10-15's garden night skipped for
it. The night manifests pass `WARM_MIN_FREE_PCT=45` to the warm-up; the
shelf's default for everything else is unchanged, and it still refuses
under memory pressure. ([#95](https://github.com/addisdev/fleet-runner/pull/95))
- **`ci-enqueue` speaks `/api/v1` and exits like `fleetr run`.** It polled the
deprecated `GET /jobs/:id` and exited 0 on `done`, which includes a job that
skipped and tested nothing, and it sent no token, so it failed against a
Expand Down
2 changes: 1 addition & 1 deletion collector/deploy/explore/night/fleet-explore-apple.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"command": [
"bash",
"-lc",
"source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-ios:com.apple.CoreSimulator.SimRuntime.iOS-26-5:com.apple.CoreSimulator.SimDeviceType.iPhone-16 --spec examples/jobs/explore-greenfolio-ios-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"bash ~/harness/shelf.sh warm pilot\""
"source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-ios:com.apple.CoreSimulator.SimRuntime.iOS-26-5:com.apple.CoreSimulator.SimDeviceType.iPhone-16 --spec examples/jobs/explore-greenfolio-ios-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"WARM_MIN_FREE_PCT=45 bash ~/harness/shelf.sh warm pilot\""
],
"priority": 86,
"needs_gb": 35,
Expand Down
2 changes: 1 addition & 1 deletion collector/deploy/explore/night/fleet-explore-tv.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"command": [
"bash",
"-lc",
"source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-tvos:com.apple.CoreSimulator.SimRuntime.tvOS-26-5:com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-3rd-generation-1080p --spec examples/jobs/explore-tvos-bench-pilot-ultra.json --spec examples/jobs/explore-tvos-bench-vision-ultra.json --spec examples/jobs/explore-dozehound-tvos-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"bash ~/harness/shelf.sh warm pilot\""
"source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-tvos:com.apple.CoreSimulator.SimRuntime.tvOS-26-5:com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-3rd-generation-1080p --spec examples/jobs/explore-tvos-bench-pilot-ultra.json --spec examples/jobs/explore-tvos-bench-vision-ultra.json --spec examples/jobs/explore-dozehound-tvos-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"WARM_MIN_FREE_PCT=45 bash ~/harness/shelf.sh warm pilot\""
],
"priority": 85,
"needs_gb": 35,
Expand Down
2 changes: 1 addition & 1 deletion collector/deploy/explore/night/fleet-explore.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"command": [
"bash",
"-lc",
"source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --spec examples/jobs/explore-garden-ultra.json --avd fleet-explore-1 --tunnel runner-ts:192.168.50.27:8788 --warm \"bash ~/harness/shelf.sh warm pilot\""
"source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --spec examples/jobs/explore-garden-ultra.json --avd fleet-explore-1 --tunnel runner-ts:192.168.50.27:8788 --warm \"WARM_MIN_FREE_PCT=45 bash ~/harness/shelf.sh warm pilot\""
],
"priority": 20,
"needs_gb": 35,
Expand Down
2 changes: 1 addition & 1 deletion docs/deploy/explore-on-ultra.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ LaunchDaemons. Explore plugs into that rather than running its own.
| The emulator | AVD `fleet-explore-1`, android-35 `google_apis`, 1080×2400 at 420 dpi |
| Holo4 | `~/.lmstudio/models/Hcompany/Holo4-35B-A3B-GGUF/`: the vendor's Q4_K_M and its vision projector, renamed to LM Studio's convention (`Holo4-35B-A3B-Q4_K_M.gguf`, `mmproj-Holo4-35B-A3B-F16.gguf`) or LM Studio lists the projector as a separate model and cannot pair them. Model key `holo4-35b-a3b`, identifier `holo4-35b-a3b-gguf` |
| The gateway name | `pilot` in the harness's `litellm.yaml`: Holo4, 300 s timeout, no fallback. The judge is the harness's `judge` role |
| The shelf | `ondemand holo4-35b-a3b@holo4-35b-a3b-gguf 65536 1 3600` in `~/harness/shelf.conf`; `bash ~/harness/shelf.sh warm pilot` loads it when at least 65% of memory is free. The night command runs it (`--warm`) after its own checks, and retries a refusal for memory for ten minutes |
| The shelf | `ondemand holo4-35b-a3b@holo4-35b-a3b-gguf 65536 1 3600` in `~/harness/shelf.conf`; `bash ~/harness/shelf.sh warm pilot` loads it when enough memory is free: 65% by default, 45% for explore, which sets `WARM_MIN_FREE_PCT=45` in its manifests and in `~/explore/env.sh` (the resident models keep ultra near 56% free, so at 65% Holo4 never loaded; at 45% its ~25 GB leaves about 29 GB). The shelf still refuses under memory pressure. The night command runs it (`--warm`) after its own checks, and retries a refusal for memory for ten minutes |
| The key | `~/.config/harness/keys/fleet.key` (roles `pilot`, `judge`, `vision`). Not the Keychain: neither SSH nor a LaunchDaemon can write the login Keychain on ultra. The workload reads `HARNESS_KEY`, then this file |
| The night | `~/night/recurring/fleet-explore.json`, run by the harness's night queue from 22:00 when memory pressure is normal, 35 GB is free and no CI job is running |
| The brain | On fleet-host, which ultra cannot route to. The night command opens `ssh -L 18788:192.168.50.27:8788 runner-ts` for the length of the run |
Expand Down
6 changes: 4 additions & 2 deletions docs/explore/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,10 @@ explore job itself, which is why it exits 0.
The shelf refused to load the model for memory, and kept refusing for
`--warm-wait` minutes (default 10). The night queue admits a job on free
memory as `vm_stat` counts it (`needs_gb` in the manifest); the shelf decides
on `memory_pressure`'s free percentage (65% for an on-demand model). They can
disagree by a point, which is how 10-14's Apple night died. Check what else
on `memory_pressure`'s free percentage (65% for an on-demand model by
default; explore's warm-up asks for 45% with `WARM_MIN_FREE_PCT`). They can
disagree by a point, which is how 10-14's Apple night died, and on 10-15 the
resident models alone held ultra at 56% free, under the old 65%. Check what else
was loaded with `~/.lmstudio/bin/lms ps`. Any other warm-up failure is a
failure, not a skip, and its last line is the job's error.

Expand Down
Loading