diff --git a/CHANGELOG.md b/CHANGELOG.md index 5aba0f7..0743d71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,6 +98,12 @@ a CI job fails when they disagree. ### Changed +- **Explore loads its model at 45% free memory, not 65%.** ultra's resident + models alone hold it near 56% free, so the shelf's default bar meant no + explore night could load Holo4 (~25 GB); 10-15's garden night skipped for + it. The night manifests pass `WARM_MIN_FREE_PCT=45` to the warm-up; the + shelf's default for everything else is unchanged, and it still refuses + under memory pressure. ([#95](https://github.com/addisdev/fleet-runner/pull/95)) - **`ci-enqueue` speaks `/api/v1` and exits like `fleetr run`.** It polled the deprecated `GET /jobs/:id` and exited 0 on `done`, which includes a job that skipped and tested nothing, and it sent no token, so it failed against a diff --git a/collector/deploy/explore/night/fleet-explore-apple.json b/collector/deploy/explore/night/fleet-explore-apple.json index 3665787..da2039a 100644 --- a/collector/deploy/explore/night/fleet-explore-apple.json +++ b/collector/deploy/explore/night/fleet-explore-apple.json @@ -4,7 +4,7 @@ "command": [ "bash", "-lc", - "source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-ios:com.apple.CoreSimulator.SimRuntime.iOS-26-5:com.apple.CoreSimulator.SimDeviceType.iPhone-16 --spec examples/jobs/explore-greenfolio-ios-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"bash ~/harness/shelf.sh warm pilot\"" + "source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-ios:com.apple.CoreSimulator.SimRuntime.iOS-26-5:com.apple.CoreSimulator.SimDeviceType.iPhone-16 --spec examples/jobs/explore-greenfolio-ios-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"WARM_MIN_FREE_PCT=45 bash ~/harness/shelf.sh warm pilot\"" ], "priority": 86, "needs_gb": 35, diff --git a/collector/deploy/explore/night/fleet-explore-tv.json b/collector/deploy/explore/night/fleet-explore-tv.json index 31fb525..c3d188a 100644 --- a/collector/deploy/explore/night/fleet-explore-tv.json +++ b/collector/deploy/explore/night/fleet-explore-tv.json @@ -4,7 +4,7 @@ "command": [ "bash", "-lc", - "source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-tvos:com.apple.CoreSimulator.SimRuntime.tvOS-26-5:com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-3rd-generation-1080p --spec examples/jobs/explore-tvos-bench-pilot-ultra.json --spec examples/jobs/explore-tvos-bench-vision-ultra.json --spec examples/jobs/explore-dozehound-tvos-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"bash ~/harness/shelf.sh warm pilot\"" + "source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --sim fleet-explore-tvos:com.apple.CoreSimulator.SimRuntime.tvOS-26-5:com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-3rd-generation-1080p --spec examples/jobs/explore-tvos-bench-pilot-ultra.json --spec examples/jobs/explore-tvos-bench-vision-ultra.json --spec examples/jobs/explore-dozehound-tvos-ultra.json --tunnel runner-ts:192.168.50.27:8788 --fresh-build --warm \"WARM_MIN_FREE_PCT=45 bash ~/harness/shelf.sh warm pilot\"" ], "priority": 85, "needs_gb": 35, diff --git a/collector/deploy/explore/night/fleet-explore.json b/collector/deploy/explore/night/fleet-explore.json index efcce5c..2ea313e 100644 --- a/collector/deploy/explore/night/fleet-explore.json +++ b/collector/deploy/explore/night/fleet-explore.json @@ -4,7 +4,7 @@ "command": [ "bash", "-lc", - "source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --spec examples/jobs/explore-garden-ultra.json --avd fleet-explore-1 --tunnel runner-ts:192.168.50.27:8788 --warm \"bash ~/harness/shelf.sh warm pilot\"" + "source ~/explore/env.sh && cd ~/fleet-runner/collector && npx tsx scripts/explore-night.ts --spec examples/jobs/explore-garden-ultra.json --avd fleet-explore-1 --tunnel runner-ts:192.168.50.27:8788 --warm \"WARM_MIN_FREE_PCT=45 bash ~/harness/shelf.sh warm pilot\"" ], "priority": 20, "needs_gb": 35, diff --git a/docs/deploy/explore-on-ultra.md b/docs/deploy/explore-on-ultra.md index bf367b0..9ef583d 100644 --- a/docs/deploy/explore-on-ultra.md +++ b/docs/deploy/explore-on-ultra.md @@ -23,7 +23,7 @@ LaunchDaemons. Explore plugs into that rather than running its own. | The emulator | AVD `fleet-explore-1`, android-35 `google_apis`, 1080×2400 at 420 dpi | | Holo4 | `~/.lmstudio/models/Hcompany/Holo4-35B-A3B-GGUF/`: the vendor's Q4_K_M and its vision projector, renamed to LM Studio's convention (`Holo4-35B-A3B-Q4_K_M.gguf`, `mmproj-Holo4-35B-A3B-F16.gguf`) or LM Studio lists the projector as a separate model and cannot pair them. Model key `holo4-35b-a3b`, identifier `holo4-35b-a3b-gguf` | | The gateway name | `pilot` in the harness's `litellm.yaml`: Holo4, 300 s timeout, no fallback. The judge is the harness's `judge` role | -| The shelf | `ondemand holo4-35b-a3b@holo4-35b-a3b-gguf 65536 1 3600` in `~/harness/shelf.conf`; `bash ~/harness/shelf.sh warm pilot` loads it when at least 65% of memory is free. The night command runs it (`--warm`) after its own checks, and retries a refusal for memory for ten minutes | +| The shelf | `ondemand holo4-35b-a3b@holo4-35b-a3b-gguf 65536 1 3600` in `~/harness/shelf.conf`; `bash ~/harness/shelf.sh warm pilot` loads it when enough memory is free: 65% by default, 45% for explore, which sets `WARM_MIN_FREE_PCT=45` in its manifests and in `~/explore/env.sh` (the resident models keep ultra near 56% free, so at 65% Holo4 never loaded; at 45% its ~25 GB leaves about 29 GB). The shelf still refuses under memory pressure. The night command runs it (`--warm`) after its own checks, and retries a refusal for memory for ten minutes | | The key | `~/.config/harness/keys/fleet.key` (roles `pilot`, `judge`, `vision`). Not the Keychain: neither SSH nor a LaunchDaemon can write the login Keychain on ultra. The workload reads `HARNESS_KEY`, then this file | | The night | `~/night/recurring/fleet-explore.json`, run by the harness's night queue from 22:00 when memory pressure is normal, 35 GB is free and no CI job is running | | The brain | On fleet-host, which ultra cannot route to. The night command opens `ssh -L 18788:192.168.50.27:8788 runner-ts` for the length of the run | diff --git a/docs/explore/troubleshooting.md b/docs/explore/troubleshooting.md index 2e4e070..237eec0 100644 --- a/docs/explore/troubleshooting.md +++ b/docs/explore/troubleshooting.md @@ -102,8 +102,10 @@ explore job itself, which is why it exits 0. The shelf refused to load the model for memory, and kept refusing for `--warm-wait` minutes (default 10). The night queue admits a job on free memory as `vm_stat` counts it (`needs_gb` in the manifest); the shelf decides -on `memory_pressure`'s free percentage (65% for an on-demand model). They can -disagree by a point, which is how 10-14's Apple night died. Check what else +on `memory_pressure`'s free percentage (65% for an on-demand model by +default; explore's warm-up asks for 45% with `WARM_MIN_FREE_PCT`). They can +disagree by a point, which is how 10-14's Apple night died, and on 10-15 the +resident models alone held ultra at 56% free, under the old 65%. Check what else was loaded with `~/.lmstudio/bin/lms ps`. Any other warm-up failure is a failure, not a skip, and its last line is the job's error.