Skip to content

Explore: survive a dropped model, judge only missions that drove the app, fail fast on Xcode's first launch - #101

Merged
addisdev merged 4 commits into
developfrom
fix/night-1016
Oct 19, 2026
Merged

addisdev merged 4 commits into
developfrom
fix/night-1016

Conversation

@addisdev

Copy link
Copy Markdown
Owner

What the first three nights on 0.9.0 showed (10-15 to 10-17, ultra)

  • The tvOS bench ran for the first time and met phase 3's bar, just: Holo4 5/10, Qwen3.8-27B 5/10 (6/10 with the end-state fix here, re-scored from its saved trees). Numbers are in docs/deploy/explore-on-ultra.md.
  • The model shelf unloaded Holo4 mid-run four times ("memory pressure: unloaded idle", 03:50 and 04:10 on 10-16, 22:33 on 10-16, 22:23 on 10-17). Later missions ended at step 0 on Invalid model identifier, and the judge filed their untouched start screens as goal findings (explore: overnight exploratory QA (workload, findings, actuators, bug garden, ultra runbook) #70, Bump fastify from 5.12.1 to 5.12.5 in /collector #71).
  • Xcode on ultra updated itself to 27.1 on 10-16 without its first-launch install. Every FleetDriver start since then exits 70 with CoreSimulator is out of date. The TV and Apple nights tried every card for about ten minutes a job, failed with "see their rows", and turned model-ladder#night red. The fix needs sudo and is WB-384.
  • On 10-17 Holo4 answered in prose with no tool call from step 1. Seven bug-garden missions took two steps between them. The judge filed the screens it abandoned as goal findings.

Changes

  • A dropped model is loaded again. explore-night.ts --warm passes its command down as FLEET_EXPLORE_WARM. When the server says the model is gone (Invalid model identifier, Model is unloaded.), the loop runs that command and takes the step again, at most twice per mission. That step does not count against the step budget or the model.
  • Only missions that drove the app get judged. A mission the model server or the device cut off is not judged, and neither is one the model abandoned (four replies with no tool call). A cut-off mission goes in a new metric, explore_cut_off (added to result.schema.json), not in explore_missions. A cut-off bench card is not scored unless it had already reached its end state.
  • A bench end state is also checked on the model's last look when nothing touched the device after it. Dozehound's channel bar hides six seconds after a press.
  • Xcode's first launch is checked up front. --sim asks xcodebuild -checkFirstLaunchStatus before warming the model, and fails with sudo xcodebuild -runFirstLaunch named as the fix. This only runs on macOS, and tests stub it with FLEET_XCODEBUILD. Inside a job, HostNotReadyError stops the device at the first such error.
  • A failed explore job puts its device's reason in its own error.
  • The explore-night log prefix is now local time.
  • changelog-coverage.mjs exempts the release's own pull requests: release/* (read from the merge subject, or from gh for a squash merge) and develop → main. Release 0.9.0 #98 had to be labelled by hand.

The S8+ wake and lock-screen fix is not in this PR. It is in #100.

Test plan

  • tsc --noEmit on ultra (~/fr-work/night-1016)
  • scripts/check-explore.ts: 239 checks, including new ones for a dropped model, a cut-off mission, an abandoned mission, an auto-hiding end state, Xcode's first launch in the night command, and a reason on a failed job
  • node --test scripts/changelog-coverage.test.mjs: 9/9. Against real history (--head origin/develop --tag v0.8.0), Release 0.9.0 #98 is exempt with no network.
  • collector npm test on ultra: exit 0, every step green (dashboard build skipped, not installed there)
  • A real night on ultra. That waits on WB-384 for TV and Apple; the bug garden runs tonight.

Not verified

  • The re-warm path has not run against the real shelf. It is covered by a fake server that drops the model.
  • The Xcode preflight has run against ultra's real Xcode only through the tests' first run, where it reported exit 69.

🤖 Generated with Claude Code

addisdev and others added 4 commits October 18, 2026 09:26
…app, fail fast on Xcode's first launch

The first three nights on 0.9.0 (10-15..10-17) on ultra:

- The model shelf unloaded Holo4 as idle under memory pressure mid-run four
  times; later missions ended at step 0 on "Invalid model identifier". The
  workload now runs the night's warm-up again (FLEET_EXPLORE_WARM, from
  explore-night --warm) and retakes the step, twice a mission at most.
- The judge filed the untouched or abandoned screens of those missions as goal
  findings (#70, #71, the bug garden's 10-17 goals). A mission cut off by the
  model server or device, or abandoned by the model, is no longer judged; a
  cut-off mission counts as explore_cut_off and an unfinished cut-off bench
  card is not scored.
- Dozehound hides its channel bar six seconds after a press, so the final
  look failed bench-focus-all-channels on a screen the model had answered
  from correctly. A bench card is also checked on the model's last look when
  nothing touched the device after it.
- Xcode 27.1 arrived without its first-launch install: every FleetDriver start
  exited 70 (CoreSimulator out of date) a minute in, card after card.
  explore-night --sim checks xcodebuild -checkFirstLaunchStatus before warming
  and fails with the fix named; the workload stops at the first such error.
- A failed explore job's error is its device's reason, not "see their rows".
- explore-night's log prefix is local time, like the times in its lines.
- changelog-coverage exempts release/* and develop->main pull requests (#98).

Runbook: the 10-15 tvOS bench (5/10 each model, bar met) and iOS night.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OLD_PID)

The night command sets SHELF_HOLD_PID to its own pid before warming; the shelf's warm-up records
that as a hold and leaves the model loaded while the process lives (addisdev/model-ladder#31).
The re-warm after a drop inherits it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@addisdev

Copy link
Copy Markdown
Owner Author

Added the hold: explore-night now sets SHELF_HOLD_PID before warming, so ultra's shelf leaves the model loaded through the night once addisdev/model-ladder#31 is deployed. The re-warm stays as the fallback. check-explore.ts: 240 checks on ultra; tsc clean.

@addisdev

Copy link
Copy Markdown
Owner Author

Self-review: CI green on every lane (collector on 4 platforms, fleet, smokes), mergeable after the #100 merge. Checked: new metric in result.schema.json and metrics.json; changelog bullets carry #101; no API or protocol shape change; docs updated (job.md, explore-on-ultra.md, CONTRIBUTING). Re-warm and the shelf hold are covered by fakes only; first real proof is tonight's nights. Merging.

@addisdev
addisdev merged commit 69d2b38 into develop Oct 19, 2026
24 checks passed
@addisdev
addisdev deleted the fix/night-1016 branch October 19, 2026 22:52
@addisdev

Copy link
Copy Markdown
Owner Author

Self-review: re-warm on a dropped model (bounded, off-budget), no judging of cut-off/abandoned missions, Xcode first-launch preflight, changelog-coverage exempting release PRs. The new explore_cut_off metric is accepted by a 0.9.1 brain (metrics allow unknown keys), so this deploys to ultra's checkout without a release. Checks green. Merging.

addisdev added a commit that referenced this pull request Oct 19, 2026
#101 merged into develop after the 0.9.1 release, and the merge placed its
bullets under the renamed [0.9.1] heading. v0.9.1 does not contain it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
addisdev added a commit that referenced this pull request Oct 19, 2026
CHANGELOG: #101 is unreleased, not part of 0.9.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant