Repository navigation
Explore: survive a dropped model, judge only missions that drove the app, fail fast on Xcode's first launch - #101
Conversation
…app, fail fast on Xcode's first launch The first three nights on 0.9.0 (10-15..10-17) on ultra: - The model shelf unloaded Holo4 as idle under memory pressure mid-run four times; later missions ended at step 0 on "Invalid model identifier". The workload now runs the night's warm-up again (FLEET_EXPLORE_WARM, from explore-night --warm) and retakes the step, twice a mission at most. - The judge filed the untouched or abandoned screens of those missions as goal findings (#70, #71, the bug garden's 10-17 goals). A mission cut off by the model server or device, or abandoned by the model, is no longer judged; a cut-off mission counts as explore_cut_off and an unfinished cut-off bench card is not scored. - Dozehound hides its channel bar six seconds after a press, so the final look failed bench-focus-all-channels on a screen the model had answered from correctly. A bench card is also checked on the model's last look when nothing touched the device after it. - Xcode 27.1 arrived without its first-launch install: every FleetDriver start exited 70 (CoreSimulator out of date) a minute in, card after card. explore-night --sim checks xcodebuild -checkFirstLaunchStatus before warming and fails with the fix named; the workload stops at the first such error. - A failed explore job's error is its device's reason, not "see their rows". - explore-night's log prefix is local time, like the times in its lines. - changelog-coverage exempts release/* and develop->main pull requests (#98). Runbook: the 10-15 tvOS bench (5/10 each model, bar met) and iOS night. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
…OLD_PID) The night command sets SHELF_HOLD_PID to its own pid before warming; the shelf's warm-up records that as a hold and leaves the model loaded while the process lives (addisdev/model-ladder#31). The re-warm after a drop inherits it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Added the hold: explore-night now sets |
|
Self-review: CI green on every lane (collector on 4 platforms, fleet, smokes), mergeable after the #100 merge. Checked: new metric in result.schema.json and metrics.json; changelog bullets carry #101; no API or protocol shape change; docs updated (job.md, explore-on-ultra.md, CONTRIBUTING). Re-warm and the shelf hold are covered by fakes only; first real proof is tonight's nights. Merging. |
|
Self-review: re-warm on a dropped model (bounded, off-budget), no judging of cut-off/abandoned missions, Xcode first-launch preflight, changelog-coverage exempting release PRs. The new explore_cut_off metric is accepted by a 0.9.1 brain (metrics allow unknown keys), so this deploys to ultra's checkout without a release. Checks green. Merging. |
#101 merged into develop after the 0.9.1 release, and the merge placed its bullets under the renamed [0.9.1] heading. v0.9.1 does not contain it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CHANGELOG: #101 is unreleased, not part of 0.9.1
What the first three nights on 0.9.0 showed (10-15 to 10-17, ultra)
docs/deploy/explore-on-ultra.md.Invalid model identifier, and the judge filed their untouched start screens as goal findings (explore: overnight exploratory QA (workload, findings, actuators, bug garden, ultra runbook) #70, Bump fastify from 5.12.1 to 5.12.5 in /collector #71).CoreSimulator is out of date. The TV and Apple nights tried every card for about ten minutes a job, failed with "see their rows", and turnedmodel-ladder#nightred. The fix needs sudo and is WB-384.Changes
explore-night.ts --warmpasses its command down asFLEET_EXPLORE_WARM. When the server says the model is gone (Invalid model identifier,Model is unloaded.), the loop runs that command and takes the step again, at most twice per mission. That step does not count against the step budget or the model.explore_cut_off(added toresult.schema.json), not inexplore_missions. A cut-off bench card is not scored unless it had already reached its end state.--simasksxcodebuild -checkFirstLaunchStatusbefore warming the model, and fails withsudo xcodebuild -runFirstLaunchnamed as the fix. This only runs on macOS, and tests stub it withFLEET_XCODEBUILD. Inside a job,HostNotReadyErrorstops the device at the first such error.explore-nightlog prefix is now local time.changelog-coverage.mjsexempts the release's own pull requests:release/*(read from the merge subject, or fromghfor a squash merge) anddevelop→main. Release 0.9.0 #98 had to be labelled by hand.The S8+ wake and lock-screen fix is not in this PR. It is in #100.
Test plan
tsc --noEmiton ultra (~/fr-work/night-1016)scripts/check-explore.ts: 239 checks, including new ones for a dropped model, a cut-off mission, an abandoned mission, an auto-hiding end state, Xcode's first launch in the night command, and a reason on a failed jobnode --test scripts/changelog-coverage.test.mjs: 9/9. Against real history (--head origin/develop --tag v0.8.0), Release 0.9.0 #98 is exempt with no network.npm teston ultra: exit 0, every step green (dashboard build skipped, not installed there)Not verified
🤖 Generated with Claude Code