Skip to content

Honest green: skipped is a job status, every failure says why, registry hygiene - #94

Merged
addisdev merged 4 commits into
developfrom
feat/honest-green
Oct 16, 2026
Merged

addisdev merged 4 commits into
developfrom
feat/honest-green

Conversation

@addisdev

Copy link
Copy Markdown
Owner

LOOP-PLAN Phase 1, all four items.

Summary

1. skipped is a job status. collector/src/skip.ts holds the one rule for how a job closes: ok: false on the final row is failed; a final row whose error begins skipped: is skipped; every device row skipped (none failed) under a plain ok: true host row is skipped; anything else is done. It is the same rule the v1 summary already used, so status and verdict cannot disagree.

  • /results closes with it and writes the reason into last_error (for failures too, from the failing device when the host row has none).
  • db.ts: a rebuild widens the status CHECK (copying whatever columns exist, by name) and, inside the same transaction, re-labels historical done jobs with a "skipped: result row. Runs once per database; result rows untouched; idempotent.
  • depends_on: a waiter on a skipped job skips with dependency X skipped, cascading; a failure outranks a skip; enqueue behind an already-skipped job is skipped on arrival, and validate warns.
  • Fan-out family status is skipped when no child ran (all skipped, or skipped + cancelled).
  • /api/jobs and /api/v1/jobs status filters, schemas (gen:api regenerated), facets, /metrics, Overview skipped_24h, dashboard grey chip + filter option + "skipped 24h" stat, digest batch wait, and a jobs line in the morning digest. CLI/MCP verdicts are unchanged (they read the summary, which already said skipped).

2. schedule-skipping alert. One alert per enabled schedule whose last FLEET_ALERT_SKIP_STREAK (3) finished runs all skipped, e.g. nightly-tvloop-hardware has skipped 3 runs in a row: no Roku developer password on this host. Runs are found by job id (<schedule>-<YYYYMMDDHHMM>, fan-out children grouped) so pre-provenance jobs count; in-flight runs are ignored; resolves on the first non-skip.

3. Every failure says why. Diagnosis of the live case: job nightly-fleet-ui-smoke-202610150230's artifact 12de2ced… was uploaded and is in the store (in_store: true, 479-byte JUnit, served fine at /artifacts/<sha>). /api/artifacts/<sha> 404'd because no such GET route existed (only DELETE and /pin live there); the brain's catch-all answered "no such endpoint". Fix: GET /api/artifacts/:sha256 now 308s to /api/v1/artifacts/:sha256/content. The missing reason was the executor never reading the report: it contained Assertion is false: "Fleet Runner" is visible.

  • junit.ts parses failing testcases; ui-test (Maestro) puts smoke: Assertion is false: … in error and test.failing_flow (added to result.schema.json). No report: Maestro's exit code + last stderr line, never the command line (it carries the test password).
  • result-reason.ts, wired into fleet-client.postResult so every host workload gets it: a device row with ok:false and no error gets its test counts; a final host row with none gets the failing devices' reasons. The dispatch catch reports uncaught exec failures as exit code + last stderr line instead of Node's message.
  • web-shots already says "N% of pixels differ" on its device rows; kept, and its final host row now inherits them.

4. Registry hygiene. (a) describeRoku and the fallback send platform: "roku"; os ~ 'roku' and tvloop (matches on the executor's Target.platform) unaffected. (b) A serial-named roku-* registration with descriptor.ip deletes roku-ip-<ip> (not when locked/claimed). (c) The lease sweep deletes rows whose descriptor declares kind or soc simulator and that have been offline > FLEET_SIMULATOR_TTL_DAYS (7); not real devices, not Android emulators, not the looser isSimulator() heuristic (it matches x86_64), not locked rows. POST /jobs/sweep returns forgotten.

Docs: concepts.md (new "Skipped is not done", chains), api.md, dashboard.md, deploy/alerts.md, deploy/index.md, protocol.md (platform list, skipped: convention, simulator TTL), lab.md, explore/findings.md (digest line), workloads/host.md (ui-test reasons), the lifecycle figure (re-rendered with npm run assets -- --only lifecycle), CHANGELOG [Unreleased].

Not verified

  • Nothing ran against the live brain or real hardware; read-only GETs only, for the diagnosis. The migration/backfill is tested on a hand-built 0.8-shaped database, not a copy of the brain's.
  • The Maestro "no report" path is unit-tested through execFailureReason, not with a real failing Maestro.
  • The 0.7.2 brain will keep showing skips as done until Phase 2 upgrades it.
  • Decision taken without asking: the morning digest still sends only when there are findings (the jobs line rides along); Android emulators are not forgotten. Easy to change if you want otherwise.

Test evidence (on ultra)

collector CI=true npm test:

ALL PASS

(1215 ok lines; new: skip: ×10, chain: skip ×5, fail: ×1, artifact: ×3, roku: registry ×3, sweep: ×1, skipped: migration ×11, stale skip: ×9, junit: ×9, exec: ×5, reasons: ×7, hygiene: ×10, Roku platform ×3, digest jobs ×3)

fleet CI=true npm test (with runner-machine deps installed so nothing skips):

ℹ tests 80
ℹ pass 80
ℹ fail 0
ℹ skipped 0

mkdocs build --strict: exit 0.

🤖 Generated with Claude Code

addisdev and others added 4 commits October 15, 2026 21:13
…ry hygiene

LOOP-PLAN Phase 1, all four items.

- The queue closes a job that ran nothing as `skipped` (one rule, skip.ts,
  shared with a one-time backfill that re-labels historical `done` skips in
  the migration that widens the status constraint). Status filters, facets,
  /metrics, the Overview, the dashboard chip and the morning digest know it.
  Waiters on a skipped dependency skip with "dependency X skipped".
- A `schedule-skipping` alert after FLEET_ALERT_SKIP_STREAK (3) skipped runs.
- ui-test reads the failing flow and step out of Maestro's JUnit report;
  every host row with ok:false gets a reason (fleet-client wrapper); a failed
  job's last_error carries it. GET /api/artifacts/:sha now 308s to the v1
  content route instead of "no such endpoint".
- Rokus register as platform roku; a serial-named Roku retires its IP-named
  duplicate; the sweep forgets declared simulators offline past
  FLEET_SIMULATOR_TTL_DAYS (7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	CHANGELOG.md
#	collector/src/api/v1/routes.ts
…hecks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@addisdev

Copy link
Copy Markdown
Owner Author

Self-review: skipped status + idempotent relabel (verified on a copy of the live brain's database: 28 historical skips relabelled, one mixed job correctly left done), stale-skip alert, failure reasons from Maestro JUnit, Roku platform and registry hygiene. Merge conflicts with #92/#93 resolved (one duplicate test import fixed). All checks green. Merging (LOOP-PLAN Phase 1).

@addisdev
addisdev merged commit 0457d21 into develop Oct 16, 2026
19 checks passed
@addisdev addisdev mentioned this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant