Skip to content

Findings reach the code: /api/v1, fleetr and MCP, and fixed findings become nightly regression tests - #96

Merged
addisdev merged 5 commits into
developfrom
feat/findings-loop
Oct 16, 2026
Merged

addisdev merged 5 commits into
developfrom
feat/findings-loop

Conversation

@addisdev

@addisdev addisdev commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

LOOP-PLAN.md Phase 4, items 1–4, plus two route-table leftovers from Phase 3.

What changed

1. Findings in the control API. Three route-table entries (new findings tag), then gen:api:

  • GET /api/v1/findings: list envelope; filters app, state (open/triaged/all), verdict (comma-separated), check, since (ISO 8601 or a span: 24h, 7d, 90m).
  • GET /api/v1/findings/{id}: the finding plus counts, evidence (role, sha256, /api/v1/artifacts/<sha>/content link, in_store), replay_flow (text inline up to 16 KiB), regression, history, links.
  • POST /api/v1/findings/{id}/verdict: real | duplicate | not_a_bug | agent_mistake | fixed | null, optional note and duplicate_of. Scope run (added to RUN_ROUTES), audited by the existing onResponse hook with the finding id as subject. fixed before real is 409 conflict.
  • The legacy /api/findings routes call the same functions (listFindings, findingDetail, parseVerdictBody, setVerdict in src/findings.ts); no logic is forked. The legacy verdict route keeps its admin requirement under tokens.

2. CLI, MCP, skill. fleetr findings, fleetr finding ID [--flow FILE|-], fleetr verdict ID real|duplicate|not-a-bug|agent-mistake|fixed|reopen [--note] [--duplicate-of]. MCP fleet_findings, fleet_finding (flow text inline), fleet_verdict (readOnlyHint false, scope run), each with an output schema. The MCP instructions mention them. SKILL.md has a "Reading a finding" section; the tool table is regenerated.

3. A finding becomes a regression test. Owner's decision implemented as: Real, then Fixed.

  • New tables regressions (keyed by finding id: flow text, kind, state candidate/active/retired, run counters) and finding_verdicts (every verdict, plus regressed events). The findings table is rebuilt once to allow fixed in its CHECK (same pattern as the jobs table for skipped), and each existing verdict gets one backfilled history row.
  • real → the replay flow is copied from the artifact store into regressions as a candidate. fixed → active (a finding marked real before this release goes straight to active). duplicate / not_a_bug / agent_mistake → retired. real on a retired one → candidate again.
  • Issue-close sync is out of scope: src/findings.ts only ever creates GitHub issues, never reads them back (by design, see its header), so fixed is the only trigger.
  • Design for the nightly: ui-test accepts params.regressions ("<app key>", or true for app.name). At run time the executor pulls the app's active flows from a new protocol route GET /findings/regressions and runs each, after the suite's own flow, on devices of the finding's platform; each run is reported to POST /findings/{id}/regression. I chose this over an auto-created nightly-regressions-<app> schedule because a collector-made schedule would have to guess targets, the build, the suite's app_id and sign-in; the existing nightly already has all of those, and with the param set once every newly fixed finding joins without editing anything. suite.flows may be omitted for a regressions-only job (a device with nothing to run is skipped, not passed).
  • Stored flows are adapted: appId → the job's suite.app_id when set; a relative runFlow (the mission's sign-in flow) is resolved under the flows dir, refusing escapes.
  • What it asserts: crash/anr on Android: logcat -b crash + am_anr read before and after the flow; a new crash of the package is reproduced. That fails the row and reopens the finding (verdict cleared, seen_count+1, build added, history line regressed on build X (job, device): ...), once per job however many devices reproduce it; the test stays active. Every other check is "replays without error" only, and a failing flow fails the row without reopening; the docs and each run's detail say this. iOS: no crash buffer read, so crash findings are "replays without error" there too. tvloop flows are stored but skipped by ui-test.

4. Morning brief. fleetr findings --since 24h --json returns the v1 list envelope of Finding objects; the shape and how to read it are in docs/agents.md ("A morning brief").

Leftovers. (a) POST /api/v1/jobs: the 400 response now names no_matching_devices, no_build and the other 400 codes; 422 names no_capable_agent and unresolved_reference, matching queue.ts. (b) The v1 Artifact document has channel: main|pr; the POST /artifacts description names x-artifact-channel.

Dashboard. A Fixed button (f, disabled until Real), a Regression test panel with state, last run and verdict history; the ? shortcut list says r d n a f. After merging #97, the Findings page's list, detail and verdict calls go through the generated v1 client (dash/src/client.ts) and the hand-written finding types are gone from api.ts. To give the page its filter menus, the v1 findings list carries facets (apps, checks, open/triaged counts) like the device and job lists. GET /api/findings, GET /api/findings/{id} and POST /api/findings/{id}/verdict are now in the deprecation table; precision, digest and issues have no v1 counterpart and are not (the id pattern is \d+ so they are not caught). The v1 verdict honours the FLEET_DASH_TOKEN speed bump the older route had. check-dash-requests: 56 requests, none deprecated.

Part of #83, not closing it: the Findings page still reads /api/findings/precision, and the issue's other dashboard-only routes (overview, visual, evals, system, ...) still need the add-to-v1-or-mark-internal decision it asks for.

Docs: explore/findings.md (Fixed, a Regression tests section, the new routes), agents.md, api.md, workloads/host.md (ui-test regressions), dashboard.md. Changelog bullets carry #96. Workload count unchanged (34); check-counts.mjs passes.

Not verified

  • No regression flow has run on a real device or a real nightly. The executor path is covered by src/workloads/regressions.test.ts against fake crash readings and a fake Maestro; Maestro's handling of a rewritten absolute runFlow path and logcat on a real phone are not exercised.
  • The findings-table rebuild is tested on a seeded old-schema database in the suite, not on the lab brain's database.
  • The dashboard changes were built (vite build passes) but not looked at in a browser.

Test evidence (ultra, Node 24)

After merging origin/develop (#95, #97), on ultra:

  • collector: CI=true npm test → typecheck, dashboard build, the dashboard calls no deprecated route (56 requests, none deprecated), workloads ALL PASS (7), counts, openapi.json and the client match the route table, auth ALL PASS, lifecycle ALL PASS, smoke: ALL PASS, api v1 contract ALL PASS. 1377 ok lines, 0 FAIL.
  • fleet: npm test → tests 83, pass 81, fail 0, skipped 2 (the two fleet up end-to-end tests skip on ultra: runner-machine deps not installed there, as before).
  • mkdocs build --strict: exit 0. node scripts/check-counts.mjs: 7 marked counts agree, 34 workloads.

🤖 Generated with Claude Code

addisdev and others added 5 commits October 15, 2026 22:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings become nightly regression tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…essions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… findings list facets; deprecate the older findings routes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@addisdev
addisdev marked this pull request as ready for review October 16, 2026 03:20
@addisdev

Copy link
Copy Markdown
Owner Author

Self-review: findings in /api/v1 sharing the store functions with the legacy routes; CLI/MCP/skill; regressions table with candidate→active→retired and reopen-on-recurrence; Findings page on the v1 client. Findings-table rebuild verified on a copy of the live brain's database (69 findings preserved, v1 list reads them). Checks green. Merging (LOOP-PLAN Phase 4).

@addisdev
addisdev merged commit 4a0f1b4 into develop Oct 16, 2026
24 checks passed
@addisdev addisdev mentioned this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant