[hivecommons-hive]

Closes #14558
Part of #14550 (tracker; #14559 backend-specific routes remain)

Targets v6.

What / why

This connects the repeat-failure pieces merged earlier to what operators see on the dashboard: the record collector (#14565), the normalizer (#14566), the detector (#14562) and the neutral BASH_ENV route (#14574). It is monitor-only: nothing here pauses, kicks or interrupts an agent.

• New status field AgentProcess.RepeatFailure (not a watchdog condition, not BusyCondition). It is refreshed at each 30 s collection by StartRepeatFailureMonitor, which starts with the other agent loops. Collected records are scrubbed and normalized, then passed to the detector. A raised note is logged once.
• Row note (C11): "Repeating failure: <command> failed N times in M min (exit S)", or (failed) when there is no exit number. When several commands repeat, the row shows the most recent plus "and K others".
• Check states (C5/C11): on / "repeat-failure check not available for this backend" / "repeat-failure check not receiving data". They appear on the agent card and in both detail-panel paths (repeatFailureLineHtml).
• C12 safeguard: ObserveRepeatFailureTool takes shell / read / edit tool events for the current turn. A covered agent is "not receiving data" only when a shell call finished at least no_data_after_s ago (default 900), no record has arrived since, and no shell call is still running. File reads and edits alone don't count, and a command that is still running doesn't start the timer. This PR adds no caller: each covered backend's signal producer comes with [scanner] repeat-failure: backend-specific routes (copilot, omp, openhands, muse, agy, aider) and real-run evidence #14559.
• Off means off (C14/C-AC12): while repeat_failure.enabled is false, each tick only clears the field and touches no file. The row shows no state and no note.
• Coverage table (C13): pkg/agent/repeat_failure_coverage.go has one row for every config.CLIBackends and config.InferenceBackends entry. Tests fail if a backend is missing, a row lacks a reason or follow-up issue, a covered row lacks committed evidence (incl. --version) or a shell-activity signal, or the docs table differs from the rendered code table.
• Docs (C15): a "Repeated-failure check" table in backend-support-tiers.md (advisor-lane style); the note and the states in dashboard-sections.md (Agents); a cross-reference in operator-reference.md. api-reference.md citations were refreshed with --fix.

Coverage decisions

No backend is marked covered. No committed real-run harness evidence exists in the tree yet, and C4 forbids claiming coverage without it.

• claude, codex, goose, pi, gemini, bob, opencode, kilo: the BASH_ENV route is installed but there is no real-run evidence. Not covered, follow-up [scanner] repeat-failure: backend-specific routes (copilot, omp, openhands, muse, agy, aider) and real-run evidence #14559.
• copilot, omp, openhands, muse, agy, aider: their commands never reach BASH_ENV, and the backend-specific route isn't built. Not covered, follow-up [scanner] repeat-failure: backend-specific routes (copilot, omp, openhands, muse, agy, aider) and real-run evidence #14559.
• vllm, llm-d, litellm, watsonx: they follow claude. Not covered, follow-up [scanner] repeat-failure: backend-specific routes (copilot, omp, openhands, muse, agy, aider) and real-run evidence #14559.

As a result, with the check on today, every row shows "not available for this backend". Notes still appear when records from the neutral route arrive.

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@Danathar

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR	Title	Relation	Files
#14716	🐛 contribute: match disabled_repos entries by exact repo name	same files	src/pkg/dashboard/static/index.html
#14724	🌱 chore: forward-merge v5 -> v6 (top-up through 7a506f7, 43 commits, hand-resolved)	CONFLICT	src/docs/api-reference.md

[hivecommons-hive]

hive/awaiting-ci: CI pending — sweep will merge when green.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

hive/awaiting-ci: pushed b567c27 — moved RepeatFailureStatus types to pkg/crux/types (crux boundary ratchet), kept the detail-panel blockers fast-path statement intact, dropped the now-referenced settings-parity allowlist entry. CI pending — sweep will merge when green.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: b567c2798

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

hive/awaiting-ci: merged v6 (post-#14399 crux-module layout) into this branch at 5b7a0a5 — repeat-failure files relocated to crux/agent, status types into package crux. CI pending — sweep will merge when green.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 386c3792d

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

hive/awaiting-ci: pushed 7f10303 restoring the RepeatFailure field on crux.Process that the v6 merge dropped (every shard failed typecheck on it). CI pending — sweep will merge when green.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 7f10303e6

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

hive/awaiting-ci: pushed the omnigent coverage row and the background-loop count fix; CI pending — sweep will merge when green.

---

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: bcad0c9b5

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

Read the new monitor against the tree at bcad0c9, not just the diff. Looks correct to me.

What I verified:

• No double counting across ticks. CollectRepeatFailures builds a fresh collector each tick, but collectFile renames the record file to a .collecting sidecar and drainCollecting removes it (src/crux/agent/repeat_failure_record.go:398, :433), so each record reaches recordRepeatFailure once.
• "Logged once" holds. RepeatFailureDetector.Record returns a note only when the threshold is first crossed and nil on every later hit for the same command/result key (src/crux/agent/repeat_failure_detect.go:117-131), so the Warn in recordRepeatFailure fires once per raise.
• Row note ordering. Notes sorts newest-first (repeat_failure_detect.go:148-153), matching FormatRepeatFailureNote's "most recent plus K others".
• Lock order. refreshRepeatFailureStatus takes m.mu then mon.mu; ObserveRepeatFailureTool releases m.mu before taking mon.mu; recordRepeatFailure takes only mon.mu. Consistent with the order documented on repeatFailureMonitor; no inversion.
• Off means off. The disabled branch of CollectRepeatFailures clears monitor state and every AgentProcess.RepeatFailure without touching the record paths, and buildAgentsWithHidden emits all three JSON fields as omitempty, so the row renders nothing (repeatFailureLineHtml returns '' on empty check and note).
• ObserveRepeatFailureTool has no caller in this PR, as the body says; repeatFailureCheckState therefore returns on/not_covered only until #14559 lands a producer.

One note for the maintainer, not a defect: repeat_failure.no_data_after_s was dropped from settings_parity_allowlist.txt on the strength of a tooltip string in index.html naming the key; there is still no dashboard control for it. Fine if that is the intended bar for parity.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

Read the new monitor against the tree at merge-base e4aa972 (status/coverage files, Manager wiring, Process snapshot, status_builder.go, families.go, the settings-parity test). The code looks correct to me; two small documentation gaps below, neither blocking.

Verified

• Lock order holds everywhere: repeatFailureRecordPaths takes only m.mu.RLock, recordRepeatFailure only mon.mu, and ObserveRepeatFailureTool / refreshRepeatFailureStatus take m.mu before mon.mu (src/crux/agent/repeat_failure_status.go:218-331). The collector callback runs with no Manager lock held.
• Off-path clears state and touches no file (repeat_failure_status.go:154-165); Process.RepeatFailure is copied in snapshot() and surfaced via status_builder.go:1259-1263 with redactTokens on the note.
• ObserveRepeatFailureTool has no production caller and every coverage row is Covered:false, so no_data is unreachable today and every row reads "not available for this backend". Both are stated in the PR body, so judged as declared scope, not a gap.

Docs-currency (low)

1. The coverage file moved into the crux module in 5b7a0a59d, but three places still point at the old path pkg/agent/repeat_failure_coverage.go: src/docs/backend-support-tiers.md:568, the failure message in src/crux/agent/repeat_failure_coverage_test.go:113, and the PR body. Actual path: src/crux/agent/repeat_failure_coverage.go.
2. src/docs/dashboard-sections.md:628 and the tooltip at src/pkg/dashboard/static/index.html:17586 explain "not available for this backend" as "Hive cannot record this backend's shell commands yet, so repeated failures are not noticed." Per the PR body's last paragraph, that is not what happens for the BASH_ENV backends (claude, codex, goose, pi, gemini, bob, opencode, kilo): records do arrive and a Repeating failure: … note renders on the same row (refreshRepeatFailureStatus computes Note independently of Check, repeat_failure_status.go:328-335). An operator will see a repeating-failure note directly above text saying repeated failures are not noticed. Suggest wording the not-covered text as "not verified for this backend (no committed real-run evidence); notes may still appear from the neutral route".

Suggestions only; I have not approved or requested changes.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

Evidence bundle `#14623bcad0c9` sha256:ed83d70ad98b2006f8aada90ca54d7c4ce08ff56d06c71435ca1504959da4f00 — download from the dashboard or `hivectl review evidence #14623`.

[github-actions]

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback