🌱 [v6] agents: notice an agent repeating the same failing command (every backend, monitor-only) #14550

Target: v6.

Target branch: v6 only. Open the PR with --base v6. hive-open-pr defaults to the repo's default branch (v5) when --base is omitted (bin/hive-open-pr.sh:36-41). The maintainer chose v6 for this set; PRs against v5 are out of scope. The v5→v6 forward-merge is currently failing (#14340), so a v5 fix would not reach v6, and the paths cited below are v6 paths (v5 keeps agent code under src/crux/agent).
Citations: path:line against v6 @ e0333c132.

Show on the dashboard when an agent keeps running the same failing command, for every backend Hive can cover, and mark plainly the ones it can't. Never pause the agent.

Follow-up to the RFC #13895 (now closed), written as a spektacular spec. Target branch: v6.

What this is

An agent sometimes runs the same failing command again and again, for example the same gh or
npm call failing the same way ten times in a row, burning tokens. Hive's existing loop breaker
only catches policy-blocked actions, and the watchdog only catches agents that are stuck. Nothing
catches an agent that is busy but going in circles. This change has Hive record each shell command
an agent runs and whether it failed, across every backend it can cover, and visibly mark the
backends it can't. When the same command keeps failing the same way, the agent's dashboard row
shows a note. It never pauses the agent. It is off by default for the first release.

Requirements

• C1. A command record.
  For each shell command an agent runs on a covered backend, Hive records: agent, backend, time,
  the command, its exit status (or just failed/succeeded where a backend gives no number), and which
  route produced the record.
• C2. A backend-neutral route.
  Backends that run each command in a fresh, non-interactive bash are covered by one Hive-owned
  route that works the same for all of them. Hive makes sure backends that pick their shell from the
  environment get bash. For a backend that would otherwise fall back to /bin/sh (bob today), this
  is an accepted, documented change of shell while the check is on; the coverage table says so.
• C3. Backend-specific routes where the neutral one can't see commands.
  Where a backend's commands never reach the neutral route (a long-lived shell, an embedded shell,
  a sandbox), Hive uses that backend's own record of the command and its result instead: its hook,
  session log or event stream. The routes suggested in the coverage table (Technical Approach) are
  the researched starting point; the implementer may choose another route for a backend as long as
  C1 holds.
• C4. A backend is "covered" only with real-run evidence.
  A backend is marked covered only when its real CLI produced a correct record in a committed,
  re-runnable harness (for example a stub model provider that makes the real CLI run one scripted
  shell command), and the evidence includes the CLI's own --version output. Hand-written test data
  does not count. The version is the one Hive pins; for a backend Hive doesn't pin or install
  (gemini, openhands, aider today), it is the version the harness installs, recorded in the coverage
  table. A backend with no such harness is marked not covered.
• C5. If a backend can't be covered, the gap is visible.
  First try a backend-specific fix (C3). If none works, the backend is marked "not covered": its
  agent rows say "repeat-failure check not available for this backend", the docs list it with the
  reason, and the implementer files a follow-up issue for it (bare, carrying that backend's row of
  the coverage table and what was tried). The PR never claims coverage it does not have.
• C6. Detection.
  When the same command fails with the same result at least N times within a window, the agent gets
  a "repeating failure" note for that command. Defaults: N = 5, window = 10 minutes. Both are
  configurable. One note per distinct command; if several commands are repeating, the row shows the
  most recent and says how many others. After a note clears it can be raised again by new failures.
• C7. "Same command" means the command, not a blur of it.
  Before comparing, Hive strips the wrapper text a backend adds around the command, collapses
  whitespace, and replaces paths under /tmp, /var/tmp or $TMPDIR and timestamps (ISO-8601,
  and 10- or 13-digit Unix times) with placeholders. Everything else is kept: gh pr view 123 and
  gh pr view 456 are different commands.
• C8. Only the agent's own commands.
  Only the command the agent itself asked to run is recorded, once, not every shell started inside
  it (scripts it runs, Hive's own wrappers and hooks). A backend wrapper that hands the command to a
  new bash with exec still produces exactly one record.
• C9. Known waiting loops do not count.
  Commands that fail by design while waiting are excluded by a configurable list. An entry matches
  a command whose normalized form starts with it. Defaults: gh pr checks (exits non-zero while
  checks are pending) and gh run watch.
• C10. Monitor only.
  The note never pauses, kicks or interrupts the agent, and the existing policy-blocked loop breaker
  keeps its current behaviour.
• C11. What the operator sees.
  • The agent's dashboard row shows "Repeating failure: <command> failed N times in M min (exit S)",
    or "(failed)" when the backend gives no exit number. The note is logged once when raised.
  • It clears when that command succeeds, or when its last failure is older than the window.
  • Each agent row shows its check state: on, "not available for this backend", or "not receiving
    data" (C12).
• C12. Runtime safeguard.
  If an agent on a covered backend has a turn in progress that shows shell-tool activity, but no
  command record has arrived for a configurable time (default 15 minutes), its row shows
  "repeat-failure check not receiving data". This catches a backend update that silently broke its
  route. File reads and edits alone don't count as shell-tool activity, and a command still running
  doesn't start the timer (a long build is not missing data). The coverage table names the
  shell-activity signal used for each covered backend; where a backend has none, the table says
  "safeguard unavailable" for it.
• C13. A CI test keeps the coverage list honest.
  A coverage table in code lists every CLI backend Hive supports, plus the inference backends (which
  run through the Claude Code route), as covered (with route and evidence) or not covered (with
  reason and follow-up issue). A test fails if any backend is missing, or if the docs table does not
  match the code table.
• C14. Off by default.
  One config switch turns the whole check on; off by default for the first release. When off, Hive
  adds nothing to the agent's environment or home, installs no route and shows no note or check
  state. Turning it off after it was on removes what Hive added at each agent's next launch, and
  until then anything left behind does nothing.
• C15. Docs.
  The backend support documentation gains a repeated-failure table (backend, route, status, evidence,
  follow-up issue) in the style of its advisor-lane table; the dashboard docs describe the note and
  the check states.

Constraints

• Target v6.
• The route must never change a command's stdout, stderr or exit status, and must not trip callers
  that start bash with -e, -u or -o pipefail or export SHELLOPTS. It leaves nothing visible
  to the command except its exit trap and one namespaced marker variable: no other variables,
  functions, option changes or IFS changes. (The only accepted difference is the documented shell
  change in C2.)
• A missing, unreadable or broken hook file, or an unwritable record file, means no record, nothing
  else. The command runs exactly as it would without the check.
• Per-command overhead from the neutral route must stay under 5 ms (median, measured in a test).
• An environment variable the agent already has for the same purpose must be chained, not
  overwritten.
• Hive must not write routes into a backend's own settings files unless Hive already writes that
  file for that backend (as it does for advisor hooks); the launch environment and Hive-owned files
  come first.
• Hive may add a guarded line to ~/.bashrc only in agent homes Hive itself creates and manages,
  never in a person's own home (contributor local mode). Where it can't, that backend is not covered
  there.
• Raw command records may contain secrets. They live in a per-agent file readable only by that
  agent's user and Hive. Each record is one line appended in a single write (long commands are cut
  to fit). Hive collects the file at least every 30 seconds by renaming it first, so new records
  go to a fresh file and none are lost or torn; it scrubs each record with its log scrubber and then
  deletes the collected raw file. Only scrubbed text is stored, logged or shown.
• No pane-text parsing for this feature.
• The existing policy-blocked loop breaker (5 hits in 60 s, then pause) must stay unchanged.

Acceptance criteria

• C-AC1. Covered backends record failures.
  With the check on, for each backend marked covered, real-run evidence (C4) shows one failing
  command (false or ls /nonexistent) recorded with that command and a failure result.
• C-AC2. Threshold and window.
  The same failing command 5 times within 10 minutes raises the note; 4 times does not; 5 times
  spread over 11 minutes does not.
• C-AC3. Different arguments stay different.
  gh pr view 123 and gh pr view 456 failing alternately, 5 times each within 10 minutes, raise
  two notes (one per command), never one merged note.
• C-AC4. Wrappers removed.
  Claude Code's and Gemini CLI's wrapped forms of ls /nonexistent each compare equal to the bare
  command.
• C-AC5. Only the outer command.
  An agent running bash ./script.sh whose script runs 3 failing inner commands produces one
  record (for bash ./script.sh), not four. A gh call through Hive's gh wrapper produces one
  record. A command run through a wrapper that ends in exec /bin/bash -c '<cmd>' (Codex's form)
  produces exactly one record, for <cmd>.
• C-AC6. Waiting loops.
  gh pr checks and gh pr checks 123 --watch failing 10 times each raise nothing with the default
  exclusion list.
• C-AC7. Never pauses.
  Raising the note never changes the agent's paused state, and the policy-blocked breaker's existing
  tests pass unchanged.
• C-AC8. Clears.
  The note clears after the command succeeds once, and a new run of 5 failures raises it again.
  With the last failure at time t and no success, the note is gone at t + 10 minutes.
• C-AC9. Not covered.
  A backend marked not covered shows "repeat-failure check not available for this backend" on its
  agent row.
• C-AC10. Not receiving data.
  A covered agent with a turn in progress, shell-tool activity and no records for 15 minutes shows
  "not receiving data". An agent that only read and edited files for 15 minutes does not, and
  neither does one whose single command has been running for 20 minutes.
• C-AC11. Coverage test.
  Removing any backend from the code coverage table, or changing a row only in the docs table, fails
  the CI test.
• C-AC12. Off means off.
  With the switch off (the default), the agent's environment and home contain nothing added by this
  feature, and no row shows a check state. The same holds after switching on, launching, switching
  off and relaunching.
• C-AC13. Secrets.
  A command containing a token-shaped string is stored, logged and shown scrubbed, and the raw entry
  is gone after Hive reads it.
• C-AC14. Never changes the command.
  For a set of commands (success, failure, a set -euo pipefail script, bash -euo pipefail -c,
  an exported SHELLOPTS=errexit:nounset, a command that sets its own EXIT trap, a command that
  lists its variables and functions, large stdout and stderr), stdout, stderr and exit status are
  byte-identical with the check on and off, apart from the trap and the one marker variable. This
  also holds when the hook file is missing, the record file is not writable, or the hook itself
  references an unset variable. Backends with the documented shell change (C2) are compared with
  the shell fixed to bash in both runs.
• C-AC14a. Records survive collection.
  1,000 failing commands run while Hive collects the record file repeatedly produce 1,000 complete
  records, none torn; the collected raw file is gone afterwards.
• C-AC15. Overhead.
  A benchmark of 200 true commands shows median added time under 5 ms per command.
• C-AC16. Existing variable chained.
  If the agent's environment already sets the variable the neutral route uses, the original file
  still runs and a record is still written.

Technical approach (direction, not binding)

• Background (premise check against v6 at e0333c132): 📝 RFC: [v6] crewAI-inspired features — learn from review feedback, count repeat lessons, extend the thrash breaker #13895 planned a second matcher on pane
  lines next to the policy-wall breaker (src/pkg/agent/manager_thrash.go). That can't work: nothing
  in v6 turns a pane line into command + exit status, and Claude Code collapses commands in the pane
  ("ran 3 shell commands", src/pkg/agent/manager_kick.go:775). Trajectory review has no evidence
  input (src/pkg/trajectory/lane.go:11) and judges safety, not retries, so that part of the RFC is
  dropped. go test and go vet are already blocked in agent pods (bin/go-wrapper.sh), so the
  RFC's own example can't happen there.
• Neutral route, tested locally on bash 5.3: setting BASH_ENV to a Hive-owned script with an
  EXIT trap records $? and $BASH_EXECUTION_STRING for bash -c and bash -lc; it does not fire
  for sh -c (dash in the node:26-slim agent image) or interactive shells. A missing BASH_ENV
  file is silent. A command that sets its own EXIT trap replaces the hook's trap, so that command
  is not recorded (acceptable; C12 catches systematic loss). Recording only when
  $BASH_EXECUTION_STRING is non-empty, plus a depth marker in the environment, keeps nested shells
  and scripts out (C8). Careful: the outer shell's EXIT trap does not fire across exec, and the
  exec'd bash inherits an exported marker, so a plain "already inside" marker would record nothing
  for Codex. Tie the marker to the shell that set it (for example its PID) or record in the shell
  whose execution string is the backend's command. The BASH_ENV file is sourced with the caller's
  options already on, so write it for set -eu. Export SHELL=/bin/bash so $SHELL-driven
  backends use bash. Set BASH_ENV in the launch environment Hive builds (agentEnvPairs,
  src/pkg/agent/manager_env.go:247; applySessionEnv, src/pkg/agent/manager_tmux.go:93);
  Claude Code rejects BASH_ENV in settings env.
• One record format and one detector; producers are the neutral route plus per-backend readers.
  The sliding window can follow recordBlockedAndCheck (src/pkg/agent/manager_thrash.go:104) with
  its own thresholds and state. Do not reuse fleetreport.NormalizeMessage
  (src/pkg/fleetreport/runtime.go:477): it collapses every number, path and quoted string, which
  would merge gh pr view 123 and 456.
• The per-agent note and check state need a new field on the agent status the dashboard renders:
  watchdog conditions are a fixed set owned and overwritten by the reconciler
  (src/pkg/watchdog/conditions.go), and BusyCondition is one recomputed string
  (src/pkg/agent/manager.go:230).
• Shell-tool activity for C12: Claude Code and Copilot session records mark tool calls; the pane
  tool-marker counter (countToolMarkers, src/pkg/agent/manager_kick.go:797-807) is a weaker
  signal for other backends. The code coverage table (C13) should carry the shell-activity signal
  chosen for each covered backend, or "safeguard unavailable".
• Patterns to follow: Hive already writes per-agent files into agent homes
  (src/pkg/agent/interactive_home.go) and installs hook files at launch for the advisor lane
  (src/pkg/agent/advisor_hook.go, Codex/Copilot/omp/Claude).
• Backend list for C13: config.CLIBackends and config.InferenceBackends
  (src/pkg/crux/config/agent_runtime.go:1254,1294). Docs: src/docs/backend-support-tiers.md
  (advisor-lane table at line ~325 is the model), dashboard docs.
• Smoke tests: .github/workflows/backend-smoke.yml runs real CLIs for two contributor backends
  today; recorded real-run test data is the alternative C4 allows.
• Suggested config block: repeat_failure.enabled (false), threshold (5), window_s (600),
  no_data_after_s (900), exclude (list); follow the config package's conventions.
• Known risks: closed-binary backends (copilot, muse, agy) may change formats between versions;
  C12 and C13 make that visible.

How we'll know it works

• On a hive with the check on, every agent row shows one of the three check states; no agent is
  silently unmonitored.
• Notes appear for real loops and rarely for normal test-fix-retest work: operators don't turn the
  check back off because of noise.

Out of scope

• Feeding the signal to trajectory review.
• Any automatic action on a repeating failure (pausing, kicking, Esc); a later phase can decide.