Repository navigation
[scanner] [shared-ci] test (agent */5) failing across hivecommons/hive — fork-PR shards on ubuntu-latest (now ubuntu-26.04 / tmux 3.6a) fail tmux restart-readiness tests #14406
Description
Shared CI incident
TestPollTmuxOutputForAgent_WiresThrashBreaker (src/crux/agent/thrash_poller_wiring_test.go:110) fails 3/3 runs (including gotestsum re-runs) on three unrelated open PRs, while it passes on the v5 default branch.
| PR | shard it landed in | diff touching src/crux/agent/ |
|---|---|---|
| #14383 | test (agent 3/5) |
adds bob_launch_shell_test.go only |
| #14391 | test (agent 4/5) |
adds ci_poll_log_read_test.go + manager_kick.go |
| #14403 | test (agent 2/5) |
adds ci_poll_wall_clock_test.go + manager_kick.go |
Evidence:
thrash_poller_wiring_test.gois byte-identical (blob51f300be) onv5, on the PRs' merge-bases and on every PR head; nothing undersrc/crux/agent/landed onv5since the merge-bases (compare/20fd2412...v5).test (agent 1-5/5)are green on the last threev5commits (d5f9b4b5,83dc25fc,cf2c8d55) and on sibling PRs 🐛 fix: backfill spoke startup probes during self-upgrade #14395, 🐛 fix: classify transient PR review fetch failures #14396, 🐛 fix(stage): write stage/040 when the claim annotation runs twice in a cycle #14398, 🐛 fix(holdguard): one branch update per head and base; stop and say why #14401, 🐛 fix(fleetreport): exclude expected commit fetch 403 warnings #14402 that do not add a_test.gotosrc/crux/agent.- The failure moves shard every time a PR adds a test file to the package (3/5 → 4/5 → 2/5), and once in that shard it is deterministic within the run.
- Failure text:
blocked-line burst through the poller never tripped the thrash breaker: pollTmuxOutputForAgent is not wired to checkBlockedThrashafter the full 20s deadline — the wiring is intact onv5, so the assertion is a false negative driven by shard composition (the test drives a real tmux sessionhive-thrashwireand expects its injected⛔lines to be new relative to the poller's first capture; a co-sharded test that shares the tmux server or shifts timing can starve it). - 🐛 fix: exclude CI log diagnosis from polling nudges #14391 additionally shows
TestSendKick_RestartsCrashedCLIThenDelivers3/3 red intest (agent 2/5), same shard-composition shape.
Why one issue, not three PR repairs
Same test, same message, three PRs whose diffs do not touch the poller or the breaker; baseline green. This is a test-isolation / shard-composition defect in the test suite, not a defect in any of the three PRs. The three fork PRs are deferred against this issue rather than retried.
Ask
- Make
TestPollTmuxOutputForAgent_WiresThrashBreakerindependent of shard composition (own tmux socket via-L, or a unique session name and a guard that no other test in the package reuses the server), or mark ittestutil.SkipfUnlessRequiredbehind the integration gate like its siblings. - Until then, treat its red as
needs-signal: the check has failed ≥3 times with no code cause.
Risk: none (test-only).
— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
Activity
hivecommons-hive commentedon Oct 26, 2026
🔒 Claimed by @thatguywob (contributor on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 30m0s without activity and renews while the work continues. Reply
/unclaimto release, or/claimto take it over if you outrank the holder.hivecommons-hive commentedon Oct 26, 2026
🔓 Claim by @thatguywob released (lease revoked).
[-][scanner] [shared-ci] test (agent */5) failing across hivecommons/hive — TestPollTmuxOutputForAgent_WiresThrashBreaker is shard-composition dependent[/-][+][scanner] [shared-ci] test (agent */5) failing across hivecommons/hive — fork-PR shards on ubuntu-latest (now ubuntu-26.04 / tmux 3.6a) fail tmux restart-readiness tests[/+]hivecommons-hive commentedon Oct 26, 2026
Root cause corrected: runner image, not shard composition
The "shard-composition dependent" reading above was wrong. The discriminator between red and green
test (agent */5)jobs is which runner the job landed on:test (agent 3/5)(job 121753920450)GitHub Actions/ubuntu-latestImage: ubuntu-26.04test (agent 1,2,4,5/5)ubuntu-latestubuntu-26.04test (agent 1-5/5)hive-runners-lke-qh55w-runner-*test (agent 1-5/5)hive-runners-lke-*v2-tests.ymlsends fork PRs to the'ubuntu-latest'fallback and everything else tovars.HIVE_RUNNER_LABELS(hive-runners-lke). GitHub rolledubuntu-latestfrom ubuntu-24.04 (tmux 3.4) to ubuntu-26.04 (tmux 3.6a-2ubuntu0.2) on 2026-10-26 (first observed ~18:00Z). Fork PRs are the only hive runs that hit it, which is why the three affected PRs are all fork PRs and every same-repo sibling andv5push is green — regardless of shard.Independent confirmation: hivecommons/crux (the
src/cruxmirror, GitHub-hosted only) went red onmainat the same hour withagent/unchanged between the green and red syncs, failingTestSendKick_RestartsCrashedCLIThenDelivers,TestRestartThenSendKick_CleanSlateThenDelivers,TestRestart_CancelsPendingAsyncKick,TestStartDrainsDeferredKickWithoutBootstrapPrompt— the tmux launch/restart readiness path (hivecommons/crux#8).What to do
v2-tests.ymlruns-onexpressions from'ubuntu-latest'to'ubuntu-24.04'(workflow edit — needs a human; the App token cannot push.github/workflows/*). crux carries the same pin in its own CI.waitForCLIReadyForAgent→capture-panedepends on so the pin can be lifted.The
needs-signalask stands, reworded: the missing signal is the runner image in the failure annotation.🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:unknown— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
hivecommons-hive commentedon Oct 26, 2026
Second symptom on the same image, logged here so the incident covers it: fork PR #14365
test (rest 4/5)(job 121798360071,ubuntu-latest= ubuntu-26.04) failsTestHiveOpenPRScript_MultilineBodyIsValidJSONWithoutPython3withbin/hive-open-pr.sh: line 59: dirname: command not found, while the identical test passes on thev5push for a67f21d (job 121813067197,hive-runners-lke). The PR does not touchbin/,src/pkg/github/or the test. Whatever lets the stub-PATH test resolvedirnameon 24.04/self-hosted does not hold on the 26.04 image; the durable fix is to adddirnameto the test's stub tool list (it is a real dependency of the script'sSCRIPT_DIRline) — PR-sized, filed for the next cycle alongside the agent-package investigation.🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:unknown— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
hivecommons-hive commentedon Oct 26, 2026
A merged PR referenced this issue without closing it. Carrying the remainder back here so the issue timeline shows what is still open.
Remainder from the PR body:
(second symptom only: the agent-package tmux failures on ubuntu-26.04 remain open on the issue)
This comment is edited in place by the task-list sweep on every cycle; it is not duplicated.
hivecommons-hive commentedon Oct 26, 2026
Fixed by #14429 (merged 0c0218b). Closing. Reply /reopen if the problem persists.
hivecommons-hive commentedon Oct 26, 2026
/reopen
The problem persists for the first symptom. #14429 fixed only the
dirnamehalf (TestHiveOpenPRScript_MultilineBodyIsValidJSONWithoutPython3), butTestPollTmuxOutputForAgent_WiresThrashBreakerstill fails 3/3 runs on every fork PR whose shard picks it up onubuntu-latest, whilev5stays green:The fix for this half is #14415 (isolates the wiring test on its own tmux socket), which is green and sitting at
stage/090-ready-for-human-merge— the merge relay refuses it under the contributor-lane tier policy, so it needs a human merge. Once #14415 lands these PRs only need a base sync. The same three tests (TestRestart_CancelsPendingAsyncKick,TestRestartThenSendKick_CleanSlateThenDelivers,TestPollTmuxOutputForAgent_WiresThrashBreaker) also fail in hivecommons/crux#6go test ./agent (ubuntu-latest), so the crux mirror needs the #14415 + #14431 test changes synced forward too.Keeping this open as the single incident until #14415 merges.
🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:unknown— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
kubestellar-prow commentedon Oct 26, 2026
@hivecommons-hive[bot]: Reopened this issue.
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
github-actions commentedon Oct 26, 2026
Reopened at the request of @hivecommons-hive[bot], who reported this issue.If this was closed by a merged PR, first check whether the fix has reached you: it only takes effect after it ships in a tagged release and your deployment updates to it. If your running version already contains the fix and the problem persists, please add whatever you are still seeing (running version, logs, steps) so it is not closed the same way again. See #6762 for why this command exists.
🐛 fix(agent): isolate thrash poller wiring test (#14415)
github-actions commentedon Oct 27, 2026
@hivecommons-hive[bot] this issue was closed by @kubestellar-prow[bot] as completed.
If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.
If you are still seeing the problem, comment
/reopenand it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.