Skip to content

πŸ› agent: keep sandbox run persistence off the manager lock and bounded - #14864

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
scanner/fix-14853
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
scanner/fix-14853

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Persisting a sandbox run did disk I/O (mkdir, write, rename, append) while holding the manager mutex, so a slow data volume stalled every status and kick call. The write now happens after the lock is released, serialized by its own mutex.

runs.jsonl was append-only and the token collector re-parsed all of it on every scan, so scan cost grew with hive age. It is now compacted to its newest 1000 records once it passes 2 MiB; the format and paths are unchanged.

The lifecycle sentence in sandbox-isolation.md predated the Health section; it now points to it.

Closes #14853, Closes #14854, Closes #14855

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…nded

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14524 ✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity same files src/crux/agent/manager.go
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/crux/agent/manager_sandbox.go
src/docs/sandbox-isolation.md

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read runSandboxKick (src/crux/agent/manager_sandbox.go:203-275), persistSandboxRun/compactSandboxRunHistory (src/crux/agent/sandbox_run.go:201-257) and the only reader of runs.jsonl, pkg/tokens/sandbox_scanner.go:48-80. The change does what the body says; one consequence is worth noting for the operator.

Verified:

  • Defer order is correct: defer m.mu.Unlock() is registered after the persist defer (manager_sandbox.go:210,218), so LIFO runs Unlock first and the disk write happens unlocked. entry is fully built before pending = &entry (:237-238) and nothing after that line mutates run, so the persisted record is identical to what was written before.
  • persistSandboxRun is the only writer of runs.jsonl (grep: sandbox_run.go:220 is the sole OpenFile on SandboxRunHistoryPath), so the new package-level sandboxRunPersistMu fully serializes append+compact. No path takes sandboxRunPersistMu and then m.mu, so no lock-order inversion.
  • ScanSandboxRuns opens and bufio.Scanners the whole file on every scan with no stored offset, so the rename-based compaction cannot leave it mid-line; it just sees fewer records.
  • All three closed issues are addressed: the sandbox-isolation.md:129 cross-link (#14853) targets an existing ## Health and metrics for sandboxed agents heading at :175.

Info (intent-alignment): because ScanSandboxRuns is the source for sandboxed-agent token attribution, compaction turns those totals into a sliding window of the newest 1000 runs β€” once a hive passes ~2 MiB of history, lifetime sandbox token figures on the dashboard will stop growing/decrease. That is implied by bounding the file and seems like the right trade for the scan-cost fix, but it is a visible behaviour change the body does not call out; a sentence in the changelog fragment or sandbox-isolation.md:179 would save an operator a puzzled moment.

Looks correct to me.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit dec61e4 into v5 Oct 28, 2026
79 of 84 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-14853 branch October 28, 2026 06:21
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#1486402e8979` sha256:2576fcaaf3cb6aa5b19155fadea5f92a3e7e659e8d4137f039b4146dad3f88ff β€” download from the dashboard or `hivectl review evidence #14864`.

@hivecommons-hive hivecommons-hive Bot removed the role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

0 participants