Skip to content

✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents - #14807

Merged
hivecommons-hive[bot] merged 4 commits into
v5from
feat/job-runtime-terminal
Oct 29, 2026
Merged

hivecommons-hive[bot] merged 4 commits into
v5from
feat/job-runtime-terminal

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #14798

Problem

Agents with sandbox.runtime: job run each kick as a Kubernetes Job in a separate pod (crux/kubejob + crux/agent/sandbox_executor.go), so no tmux session exists in the hive pod. The dashboard ▶ terminal (ttyd via terminal_proxy.go + deploy/ttyd-tmux.sh) therefore shows "waiting for agent … tmux session not created yet" forever. Operators on job-runtime spokes (a1-vllm-spyre deps/quality agents) cannot see what the sandboxed agent is doing.

What changes

When the selected agent's effective sandbox runtime is job, the SPA's ▶ terminal link opens /terminal/job/<agent> instead of the ttyd pane. That page's /stream endpoint:

  • follows the newest Job pod's log (hive.kubestellar.io/agent=<launch-name> selector) while the kick runs, polling while the pod is Pending (e.g. waiting for an accelerator node);
  • prints the container exit code when it ends;
  • between kicks (the launcher deletes a completed Job at once) shows the last run's persisted transcript <workspace>/.hive/sandbox-transcript.log and its exit code, or a clear "no sandbox run yet" state;
  • is read-only; output is redacted with the same redactTokens seam as the tmux pane.

Pieces:

  • crux/kubejob/podlog.go: LatestPod, PodStatus, StreamLog over the launcher's existing plain-HTTP in-cluster client (no client-go added), plus LineFilterWriter.
  • crux/sandbox/naming.go: shared naming (LaunchName, WorkspaceDir, LatestTranscript, TranscriptExitCode) so executor and dashboard agree without pkg/dashboard importing crux/agent. The executor now appends exit code: N to the transcript.
  • pkg/dashboard/job_terminal.go: GET /terminal/job/{name} (viewer) and GET /terminal/job/{name}/stream (text/plain, flushed per line). Routes live under /terminal so the existing handoff code, Path=/terminal assertion cookie and terminal role check apply unchanged. Agent status gains sandboxRuntime; index.html routes the link.
  • Docs: docs/sandbox-isolation.md, docs/api-reference.md (citations script run), changelog fragment added-14798-job-runtime-terminal.md.
  • internal/testutil/dashboard_import_allowlist.txt: adds crux/kubejob (the launcher client this feature deliberately reuses) and crux/sandbox (tiny shared naming package). Log scrubbing goes through the existing redactTokens seam rather than a new crux/logscrub import.

Design choice: Go endpoint (b) over kubectl logs in ttyd-tmux.sh (a)

  • The image has no kubectl; kubejob already talks to the API server with a plain HTTP client, so (b) adds no binary or dependency.
  • ttyd-tmux.sh runs as the unprivileged ttyd user without hive.yaml, so it cannot resolve the effective per-agent runtime, launch name, or transcript path; the dashboard already has all of that.
  • Same auth path as the tmux pane (handoff code + assertion cookie + role) with no new surface for ttyd.
  • Same RBAC (pods/log get) the launcher already needs; no new grants.

Tests

  • go test ./crux/kubejob/ ./crux/sandbox/ ./pkg/dashboard/ ./internal/testutil/ — pass (full pkg/dashboard suite 158s).
  • go test ./crux/agent/ -run Sandbox — pass. Two unrelated TestApplySessionEnv_* tests fail identically on a clean v5 checkout in this environment (inherited env), not touched here.
  • New tests use time.After/gated fakes only — no time.Sleep (sleep ratchet).
  • bash scripts/check-api-reference-citations.sh clean; go vet clean; gofmt clean on touched files.
  • No shell changes (ttyd-tmux.sh untouched).

Operator notes

No new RBAC: deploy/k8s/sandbox-job-rbac.yaml (and the hub's hive-sandbox-jobs) already grant pods/pods/log get+list. A spoke whose ServiceAccount lacks that Role will see a ── cannot list pods … line in the pane pointing at it.

@kubestellar-prow kubestellar-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the DCO. labels Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15206 🐛 Activity Monitor: delegated adopt-supersede reports executing → adopted/could-not-adopt, routes workflow-path to the workflows blocker, re-adoption survives the mutation journal same files src/pkg/dashboard/static/index.html
#15251 🐛 tracker: show Merging in the merge gate for at least the toast lifetime before ✓ CONFLICT src/docs/api-reference.md
#15241 🐛 tracker: keep a closed linked issue as the row head instead of promoting the PR same files src/pkg/dashboard/static/index.html
#15240 🐛 Never adopt a person's PR: owner/human-authored PRs return to their author instead of the contributor-offline lane same files src/pkg/dashboard/server.go
src/pkg/dashboard/static/index.html
#15253 🐛 decision toast: issue decisions no longer read as 'approved for merge' same files src/pkg/dashboard/static/index.html
#15256 🐛 clanker PRs with a human changes-requested review show Returned to contributor, not Sentinel alert same files src/docs/api-reference.md
src/pkg/dashboard/static/index.html
#15208 🐛 fix: decidePR answers capability-gap, not adopt-supersede, for heads the App cannot adopt same files src/pkg/dashboard/static/index.html
#15247 📖 docs: refresh operator decision API reference same files src/docs/api-reference.md
#15223 ✨ Soak countdown on every soak pill, with the live timer on hover same files src/pkg/dashboard/static/index.html
#15249 ✨ dashboard: merged ledger rolls like an odometer and the rising glyph eases slower same files src/pkg/dashboard/static/index.html
#15104 🐛 Activity Monitor: Rebase, then merge as maintainer; fixer owns parked conflicting PRs; dismiss self-inflicted changes_requested same files src/docs/api-reference.md
src/internal/testutil/dashboard_import_allowlist.txt
src/pkg/dashboard/server.go
src/pkg/dashboard/static/index.html
#15219 🐛 fix: preserve human and owner PRs during delegated decisions same files src/internal/testutil/dashboard_import_allowlist.txt
src/pkg/dashboard/static/index.html
#15122 🐛 Write-back containment: per-kind gate for every unsolicited GitHub write, with backfill guard CONFLICT src/docs/api-reference.md
#15134 🐛 Delegated decision on a capability gap: model the blocker, stop the un-park/re-park loop, name the missing App permission CONFLICT src/docs/api-reference.md
#15186 🐛 Activity Monitor: one coherent post-decision state for Let hive decide rows; 060 reviewed in CI / Review same files src/docs/api-reference.md
src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor
068c834

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 068c83448033) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • src/pkg/dashboard/job_terminal_test.go

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
022083f

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 022083f9aa75) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • src/pkg/dashboard/job_terminal_test.go

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive hivecommons-hive Bot added the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
5954ee4

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 5954ee458404) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • src/pkg/dashboard/job_terminal_test.go

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

…time: job agents

Agents with sandbox.runtime: job run each kick as a Kubernetes Job in
another pod, so there is no tmux session in the hive pod and the ▶
terminal (ttyd) waited forever on "tmux session not created yet".

For a job-runtime agent the SPA now routes ▶ terminal to a read-only
viewer at /terminal/job/{name} whose /stream endpoint follows the newest
Job pod's log through the kubejob launcher's existing in-cluster client
and pods/log RBAC, polls while the pod is Pending, prints the container
exit code at the end, and between kicks (completed Jobs are deleted at
once) shows the last persisted sandbox transcript — which now ends with
an "exit code: N" line — or a clear "no sandbox run yet" state.

- crux/kubejob: LatestPod / PodStatus / StreamLog over the plain HTTP
  client, plus LineFilterWriter.
- crux/sandbox: shared naming helpers (LaunchName, WorkspaceDir,
  LatestTranscript, TranscriptExitCode) used by the executor and the
  dashboard so neither hard-codes the other's layout.
- pkg/dashboard: /terminal/job routes under the existing /terminal
  handoff + role check; agent status gains sandboxRuntime.
- RBAC unchanged: sandbox-job-rbac.yaml already grants pods/log.

Fixes #14798

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
… to repo root

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…ge cases

Resolve conflicts with the sandbox-health and session-age status fields,
refresh api-reference citations, and add LatestTranscript tests for the
directory-skip and equal-mtime tie-break branches so crux/sandbox stays
above its 98% coverage floor.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the feat/job-runtime-terminal branch from 5954ee4 to 8495dee Compare October 29, 2026 02:19
@kubestellar-prow kubestellar-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 29, 2026
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
8495dee

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 8495dee31b5d) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • src/pkg/dashboard/job_terminal_test.go

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#148078495dee` sha256:3411c52df32c881781407ffae0ab50e6df24d3a8f148c26a2c0bd87e996711ed — download from the dashboard or `hivectl review evidence #14807`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 8495dee.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves conflicts-with-open-pr Open PR has a merge conflict with another open PR dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard ▶ terminal only shows 'tmux session not created yet' for sandbox.runtime: job agents

1 participant