Repository navigation
✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents - #14807
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
068c834
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
068c834 to
022083f
Compare
022083f
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
|
@clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
5954ee4
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
…time: job agents
Agents with sandbox.runtime: job run each kick as a Kubernetes Job in
another pod, so there is no tmux session in the hive pod and the ▶
terminal (ttyd) waited forever on "tmux session not created yet".
For a job-runtime agent the SPA now routes ▶ terminal to a read-only
viewer at /terminal/job/{name} whose /stream endpoint follows the newest
Job pod's log through the kubejob launcher's existing in-cluster client
and pods/log RBAC, polls while the pod is Pending, prints the container
exit code at the end, and between kicks (completed Jobs are deleted at
once) shows the last persisted sandbox transcript — which now ends with
an "exit code: N" line — or a clear "no sandbox run yet" state.
- crux/kubejob: LatestPod / PodStatus / StreamLog over the plain HTTP
client, plus LineFilterWriter.
- crux/sandbox: shared naming helpers (LaunchName, WorkspaceDir,
LatestTranscript, TranscriptExitCode) used by the executor and the
dashboard so neither hard-codes the other's layout.
- pkg/dashboard: /terminal/job routes under the existing /terminal
handoff + role check; agent status gains sandboxRuntime.
- RBAC unchanged: sandbox-job-rbac.yaml already grants pods/log.
Fixes #14798
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
… to repo root Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…ge cases Resolve conflicts with the sandbox-health and session-age status fields, refresh api-reference citations, and add LatestTranscript tests for the directory-skip and equal-mtime tie-break branches so crux/sandbox stays above its 98% coverage floor. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
5954ee4 to
8495dee
Compare
8495dee
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 8495dee. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #14798
Problem
Agents with
sandbox.runtime: jobrun each kick as a Kubernetes Job in a separate pod (crux/kubejob+crux/agent/sandbox_executor.go), so no tmux session exists in the hive pod. The dashboard ▶ terminal (ttyd viaterminal_proxy.go+deploy/ttyd-tmux.sh) therefore shows "waiting for agent … tmux session not created yet" forever. Operators on job-runtime spokes (a1-vllm-spyre deps/quality agents) cannot see what the sandboxed agent is doing.What changes
When the selected agent's effective sandbox runtime is
job, the SPA's ▶ terminal link opens/terminal/job/<agent>instead of the ttyd pane. That page's/streamendpoint:hive.kubestellar.io/agent=<launch-name>selector) while the kick runs, polling while the pod isPending(e.g. waiting for an accelerator node);<workspace>/.hive/sandbox-transcript.logand its exit code, or a clear "no sandbox run yet" state;redactTokensseam as the tmux pane.Pieces:
crux/kubejob/podlog.go:LatestPod,PodStatus,StreamLogover the launcher's existing plain-HTTP in-cluster client (no client-go added), plusLineFilterWriter.crux/sandbox/naming.go: shared naming (LaunchName,WorkspaceDir,LatestTranscript,TranscriptExitCode) so executor and dashboard agree withoutpkg/dashboardimportingcrux/agent. The executor now appendsexit code: Nto the transcript.pkg/dashboard/job_terminal.go:GET /terminal/job/{name}(viewer) andGET /terminal/job/{name}/stream(text/plain, flushed per line). Routes live under/terminalso the existing handoff code,Path=/terminalassertion cookie andterminalrole check apply unchanged. Agent status gainssandboxRuntime;index.htmlroutes the link.docs/sandbox-isolation.md,docs/api-reference.md(citations script run), changelog fragmentadded-14798-job-runtime-terminal.md.internal/testutil/dashboard_import_allowlist.txt: addscrux/kubejob(the launcher client this feature deliberately reuses) andcrux/sandbox(tiny shared naming package). Log scrubbing goes through the existingredactTokensseam rather than a newcrux/logscrubimport.Design choice: Go endpoint (b) over
kubectl logsin ttyd-tmux.sh (a)kubectl;kubejobalready talks to the API server with a plain HTTP client, so (b) adds no binary or dependency.ttyd-tmux.shruns as the unprivileged ttyd user withouthive.yaml, so it cannot resolve the effective per-agent runtime, launch name, or transcript path; the dashboard already has all of that.pods/log get) the launcher already needs; no new grants.Tests
go test ./crux/kubejob/ ./crux/sandbox/ ./pkg/dashboard/ ./internal/testutil/— pass (fullpkg/dashboardsuite 158s).go test ./crux/agent/ -run Sandbox— pass. Two unrelatedTestApplySessionEnv_*tests fail identically on a cleanv5checkout in this environment (inherited env), not touched here.time.After/gated fakes only — notime.Sleep(sleep ratchet).bash scripts/check-api-reference-citations.shclean;go vetclean;gofmtclean on touched files.Operator notes
No new RBAC:
deploy/k8s/sandbox-job-rbac.yaml(and the hub'shive-sandbox-jobs) already grantpods/pods/logget+list. A spoke whose ServiceAccount lacks that Role will see a── cannot list pods …line in the pane pointing at it.