Skip to content

Run tasks as Kubernetes pods when there is no Agent Substrate - #10

Merged
h0ffmann merged 1 commit into
mainfrom
claude/rax-pod-runner
Oct 31, 2026
Merged

h0ffmann merged 1 commit into
mainfrom
claude/rax-pod-runner

Conversation

@h0ffmann

Copy link
Copy Markdown
Collaborator

What

rax-server --runner=pod runs tasks as plain Kubernetes pods instead of actors on Agent Substrate. It is for clusters that cannot host Substrate, such as piabrota/nix-os's single-node KubeSolo VPS (1 vCPU, 4 GB, no gVisor), where rax-server already runs with its store in PGX but cannot run a task.

It is RAX's own mode, not AX's: no sandbox, snapshot, workspace setup or ssh. Each resume runs the task once:

Task Pod
created, or suspended no pod running; a running one is deleted
resumed a new pod; the task's older pods are deleted first
Running, Resuming the pod is Running, Pending
Suspended the pod exited 0, or there is none
Failed the pod exited non-zero or cannot start (its image, a root user); Ready says why
deleted its pods are deleted
  • crates/rax-server/src/pods.rs: PodRunner, which implements the existing Reconciler and Actors traits, and KubeClient, a few JSON calls over the TLS dialer rax-substrate already has (in-cluster token and CA, the token re-read on every request). No kube-rs: the build gains no new crates.
  • Pods get the task's spec.env, AX_TASK_NAME, AX_TASK_ATESPACE, AX_TASK_YAML, limits (requests equal) from spec.resources.limits or the server's defaults, restartPolicy: Never, no privilege escalation, no capabilities, RuntimeDefault seccomp and runAsNonRoot.
  • Secrets reach tasks only through --pod-env-from-secret, chosen by the server; a task's env cannot name one.
  • A quota refusal (exceeded quota), a conflict, 429, 5xx or an unreachable API leaves the task Pending, to retry.
  • Actors::absent_actor (default None): a task with no pod reads as Suspended under the pod runner, not Failed; Substrate keeps its behaviour.
  • New flags: --runner (RAX_RUNNER), --pod-namespace, --pod-env-from-secret, --pod-service-account, --pod-cpu-limit, --pod-memory-limit. The default stays substrate.
  • README: "Running without Substrate: the pod runner".

Tests

  • Unit tests with a fake Kubernetes API (axum, pods in memory): the pod manifest, pod phases to actor states (including ImagePullBackOff and exit codes), which errors are transient, a task's lifecycle across create, resume, finish, resume again and suspend, listing and deleting by label, and an unreachable API leaving the task Pending.
  • End to end against k3s v1.31 in Docker: rax-server with --runner=pod --store=memory://, the in-cluster token of a service account with a Role on pods in ax-tasks, and the rax CLI:
    • a task that echoes its environment finished Suspended, and its log showed AX_TASK_NAME, AX_TASK_ATESPACE and a key from --pod-env-from-secret;
    • exit 3 → Failed, "The task's pod exited with code 3 (Error)";
    • an image running as root → Failed (runAsNonRoot);
    • resuming the finished task replaced its pod; deleting tasks deleted theirs.
  • cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test --all pass.

What comes next

In piabrota/nix-os: a namespace ax-tasks with a quota and RBAC, rax-server with --runner=pod, and two agents run as tasks (a failed-deploy diagnosis and a daily health digest).

rax-server --runner=pod implements the server's Reconciler and Actors over
plain pods in one namespace, for clusters that cannot host Substrate (a small
single node, no gVisor). Each resume runs the task once in a new pod; a pod
that exits 0 is the task suspended again, a non-zero exit or a pod that cannot
start fails it with the reason. Secrets reach tasks only through
--pod-env-from-secret, chosen by the server.

The Kubernetes client is a few JSON calls over the TLS dialer rax-substrate
already has, not kube-rs. Actors gains absent_actor, so a task with no pod
reads as suspended rather than failed; Substrate keeps the old behaviour.
@h0ffmann
h0ffmann merged commit 458aae3 into main Oct 31, 2026
3 checks passed
h0ffmann added a commit to piabrota/nix-os that referenced this pull request Oct 31, 2026
…m the box (#36)

k8s/ax-tasks holds the task namespace, a quota (two 768Mi agents) and a Role
for rax-server's service account on pods there. rax-server moves to the pod
runner (piabrota/rax#10, image sha-458aae3). piabrota.kubesolo.secrets turns a
root-only env file on the box into a Kubernetes Secret, re-applied when the file
changes; the agents' credentials go in ax-tasks/agents that way.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant