Skip to content

RFC: K3s backend for persistent T3 hosting and isolated agent workers #45

Open
Open
RFC: K3s backend for persistent T3 hosting and isolated agent workers#45

Description

@kvnloo
kvnloo
opened on Oct 28, 2026

Goal

Make self-hosted K3s a practical execution backend for T3 Code, using T3's existing remote-environment and delegation architecture. One persistent T3 control environment should be able to launch, monitor, resume, cancel, and clean up isolated agent environments on a local K3s cluster, with web/desktop/mobile clients staying usable.

Scope / updated authorization (October 28, 2026): User explicitly authorized exactly one upstream design issue, pingdotgg/t3code#26535, which maintainers converted to an Ideas discussion on 2026-10-28; the original issue remains a read-only provenance link. All continuing engineering and coordination remain downstream in kvnloo/t3code; no additional upstream issues, PRs, comments, default-branch writes or merges without a new human authorization. Preserve contributor credit. This is a capability proposal, not a claim of native K3s support.

Problem / distinctions

There are two independent questions to validate:

  1. Host T3 on K3s: headless server, stable public/private endpoint, persistence, pairing/auth, upgrades and restart recovery.
  2. Use K3s for worker execution: provision a separate sandbox/T3 environment per approved task, route a remote agent invocation to it, retain task state, and safely reclaim resources.

Packaging a container does not automatically provide scheduling; T3 remote delegation does not automatically create or manage pods.

Relevant upstream work (recheck exact heads/status before coding)

Proposed minimal architecture

Web / Desktop / Mobile
         |
persistent T3 owner (threads, approvals, durable orchestration)
         |
optional external K3s environment broker (idempotent request/lease)
         |
K3s API --> per-task isolated T3 server + workspace/PVC + agent runtime
         |
connected environment / scoped remote delegation
         |
result, logs, status, artifacts --> owning T3 thread
  • Owner authority stays singular. No distributed SQLite, shared writable checkout, or cross-pod takeover of an owning thread.
  • External adapter first. Implement a narrow K3s broker/sidecar and declarative manifests before changing core T3 contracts/UI. Avoid a second scheduler and unnecessary hosted-cloud dependencies.
  • Task identity is durable. Stable idempotency key, exact repo/commit/branch, pod/environment ID, TTL, cancellation/restart states and recovered receipts. Never treat pod completion alone as an accepted agent result.
  • Security by default. Namespace/service-account/RBAC least privilege; pod security, network policies and CPU/RAM/GPU limits; scoped Git/provider credentials; no public ingress by default; explicit pairing/approval. Do not log secrets.
  • Data lifecycle. Separate owner state from per-task workspace/PVC. Define stop/resume/archive/delete explicitly. Cancellation and idle cleanup must not delete user work without consent.
  • Performance. No extra work when disabled, bounded polling/WS updates, report cold-start/provisioning latency and resource overhead. Local-only K3s must work without cloud GPUs.

Acceptance / staged proof

  • Inventory: check current upstream main and relevant feature branches; document existing remote/server, container, pairing and authentication capabilities, blockers and non-goals with links.
  • P0 — hosting: on a real single-node local K3s cluster, deploy a persistent headless T3 server with a stable reachable endpoint and storage; pair a client; restart/upgrade the pod and verify thread/state survives. Provide manifests and rollback instructions. Record exact image SHA/tag, K3s version and host.
  • P1 — one worker: from the owning T3 environment, submit an approved test task against a pinned repo revision; provision exactly one isolated K3s worker; connect/execute; deliver its actual result to the owner; show status from at least web or mobile.
  • Failure tests: duplicate request, lost launch response, pod eviction/restart, owner restart, unreachable worker, cancel, expired lease and credential revocation each have defined recovery and no duplicate accepted completion.
  • Isolation: demonstrate resource caps, scoped auth/network access, no secret leakage in logs, and no shared mutable owner DB/workspace.
  • Evidence: capture exact-head checks, focused tests, runtime transcripts, observed provisioning/cold-start and cleanup times, remaining unsupported modes and unresolved upstream dependencies.

Execution / return

  1. Start with read-only architecture and overlap audit; classify any dependency blocked by unmerged/rebased upstream work.
  2. Make the smallest independently testable fork-only implementation slice, ideally out-of-tree K3s adapter/manifests before core UI changes.
  3. Report tests and artifacts on this issue, with PASS / FAIL / NOT RUN, tested commit and exact K3s environment. Keep any eventual fork PR separate and in draft; do not merge or post upstream.
  4. Add only proven, qualified implementation to the existing T3Code promotion queue #42. A READY classification is not upstream authorization.

Expansion: PAIR-informed inference and private multi-host mesh (2026-10-28)

Human objective: expand this program with prior NVIDIA Personal AI Router (PAIR), heterogeneous/local GPU affinity and eventually scale a bounded T3/K3s worker fabric across Tailscale or other authenticated private VPNs. This is deferred roadmap scope; it does not relax the P0/P1 proof gates or authorize new upstream writes, live deployment, host changes or competing task schedulers. Keep Draft PR #46 strictly P0.

Ownership boundaries and minimal interfaces

T3 owner (task state, approvals, delegation, result verification)
       |
optional K3s environment broker (idempotent provision/lease/delete)
       |
private address + attested T3 environment identity + user-approved /mcp peer link
       |
T3 worker environments on approved hosts or independent K3s clusters
       | (optional; inference only)
PAIR-compatible inference gateway -> eligible model-serving host
       |
verified owner result + bounded telemetry -> existing evaluation/policy systems
  • T3 remains the only agent/task authority; T3's saved client connection, peer-link OAuth grant and runtime provisioning lease are three distinct identities/authorizations.
  • K3s/Kubernetes schedules pods; do not stretch one K3s control plane across intermittent WAN hosts by default. Begin with separate clusters and expose only approved workloads/endpoints across the private network. No Karmada/Nomad/new scheduler in the baseline.
  • PAIR routes independent inference requests to one compatible serving node (model inventory, available GPU/RAM, workload, observed affinity); it is not a K3s worker scheduler and does not pool 12 GB + 8 GB into 20 GB or shard a live model across hosts. It must be optional: preserve the existing provider endpoint when disabled.
  • Network is a replaceable transport: a private HTTPS endpoint, stable T3 environment ID, verified peer identity, approval and authorization boundaries. Tailscale Kubernetes Operator ingress/egress is the first supported candidate. Later independently assess Headscale + Tailscale clients, NetBird, and configured WireGuard/other private VPN routes; do not assume Tailscale Operator OAuth/Ingress automatically works with those replacements. Manual DNS/endpoint discovery is acceptable; multicast is not assumed. No public Funnel or unrestricted cluster API.
  • Isolation and learning remain separate: no provider secrets in model/worker telemetry; the overlay network alone does not authorize a T3 peer or PAIR node; no automatic scaling/placement changes without deterministic budget/security limits and human opt-in.

P2 — multi-host/private-network expansion (after verified P0 and P1)

  • Test two explicitly approved independent environments over LAN then tailnet, then two independent K3s clusters via Tailscale Operator cross-cluster Service ingress/egress (stable private FQDN, HTTPS, no exposed kube API). Record packet route (direct vs relay), DNS behavior, RTT, throughput, handshake/reconnect and negative ACL proof. Do not mistake connected for authorized.
  • Define a minimal VPN-neutral PrivateEnvironmentEndpoint record: environmentId, authenticated httpsOrigin, identity verification, transport label, tested reachability status, last probe and lease reference. Keep all credentials out of record, Git and logs.
  • Demonstrate remote owner->worker link/delegation with parent/worker restarts, duplicate request, route churn, approval revocation, offline/disconnected cluster, cancellation, exact result/receipt identity and no duplicate accepted completion. No automatic enrollment of transient workers in client auto-balance.
  • Compare private options only on real supported setups (Tailscale first; Headscale/NetBird/WireGuard after matching gateway/identity capabilities are verified). Leave unsupported transport NOT RUN, not implicitly compatible.

P3 — PAIR-informed inference gateway (separate from task placement)

  • Inventory kvnloo/Personal-AI-Router and the official NVIDIA/Personal-AI-Router, read PAIR's current runtime/engine and actual gateway/protocol capabilities, and find a compatible T3-controlled harness endpoint (e.g. explicitly configured OpenAI-compatible model gateway if supported). Never assume all T3 harnesses accept a PAIR endpoint.
  • Explicitly account for PAIR's existing cross-VPN discovery limitations: #85, #91, #92; PAIR uses mDNS on LAN and must have a verified unicast/manual-node or fixed implementation for VPN-only peers. Origin PR #10 proposes address-based overlay support but was still open in this review. No claim it shipped.
  • Reuse the already researched node/engine/model eligibility, warm-model affinity, GPU class/VRAM headroom, cold-start penalty, current queue and tailnet RTT; route each independent request to a single eligible node, never pool VRAM or interrupt/reschedule a streaming response.
  • Verify which backend actually served a request. Previous downstream PAIR PR #2 proposes authoritative X-PAIR-Served-By after failover but is closed/unmerged; do not rely on this header without confirming upstream behavior. Collect verified scheduler/engine receipts or mark attribution unproven.
  • Compare explicit local-only baseline against two-node eligible routing using p50/p95 TTFT, tokens/sec, p95 tail latency, model load/cold start, tokens/J or watt-hours when available, GPU memory/power, request failures, and network/relay overhead. Record tested model, engine, node, network and exact versions. Compare aggregate throughput and utilization, not claim faster single-request inference.

P3b — optional KAI GPU fractioning (gated, not part of P0/P1)

Decision: HOLD / read-only compatibility research until a measured need exists. GPU fractioning is a node-level enforcement adapter for KAI-assigned, same-physical-GPU fractions, not a T3 placement system, PAIR model router, cross-host memory pool, or new scheduler. Keep existing CPU-only P0 and single-worker P1 manifests unchanged; do not install the operator, touch the NVIDIA driver/container runtime, evict active GPU tenants, or create GPU pods on a working host without a separate explicit owner-approved rollback/maintenance plan.

  • Ownership and contract: T3 owns the user-approved task/peer link and verified result; z0intelligence validates semantic eligibility and policies; Kerdoios may offer an opt-in bounded capacity ranking; Kubernetes/Kueue owns resource admission; KAI Scheduler chooses the GPU and fractional memory allocation; gpu-fractioning manages node NRI/fractiond and MPS/mpsd memory enforcement plus metricsd per-pod observation; PAIR routes independent inference calls only. Existing z0.capacity.snapshot.v1 z0 #131, #133, #134 is the optional read-only capacity bridge; unknown is not zero and a capacity offer is not runtime authorization.
  • Compatibility gates (verify live, do not assume): current source at dbf2d4e requires Kubernetes >=1.28; containerd >=2.0 with NRI enabled (or supported CRI-O); NVIDIA GPU Operator >=26.7.0, toolkit >=1.20.1, device plugin >=0.20.1, and NVIDIA driver r615+ / CUDA 13.4. GPU Operator's NRI mode changes RuntimeClass and requires management CDI namespace allowlisting. Check actual K3s-bundled containerd/CRI/NRI, driver, toolkit, device-plugin versions and GPU support on each node before even drafting an install command. The working RTX 3080 Ti 12 GB and RTX 2080 Super 8 GB do not form one 20 GB VRAM pool, and do not have MIG; a fractional share is within an individual compatible physical GPU.
  • Separate two generations: existing GPU memory fractions with time-slicing or sm-sharing are distinct from planned guaranteed compute fraction requests/ceilings. The latter is KAI #2191, #2192, gpu-fractioning #157 and #163, Alpha / driver r620+ / CUDA 13.5, not assumed production-ready or usable on r615 alone.
  • Known live upstream risks to review before any real fractioning canary: #165 documents stale container/metrics mapping after reconnect; PR #166 has a focused unmerged patch. #167 documents GpuFractioningConfig Helm hook upgrade/uninstall lifecycle pitfalls. Treat "pod Ready" and nonzero metric output as insufficient proof of actual memory enforcement. Verify driver/MPS failures, hard memory cap/OOM, container restart, device placement, metrics cleanup, driver upgrade/drain and uninstall/rollback on an isolated disposable GPU test host before promotion.
  • Experiment path: (A) read-only version/physical GPU probe and frozen baseline; (B) ordinary exclusive whole-GPU K3s Job on one eligible node, with normal provider throughput/performance and no user-impact regression; (C) on a separate approved isolated GPU test node, two fixed, safe, low-memory test pods on the same GPU with KAI-issued fraction annotations, driver/NRI limits, instrumented OOM negative control and independent metrics; (D) only if worthwhile, compare whole-GPU vs time-slicing vs sm-sharing on verified useful work per GPU-second, prefill/decode p50/p95, model memory, watts, crash/error rate, and tailnet cost. Guard all writes and changes behind human approval. No new CRD or second cluster scheduler.
  • Attention gate: actively research prereqs and upstream bug resolution now, but no GPU fractioning PR or live deployment as part of this T3 lane until P0 live storage/route acceptance, P1 one isolated worker, and exclusive GPU baseline each pass. Keep source-specific issues/PRs in their existing repos; consolidate cross-project status in this issue and keep the upstream T3 Ideas discussion high-level.

P4 — optional verified adaptive placement (not in T3 core)

  • Reuse the existing Evolution Lab -> z0evals -> z0intelligence -> Kerdoios -> Kueue/KAI/KEDA governance from prior ABAB research. Learn only bounded legal actions (route, warm/cold policy, queue/replicas/placement hints) after deterministic eligibility/security and quota checks. Preserve Kueue admission, KAI GPU placement and KEDA elasticity; never add a second privileged learning scheduler to T3.
  • Require offline replay -> shadow -> limited canary -> separately authorized promotion, exact experimental lineage, independent acceptance and fail-safe rollback. Do not train weights or automate new infrastructure during the P0 pilot.

References — consolidated evidence and ownership index (2026-10-28)

This section is the single cross-project reference index for this T3/K3s proposal. The linked upstream idea is for architecture discussion, not a second execution queue. Keep detailed implementation/measurements in their original owning repositories. Source links denote the current published design or reported defect, not confirmation that the related cluster has been tested.

1. T3 execution and discussion

  • T3 Code K3s work order, this issue #45 — canonical intent, gates, and dependency/attention ledger; Draft PR #46 — CPU-only, persistent T3 P0 deployment and restart-identity acceptance, LIVE K3s / Tailscale NOT RUN (last inspected head 8da283b04954a02b57c1b0234e27032731fa7fb2; re-read before next worker action).
  • T3 upstream issue #26535 — converted by maintainer to an Ideas discussion on October 28; original issue is locked, direct discussion URL not supplied by the issue REST view. No upstream PR approval; treat converted thread as a concise design question, not another progress-report destination.
  • T3 remote architecture + remote access — stable environment identity, private routes, peer-link OAuth and delegated tasks; #9895 — existing connected-machine load balancing; #16719 — remote delegation; #17051 — record-first launch recovery still needs independently validated durability; #15350 — separate same-host Docker sandbox implementation, avoid overlap.

2. Existing z0 / fleet planning (not a second scheduler)

  • z0intelligence #131 — canonical federated capacity plane: typed CapabilityDemand -> CapacityOffer -> PlacementLease; semantic eligibility in z0, capacity economics in Kerdoios, actual execution/admission in existing runtimes. Fleet migration research comment explicitly rejects fleet-wide GPU/cluster migration without host safety and runtime evidence (its historic fleet report is owner-supplied, not independently re-probed).
  • z0 #133 / #134 — read-only z0.capacity.snapshot.v1 hardware, Tern and Kerdoios inputs, stable identities and sticky/nonmigratable active sessions; Kerdoios #50 — hardware-aware local SLM resource placement; Kerdoios #51 — quota/offer economics.
  • .files #23 — actual explicit K3s host/cluster onboarding, CPU-first and GPU-fractioning deferred, no automatic host/driver edits; OSS Factory #127 / Draft #128 — existing bounded K8s Job renderer and offline safety tests, live kind/K3s not independently proven. Reuse fixtures, not scheduling authority.

3. GPU fractioning, KAI and release risks

4. Private-network and inference separation

  • Tailscale Kubernetes Operator — first private-cluster ingress/egress candidate; service mirroring for future independent K3s clusters. Headscale and NetBird/WireGuard are alternative transport research, not claimed drop-in Tailscale Operator compatibility.
  • NVIDIA Personal AI Router / downstream fork — optional model-serving request routing, not GPU resource scheduler; PAIR #85, #91, #92 and PR #10 track non-multicast VPN node discovery. Fork PR #2 proposed actual serving-node attribution but was closed unmerged, so prove receipts independently.
  • Kueue, KAI, KEDA remain admission / GPU placement / elastic scaling owners. Existing z0/Evolution Lab/Tokenomics/z0evals provide only policy research and verified performance evidence; no autonomous routing/driver/platform changes.

5. Attention / next-action contract

ACTIVE NOW: validate current draft #46 on a disposable K3s host: exact-head CI, immutable image, PVC/environment identity across real Pod restart, authenticated private HTTPS/ACL, client pairing and negative controls (NOT RUN until executed). Then isolate/provision one P1 worker and prove existing T3 peer-delegation lifecycle. Next prove exclusive GPU scheduling only when necessary and explicitly approved. WAIT: Tailscale multi-cluster, PAIR engine routing, GPU fractioning, r620 compute fractions, and ABAB autonomous advice are future lanes with independent acceptance; they must not preempt the live P0/P1 blocker or manufacture separate work queues. Do not open duplicate upstream/fork issues or PRs for design-only references. Promotion queue #42 receives verified exact-head outcomes only.

Activity

  1. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Current-head architecture triage — LIMITED (source/issue review; K3s runtime NOT RUN)

    New origin design request: pingdotgg/t3code#26535, user-authorized on October 28. This approval covers the one upstream RFC issue only, not recurring origin comments or upstream PRs. Current upstream main audited at 5211acfe95a0c6d82a60e2524b5e46b47fe67a3e.

    Decision: adapter outside T3, provision-then-link

    1. What already exists on current main: persistent environment IDs and ordered LAN/tailnet/relay/SSH routes in remote.md; #9895 merged opt-in CPU/memory-informed load balance of existing connected environments. PeerLinks, PeerLinkRequests and RemoteDelegation now exist on main; Linked environments remains beta/off by default and requires explicit authorization on both sides. t3 environment link uses a peer's pairing code and selectable access.
    2. What K3s must supply: create/reuse a containerized T3 environment, stable PVC-backed ID and workspace/repo clone, Service/private endpoint, readiness, lease/cancel/reconcile and deferred deletion. The source owner remains authoritative for task state. A client-saved environment is NOT itself a server-to-server PeerLinks grant.
    3. Best seam: independent opt-in K3s broker/CLI/sidecar that returns a normal T3 endpoint, plus existing user-approved peer-link/delegation. Only after a running proof and human design approval consider an optional narrow provisioning hook at the environment lifecycle boundary; do not add a parallel T3 scheduler or alter RemoteDelegation to create Pods.
    4. Private route: Tailscale Kubernetes Operator/Ingress can present a stable private HTTPS DNS name. Avoid plain 100.64/10 security assumptions: bug #26450 and open PR #26475 document a tailnet-linking mismatch. Prefer HTTPS, require peer environment ID verification, never pass credentials to an unverified endpoint. Per-worker T3 Connect links may also run into machine-account limits.
    5. Do not overlap #15350: open 60-file Docker sandbox PR isolates same-host worktrees with host path mounts and a Docker stdio/port relay. Its maintainer review reports outstanding mobile offline rollback, removal races, remote port reachability, and macOS native proof. K3s environments are separate networked servers with their own project/FS; they're a different lifecycle. Do not base the first K3s prototype on this unmerged branch or take over its files.
    6. Specific failure boundaries: #17051 remote launch record-before-send; #11849 container identity; #13582 public URL vs pod IP. A pod going Ready or a remote delegate_task returning accepted does not prove exactly-once effect or durable recovery. Assign one persisted, scoped provision ID; reconcile by identity and verify receipts on owner.

    Prior-contribution replay: what improved acceptance

    The linked account's recent T3 upstream PR inventory returns 48 PRs, 5 merged and 1 still open as of this triage. Positive examples: #20299, #20736, #14139, #14128, #14140: small fixes, adapted to current main, executable regressions / CI and reviewer feedback. Failure/hold patterns are concrete:

    Earlier K3s work: Factory #127 / draft #128 offers a bounded Kubernetes Job manifest renderer, red/green offline contract tests (22 PASS after its repair) and CI evidence. No real kind or K3s run was verified; reuse the security/manifest fixture and avoid creating a competing scheduler.

    Next staged downstream implementation

    • P0: one persistent headless T3 environment on a single-node local K3s cluster, PVC-backed environment-id, private HTTPS route, client pairing and restart survival; pinned image and measured first-connection latency.
    • P1: opt-in broker provisions a distinct isolated worker with idempotent lease; manual peer link and one task using existing delegate_task; result recorded/verified by originating environment.
    • P2: owner/worker restart, lost launch response, duplicate requests, OAuth/peer revocation, cancel/retention/cleanup, no leaked credentials or filesystem cross-over.
    • P3: only if justified and explicitly approved, a small T3 integration seam for broker discovery/status; do not add new UX on all surfaces speculatively.

    Recurring worker: newly created enabled hourly automation T3 K3s Downstream Build (ID 6ae216c385c88191bf2581238548dee2, around minute :10, America/Chicago). It reads the current fork/origin state and contributes only to fork-owned branches/issues/PRs, with exact-head CI and source checks. No local K3s, device, image build, end-to-end agent execution or performance benchmark was run in this triage. Status: LIMITED / architecture established; implementation and runtime proof pending.

  2. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    P0 code now published (LOCAL_ONLY / LIMITED). Draft fork PR #46, branch infra/k3s-p0-owner-20261028, exact head 0e70b0ddd7221fce5cd494a03ec1200c28b784d7 against fork main. Six new out-of-tree files: checksum-verified T3 v0.1.3 nonroot image recipe; restricted single-Pod namespace/SA/quota/PVC/Deployment/Service; private Tailscale HTTPS Ingress + ingress NetworkPolicy; strict digest renderer; offline checks/CI workflow; human-only deployment, pairing, restart, retention and rollback instructions. No T3 core edits, cluster deployment or upstream writes.

    PASS (observed, exact-head local): checked-in Python contract suite 5/5, plus 26 independent assertions on resource/security bounds, PVC identity, private ingress and malformed digest input; rendered YAML parse, Python compile and shell checksum-filename fixture PASS. All four exact executed source files matched GitHub blob SHAs: k3s.yaml b975ecb9b1ca65918381ca7b3b5c9aada9bb091d, render.py 79674ec421202de2e47fd4302211fe849f1697b2, test_contract.py 9bf71dbf0d2eadb58aedea191abd3e4aa69a17eb, Dockerfile 3552af7432aaa8fa74a68488db77a406be087dd4. Python 3, PyYAML 6.0.3. NOT RUN: hosted CI (no exact-head run returned yet); Docker image build; kubectl validation; K3s/pod/storage; tailnet ACL/HTTPS; actual T3 pairing, restart persistence, provider/delegation or latency. TCP readiness is not application auth, and actual NetworkPolicy enforcement must be tested.

    Maintainer scope: origin #26535 was closed and converted to an Ideas discussion Oct 28, with no issue comment/implementation approval; read-only now. Version split remains: upstream main 5211acfe95a0c6d82a60e2524b5e46b47fe67a3e vs CLI release v0.1.3. Origin #17051 remote launch durability and #26475 Tailscale linking remain unproven. Reused safety lessons from independent Factory #127/draft #128, not its job scheduler or unverified cluster proof.

    Next: get exact-head CI for #46 or fix workflow gate; have a person opt into a disposable K3s P0 run and record digest/K3s version, stable persisted T3 environment ID, private HTTPS client pairing and after-restart data/negative ACL checks. Only then P1 idempotent isolated worker broker and manual OAuth peer link. No promotion authorization or claim of live K3s success.

  3. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    P0 identity-restart acceptance improved; LOCAL_ONLY / LIMITED. Continued existing Draft PR #46, branch infra/k3s-p0-owner-20261028, exact head 87b91feaed9db5cf81620eebfbf11335fe592bc4 (fork main target). Added read-only examples/k3s-p0/acceptance.py and 4 fake-kubectl regression tests; README includes private baseline-before/after-human-restart instructions. Probe verifies digest-pinned Ready single server, bound PVC UID, private Tailscale Ingress + ClusterIP, and persistent hashed T3 environment ID/hostname/PVC UID. It neither deploys nor logs raw identity.

    PASS: 4/4 new offline mocked acceptance tests on source matching fetched GitHub blob SHA. Existing 5/5 contract tests + 26 security assertions passed previously at 0e70b0d, not rerun at this head. NOT RUN: hosted CI (0 exact-head Actions runs/statuses observed), Docker image, actual kubectl/K3s/PVC, Tailscale HTTPS/ACL, pairing/client, threads/provider/restart/delegation. No real K3s execution or performance assertion.

    Current upstream main@125b61954dbba5822a8108c8ca47a4846b4b4328 (one new unrelated PR-sync change); linked-work blockers remain #17051 and #26475. Upstream design #26535 closed/converted to Ideas; no approval or upstream writes.

    Next blocker: exact-head CI then human isolated/disposable-cluster image + restart/client/private-route negative acceptance; do not proceed to claiming P1 delegation until this evidence exists. Existing live installation untouched.

  4. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    P0 restart proof corrected (LOCAL_ONLY / LIMITED). Continued existing Draft #46, exact fork head 8da283b04954a02b57c1b0234e27032731fa7fb2, base fork main. Old probe could report success after a second read of the same Pod without a restart. New v2 baseline collects selected Running/Ready Pod UID; the post-restart check requires a different UID, same hashed environment ID/PVC UID/private hostname and same digest-pinned image. Rejects terminating, duplicate or unready Pods; named-Pod read avoids targeting a different replica. Documentation requires a freshly captured v2 baseline. No T3 core/deployment changed.

    PASS (executed offline at matching blob SHAs): Python 3.13.5, 7/7 focused fake-kubectl tests and compilation. acceptance.py blob e216836316a4c7bba28fcbaeea31e848fdc138fc; test_acceptance.py blob c81f62958f450a60b593623d70caf0b779fabf11. Prior five manifest contract tests were not rerun at this exact head. NOT RUN: hosted Actions (0 exact-head runs/statuses), Docker image, real kubectl/K3s, pod restart/client pairing, PVC/thread persistence, Tailscale HTTPS+negative ACL, provider or remote delegation.

    Current origin: main@30eb7e7bb1ef103fdd6f54081b6a4acf66b3ee4c checked; #15350, #26475 still open. New relevant #26344 is OPEN: remote delegated task approvals/questions are not surfaced on the owner parent today. #17051 record-before-launch remains open. No origin approval from closed #26535.

    Next gate: exact-head hosted CI (if enabled), then human-opted disposable K3s image/PVC/restart/private-route+client proof. Do not claim live P0 or P1 success; no change to existing infrastructure and no origin writes.

  5. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Roadmap expansion: NVIDIA PAIR + private multi-host mesh — DESIGN ONLY

    Human scope update (Oct 28): added an explicit expansion section in the issue body for (P2) independent K3s hosts/clusters over Tailscale or separately validated Headscale/NetBird/WireGuard routes, (P3) NVIDIA PAIR-informed inference routing, and (P4) bounded ABAB shadow/canary policy from existing z0 intelligence/evaluation systems. Existing T3 task ownership, manual peer authorization, K3s pod scheduling and no-cloud-GPU requirements remain unchanged. The existing hourly T3 K3s Downstream Build automation was updated, not duplicated, to see these dependencies after P0/P1.

    Verified design blockers/overlap: upstream #26535 was converted to an Ideas discussion (verified GitHub timeline) and grants no PR approval. NVIDIA PAIR's #85/#91/#92 document mDNS/manual-node limitations over VPN; PR #10 is open, not established as shipped. Our prior PAIR draft #2 is closed unmerged: do not assume X-PAIR-Served-By attribution exists. Tailscale documents multi-cluster service mirroring. PAIR routes one independent inference request to one eligible node; it neither pools VRAM nor allocates T3 agent tasks.

    Active implementation unchanged: fork Draft #46 remains open at head 8da283b04954a02b57c1b0234e27032731fa7fb2 (last read here); P0 live cluster/storage/Tailscale/client verification still NOT RUN this turn. No code edits, PRs, tests, new tasks, upstream comments, or production infrastructure changes during this roadmap update. Keep P0 proof and P1 one-worker broker first; do not promote design-only roadmap work to queue #42.

  6. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    T3K3S-GPUFRAC-REFS-20261028 — Design/reference consolidation; implementation NOT RUN.

    Added P3b GPU fractioning (HOLD) and a grouped References section to the existing #45 issue body; no second queue or competing worker. Reused prior z0intelligence #131, #133, #134, host plan .files #23, Factory #127/#128, and existing GPU Fractioning README.

    Decision: T3 task control → K3s provisioning → optional, later KAI GPU-fraction placement → GPU Fractioning NRI/MPS enforcement → independent evidence; PAIR remains inference-only. Current prerequisites include driver r615+; new compute fractions KAI #2191/GPU #157 call for r620+ and remain Alpha. Upstream #165/PR #166 stale mappings and #167 Helm lifecycle remain review gates. Neither the current two GPUs' VRAM nor remote hosts become one memory device.

    Attention: focus only on measured P0 persistent server / P1 single-worker proof first; postpone multi-host/VPN, PAIR and GPU fractioning changes. Existing hourly automation updated in place, no new task. No cluster, GPU, driver, infra, default branch or upstream changes; NO live GPU/K3s acceptance in this update. Upstream #26535 is converted to a Discussion, and the GitHub connector used here has no Discussions write action; do not misreport this downstream reference update as an upstream discussion post.

  7. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    P0 acceptance hardened — LOCAL_ONLY / LIMITED. Continued existing Draft #46, branch infra/k3s-p0-owner-20261028, exact head 4cbc64517a1da36d86ff666cc6d53a4b4d44e090. No competing PR, upstream edits, deployments, or new infrastructure.

    Change: the read-only restart probe now rejects a Ready but wrong-image, wrong-PVC, API-token-mounted or hostPath-mounted Pod, and rejects a ClusterIP Service or Tailscale Ingress misrouted to another workload/port/public host. Two added offline tests exercise 11 fail-closed mutations. Old Pod UID + identity checks remain.

    PASS: independently reconstructed source matches the GitHub blob SHA for every executed file (new acceptance/test files + unchanged template, renderer, Dockerfile and contract tests). Python 3.13.5 / PyYAML 6.0.3 local unittest discover: 14/14 PASS; compilation PASS. Tests execute mock kubectl, not a cluster. NOT RUN: exact-head hosted Actions (0 runs/checks observed), image build, real K3s PVC/Pod restart, private HTTPS/tailnet ACL/NetworkPolicy negative checks, T3 client pairing/thread persistence, OAuth peer link or delegated result.

    Live blocker: human-approved disposable cluster and private-route/client proof; do not advance P1 proof or treat this as upstream-ready. Relevant origin remains open #15350, #26475, #26344, plus new read-only overlap #26567, #26551 and #26556. GPU Fractioning, PAIR and multi-cluster remain HOLD. Canonical promotion queue #42 not promoted.

  8. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    T3K3S-P0-ENDPOINT-SAFETY-005 — P0 false-positive routing proof hardened; LOCAL_ONLY / LIMITED.

    • Continued existing Draft #46 on infra/k3s-p0-owner-20261028, exact head 8514ad87c0addaee297d080f37d91d71645fce99. Three changed files: acceptance.py, fake-kubectl tests and README. No new PR, default-branch/upstream writes, cluster changes or worker scheduler.
    • New gate: fail closed on stale/unobserved Deployment generation or incomplete updated/available replicas; verify the live Service EndpointSlice targets the same Ready Pod UID/IP and T3 port 3773 (including dual stack). A Ready Pod plus Service selector/Ingress is no longer sufficient evidence of its live backend. Added two integration regression methods (8 endpoint and 4 deployment failure mutations).
    • PASS, scoped: an independent local Python run exercised the exact new endpoint-verifier function, 2/2 tests, 10 adverse fixtures and a dual-stack positive fixture; compile PASS. Published helper and all three changed GitHub source files were re-read and match the commit. NOT RUN at this head: full 16-test Python suite and hosted Actions (0 workflow/check runs observed); Docker build, real K3s/EndpointSlice data plane, tailnet HTTPS/ACL negative tests, T3 client pairing, PVC/thread persistence after restart, OAuth links and delegation. Earlier 14/14 offline PASS is prior head 4cbc645, not silently carried forward.
    • Next blocker: execute full exact-head suite/CI, then separately authorized disposable K3s+Tailscale P0 identity/client/data/restart acceptance. Origin #17051 and #26475 still gate P1. GPU fractioning, PAIR and multi-host remain HOLD/design. Nothing qualifies for promotion queue T3Code — single canonical promotion queue (#42) #42.

  9. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    T3K3S-P0-PRIVATE-ROUTES-006 — P0 offline route guard advanced; LOCAL_ONLY / LIMITED.

    Continued existing Draft #46 on infra/k3s-p0-owner-20261028, exact head 8b2d42989918b417b71cfd16338921a82a490499 (parent 8514ad87), still targeting fork main. Changed three out-of-tree files only (acceptance.py, test_acceptance.py, README). No K3s/host/provider install, origin write, default merge, or new PR.

    • New fail-closed checks: require only one status-advertised, exact t3-k3s-p0.<tailnet>.ts.net Ingress route (reject arbitrary *.ts.net, extra addresses, IP); list all namespace NetworkPolicies and reject absent, extra/additive or widened owner ingress allowances. This is API-state evidence only, not proof that the CNI enforces NetworkPolicy or that Tailscale ACL/TLS works.
    • PASS: GitHub commit/readback shows only three changed paths, exact PR head 8b2d42989918b417b71cfd16338921a82a490499. Locally reproduced new helper logic passed 4 focused unit tests, including 8 ingress + 7 policy negative mutation checks; standalone py_compile passed. This is not a full run of the repo's source test suite.
    • NOT RUN: full 18-test (11 earlier acceptance + 5 contract + 2 new) in-repo suite and hosted workflow (0 observed runs/checks on exact head); Docker/image build, disposable K3s, CNI negative path, HTTPS/TLS, tailnet ACL, client pairing, thread/data survival after Pod UID change, and delegate_task.
    • Disposition: HOLD for promotion (T3Code — single canonical promotion queue (#42) #42 unchanged). Next: execute the complete source-tree offline suite/enable CI, then human-approved disposable-cluster P0 restart/identity/data and negative-network acceptance before starting P1. P2/PAIR/KAI GPU fractioning remain parked.

  10. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    T3K3S-P0-HTTPS-DESCRIPTOR-007 — P0 private-route acceptance now checks the real HTTPS descriptor (LOCAL_ONLY / LIMITED).

    • Continued existing Draft PR #46 only, head 6cb505417dff863af900bb0661fdc8aa891f5c05 targeting fork main. Changed 3 out-of-tree paths: new descriptor_probe.py, test_descriptor_probe.py, and README. No new branch/PR, origin write, default merge, hardware/cluster/network edit or credentials.
    • What changed: an authorized client can now send a credential-free, certificate/hostname-verified HTTPS GET to T3's public /.well-known/t3/environment endpoint, comparing the returned identity only as a SHA-256 hash against the private acceptance.py v2 Kubernetes baseline. Fail closed on wrong tailnet origin, wrong ID, TLS/connect failure, redirect/non-200, unexpected content type, malformed/oversized JSON, and nonprivate/symlink/stale baseline. No proxy, TLS bypass, OAuth token or scheduler added. Upstream released v0.1.3 contract independently confirms this endpoint.
    • PASS — actual offline execution: Python 3.13.5, new focused 7/7 tests and compilation. Independently reconstructed new source/test files match current GitHub blob SHAs 067ae2956b1e0c72ef519dd9c9af86e2a39f222c and fe105d2cc0c5e39ae8eb2864c7c1a7471431ad3c. These are fake TLS responses, not a live tailnet test.
    • NOT RUN: older 18-test suite on new head, hosted CI (0 exact-head check/workflow runs), real image/K3s/EndpointSlice/network policy, DNS/TLS/ACL, paired web/mobile client, Pod UID restart with thread/data preservation, provider and OAuth delegate_task. Current origin main 9826dbb; related #17051, #26475, #26344, #26665 remain separate review gates. Origin converted Ideas #26535 is unchanged/read-only.

    Next human-proof gate: on a deliberately approved disposable K3s cluster, run existing v2 baseline before/after one human Pod restart, and the documented descriptor probe from the approved tailnet client both times. Independently verify negative ACLs, client auth, saved threads/data. P1/PAIR/multi-host/KAI GPU fractioning stay HOLD. Promotion queue #42 remains NOT READY.

  11. kvnloo commented on Oct 28, 2026

    @kvnloo
    kvnloo
    on Oct 28, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    T3K3S-P0-PREFLIGHT-008 — material code progress, LOCAL_ONLY / LIMITED. Continued existing Draft PR #46 (no competing P0 PR), exact head 8a65a319e315726896693c17464c92872b9ca8ca into fork main. Three out-of-tree files only: examples/k3s-p0/preflight.py, test_preflight.py, README.

    New operator safety gate: explicit-context, GET-only Kubernetes preflight before human P0 apply; fail closed if an existing t3-k3s-p0 namespace could be overwritten, if not exactly one Ready/untainted node or one named default StorageClass, if Kubernetes <1.28, or if the Tailscale namespace/IngressClass controller is absent or wrong. Query errors suppress stderr; no secrets, live apply, infrastructure or provider change. Reference: Tailscale ingress operator, controller tailscale.com/ts-ingress (source).

    PASS (real offline Python execution, exact Git blob SHA): 4/4 new mock-kubectl tests + compilation, including 9 invalid/mismatched cluster fixture groups and malformed-query cases; local git hash-object matches fetched upstream fork blobs: preflight f362d80927d9f80646f10500e35fddb581e9a995, tests 1efc74e96531da9294c0014cf010757c3b4b3cef. README readback verified. NOT RUN at this head: full combined P0 suite, GitHub Actions (0 runs observed on exact head), Docker image/K3s commands, Tailscale ingress TLS/negative ACL/dataplane, client pairing, thread/PVC identity across real restart, P1 OAuth or delegated result. A passing preflight does not prove the cluster is disposable, owner-approved or actually network-isolated.

    Origin drift review: main@e34067edef2eba03c1e40bca7d2b2d933ae12b85; #15350, #26475 still open. New linked-task repair candidates #26665 (correct environment for remote approval navigation) and #26704 (delegated child PR-watch result delivery) are open/unmerged, not T3 P0 proof. #17051 remains a delegated-task durability concern.

    Next: owner-approved disposable-cluster apply, private client/data/restart/denied-route evidence. Keep P1/P2/PAIR/GPU fractions HOLD, do not mark promotion READY. No upstream/default-branch/cluster writes.

  12. kvnloo commented on Oct 29, 2026

    @kvnloo
    kvnloo
    on Oct 29, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    T3K3S-P1-PLAN-009 — real downstream code, LOCAL_ONLY / P1 HOLD for physical proof. Created stacked Draft #47 from P0 Draft #46 head, not fork main; branch infra/k3s-p1-worker-plan-20261028, exact head 97e42b7ba439e2277aa5d68993e2d0eb372966f1, base P0 8a65a31. Exactly 4 added out-of-tree files; no P0/core/default-branch/other-repo edits or competing P1 PR.

    Implemented: inert deterministic hashed-key worker plan for 8 bounded K8s objects (restricted namespace, SA without token, 1-Pod quota, distinct PVC, Recreate server Deployment, ClusterIP, Tailscale ingress + ingress-only policy), private 0600 create-once local receipt, stable repeat output, changed same-key image/expiry conflict rejection. No Kubernetes writes, no real cluster lease, no expiry controller/auto-delete, no pairing/peer consent or task scheduler. P0 live acceptance remains first; original v0.1.3 image peer beta compatibility is unproven.

    PASS (executed offline): 5/5 Python unit tests, compilation; CLI process exercised identical-byte YAML retry (8 YAML docs), conflict expiry rejection. Exact GitHub blob SHA matched local executed files: plan 8af66bd933be39c5ceae202639f5ae48e2585295, tests fd407acfd99b4ab9ab13e61908fbd97e0e07c817. PR compare P0→P1 verified exactly 4 added paths. NOT RUN: hosted Actions (0 observed for new head), image build, K3s+PVC, HTTPS/ACL/CNI/remote client, paired owner/restart/data persistence, P1 provision/TTL/peer-delegation/failure injection. No claim of runtime success.

    Origin watch: main b287102; sandbox #15350, Tailscale peer #26475, question relay #26344, child result #26704 and workflows #26673 remain open, not P0/P1 proof. #17051 still open. Existing Ideas #26535 unchanged/read-only.

    Next: real separately approved disposable K3s P0 identity+data+private-client restart acceptance; only then opt-in P1 lease reconciliation and one actual verified remote delegated result. PAIR/multi-cluster/GPU fractions HOLD. Promotion queue #42 unchanged; nothing READY.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions