Skip to content

[17.0-stable] pkg/kube: port cluster-init.sh to Go daemon - #7058

Open
rucoder wants to merge 71 commits into
lf-edge:17.0-stablefrom
rucoder:rucoder/17.0-kube-init
Open

rucoder wants to merge 71 commits into
lf-edge:17.0-stablefrom
rucoder:rucoder/17.0-kube-init

Conversation

@rucoder

@rucoder rucoder commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Backport kube-init, the Go daemon that replaced cluster-init.sh on master (#5971), to 17.0-stable. 17.0-stable still runs the shell version. Every kube fix on master since #5971 lands in kube-init, so each one has had to be rewritten in shell to reach 17.0: #6584, #6727, #6753, #6832, #6886, #6887 and #6917 all were. 17.0 also misses the kube-init fixes nobody rewrote. With kube-init on 17.0-stable, master's kube fixes cherry-pick as they are.

Backport of #6116, #5971, #6309, #6449, #6456, #6499, #6507, #6542, #6590, #6599, #6614, #6652, #6665, #6702, #6704, #6755, #6771, #6794, #6796, #6860, #6861, #6897, #6934

This is a single PR. The earlier plan (#7055 and a re-vendoring PR first) is dropped together with #6697, which is not being backported: kube-init does not use cnirpc or anything #6697 added.

What it contains, in order:

  1. eve-k: disable apiserver watch cache on multi-node clusters #6116, disable the apiserver watch cache on multi-node clusters. kube-init stages that manifest, so it goes first.
  2. pkg/kube: port cluster-init.sh to Go daemon #5971, the daemon itself, and the master fixes to it that 17.0-stable has no version of: eve-k: add cluster tie-breaker support to evetest, fix pkg/kube kubeclient regression #6309, kube-init: fix eve-k VNC proxy regressions from the bash rewrite #6456, kube-init: raise the k3s start deadlines to 30 minutes #6599, kube-init: address the k8s node by its normalized name when re-applying labels #6704, eve-k: fix app deployment on arm64 #6861, Longhorn fixes: Node CR spec.name on recreate and disk-status readiness gate #6860, kube-init: bound API calls and the Longhorn pod exec with timeouts #6897, kube: expire events after 1h, drop the 72h backlog #6934, plus the dependency bumps build(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 in /pkg/kube/kube-init #6449, build(deps): bump google.golang.org/grpc from 1.83.1 to 1.83.2 in /pkg/kube/kube-init #6499, build(deps): bump github.com/containerd/containerd/v2 from 2.2.5 to 2.2.8 in /pkg/kube/kube-init #6507, build(deps): bump github.com/containerd/containerd/v2 from 2.2.8 to 2.2.9 in /eve-tools/bpftrace-compiler #6652 and build(deps): bump github.com/moby/sys/user from 0.4.0 to 0.4.1 in /pkg/kube/kube-init #6755.
  3. The kube-init half of master PRs whose shell or pillar half is already on 17.0-stable: mgmtproxy: require proxy-auth token, TLS, and a destination policy on the cni0 listener #6542 (on 17.0 as [17.0-stable] mgmtproxy: require proxy-auth token, TLS, and a destination policy on the cni0 listener #6584), evetest: port scans; fix the exposure they found on EVE and EVE-k #6665 ([17.0-stable] evetest: port scans; fix the exposure they found on EVE and EVE-k #6753), kube-init: recycle controller pods after a single→cluster datastore migration #6702 ([17.0-stable] kube-init: recycle controller pods after a single→cluster datastore migration #6917), kube-init: fix the SR-IOV device plugin gate and the system-pod readiness livelock #6590 ([17.0-stable] kube-init: fix the SR-IOV device plugin gate and the system-pod readiness livelock #6886), Fix CDI HTTP DataVolume imports through the mgmtproxy cni0 TLS proxy #6794 ([17.0-stable] Fix CDI HTTP DataVolume imports through the mgmtproxy cni0 TLS proxy #6887), kube: clear the retired pod subnet when a node joins #6614 ([17.0-stable] kube: clear the retired pod subnet when a node joins #6727), dom0-ztools: confine containerd to ctrd_max_vcpus #6771 ([17.0-stable] dom0-ztools: confine containerd to ctrd_max_vcpus #6832). Where the 17.0 version already has a commit's pillar or shell part, the commit here carries only its kube-init files: the mgmtproxy cni0 listener (mgmtproxy: require proxy-auth token, TLS, and a destination policy on the cni0 listener #6542), the cluster status server authentication (evetest: port scans; fix the exposure they found on EVE and EVE-k #6665), holding app volume creation until controllers are recycled (kube-init: recycle controller pods after a single→cluster datastore migration #6702) and the CDI bump (Fix CDI HTTP DataVolume imports through the mgmtproxy cni0 TLS proxy #6794).
  4. The rest of those PRs, so they come over whole:
  5. eve-k: fix VM failover held back by the stats lease and a false portmap overlap #6796's two zedkube commits, added on review so they are not forgotten. [17.0-stable] zedrouter: ignore portmaps on Switch NIs in the overlap check #6797 backported only its zedrouter commit, because these two needed the backup-DNID series (eve-k: let a healthy peer act for a downed node's designated app (backup DNID) #6505), which reached 17.0-stable later in [17.0-stable] eve-k: let a healthy peer act for a downed node's designated app (backup DNID) #6867. They apply cleanly here:
    • the VMIRS replica reset follows the 60s eve-app-op lease instead of the 300s stats lease, so failover is not held back for up to five minutes when the dead node was the stats leader.
    • the failover branch runs only when a pod is terminating.
  6. 17.0-only commits:

Not included:

tools/bump-and-commit.sh finds no package hash to update.

Backported commits

Cherry-picked with -x from master, grouped by master PR:

#6116

  • 27496ff pkg/kube: disable apiserver watch cache on multi-node clusters

#5971

  • 66f1559 pkg/kube/kube-init: vendor k8s.io v0.34.1 and containerd v2.2.5
  • f89ab6b pkg/kube/kube-init/state: markers, atomic writes, reboot
  • a96859d pkg/kube/kube-init/state: /var/lib snapshot and rollback
  • c6521d1 pkg/kube/kube-init/prereqs: kernel modules, cgroup, iSCSI, vault + containerd wait
  • 8803295 pkg/kube/kube-init/prereqs: lend the control plane idle CPUs on first boot
  • 9f3cf92 pkg/kube/kube-init/k3s: config drop-ins, supervisor, readiness, node-password, install
  • 6e6aabe pkg/kube/kube-init/update: k3s and cluster-component upgrade flow
  • c9144fa pkg/kube/kube-init/kubeclient: process-wide client-go bundle with Default()
  • 3745588 pkg/kube/kube-init/kubectlx: typed client-go and containerd convenience layer
  • 9a5dbe0 pkg/kube/kube-init/deploy: Component API, work-queue scheduler, BestEffort retry loop
  • 64570bd pkg/kube/kube-init/pubsubclient: shared pillar pubsub subscription manager
  • d0c1251 pkg/kube/kube-init: pubsub subscriptions and lifecycle publication
  • 335af07 pkg/kube/kube-init/images: image tarball import into containerd
  • 856c364 pkg/kube/kube-init/mgmtproxy: cni0 anchor IP + CDI ImportProxy + containerd env
  • 794a58a pkg/kube/kube-init/clustermode: single↔HA transitions, master leases, startup stagger
  • 3355cc4 pkg/kube/kube-init/monitor: RUNNING-state watchers
  • ee13ef0 pkg/kube/kube-init/tiebreaker: three-node HA tie-breaker configuration
  • e3cef7c pkg/kube/kube-init/components: Multus, KubeVirt, CDI, Longhorn, descheduler, RBAC, SR-IOV, kube-vip
  • 8d45ff2 pkg/kube/kube-init/vnc: VNC proxy with caller-PID watchdog
  • 9abf8cc pkg/kube/kube-init: FSM entry point
  • 933cef7 pkg/kube/kube-init/cmd/k3s-sctl: operator CLI for the control socket
  • 7487742 pkg/kube: Dockerfile entrypoint, k3s-control wrapper, kubevip-delete helper
  • c0cfc4b pkg/kube: retire shell library files replaced by the Go daemon
  • 3b3e9aa pkg/kube: kubevirt-features.yaml declare empty arrays for permittedHostDevices
  • 84b63ae images: pre-load KubeVirt v1.7.3, not v1.6.0
  • a0dd81e kube-init: one declaration per component version
  • 6f72bdd kube-init: migrate the legacy K3sBase marker after /var/lib is mounted
  • 259f9de kube-init: finish an update the k3s step left open
  • ec4c168 kube-init: guard the multus CRD-before-instance precondition
  • e946941 kube-init: fix yetus spelling and comment nits
  • 83b1382 kube-init: fix external-boot-image tarball path
  • a7cdcba Makefile: build kube-init binary + linuxkit package
  • d1b6f47 .spdxignore: skip pkg/kube/kube-init/vendor from SPDX scan
  • 3ef0d4a Makefile: run the kube-init tests in make test

#6309

  • 30cbb56 pkg/kube: rebuild the kubeclient when the k3s kubeconfig changes

#6449

  • 350e8ed build(deps): bump google.golang.org/grpc in /pkg/kube/kube-init

#6456

  • 7fdde36 kube-init: fix eve-k VNC proxy regressions from the bash rewrite

#6499

  • 3b31222 build(deps): bump google.golang.org/grpc in /pkg/kube/kube-init

#6507

  • 741f541 build(deps): bump github.com/containerd/containerd/v2

#6590

  • 25c86e8 kube-init: only deploy the SR-IOV device plugin when network VFs exist
  • 7314b4c kube-init: stop one wedged system pod from pinning the node out of RUNNING

#6542

  • 5a86d93 kube-init: route the k3s binary download through mgmtproxy
  • c7daf10 kube-init: route kubectlx manifest fetches through mgmtproxy
  • 5327945 mgmtproxy: require proxy-auth token, TLS, and a destination policy on the cni0 listener

#6599

  • f8b3509 kube-init: raise the k3s start deadlines to 30 minutes

#6614

  • adb213c kube: clear the retired pod subnet when a node joins

#6652

  • 327e998 Bump containerd/v2 to 2.2.9 in kube-init

#6665

  • 042622f zedkube: authenticate the cluster status server
  • 7fddaf2 kube: keep servicelb and traefik disabled

#6704

  • 5e153e0 kube-init: address the k8s node by its normalized name when re-applying labels

#6755

  • a4cbd44 build(deps): bump github.com/moby/sys/user in /pkg/kube/kube-init

#6771

  • d0f8eb9 kube-init: widen containerd's cpuset for the first boot

#6794

  • c5248c8 pkg/kube: bump CDI v1.57.1 -> v1.66.1

#6796 (zedrouter commit already on 17.0-stable via #6797)

  • 902a643 zedkube: gate the VMIRS replica reset on the eve-app-op lease
  • 007a6ed zedkube: start a failover only when a pod is terminating

#6861

  • 6a33fb4 kube-init: deploy CDI on arm64
  • e700961 domainmgr: don't wait for KubeVirt on non-amd64
  • 1e3232b volumemgr: keep the image of a native container on eve-k

#6702

  • c8414b0 kube-init: recycle controller pods after a single→cluster datastore migration
  • a407274 kube-init, pillar: hold app volume creation until controllers are recycled

#6860

  • 07070ee kube-init: set spec.name when recreating a missing Longhorn Node object
  • 638874a kubeapi: gate Longhorn readiness on recorded disk status

#6897

  • a32ead2 kube-init: bound kubeclient API calls with a request timeout
  • ed0b52e kube-init: put a deadline on the Longhorn pod exec

#6934

  • 01073b8 kube: expire Kubernetes events after 1h
  • 6babf29 kube: drop events left from the 72h event-ttl

17.0-only commits, not cherry-picks:

  • pkg/kube: retire the 17.0-only helpers kube-init replaces
  • pillar: point the kube-side contract comments at kube-init
  • kube-init: declare go 1.25.0 like the other EVE modules
  • kube-init: pin pillar at a 17.0-stable commit

How to test and validate this PR

Automated, in CI: make test now runs kube-init's unit tests, and the eve-k build compiles the daemon.

Done locally:

  • make pkg/kube and make pkg/pillar build.
  • kube-init unit tests pass under -race (the command make test runs). The pillar unit tests for kubeapi, volumemgr, domainmgr and zedkube pass, including the new Longhorn disk-status and native-container tests and eve-k: fix VM failover held back by the stats lease and a false portmap overlap #6796's TestIsDecisionNodeFollowsAppOpLease.
  • go vet -tags k ./... on pillar is clean.
  • mini-yetus: clean apart from two revive warnings on lines that are unchanged from master and 17.0-stable.
  • tools/bump-and-commit.sh finds no package hash to update.

Still to do on eve-k hardware (amd64 and arm64):

  1. Fresh install, single node: k3s, KubeVirt, CDI and Longhorn come up and the node reaches RUNNING; deploy a VM app and a native container app.
  2. Upgrade a node running a 17.0-stable build (with cluster-init.sh) to this build: kube-init takes over the existing k3s state without reinstalling, and the apps keep running.
  3. Three-node cluster: form the cluster, convert a single node to a cluster member, fail over an app, and reboot a node.
  4. arm64: a native container app deploys and survives a reboot (eve-k: fix app deployment on arm64 #6861).

Changelog notes

eve-k: cluster bring-up and update now run in the Go kube-init daemon instead of cluster-init.sh, as on master. It brings the fixes made to kube-init on master since, plus Longhorn readiness and arm64 app deployment fixes.

Fixed VM app failover on eve-k clusters being delayed by up to five minutes when the failed node was the cluster's stats leader.

Checklist

  • I've provided a proper description
  • I've added the proper documentation (kube-init's README comes with pkg/kube: port cluster-init.sh to Go daemon #5971; nothing new)
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

europaul and others added 30 commits November 6, 2026 15:18
During the single-node to multi-node transition, and on any later leader
election such as a node failover, k3s restarts and the embedded etcd
reforms. A transiently-behind kube-apiserver watch cache can then serve a
reconnecting informer a successful but incomplete watch stream, silently
dropping a window of events with no "410 Gone" to force a re-list. CoreDNS
runs its kubernetes informer with resync disabled (resyncPeriod = 0), so the
lost Service events are never backfilled: Services created in that window
stay permanently NXDOMAIN, which breaks CDI upload and app volume deployment
on the freshly formed cluster.

Serving LIST/WATCH straight from etcd removes the stale-serve path: a
reconnect either resumes cleanly or gets an honest 410 that forces a
re-list. Ship a config.yaml.d drop-in that appends watch-cache=false to
kube-apiserver-arg (the "+" suffix appends rather than replacing the base
args), installed by cluster-init.sh only in edge-node-cluster mode so
single-node keeps the cache. Edge clusters are small, so the cache buys
little; the control-plane cost was benchmarked and found negligible at
realistic load.

Signed-off-by: Paul Gaiduk <paulg@zededa.com>
(cherry picked from commit 27496ff)
Direct deps kube-init's typed client-go and containerd usage
relies on:

  k8s.io/api                    v0.34.1
  k8s.io/apimachinery           v0.34.1
  k8s.io/apiextensions-apiserver v0.34.1
  k8s.io/client-go              v12.0.0+incompatible (aliased to v0.34.1
                                                     via replace)
  github.com/containerd/containerd/v2   v2.2.5
  github.com/containerd/errdefs         v1.0.0

k8s.io/kube-openapi's own go.mod aliases v0.31.0 as its module
version, but the repo never publishes that as a tag — the pseudo
below is what actually exists on the repo and is what our replace
directive points at.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 66f1559)
The primitives kube-init's FSM writes to persistent storage:

 - Marker: typed filesystem-path values (LonghornInitialized,
   NativeKubernetesMode, MultusInitialized, …). IsMarked / Mark /
   Unmark route through AtomicWriteFile so partial writes never
   leave a half-marker.
 - AtomicWriteFile: temp-file + rename atop the destination, with
   an fsync in between so the on-disk state matches the return
   value even across a crash.
 - Reboot: the daemon's controlled reboot path. Writes a marker
   naming the reason before invoking the reboot binary so the
   next boot can consult the reason without racing the boot
   sequence.
 - K3sKubeconfig / ContainerdSocket / SealedDirName: canonical
   paths every subsystem shares. Constants (not vars) so a
   subsystem cannot accidentally rewrite them at test time.
 - ToK8sName: kubernetes-node-name transform (lowercased device
   UUID). Load-bearing for cross-subsystem consistency — the
   FSM's node-uuid label, monitor's reapply, tiebreaker's
   node-lookup, and Longhorn's node-object all agree on this
   spelling.

The /var/lib snapshot itself lives in its own commit; what is here are
the primitives it is built on.

state also carries the operator wait items: a named point the FSM will
block at until released, used to hold a node mid-transition while it is
inspected.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit f89ab6b)
A node that joins an HA cluster has its /var/lib overwritten with
cluster-mode state: different certs, a different node identity and a
different datastore. Converting it back to a standalone node is
therefore not a reconfiguration but a restore, so each node captures its
pre-cluster /var/lib during its own first boot, while it is still
standalone, and the conversion puts that tree back.

What is captured is node identity, not the whole tree: server/tls,
server/cred, the cluster token and agent/. Everything else under
/var/lib is re-derivable — the unpacked k3s bundle, CNI binaries and k3s
binaries alone account for ~419MB of a ~443MB tree and are reproduced by
the install path. kubelet/ is deliberately excluded: restoring it is
worse than not, since a stale cpu_manager_state breaks kubelet on the
next boot.

The datastore is the one file written continuously, so it cannot be
copied byte-for-byte underneath a live writer. It is captured with
SQLite's VACUUM INTO, which produces a single self-contained,
online-consistent file, so the snapshot is taken with k3s running and
nothing has to be stopped or quiesced to get a usable copy. Note this is
kine/SQLite rather than etcd: server/db/etcd/ is a vestigial stub on a
single-node server, so `k3s etcd-snapshot` does not apply.

Restoring is an overlay copy, which means anything the snapshot does not
contain survives it, and cluster-mode leftovers have to be removed
explicitly:

  server/db/etcd     a real etcd member directory next to a restored kine
                     datastore gives k3s two conflicting datastores, and
                     makes etcdClusterInitialized() report a bootstrapped
                     cluster that no longer exists.
  state.db-wal/-shm  SQLite treats a -wal beside a database as that
                     database's own journal and replays it on open, so a
                     cluster-mode journal feeds pages into the restored
                     file: it grows past its snapshot size,
                     integrity_check reports "invalid page number", and
                     k3s dies with "database disk image is malformed" on
                     every start with no way back. VACUUM INTO never
                     writes these, so removing them is unconditional.
  Longhorn replicas  replicas under /persist/vault/volumes belonging to
                     volumes the restored cluster knows nothing about
                     would otherwise be adopted by a single node that
                     cannot serve them.

Component markers are part of the restored state. The snapshot is taken
before they are written, so a restored tree reads "components installed,
not yet initialised" and the next boot re-runs initialisation against
the datastore it actually has.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit a96859d)
…ntainerd wait

Everything that must be ready before k3s can start. The FSM
sits in INSTALLING → STARTING_CTRD while this package runs; a
failure here aborts the boot rather than starting k3s on a broken
substrate.

 - Kernel modules loaded via modprobe (bridge, iptables_nat,
   iscsi_tcp, etc.). Absent kernel bits surface with the specific
   module name so operators can spot a kernel-config regression.
 - cpu_manager_state cleanup: kubelet sometimes leaves a stale
   cpu_manager_state under /var/lib/kubelet after a reboot with a
   changed --cpu-manager-policy. Deleting it on every boot before
   containerd starts is upstream-blessed cleanup.
 - Cgroup mount: /sys/fs/cgroup, kubepods.slice hierarchy.
 - iSCSI daemon start (needed for Longhorn's block-mode volumes).
 - Vault wait: pillar's vaultmgr signals "ready" via a fs marker;
   the daemon blocks here until the marker is present so k3s does
   not race the vault-mount.
 - EdgeNodeInfo wait: blocks until the pubsub subscription
   delivers the device identity (DeviceName / DeviceID).
 - Containerd start + reachability probe.

Everything is idempotent — the FSM re-enters INSTALLING on
recycle without expecting a clean state.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit c6521d1)
… boot

First boot is the most CPU-hungry moment in a node's life and the one
with the least to lose: k3s unpacks, imports images, applies every
component manifest and waits for kube-system to converge, while no user
workload is running yet. Steady state is the opposite — dom0 is capped
so guests get the rest of the machine, and that cap is what the node
must run under once it is serving.

Rather than choose one of those, borrow for the duration. This computes
the CPU set dom0 may use while bringing the cluster up, hands the
control plane the CPUs that would otherwise sit idle, and returns them
when first boot completes, so the steady-state cap is what survives.

The set is derived rather than declared: the loan is bounded by what
dom0_max_vcpus actually permits, so a node configured for a small dom0
does not end up with a control plane wider than its own vcpu budget.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 8803295)
…password, install

Everything about running the k3s process:

 - binary.go / install.go: k3s binary download with SHA-256
   verify + symlink rebuild for the multi-call subcommands
   (kubectl / ctr / crictl → k3s). Idempotent — a re-install
   picks up unchanged bytes cheaply.
 - config.go: drop-in files under /etc/rancher/k3s/config.yaml.d
   with numeric prefixes deciding merge order (00-nodename,
   01-clusterconfig, 03-enc-disable-local-path, 99-user-override).
   Slot 02 is reserved. Byte-stable rendering keeps k3s's
   drop-in-hash restart trigger quiet on no-op reconciles.
 - paths.go: absolute paths for the k3s binary and every state
   file the daemon writes.
 - supervisor.go: Setpgid + orphan-sweep supervisor for the k3s
   process. Ports its own PID stays alive until the child's
   process group is fully reaped; port-listening probe rules out
   an outer-shell-still-running scenario before signalling
   "started".
 - token.go: k3s node-token rotation on cluster-mode change.
 - node_password.go: FixNodePasswordSecret clears a stale
   node-password secret when the file marker StaleNodePasswdFlag
   is present. Delete via typed clientset (kube-system Secrets).
 - readiness.go: WaitKubeconfig (poll for /etc/rancher/k3s/k3s.yaml
   with ctx cancel), then WaitReady blocks until Node's Ready
   condition + kube-system pods are Ready + node-uuid label is
   applied. Uses a client-go client built from the kubeconfig
   file — dial errors during the boot window are tolerated as
   "keep polling", so no shell-out is needed for the chicken-
   and-egg case of probing an API that isn't up yet.

Every step is bounded by the caller's ctx (INSTALLING /
STARTING_K3S / WAIT_K3S_READY handlers set the deadline).
Restart hooks live under /etc/k3s-supervisor/hooks.d and fire
between stop and start on operator-driven restart (RT PerformanceProfile
changes are the canonical consumer).

Bringing a node into a cluster retires its whole PKI: everything under
server/tls goes, along with cred/ipsec.psk and the kubeconfigs derived
from it. A kubeconfig embeds the CA, so one that outlives the CA it
describes is not inert — the readiness wait would adopt it, build a
client against an authority that no longer exists, and every call would
fail the TLS handshake against a healthy apiserver. dynamic-cert.json is
the file that bites hardest: it caches the whole kube-system/k3s-serving
Secret and is preloaded before the datastore is up, so a survivor makes
the supervisor serve a cert the adopted cluster CA cannot verify.

The readiness wait is built to be diagnosable, because every distinct
cause of "no answer from the apiserver" otherwise looks like the same
hanging call. It records what the client is aimed at and the fingerprint
of the CA it trusts, then on each failure separates the layers: whether
anything is accepting on the API port, what /readyz says is outstanding
(etcd, informer sync, a poststarthook), and whether the kubeconfig moved
underneath the client — in which case the client is rebuilt rather than
polled to death. Every call is individually bounded: rest.Config.Timeout
is deliberately unset because it would cut informer watches, so a
per-call context is what stops one unanswered request from consuming the
whole readiness budget.

Readiness is scoped to this node. Pod readiness is counted with a field
selector for the local node, so a cluster peer that is unhealthy or
mid-join cannot hold up a node that is itself ready.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 9f3cf92)
k3s version override + component-by-component upgrade driven by
the KubeConfig pubsub subscription. Runs as a background loop
during RUNNING; a version bump on the wire triggers download +
SHA-verify + install of the new k3s binary followed by rolling
component upgrades.

 - update.go: main loop. Pulls the desired k3s version from the
   KubeConfig subscription cache, downloads via curl into a
   /tmp path with a per-attempt timeout, SHA-256-verifies against
   the version-specific expected hash, atomically installs into
   /var/lib/k3s/bin/k3s.
 - cluster.go: checkClusterReady probes the API server via
   discovery.ServerVersion; componentIsInstalled probes each
   component's namespace via a typed Namespaces().Get,
   distinguishing "not installed" (NotFound → false, nil) from
   "cannot tell" (err → false, err) so a transient API outage
   isn't misread as "component gone".
 - k3s_update.go: k3s download / verify / install flow.
 - status.go: publishes update progress to zedagent via
   pubsub so the controller sees each component's version.

Retry gate: the daemon consults the KubeClusterUpdateStatus
subscription to skip retrying a version the controller has
already reported as failed — matches the "one attempt per version"
semantic pillar expects.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 6e6aabe)
…ault()

Holds the typed clientset, apiextensions clientset, dynamic
client, discovery + deferred RESTMapper, and shared informer
factories (typed + dynamic). Constructed once at daemon boot from
state.K3sKubeconfig and exposed via a package-level singleton
(SetDefault / Default) so subsystems don't have to thread a
*Client parameter through every function.

Default() panics if consulted before SetDefault — that path
indicates a lifecycle bug and should crash loud rather than
nil-deref deeper.

ResetMapper() invalidates the discovery cache so kubectlx.applyOne
picks up newly-installed CRDs on retry. WaitForKubeconfig polls
for state.K3sKubeconfig with ctx cancellation, exposed for
callers that need to gate on file existence explicitly.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit c9144fa)
…ce layer

Every operation the daemon performs against Kubernetes or
containerd goes through this package. No argv assembly, no
stderr parsing, no subprocess exec.

apply.go — Apply / ApplyFile / ApplyURL / Get + DeleteFile /
DeleteURL / DeleteBytes. Server-side apply via the dynamic
client, multi-document YAML parsing, mapper-reset + retry-once
on meta.NoKindMatchError closes the CR-before-CRD race in one
place. Retry via client-go/util/retry.OnError with an
apierrors-based classifier (transient vs permanent). Force-
conflict-override is on by default (ApplyOptions.NoForce inverts
the flag) because kube-init owns the components it applies;
context.Canceled short-circuits so peer-cancel doesn't drag
30s+ of backoff sleeps.

wait.go — WaitCRDEstablished / WaitDeploymentReady /
WaitDaemonSetReady / WaitJobComplete / WaitForCondition, all
informer-driven via watchtools.UntilWithSync. WaitForCondition
evaluates a JSONPath against the target object on every event —
CR-defined phase fields (KubeVirt / CDI status.phase, etc.) fit
this shape.

containerd.go — ContainerdClient wraps containerd.Client (v2)
with namespaces.WithNamespace(k8s.io) baked into every operation.
ImportImage / ImageExists / DeleteImage / ListImages / TagImage.
A health probe at construction turns "containerd is down" into
one clear error instead of per-operation failure spam. No
cri-api dependency — CRI's image operations are containerd
image-service operations under the k8s.io namespace, which the
Go client already exposes.

CRs are gated on their CRD actually being established rather than merely
created: each CRD document is applied and awaited before the CRs that
depend on it. Establishment is judged from the object, not from a bare
condition read — a CRD that is terminating still reports
Established=True, so a deletion timestamp disqualifies it. That closes
the window where a CR would be applied against a CRD about to disappear.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 3745588)
…ffort retry loop

A component starts the moment its declared deps are Ready — no
per-wave synchronization barrier. A fast component whose deps
are satisfied never waits for a slow unrelated peer.

types.go — Component (Manifests + Apply + Ready + PolicyDeps +
BestEffort + ReadyTimeout), Manifest (File / URL / Bytes), Edge
(structural-derivation output tagged with rule: crd / namespace /
webhook / sa / policy), RetryPolicy, RetryCallback.

deploy.go — Graph + Run + Edges. plan() validates the graph and
resolves edges. The scheduler runs a single-threaded coordinator
loop that tracks launched/completed counters directly (a
WaitGroup-based helper is a runtime panic when a completing
goroutine's result enqueues children mid-drain).

retry.go — Graph.RetryCtx is the daemon-scope context that keeps
BestEffort background retries alive after Run's per-invocation
ctx ends. Exponential backoff with jitter (defaults: 30s →
5min × 6 = ~30min budget). Re-arms Apply → Ready on Ready-
timeout too, not just Apply-failure. Retries observe RetryCtx
cancel (daemon shutdown).

rules.go — placeholder for the four structural-dep rules
(CRD→CR, Namespace→resource, webhook-Service→webhooked-resource,
ServiceAccount→Pod-spec) that fire once callers populate
Component.Manifests with real bytes. Today Manifests is empty
across the shipped graph; ordering rides entirely on PolicyDeps.

deploy_test.go — 25 tests covering plan validation, edge
tagging, edge sorting, dependency-order execution, parallel-
execution property (two-channel handshake, not sleep), error
propagation with alphabetical head + rest, ctx cancellation,
BestEffort semantics, Ready-only-on-Apply-success, MaxParallel
concurrency cap, Apply=nil (Ready-only) shape, and four
retry-loop tests (eventually-succeeds, exhausts, ctx-cancel-mid-
loop, one-shot-without-RetryCtx). All pass -race across 50
iterations.

Readiness is bounded by progress rather than by a fixed timeout, so a
component that is still making headway is not cut off while one that has
genuinely stalled does not hold the graph indefinitely. What the progress
probe observed is logged, because "not ready yet" and "not ready and no
longer moving" are the same line otherwise.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 9a5dbe0)
…nager

Single-goroutine dispatch of pillar pubsub topics into per-topic
subscriber caches. Modelled on pkg/pillar/cmd/monitor/subscriptions.go —
one map of label → Subscription, deferred Activate, one goroutine
pumping every topic via pubsub.MultiChannelWatch.

Subscribers across kube-init packages (edgenodeinfo, encconfig,
encstatus, kubeconfig, kcus) register their topics with the
Manager during daemon boot; the Manager owns the connection and
the pump goroutine. Registering after the pump has started is
supported — new subscriptions activate lazily.

Replaces file-polling under /run/zedagent/... / /persist/status/
zedkube/... — every subscriber owns its own decoded cache with
the canonical pillar types (no local JSON re-declaration), so
JSON-tag skew between pillar and kube-init cannot silently drop
a field's value.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 64570bd)
Six typed subscribers wired to the shared pubsubclient.Manager;
one publisher for the daemon's own lifecycle status.

 - edgenodeinfo: EdgeNodeInfo subscription with WaitForFirst so
   the boot path can block on device identity (DeviceName,
   DeviceID). Exposes Get() for read-only consumers on the FSM
   loop.
 - encconfig: EdgeNodeClusterConfig subscription. Drives cluster
   shape (single vs HA), TieBreakerNodeID, cluster interface.
 - encstatus: EdgeNodeClusterStatus subscription. Present() folds
   both file deletion and the zero-UUID-delete sentinel into a
   single check so a cluster delete on the controller side is
   detected the same regardless of how zedkube signals it.
 - kubeconfig: KubeConfig subscription with K3sVersion accessor
   for the update loop. Tagged consistently with pillar's
   canonical json tags (capital K, no override).
 - kcus: KubeClusterUpdateStatus subscription gating upgrade
   retries. DestinationKubeUpdateVersion is uint32 (matching
   pillar), so a "cluster already tried and failed at version X"
   record correctly gates the next attempt.
 - kubeinitstatus: publishes the daemon's lifecycle events
   (transition-step, phase, etc.) so operators observing the
   controller see kube-init state transitions without needing to
   tail the kube container log.

Every subscription decodes to pillar's canonical types (via
pkg/pillar/types) — no local *JSON structs that can drift.

The node also publishes what it is actually doing, so the controller and
pillar can see the daemon's own view rather than infer it: the FSM state
and phase, derived cluster membership (restoring, leaving, joining,
member, join-pending or single), the current transition step, whether the
join watchdog is armed, and the last error. Membership is derived from
the markers and configuration on disk in one place, with a defined
precedence, so two agents cannot disagree about whether this node is in
a cluster.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit d0c1251)
First-boot import of pre-packaged image tarballs (KubeVirt, CDI,
Longhorn, Multus, SUC, the EVE-authored external-boot-image)
into the user-side containerd's k8s.io namespace, so kubelet
pods reference images without a first-boot registry pull.

 - Import via kubectlx.ContainerdClient (containerd v2 Go client
   under the k8s.io namespace).
 - ImportUpstreamImage is a no-op when the image is already
   present or the tarball is absent; kubelet falls back to a
   registry pull for anything missing.
 - ImportExternalBootImage re-tags the EVE-authored image
   (whose tarball name isn't the runtime tag) to the running
   EVE release's tag, then cleans up prior EVE-authored images
   for the same base name via ListImages + DeleteImage.
 - Manifest.json is parsed via `tar` (fatal-line-buffered) with
   the RepoTags extraction in Go — no jq dependency.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 335af07)
…ainerd env

Wires kube-init into pillar's cost-aware management proxy so
in-cluster traffic (CDI's image importer, containerd's image
pulls) routes through pillar rather than out the WAN.

 - cni0.go: SetupCNI0ProxyIP assigns a well-known anchor IP on
   the flannel-created cni0 bridge — CDI pods reach the
   management proxy at that IP without a per-node hostname
   lookup. Enabled() only fires when pillar's mgmtproxy config
   file is present.
 - The CDI CR's spec.config.importProxy fields (HTTPSProxy /
   noProxy) are reconciled every steady-state tick: a dynamic
   Get reads the current values, and a merge-patch fixes any
   drift. In-Go JSONPath extraction reads the field values —
   no `-o jsonpath=…` shell-out.
 - containerd-launch env injection is a separate concern
   (pkg/kube/kube-init/prereqs) and pre-dates this package.

Wildcard-webhook rules (rules[].apiGroups: ["*"]) are treated as
"no derived edge from this webhook" in the deploy planner — a
catch-all webhook shouldn't sequence the whole graph. Not the
concern of this package but noted here so the wiring is
consistent.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 856c364)
… startup stagger

Every state change between "this node is a single-node cluster"
and "this node is a member of an HA cluster" — including the
recovery paths — lives here.

 - transition.go: the ordered step runner. Single→cluster
   rotates the k3s node-token, applies encstatus-derived
   configuration, and updates supervisor args. Cluster→single
   reverts. Progress callback surfaces the current step name to
   the FSM's control socket so `k3s-sctl status` reports
   `transition-step=<step>`.
 - masterleases.go: stale etcd master-lease cleanup. After a
   single→HA conversion, etcd carries a lease from the
   pre-conversion single-node identity; the deploy graph's
   Longhorn readiness stalls if that lease's expiry sits ahead
   of the new node's registration. Cleanup deletes the stale
   lease from etcd directly (etcdctl invocation — outside the
   client-go surface because etcd doesn't ship a stable Go SDK).
 - startup_rank.go: on HA startup, computes the local node's
   rank (0-based position in the sorted control-plane node
   list) and sleeps rank × 25s before launching k3s. Fans out
   simultaneous power-cycles into a staggered sequence so etcd
   quorum re-forms cleanly without three nodes fighting for the
   leader lease at t=0.

The multus reset performed during a transition is best-effort. The
NetworkAttachmentDefinition CRD is cluster-scoped, so tearing it down to
re-address one node would take the definition away from every other node
in the cluster; the node re-applies its own configuration and restarts
the DaemonSet instead, and a failure to do so is logged rather than
aborting a transition that has already moved on.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 794a58a)
The steady-state loop the FSM enters after DEPLOYING. Every 15s
health tick runs a fixed sequence of housekeeping + reconcile
tasks; separate goroutines watch containerd exit, kubeconfig
drift, log rotation, cluster-config-changed events, and the
user-override drop-in for k3s config.

Tasks the health tick runs (in order):
 - DHCP CNI daemon liveness (respawn on exit).
 - CNI plugin binary drift (copy from stage dir when needed).
 - Multus re-render + apply if the initialised marker is absent.
 - Node label + annotation reapply (node-uuid, longhorn disk
   config) when the labels-initialised marker is absent — one
   merge-patch does labels and annotations together, then a
   readback verifies both landed.
 - Debug-user RBAC re-application.
 - Kubeconfig sync (copy /etc/rancher/k3s/k3s.yaml to the
   kube-init-visible location, atomic).
 - Image reimport (external-boot-image via kubectlx.
   ContainerdClient — ImageExists check, tarball import only if
   the image is genuinely absent).
 - Longhorn post-install config (via components.
   LonghornPostInstallConfig).
 - Registration manifest reconcile.

transition.go — countReadyNodes counts Nodes with Ready=True
via typed clientset. Used by the cluster-mode monitor to detect
"we're on the HA majority" transitions. parseTransitionMarker
covers the FSM's cluster-transition marker file (unix-ts +
reboot-count).

Steady-state ticks are gated on state markers so a tick after a
successful bootstrap is silent — no log line, no API call.

The join watchdog is driven by progress, not by a deadline. How long a
join takes is unknowable — it covers a k3s restart, an image import,
registration and every kube-system pod — so any budget for it is a guess
that some slower device will fail. How long a join may make no progress
at all is a stable quantity, so every advance to a stage this join has
not reached before kicks the timer: a slow-but-advancing join is never
rebooted, while a wedged one trips promptly. A crash loop does not count
as progress, because it revisits stages rather than advancing.

The cluster-config watch runs in every state, not only in RUNNING. A
controller can withdraw a node's cluster configuration at any moment,
including seconds after a join completed while the node is still working
through readiness. A zero-UUID or otherwise empty status is treated as
"no live cluster" only when it really is one, so a status that arrives
late or half-written cannot be mistaken for a withdrawal.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 3355cc4)
The tie-breaker is the third control-plane node in an EVE-K HA
cluster whose only job is to give etcd a third vote so single-
node loss doesn't strand the cluster. This package configures
that role.

ConfigApply is the phase entry point; it runs when the cluster
first reaches three nodes AND the local node is the designated
tie-breaker (per encconfig's TieBreakerNodeID). Steps:

 - nodesConfigApply — label every node with tie-breaker-node=
   true/false and cordon/uncordon accordingly. Merge patches
   via Nodes().Patch keep both label and spec.unschedulable
   updates atomic per node.
 - kubevirtConfig — scale virt-operator + KubeVirt CR to 2
   replicas (tie-breaker excluded). Deployment scale via typed
   AppsV1 merge-patch; CR replica field via dynamic Resource
   Patch.
 - kubevirtTieBreakerConfigApply — patch every DaemonSet in the
   kubevirt namespace with a nodeSelector keeping pods off the
   tie-breaker.
 - cdiConfig — same nodeSelector patch on every Deployment in
   the cdi namespace.
 - longhornNodeSetSched — flip allowScheduling +
   evictionRequested on the tie-breaker's longhorn.io Node CR
   and every disk it owns. JSON-patch operations via dynamic
   Resource Patch.
 - longhornRescale — scale Longhorn CSI sidecar Deployments to
   2 replicas; patch every longhorn-system DaemonSet with the
   nodeSelector (retries 5× because longhorn-manager
   occasionally races us during install).
 - drainNode — cordon + evict every non-DaemonSet pod on the
   tie-breaker via policy/v1.Eviction (matches kubectl drain
   --ignore-daemonsets --delete-emptydir-data --force semantics)
   + wait until pods leave the node.
 - StatusSet — stamp every node with tie-breaker-config-applied=1
   as the "phase complete" marker so a subsequent reboot skips
   the whole flow.

Every operation is idempotent — a partial ConfigApply that gets
interrupted mid-way completes on the next FSM entry.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit ee13ef0)
…eduler, RBAC, SR-IOV, kube-vip

Everything the deploy graph runs during DEPLOYING and everything
the monitor's steady-state ticks reconcile against.

components.go — DeployAll builds a deploy.Graph and runs it. The
graph has ~10 Components; only Longhorn declares a PolicyDep
(longhorn depends on manifests, which stages storage-classes.yaml
into the auto-deploy dir). Every Install* function:
 - Longhorn: applyLonghornDiskConfig labels + annotates the node
   via one merge-patch; the longhorn config YAML applies as a
   drop-in.
 - KubeVirt: InstallKubeVirt applies the operator manifest via
   SSA, waits for virt-operator Ready, applies the CR via SSA,
   applies feature-gate CR via SSA. kubeVirtConfigReplicas
   writes .spec.replicas + .spec.infra.replicas via SSA with a
   distinct field manager ("kube-init-replicas") so subsequent
   applies of the operator manifest don't fight for ownership.
 - CDI: apply operator, wait for cdi-operator Ready, apply CR.
   WaitCDIReady goes through kubectlx.WaitForCondition on
   .status.phase == "Deployed".
 - Multus: template rendering (IP address substitution) then
   applyMultus + linkMultusIntoK3s.
 - descheduler / debug-rbac / manifests / kube-vip / SR-IOV
   manifest staging / stale-mount-cleanup daemon launcher —
   one Component each.
 - WaitLonghornReady is informer-driven (via a wait-scoped
   dynamic factory over longhorn.io/Node and typed DaemonSets in
   longhorn-system). No polling; the compound predicate is
   re-evaluated on every event.

kubevirt.go — MigrateKubeVirtFeatureGates reads the running CR
via dynamic Get, patches with dynamic Patch, marks the migration
done. buildFeatureGatesPatch is table-driven so a new gate is a
one-line addition.

longhorn.go — LonghornIsReady is the compound predicate the
monitor consults on every tick. Uses typed clientsets for
DaemonSet readiness and dynamic client for the Longhorn Node CR
and engine images. longhornEngineDeployedOnNode distinguishes
"status.nodeDeploymentMap absent" (fresh CR — not-ready-yet)
from "map present but this node is false" (recycle engine +
manager pods to force reconcile). CheckOverwriteNsmounter uses
client-go remotecommand.NewSPDYExecutor (via podExec helper) to
copy a fixed nsmounter binary into every longhorn-csi-plugin
pod that doesn't already carry the marker file.

registration.go — RegistrationApplyIfReady stages the
controller-supplied registration manifest into the k3s
server-manifests dir. RegistrationApplied checks the AddOn CR
via dynamic Get.

uninstall.go — the K3sBase conversion flow. Uninstalls
descheduler / Longhorn / CDI / KubeVirt / Multus in reverse
dependency order. Longhorn uninstall is bounded by a 1000-poll
budget (~83 min) covering the worst-case data-shred workload;
timeout aborts the conversion rather than declaring
NativeKubernetesMode with volumes half-shredded.

sriov.go / stale_mount_cleanup.go — hardware-conditional
manifest staging and background cleanup daemon launcher.

Every Install has an idempotence gate on a state.Marker; a
re-entry to DEPLOYING or a re-tick of the monitor is a cheap
noop when the marker is already set.

Longhorn's readiness is taken from its instance managers rather than
from the deployment alone: the CSI plugin can report available while the
per-node instance managers are still coming up, and a volume attach in
that window fails. Multus is restarted through its DaemonSet, so a
re-applied configuration is actually picked up by the pods already
running.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit e3cef7c)
Launches a virtctl vnc proxy on demand for a KubeVirt VMI and
tears it down when the caller process exits. Caller-PID
watchdog polls /proc/<pid> so an orphaned proxy from a
disconnected debug session doesn't linger.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 8d45ff2)
14 states, 27 events. The daemon boots into INIT, walks through
INSTALLING → STARTING_CTRD → CONFIGURING → STARTING_K3S →
IMPORTING (first-boot only) → WAIT_K3S_READY → DEPLOYING (first-
boot only) → RUNNING, and handles restart / recycle / cluster-
mode-transition events without dropping back into an ad-hoc
respawn loop.

State entries are `enterStateFn` dispatched from `transition`;
async work runs on a per-attempt workCtx via startAsync so
transition cancellation drains the work goroutine cleanly. The
control socket at /run/k3s-supervisor.sock accepts `status`,
`restart`, `stop`, `graph` commands; every operator-facing k3s-
sctl invocation dispatches through the socket rather than
touching flag files.

initKubeclient wires kubeclient.Default at WAIT_K3S_READY's
transition (test-safe when the kubeconfig file is absent —
transition-table tests exploit this). components.DeployAll takes
a retryCtx that main.go supplies as d.rootCtx, so BestEffort
background retries in the deploy graph survive per-invocation
workCtx while still dying on daemon shutdown.

Restart-reason is a typed field carried through the STOPPING_K3S
→ RUNNING_HOOKS → STARTING_K3S transition so restart hooks
(/etc/k3s-supervisor/hooks.d) fire once for operator-driven
restarts and never for crash recovery. The RT operator's
PerformanceProfile-change flow depends on this — CPU isolation /
RDT partitioning need to be reconfigured between stop and start,
not on every crash.

README.md sketches the architecture, package boundaries, and
the future-work list (bootstrap-cert fingerprint pinning,
config-package retirement, and so on).

Two behaviours in the loop are worth calling out. A cluster transition
that fails retries in place instead of falling through to a state that
assumes it succeeded — falling through leaves a node half-transitioned,
which is not recoverable by pushing new configuration. And images are
imported on every k3s start rather than only on first boot, because a
restart can find the content store missing entries that the running
cluster still references.

The operator breakpoints are plumbed through here: a wait item placed in
the FSM's path lets a test or an engineer hold a node at a chosen point
and release it, which is how the multi-node transitions are inspected on
real hardware.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 9abf8cc)
One-shot tool that dials /run/k3s-supervisor.sock and prints the
reply. Reads until EOF so multi-line responses (graph) print
naturally alongside single-line ones (status / restart / stop).
K3S_SUPERVISOR_SOCKET overrides the path for tests.

Exit code follows the last line: non-zero when it starts with
ERR.

Subcommands:
 - restart — graceful k3s restart (pre-restart hooks fire).
 - status — one-line status report (state, k3s pid, uptime,
            backoff, restart count, phase, transition-step).
 - stop    — stop the kube-init daemon.
 - graph   — resolved deploy-graph edges (one per line, each
             tagged with its rule source).

Breakpoints are managed from the same CLI (break set / clear / list), so
holding a node at a chosen point does not require reaching into
/persist by hand.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 933cef7)
…helper

pkg/kube/Dockerfile — the kube linuxkit container's entrypoint
is /usr/bin/kube-init. Everything else the container needs to
ship (k3s multi-call binary, longhorn nsmounter, CNI plugin
staging, longhorn/kubevirt/CDI/multus tarball drops under
/images/) is installed via the existing build.yml conventions.

pkg/kube/k3s-control.sh — small wrapper that dispatches to
k3s-sctl. Kept so external tooling (eden / eve exec kube …)
that used to invoke this path continues to work; every
subcommand it takes now goes over the control socket rather
than touching flag files.

pkg/kube/kubevip-delete.sh — minimal helper that KubeVIPDelete
calls to remove leftover kube-vip state files after component
deletion (leftover static-pod manifests, arp cache). Not part
of the deploy graph; runs from the uninstall path.

sqlite is included in the image because the datastore snapshot is taken
with SQLite's VACUUM INTO, which is what makes an online-consistent copy
possible without stopping k3s.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 7487742)
Eleven files under pkg/kube/ implemented the previous
cluster-init flow in POSIX shell + kubectl. Every operation
they exposed lives in pkg/kube/kube-init/ now — no external
caller depends on them, and keeping them alongside the Go
daemon would leave two divergent implementations of the same
paths that operators could accidentally invoke.

Deleted:
 cluster-init.sh cluster-update.sh cluster-utils.sh
 descheduler-utils.sh kubevirt-utils.sh longhorn-utils.sh
 pubsub.sh registration-utils.sh tie-breaker-utils.sh
 utils.sh vnc-proxy.sh lib/config.sh
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit c0cfc4b)
…stDevices

The KubeVirt CRD requires spec.configuration.permittedHostDevices.
{pciHostDevices,mediatedDevices,usb} to be arrays. Bare keys
parse as `null` in YAML; server-side apply forwards `null` as an
explicit set-to-null, which the schema rejects with:

  KubeVirt.kubevirt.io "kubevirt" is invalid:
  spec.configuration.permittedHostDevices.mediatedDevices:
  Invalid value: "null": ... must be of type array

Explicit `[]` satisfies the schema.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 3b3e9aa)
The KubeVirt image catalog pinned the five virt-* images at v1.6.0
while every other declaration of the version says v1.7.3: the CR URL
applied by InstallKubeVirt, the operator manifest that names the
image and sets KUBEVIRT_VERSION, and expected_versions.yaml, which
drives the update-component drift check. A node pre-loading v1.6.0
would never satisfy a v1.7.3 pod spec, so the operator's pods would
pull from quay anyway and the pre-load would buy nothing.

The mismatch is currently latent: no rule produces the /images/*.tar
tarballs yet, and a missing tarball is a logged no-op that defers to
kubelet's pull-on-first-use. Correcting it now keeps the catalog
honest for whenever the tarballs start shipping.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 84b63ae)
Component versions were spread across five files: the Longhorn config
path baked the version into a filename, the KubeVirt CR URL inlined
it, CDI and k3s each had their own const, and the pre-loaded image
catalog repeated all of them once per image. Nothing tied those
copies together, which is how the KubeVirt tags came to disagree with
the operator manifest.

Introduce a leaf versions package holding one const per component and
derive every other site from it. All const expressions, so this is a
compile-time rearrangement with no runtime change. The k3s const stays
a default rather than an authority: the controller's k3s.version
config item still overrides it at runtime.

Two declarations remain outside the Go tree and cannot reference the
package — expected_versions.yaml and the vendored kubevirt-operator
manifest — so the package doc names them as sites a bump must also
touch.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit a0dd81e)
A device converted to native Kubernetes under an older EVE image
records that in a marker whose name has since changed. The rename ran
from main(), before the prereqs pass bind-mounts /var/lib, so it
inspected the container's own empty directory, found nothing, and
returned successfully having done nothing. The real marker was still
under its legacy name once the mount appeared, so the daemon read the
device as un-converted and reinstalled KubeVirt and CDI on a node that
had deliberately had them removed.

Run the migration inside the init work step instead, right after the
prereqs pass returns and well before the marker is read. A failure now
returns rather than calling log.Fatalf: at this point the daemon has a
state machine that can back off and retry, which it did not when the
call ran from main().

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 6f72bdd)
zedagent holds the device in BaseOsUpdating for as long as the cluster
update status names an unfinished component. A k3s update reports its
own step complete and then reboots; on the way back up the component
pass finds everything already converged, returns early, and publishes
nothing. The status stays parked on the k3s step and the device reports
BaseOsUpdating indefinitely, which the shell avoided by advancing the
status itself on that path.

Advance it the same way: on the converged path, a status sitting on a
completed k3s step is retroactively finished. Both halves of that
condition matter — a status naming another component, or naming k3s at
any stage short of completed, belongs to a pass still in flight and is
left alone.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 259f9de)
The multus manifest declares the NetworkAttachmentDefinition CRD and an instance
of that kind in the same file, so applying it depends on ApplyFile waiting for
establishment between documents. Nothing checked that the manifest still ships
both halves: drop the instance or rename the CRD and the ordering silently stops
mattering, or silently starts failing. Read the manifest and assert both.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit ec4c168)
andrewd-zededa and others added 14 commits November 6, 2026 15:18
kubeclient set no rest.Config.Timeout, because that timeout also cuts
open watch streams. Without it, an API call whose context has no
deadline waits forever on an apiserver that accepts the connection and
never answers.

Give the request clients a 90s Timeout. This is longer than the
apiserver's own 60s request timeout, so it only stops a call that gets
no answer. Add watch clients without the timeout. Use them for the
informer factories, the Longhorn wait informers and the kubectlx waits.
The discovery client keeps the 32s default that client-go applies when
the config has no Timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
(cherry picked from commit a32ead2)
podExec runs from the monitor tick, whose context has no deadline.
kubeclient's requestTimeout does not cover exec, so a stuck exec can
stop the health checks.

Give each podExec a 60s deadline. The SPDY executor ignores the context
while it waits for the upgrade response, so try the WebSocket executor
first. It applies the deadline to the handshake too. SPDY stays as the
fallback for an apiserver that refuses the WebSocket upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
(cherry picked from commit ed0b52e)
On a single-node EVE-k device, k3s stores its state in sqlite through
kine. kine keeps lease expiry timers in memory and, at every k3s
start, sets each key's expiry to the start time plus its lease. With
event-ttl at 72h, a device that reboots more often than every three
days never expires an event. On sc-sm-e300-10, which reboots several
times a day, 26 days of events accumulated: 295600 of 298249 rows and
236 MiB of the 644 MB datastore. kine lists scan the whole name index,
so every list and every compaction gets slower as the table grows, and
listing the events no longer finishes within two minutes.

A 1h TTL is shorter than a typical uptime, so events expire within an
hour of each k3s start, including those carried over from before a
reboot.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
(cherry picked from commit 01073b8)
Lowering event-ttl only affects events written afterwards: kine stores
each event's lease in its row and re-arms expiry from that lease at
every k3s start. On a device that reboots more often than every three
days, events written under the former 72h TTL therefore never expire,
and an upgraded device keeps its whole backlog.

Before starting k3s, kube-init now deletes the events that carry the
72h lease and vacuums the datastore if any were removed. Events from a
user override with a different TTL are left alone. On sc-sm-e300-10 the
same deletion, done by hand, removed 297835 events in 42 s and shrank
the datastore from 644 MB to 81 MB.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
(cherry picked from commit 6babf29)
17.0-stable added two shell-era helpers that master never had, because
master already ran kube-init when the matching fixes landed:

- cluster-status-probe, the client half of the authenticated cluster
  status server (lf-edge#6753). kube-init's k3s/statusauth.go derives the same
  credentials in-process.
- cni-state-utils.sh, which clears the retired pod subnet when a node
  joins a cluster (lf-edge#6727), with its host-side test and the
  `make -C pkg/kube test` target that ran it. kube-init's clustermode
  does the same cleanup.

Nothing in the kube container runs either one any more; remove them,
the Dockerfile stage that built the probe, and the Makefile and README
references.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Three earlier 17.0 backports carried master commits whose pillar half
names kube-init as the other side of a contract, and reworded those
comments to name cluster-init.sh or cluster-status-probe, which ran on
this branch then:

- lf-edge#6584 (master 5327945): mgmtproxy's cni0 token and TLS cert, read
  by kube-init's mgmtproxy.PatchCDIProxyConfig.
- lf-edge#6753 (master 042622f): the cluster status server credentials,
  derived again in kube-init's k3s/statusauth.go.
- lf-edge#6917 (master a407274): the controller-restart marker, written by
  kube-init's clustermode.ControllerRestartPending.

kube-init now owns all three, so restore master's wording. Comments,
docs and one test failure message only; no code change.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's go.mod says go 1.25.12, the toolchain master's eve-alpine
ships. 17.0-stable's eve-alpine has Go 1.25.11, so the kube image build
stops with "go.mod requires go >= 1.25.12 (running go 1.25.11;
GOTOOLCHAIN=local)".

Nothing needs 1.25.12: the go directive is the minimum language
version, every other module in the tree declares go 1.25.0 on both
master and 17.0-stable, and no kube-init dependency asks for more than
1.25.0. Declare the same.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init came over from master with pillar pinned at a master commit,
which is not on 17.0-stable. Pin pillar at the current 17.0-stable head
instead, using go get, go mod tidy and go mod vendor only. kube-init
uses nothing from pillar that 17.0-stable lacks, so no pillar change has
to land first.

go get also moves eve-api, otel, logr and genproto to the versions this
pillar requires. tidy marks golang.org/x/sys as a direct dependency,
since kube-init imports it itself.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The instance-manager gate is satisfied as soon as its CR reports
Running, often seconds after the pod starts -- before Longhorn's disk
monitor has ever run for that node. For a v1 (filesystem) disk, that
monitor itself requires a running instance-manager, so a fresh one can
pass the existing gate while Status.DiskStatus stays nil indefinitely,
since nothing retriggers the monitor once the window passes.

Observed live: a descheduler-driven VMI move landed a volume on a node
4 seconds after its instance-manager started, and the replica controller
failed forever with "cannot find disk name for replica" -- even though
longhorn-disk.cfg on disk held the correct UUID throughout.

Add a gate requiring every disk in Spec.Disks to have a
Status.DiskStatus entry with a recorded DiskUUID, and at least one of
them to have a True Ready condition, the exact condition that stayed
unmet. Only one ready disk is required so that a recorded but degraded
disk does not strand a node that can still host replicas, matching the
any-disk semantics of checkLonghornSchedulable.

The gate lives in checkLonghornReady, so it applies to all of its
callers, not only the descheduler: ClusterStorageReadyForVolumes (volume
creation), WaitForLonghornReady (the CSI attach wait) and the
components-ready poll. All of them need the node's disks resolved before
replicas can be placed or attached.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit 638874a)
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube linuxkit package now ships a Go daemon in place of the
shell path. Adds a kube-init build rule that DOCKER_GO-invokes
`go build ./cmd/... .` inside pkg/kube/kube-init/, staged into
pkg/kube/dist/ before the linuxkit pkg build. pillar-* targets
are unchanged.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit a7cdcba)
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's vendor tree is upstream Go modules with well-known
licenses; the SPDX generator's per-file scan blows out on the
tree size without adding value.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit d1b6f47)
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube-init daemon is its own Go module, and make test enumerates
its modules explicitly, so none of kube-init's tests have ever been
executed by the target — the package ships a test suite that no CI
path runs. The container build only compiles the binary, and no
workflow names the module either, so the gap is invisible short of
noticing the absence in the log.

Add it alongside the other per-module invocations. The suite needs
./... because kube-init's tests live in its subpackages rather than
at the module root, unlike the newlog and edgeview entries above.
Runs green under -race in about 75 seconds.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 3ef0d4a)
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
domainmgr retries WaitForKubernetes forever while the KubeVirt CR is
not Available on single node and replicated storage clusters. KubeVirt
is only installed on amd64, so on arm64 eve-k domainmgr never gets past
startup, never subscribes to DomainConfig, and every app stays in
Booting.

Apps on those arches run as pods, so skip the KubeVirt readiness gate
there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
(cherry picked from commit e700961)
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A NOHYPER app on eve-k runs as a pod whose container image is loaded
locally into containerd (PullNever). The pod does not read the app's
PVC at all.

After a reboot, contentTreeSatisfiedByPVCs sees the app's PVC already
filled and accepts the content tree as REMOTELOADED, so the image is
never downloaded and ingested again. The volume adopts the PVC, the pod
is started and fails forever with ErrImageNeverPull.

Never accept a content tree from PVCs when a native container volume
references it, and don't take the PVC adoption fast path for such a
volume either: it waits for the image like before, and the CSI handler
reuses the existing, fully uploaded PVC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
(cherry picked from commit 1e3232b)
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
// https://github.com/lf-edge/eve/pull/5971.
httpsClient := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // lgtm[go/disabled-certificate-check]
@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 24.17%. Comparing base (1797593) to head (9cd7b72).
⚠️ Report is 1 commits behind head on 17.0-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           17.0-stable    #7058      +/-   ##
===============================================
+ Coverage        23.53%   24.17%   +0.63%     
===============================================
  Files              517      528      +11     
  Lines            95834    97450    +1616     
===============================================
+ Hits             22559    23555     +996     
- Misses           71358    72016     +658     
+ Partials          1917     1879      -38     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kicking off tests....

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Half of #6796 (pillar, not kube-init) never reached 17.0-stable since it depended on this go daemon. It makes sense to add those commits in this PR to avoid it getting forgotten (since it looks like it has already been backported).

rene and others added 2 commits November 16, 2026 09:42
When a node goes down, the virt-launcher pods of its VM apps stay in
Terminating and KubeVirt often does not create a replacement. After two
minutes checkAppsFailover() then scales the VMIRS to 0 and back to 1 to
get one scheduled, but only on the node for which isDecisionNode() is
true, which followed the eve-kube-stats-leader lease.

That lease runs with a 300s duration. If the node that died was the
stats leader, no survivor can acquire it for up to five minutes, and no
node is allowed to do the reset in the meantime. This was hit on a
3-node cluster where the powered-off node had been the stats leader for
two days: the pods went Terminating at 08:47:57 and the reset was due
from about 08:50:50, but dell-5810-2 only took the lease at 08:52:17,
reset the VMIRS at 08:53:04, and the replacement pod was created at
08:53:17, almost six minutes after the node was lost.

The eve-app-op election was added for exactly this: deciding the one
node allowed to act for an app whose node is down, with a 60s lease so
the handover never dominates the outage, and with the tie-breaker
excluded. Make isDecisionNode() follow it. The stats lease remains in
use for the stats collector, the node-prune sweep and the other
existing readers, none of which need a fast handover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
(cherry picked from commit 902a643)
checkAppsFailover() enters its "failover start" branch for every app
scheduled on this node that is not yet running, whether or not an old
copy of its pod is Terminating anywhere. For an app that is just
booting (pod in Init, for example) it then calls DetachOldWorkload()
with an empty node and app domain name, which returns at once with
"a node name is required", and logs "failover start/complete for
appDomainName: " with an empty name every 10 seconds. It also keeps
checkStuckPendingVMI() suppressed for that app although no failover is
landing on this node.

These spurious lines showed up on the failover node while debugging a
slow failover, and made the real failover activity harder to find.
Require a terminating pod before taking this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
(cherry picked from commit 007a6ed)
@rucoder

rucoder commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Half of #6796 (pillar, not kube-init) never reached 17.0-stable since it depended on this go daemon. It makes sense to add those commits in this PR to avoid it getting forgotten (since it looks like it has already been backported).

@eriknordmark good catch! I added missing commits to the PR

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants