Repository navigation
Conversation
During the single-node to multi-node transition, and on any later leader election such as a node failover, k3s restarts and the embedded etcd reforms. A transiently-behind kube-apiserver watch cache can then serve a reconnecting informer a successful but incomplete watch stream, silently dropping a window of events with no "410 Gone" to force a re-list. CoreDNS runs its kubernetes informer with resync disabled (resyncPeriod = 0), so the lost Service events are never backfilled: Services created in that window stay permanently NXDOMAIN, which breaks CDI upload and app volume deployment on the freshly formed cluster. Serving LIST/WATCH straight from etcd removes the stale-serve path: a reconnect either resumes cleanly or gets an honest 410 that forces a re-list. Ship a config.yaml.d drop-in that appends watch-cache=false to kube-apiserver-arg (the "+" suffix appends rather than replacing the base args), installed by cluster-init.sh only in edge-node-cluster mode so single-node keeps the cache. Edge clusters are small, so the cache buys little; the control-plane cost was benchmarked and found negligible at realistic load. Signed-off-by: Paul Gaiduk <paulg@zededa.com> (cherry picked from commit 27496ff)
Direct deps kube-init's typed client-go and containerd usage
relies on:
k8s.io/api v0.34.1
k8s.io/apimachinery v0.34.1
k8s.io/apiextensions-apiserver v0.34.1
k8s.io/client-go v12.0.0+incompatible (aliased to v0.34.1
via replace)
github.com/containerd/containerd/v2 v2.2.5
github.com/containerd/errdefs v1.0.0
k8s.io/kube-openapi's own go.mod aliases v0.31.0 as its module
version, but the repo never publishes that as a tag — the pseudo
below is what actually exists on the repo and is what our replace
directive points at.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 66f1559)
The primitives kube-init's FSM writes to persistent storage: - Marker: typed filesystem-path values (LonghornInitialized, NativeKubernetesMode, MultusInitialized, …). IsMarked / Mark / Unmark route through AtomicWriteFile so partial writes never leave a half-marker. - AtomicWriteFile: temp-file + rename atop the destination, with an fsync in between so the on-disk state matches the return value even across a crash. - Reboot: the daemon's controlled reboot path. Writes a marker naming the reason before invoking the reboot binary so the next boot can consult the reason without racing the boot sequence. - K3sKubeconfig / ContainerdSocket / SealedDirName: canonical paths every subsystem shares. Constants (not vars) so a subsystem cannot accidentally rewrite them at test time. - ToK8sName: kubernetes-node-name transform (lowercased device UUID). Load-bearing for cross-subsystem consistency — the FSM's node-uuid label, monitor's reapply, tiebreaker's node-lookup, and Longhorn's node-object all agree on this spelling. The /var/lib snapshot itself lives in its own commit; what is here are the primitives it is built on. state also carries the operator wait items: a named point the FSM will block at until released, used to hold a node mid-transition while it is inspected. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit f89ab6b)
A node that joins an HA cluster has its /var/lib overwritten with
cluster-mode state: different certs, a different node identity and a
different datastore. Converting it back to a standalone node is
therefore not a reconfiguration but a restore, so each node captures its
pre-cluster /var/lib during its own first boot, while it is still
standalone, and the conversion puts that tree back.
What is captured is node identity, not the whole tree: server/tls,
server/cred, the cluster token and agent/. Everything else under
/var/lib is re-derivable — the unpacked k3s bundle, CNI binaries and k3s
binaries alone account for ~419MB of a ~443MB tree and are reproduced by
the install path. kubelet/ is deliberately excluded: restoring it is
worse than not, since a stale cpu_manager_state breaks kubelet on the
next boot.
The datastore is the one file written continuously, so it cannot be
copied byte-for-byte underneath a live writer. It is captured with
SQLite's VACUUM INTO, which produces a single self-contained,
online-consistent file, so the snapshot is taken with k3s running and
nothing has to be stopped or quiesced to get a usable copy. Note this is
kine/SQLite rather than etcd: server/db/etcd/ is a vestigial stub on a
single-node server, so `k3s etcd-snapshot` does not apply.
Restoring is an overlay copy, which means anything the snapshot does not
contain survives it, and cluster-mode leftovers have to be removed
explicitly:
server/db/etcd a real etcd member directory next to a restored kine
datastore gives k3s two conflicting datastores, and
makes etcdClusterInitialized() report a bootstrapped
cluster that no longer exists.
state.db-wal/-shm SQLite treats a -wal beside a database as that
database's own journal and replays it on open, so a
cluster-mode journal feeds pages into the restored
file: it grows past its snapshot size,
integrity_check reports "invalid page number", and
k3s dies with "database disk image is malformed" on
every start with no way back. VACUUM INTO never
writes these, so removing them is unconditional.
Longhorn replicas replicas under /persist/vault/volumes belonging to
volumes the restored cluster knows nothing about
would otherwise be adopted by a single node that
cannot serve them.
Component markers are part of the restored state. The snapshot is taken
before they are written, so a restored tree reads "components installed,
not yet initialised" and the next boot re-runs initialisation against
the datastore it actually has.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit a96859d)
…ntainerd wait Everything that must be ready before k3s can start. The FSM sits in INSTALLING → STARTING_CTRD while this package runs; a failure here aborts the boot rather than starting k3s on a broken substrate. - Kernel modules loaded via modprobe (bridge, iptables_nat, iscsi_tcp, etc.). Absent kernel bits surface with the specific module name so operators can spot a kernel-config regression. - cpu_manager_state cleanup: kubelet sometimes leaves a stale cpu_manager_state under /var/lib/kubelet after a reboot with a changed --cpu-manager-policy. Deleting it on every boot before containerd starts is upstream-blessed cleanup. - Cgroup mount: /sys/fs/cgroup, kubepods.slice hierarchy. - iSCSI daemon start (needed for Longhorn's block-mode volumes). - Vault wait: pillar's vaultmgr signals "ready" via a fs marker; the daemon blocks here until the marker is present so k3s does not race the vault-mount. - EdgeNodeInfo wait: blocks until the pubsub subscription delivers the device identity (DeviceName / DeviceID). - Containerd start + reachability probe. Everything is idempotent — the FSM re-enters INSTALLING on recycle without expecting a clean state. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit c6521d1)
… boot First boot is the most CPU-hungry moment in a node's life and the one with the least to lose: k3s unpacks, imports images, applies every component manifest and waits for kube-system to converge, while no user workload is running yet. Steady state is the opposite — dom0 is capped so guests get the rest of the machine, and that cap is what the node must run under once it is serving. Rather than choose one of those, borrow for the duration. This computes the CPU set dom0 may use while bringing the cluster up, hands the control plane the CPUs that would otherwise sit idle, and returns them when first boot completes, so the steady-state cap is what survives. The set is derived rather than declared: the loan is bounded by what dom0_max_vcpus actually permits, so a node configured for a small dom0 does not end up with a control plane wider than its own vcpu budget. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 8803295)
…password, install Everything about running the k3s process: - binary.go / install.go: k3s binary download with SHA-256 verify + symlink rebuild for the multi-call subcommands (kubectl / ctr / crictl → k3s). Idempotent — a re-install picks up unchanged bytes cheaply. - config.go: drop-in files under /etc/rancher/k3s/config.yaml.d with numeric prefixes deciding merge order (00-nodename, 01-clusterconfig, 03-enc-disable-local-path, 99-user-override). Slot 02 is reserved. Byte-stable rendering keeps k3s's drop-in-hash restart trigger quiet on no-op reconciles. - paths.go: absolute paths for the k3s binary and every state file the daemon writes. - supervisor.go: Setpgid + orphan-sweep supervisor for the k3s process. Ports its own PID stays alive until the child's process group is fully reaped; port-listening probe rules out an outer-shell-still-running scenario before signalling "started". - token.go: k3s node-token rotation on cluster-mode change. - node_password.go: FixNodePasswordSecret clears a stale node-password secret when the file marker StaleNodePasswdFlag is present. Delete via typed clientset (kube-system Secrets). - readiness.go: WaitKubeconfig (poll for /etc/rancher/k3s/k3s.yaml with ctx cancel), then WaitReady blocks until Node's Ready condition + kube-system pods are Ready + node-uuid label is applied. Uses a client-go client built from the kubeconfig file — dial errors during the boot window are tolerated as "keep polling", so no shell-out is needed for the chicken- and-egg case of probing an API that isn't up yet. Every step is bounded by the caller's ctx (INSTALLING / STARTING_K3S / WAIT_K3S_READY handlers set the deadline). Restart hooks live under /etc/k3s-supervisor/hooks.d and fire between stop and start on operator-driven restart (RT PerformanceProfile changes are the canonical consumer). Bringing a node into a cluster retires its whole PKI: everything under server/tls goes, along with cred/ipsec.psk and the kubeconfigs derived from it. A kubeconfig embeds the CA, so one that outlives the CA it describes is not inert — the readiness wait would adopt it, build a client against an authority that no longer exists, and every call would fail the TLS handshake against a healthy apiserver. dynamic-cert.json is the file that bites hardest: it caches the whole kube-system/k3s-serving Secret and is preloaded before the datastore is up, so a survivor makes the supervisor serve a cert the adopted cluster CA cannot verify. The readiness wait is built to be diagnosable, because every distinct cause of "no answer from the apiserver" otherwise looks like the same hanging call. It records what the client is aimed at and the fingerprint of the CA it trusts, then on each failure separates the layers: whether anything is accepting on the API port, what /readyz says is outstanding (etcd, informer sync, a poststarthook), and whether the kubeconfig moved underneath the client — in which case the client is rebuilt rather than polled to death. Every call is individually bounded: rest.Config.Timeout is deliberately unset because it would cut informer watches, so a per-call context is what stops one unanswered request from consuming the whole readiness budget. Readiness is scoped to this node. Pod readiness is counted with a field selector for the local node, so a cluster peer that is unhealthy or mid-join cannot hold up a node that is itself ready. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 9f3cf92)
k3s version override + component-by-component upgrade driven by the KubeConfig pubsub subscription. Runs as a background loop during RUNNING; a version bump on the wire triggers download + SHA-verify + install of the new k3s binary followed by rolling component upgrades. - update.go: main loop. Pulls the desired k3s version from the KubeConfig subscription cache, downloads via curl into a /tmp path with a per-attempt timeout, SHA-256-verifies against the version-specific expected hash, atomically installs into /var/lib/k3s/bin/k3s. - cluster.go: checkClusterReady probes the API server via discovery.ServerVersion; componentIsInstalled probes each component's namespace via a typed Namespaces().Get, distinguishing "not installed" (NotFound → false, nil) from "cannot tell" (err → false, err) so a transient API outage isn't misread as "component gone". - k3s_update.go: k3s download / verify / install flow. - status.go: publishes update progress to zedagent via pubsub so the controller sees each component's version. Retry gate: the daemon consults the KubeClusterUpdateStatus subscription to skip retrying a version the controller has already reported as failed — matches the "one attempt per version" semantic pillar expects. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 6e6aabe)
…ault() Holds the typed clientset, apiextensions clientset, dynamic client, discovery + deferred RESTMapper, and shared informer factories (typed + dynamic). Constructed once at daemon boot from state.K3sKubeconfig and exposed via a package-level singleton (SetDefault / Default) so subsystems don't have to thread a *Client parameter through every function. Default() panics if consulted before SetDefault — that path indicates a lifecycle bug and should crash loud rather than nil-deref deeper. ResetMapper() invalidates the discovery cache so kubectlx.applyOne picks up newly-installed CRDs on retry. WaitForKubeconfig polls for state.K3sKubeconfig with ctx cancellation, exposed for callers that need to gate on file existence explicitly. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit c9144fa)
…ce layer Every operation the daemon performs against Kubernetes or containerd goes through this package. No argv assembly, no stderr parsing, no subprocess exec. apply.go — Apply / ApplyFile / ApplyURL / Get + DeleteFile / DeleteURL / DeleteBytes. Server-side apply via the dynamic client, multi-document YAML parsing, mapper-reset + retry-once on meta.NoKindMatchError closes the CR-before-CRD race in one place. Retry via client-go/util/retry.OnError with an apierrors-based classifier (transient vs permanent). Force- conflict-override is on by default (ApplyOptions.NoForce inverts the flag) because kube-init owns the components it applies; context.Canceled short-circuits so peer-cancel doesn't drag 30s+ of backoff sleeps. wait.go — WaitCRDEstablished / WaitDeploymentReady / WaitDaemonSetReady / WaitJobComplete / WaitForCondition, all informer-driven via watchtools.UntilWithSync. WaitForCondition evaluates a JSONPath against the target object on every event — CR-defined phase fields (KubeVirt / CDI status.phase, etc.) fit this shape. containerd.go — ContainerdClient wraps containerd.Client (v2) with namespaces.WithNamespace(k8s.io) baked into every operation. ImportImage / ImageExists / DeleteImage / ListImages / TagImage. A health probe at construction turns "containerd is down" into one clear error instead of per-operation failure spam. No cri-api dependency — CRI's image operations are containerd image-service operations under the k8s.io namespace, which the Go client already exposes. CRs are gated on their CRD actually being established rather than merely created: each CRD document is applied and awaited before the CRs that depend on it. Establishment is judged from the object, not from a bare condition read — a CRD that is terminating still reports Established=True, so a deletion timestamp disqualifies it. That closes the window where a CR would be applied against a CRD about to disappear. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 3745588)
…ffort retry loop A component starts the moment its declared deps are Ready — no per-wave synchronization barrier. A fast component whose deps are satisfied never waits for a slow unrelated peer. types.go — Component (Manifests + Apply + Ready + PolicyDeps + BestEffort + ReadyTimeout), Manifest (File / URL / Bytes), Edge (structural-derivation output tagged with rule: crd / namespace / webhook / sa / policy), RetryPolicy, RetryCallback. deploy.go — Graph + Run + Edges. plan() validates the graph and resolves edges. The scheduler runs a single-threaded coordinator loop that tracks launched/completed counters directly (a WaitGroup-based helper is a runtime panic when a completing goroutine's result enqueues children mid-drain). retry.go — Graph.RetryCtx is the daemon-scope context that keeps BestEffort background retries alive after Run's per-invocation ctx ends. Exponential backoff with jitter (defaults: 30s → 5min × 6 = ~30min budget). Re-arms Apply → Ready on Ready- timeout too, not just Apply-failure. Retries observe RetryCtx cancel (daemon shutdown). rules.go — placeholder for the four structural-dep rules (CRD→CR, Namespace→resource, webhook-Service→webhooked-resource, ServiceAccount→Pod-spec) that fire once callers populate Component.Manifests with real bytes. Today Manifests is empty across the shipped graph; ordering rides entirely on PolicyDeps. deploy_test.go — 25 tests covering plan validation, edge tagging, edge sorting, dependency-order execution, parallel- execution property (two-channel handshake, not sleep), error propagation with alphabetical head + rest, ctx cancellation, BestEffort semantics, Ready-only-on-Apply-success, MaxParallel concurrency cap, Apply=nil (Ready-only) shape, and four retry-loop tests (eventually-succeeds, exhausts, ctx-cancel-mid- loop, one-shot-without-RetryCtx). All pass -race across 50 iterations. Readiness is bounded by progress rather than by a fixed timeout, so a component that is still making headway is not cut off while one that has genuinely stalled does not hold the graph indefinitely. What the progress probe observed is logged, because "not ready yet" and "not ready and no longer moving" are the same line otherwise. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 9a5dbe0)
…nager Single-goroutine dispatch of pillar pubsub topics into per-topic subscriber caches. Modelled on pkg/pillar/cmd/monitor/subscriptions.go — one map of label → Subscription, deferred Activate, one goroutine pumping every topic via pubsub.MultiChannelWatch. Subscribers across kube-init packages (edgenodeinfo, encconfig, encstatus, kubeconfig, kcus) register their topics with the Manager during daemon boot; the Manager owns the connection and the pump goroutine. Registering after the pump has started is supported — new subscriptions activate lazily. Replaces file-polling under /run/zedagent/... / /persist/status/ zedkube/... — every subscriber owns its own decoded cache with the canonical pillar types (no local JSON re-declaration), so JSON-tag skew between pillar and kube-init cannot silently drop a field's value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 64570bd)
Six typed subscribers wired to the shared pubsubclient.Manager; one publisher for the daemon's own lifecycle status. - edgenodeinfo: EdgeNodeInfo subscription with WaitForFirst so the boot path can block on device identity (DeviceName, DeviceID). Exposes Get() for read-only consumers on the FSM loop. - encconfig: EdgeNodeClusterConfig subscription. Drives cluster shape (single vs HA), TieBreakerNodeID, cluster interface. - encstatus: EdgeNodeClusterStatus subscription. Present() folds both file deletion and the zero-UUID-delete sentinel into a single check so a cluster delete on the controller side is detected the same regardless of how zedkube signals it. - kubeconfig: KubeConfig subscription with K3sVersion accessor for the update loop. Tagged consistently with pillar's canonical json tags (capital K, no override). - kcus: KubeClusterUpdateStatus subscription gating upgrade retries. DestinationKubeUpdateVersion is uint32 (matching pillar), so a "cluster already tried and failed at version X" record correctly gates the next attempt. - kubeinitstatus: publishes the daemon's lifecycle events (transition-step, phase, etc.) so operators observing the controller see kube-init state transitions without needing to tail the kube container log. Every subscription decodes to pillar's canonical types (via pkg/pillar/types) — no local *JSON structs that can drift. The node also publishes what it is actually doing, so the controller and pillar can see the daemon's own view rather than infer it: the FSM state and phase, derived cluster membership (restoring, leaving, joining, member, join-pending or single), the current transition step, whether the join watchdog is armed, and the last error. Membership is derived from the markers and configuration on disk in one place, with a defined precedence, so two agents cannot disagree about whether this node is in a cluster. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit d0c1251)
First-boot import of pre-packaged image tarballs (KubeVirt, CDI, Longhorn, Multus, SUC, the EVE-authored external-boot-image) into the user-side containerd's k8s.io namespace, so kubelet pods reference images without a first-boot registry pull. - Import via kubectlx.ContainerdClient (containerd v2 Go client under the k8s.io namespace). - ImportUpstreamImage is a no-op when the image is already present or the tarball is absent; kubelet falls back to a registry pull for anything missing. - ImportExternalBootImage re-tags the EVE-authored image (whose tarball name isn't the runtime tag) to the running EVE release's tag, then cleans up prior EVE-authored images for the same base name via ListImages + DeleteImage. - Manifest.json is parsed via `tar` (fatal-line-buffered) with the RepoTags extraction in Go — no jq dependency. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 335af07)
…ainerd env Wires kube-init into pillar's cost-aware management proxy so in-cluster traffic (CDI's image importer, containerd's image pulls) routes through pillar rather than out the WAN. - cni0.go: SetupCNI0ProxyIP assigns a well-known anchor IP on the flannel-created cni0 bridge — CDI pods reach the management proxy at that IP without a per-node hostname lookup. Enabled() only fires when pillar's mgmtproxy config file is present. - The CDI CR's spec.config.importProxy fields (HTTPSProxy / noProxy) are reconciled every steady-state tick: a dynamic Get reads the current values, and a merge-patch fixes any drift. In-Go JSONPath extraction reads the field values — no `-o jsonpath=…` shell-out. - containerd-launch env injection is a separate concern (pkg/kube/kube-init/prereqs) and pre-dates this package. Wildcard-webhook rules (rules[].apiGroups: ["*"]) are treated as "no derived edge from this webhook" in the deploy planner — a catch-all webhook shouldn't sequence the whole graph. Not the concern of this package but noted here so the wiring is consistent. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 856c364)
… startup stagger Every state change between "this node is a single-node cluster" and "this node is a member of an HA cluster" — including the recovery paths — lives here. - transition.go: the ordered step runner. Single→cluster rotates the k3s node-token, applies encstatus-derived configuration, and updates supervisor args. Cluster→single reverts. Progress callback surfaces the current step name to the FSM's control socket so `k3s-sctl status` reports `transition-step=<step>`. - masterleases.go: stale etcd master-lease cleanup. After a single→HA conversion, etcd carries a lease from the pre-conversion single-node identity; the deploy graph's Longhorn readiness stalls if that lease's expiry sits ahead of the new node's registration. Cleanup deletes the stale lease from etcd directly (etcdctl invocation — outside the client-go surface because etcd doesn't ship a stable Go SDK). - startup_rank.go: on HA startup, computes the local node's rank (0-based position in the sorted control-plane node list) and sleeps rank × 25s before launching k3s. Fans out simultaneous power-cycles into a staggered sequence so etcd quorum re-forms cleanly without three nodes fighting for the leader lease at t=0. The multus reset performed during a transition is best-effort. The NetworkAttachmentDefinition CRD is cluster-scoped, so tearing it down to re-address one node would take the definition away from every other node in the cluster; the node re-applies its own configuration and restarts the DaemonSet instead, and a failure to do so is logged rather than aborting a transition that has already moved on. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 794a58a)
The steady-state loop the FSM enters after DEPLOYING. Every 15s health tick runs a fixed sequence of housekeeping + reconcile tasks; separate goroutines watch containerd exit, kubeconfig drift, log rotation, cluster-config-changed events, and the user-override drop-in for k3s config. Tasks the health tick runs (in order): - DHCP CNI daemon liveness (respawn on exit). - CNI plugin binary drift (copy from stage dir when needed). - Multus re-render + apply if the initialised marker is absent. - Node label + annotation reapply (node-uuid, longhorn disk config) when the labels-initialised marker is absent — one merge-patch does labels and annotations together, then a readback verifies both landed. - Debug-user RBAC re-application. - Kubeconfig sync (copy /etc/rancher/k3s/k3s.yaml to the kube-init-visible location, atomic). - Image reimport (external-boot-image via kubectlx. ContainerdClient — ImageExists check, tarball import only if the image is genuinely absent). - Longhorn post-install config (via components. LonghornPostInstallConfig). - Registration manifest reconcile. transition.go — countReadyNodes counts Nodes with Ready=True via typed clientset. Used by the cluster-mode monitor to detect "we're on the HA majority" transitions. parseTransitionMarker covers the FSM's cluster-transition marker file (unix-ts + reboot-count). Steady-state ticks are gated on state markers so a tick after a successful bootstrap is silent — no log line, no API call. The join watchdog is driven by progress, not by a deadline. How long a join takes is unknowable — it covers a k3s restart, an image import, registration and every kube-system pod — so any budget for it is a guess that some slower device will fail. How long a join may make no progress at all is a stable quantity, so every advance to a stage this join has not reached before kicks the timer: a slow-but-advancing join is never rebooted, while a wedged one trips promptly. A crash loop does not count as progress, because it revisits stages rather than advancing. The cluster-config watch runs in every state, not only in RUNNING. A controller can withdraw a node's cluster configuration at any moment, including seconds after a join completed while the node is still working through readiness. A zero-UUID or otherwise empty status is treated as "no live cluster" only when it really is one, so a status that arrives late or half-written cannot be mistaken for a withdrawal. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 3355cc4)
The tie-breaker is the third control-plane node in an EVE-K HA cluster whose only job is to give etcd a third vote so single- node loss doesn't strand the cluster. This package configures that role. ConfigApply is the phase entry point; it runs when the cluster first reaches three nodes AND the local node is the designated tie-breaker (per encconfig's TieBreakerNodeID). Steps: - nodesConfigApply — label every node with tie-breaker-node= true/false and cordon/uncordon accordingly. Merge patches via Nodes().Patch keep both label and spec.unschedulable updates atomic per node. - kubevirtConfig — scale virt-operator + KubeVirt CR to 2 replicas (tie-breaker excluded). Deployment scale via typed AppsV1 merge-patch; CR replica field via dynamic Resource Patch. - kubevirtTieBreakerConfigApply — patch every DaemonSet in the kubevirt namespace with a nodeSelector keeping pods off the tie-breaker. - cdiConfig — same nodeSelector patch on every Deployment in the cdi namespace. - longhornNodeSetSched — flip allowScheduling + evictionRequested on the tie-breaker's longhorn.io Node CR and every disk it owns. JSON-patch operations via dynamic Resource Patch. - longhornRescale — scale Longhorn CSI sidecar Deployments to 2 replicas; patch every longhorn-system DaemonSet with the nodeSelector (retries 5× because longhorn-manager occasionally races us during install). - drainNode — cordon + evict every non-DaemonSet pod on the tie-breaker via policy/v1.Eviction (matches kubectl drain --ignore-daemonsets --delete-emptydir-data --force semantics) + wait until pods leave the node. - StatusSet — stamp every node with tie-breaker-config-applied=1 as the "phase complete" marker so a subsequent reboot skips the whole flow. Every operation is idempotent — a partial ConfigApply that gets interrupted mid-way completes on the next FSM entry. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit ee13ef0)
…eduler, RBAC, SR-IOV, kube-vip
Everything the deploy graph runs during DEPLOYING and everything
the monitor's steady-state ticks reconcile against.
components.go — DeployAll builds a deploy.Graph and runs it. The
graph has ~10 Components; only Longhorn declares a PolicyDep
(longhorn depends on manifests, which stages storage-classes.yaml
into the auto-deploy dir). Every Install* function:
- Longhorn: applyLonghornDiskConfig labels + annotates the node
via one merge-patch; the longhorn config YAML applies as a
drop-in.
- KubeVirt: InstallKubeVirt applies the operator manifest via
SSA, waits for virt-operator Ready, applies the CR via SSA,
applies feature-gate CR via SSA. kubeVirtConfigReplicas
writes .spec.replicas + .spec.infra.replicas via SSA with a
distinct field manager ("kube-init-replicas") so subsequent
applies of the operator manifest don't fight for ownership.
- CDI: apply operator, wait for cdi-operator Ready, apply CR.
WaitCDIReady goes through kubectlx.WaitForCondition on
.status.phase == "Deployed".
- Multus: template rendering (IP address substitution) then
applyMultus + linkMultusIntoK3s.
- descheduler / debug-rbac / manifests / kube-vip / SR-IOV
manifest staging / stale-mount-cleanup daemon launcher —
one Component each.
- WaitLonghornReady is informer-driven (via a wait-scoped
dynamic factory over longhorn.io/Node and typed DaemonSets in
longhorn-system). No polling; the compound predicate is
re-evaluated on every event.
kubevirt.go — MigrateKubeVirtFeatureGates reads the running CR
via dynamic Get, patches with dynamic Patch, marks the migration
done. buildFeatureGatesPatch is table-driven so a new gate is a
one-line addition.
longhorn.go — LonghornIsReady is the compound predicate the
monitor consults on every tick. Uses typed clientsets for
DaemonSet readiness and dynamic client for the Longhorn Node CR
and engine images. longhornEngineDeployedOnNode distinguishes
"status.nodeDeploymentMap absent" (fresh CR — not-ready-yet)
from "map present but this node is false" (recycle engine +
manager pods to force reconcile). CheckOverwriteNsmounter uses
client-go remotecommand.NewSPDYExecutor (via podExec helper) to
copy a fixed nsmounter binary into every longhorn-csi-plugin
pod that doesn't already carry the marker file.
registration.go — RegistrationApplyIfReady stages the
controller-supplied registration manifest into the k3s
server-manifests dir. RegistrationApplied checks the AddOn CR
via dynamic Get.
uninstall.go — the K3sBase conversion flow. Uninstalls
descheduler / Longhorn / CDI / KubeVirt / Multus in reverse
dependency order. Longhorn uninstall is bounded by a 1000-poll
budget (~83 min) covering the worst-case data-shred workload;
timeout aborts the conversion rather than declaring
NativeKubernetesMode with volumes half-shredded.
sriov.go / stale_mount_cleanup.go — hardware-conditional
manifest staging and background cleanup daemon launcher.
Every Install has an idempotence gate on a state.Marker; a
re-entry to DEPLOYING or a re-tick of the monitor is a cheap
noop when the marker is already set.
Longhorn's readiness is taken from its instance managers rather than
from the deployment alone: the CSI plugin can report available while the
per-node instance managers are still coming up, and a volume attach in
that window fails. Multus is restarted through its DaemonSet, so a
re-applied configuration is actually picked up by the pods already
running.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit e3cef7c)
Launches a virtctl vnc proxy on demand for a KubeVirt VMI and tears it down when the caller process exits. Caller-PID watchdog polls /proc/<pid> so an orphaned proxy from a disconnected debug session doesn't linger. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 8d45ff2)
14 states, 27 events. The daemon boots into INIT, walks through INSTALLING → STARTING_CTRD → CONFIGURING → STARTING_K3S → IMPORTING (first-boot only) → WAIT_K3S_READY → DEPLOYING (first- boot only) → RUNNING, and handles restart / recycle / cluster- mode-transition events without dropping back into an ad-hoc respawn loop. State entries are `enterStateFn` dispatched from `transition`; async work runs on a per-attempt workCtx via startAsync so transition cancellation drains the work goroutine cleanly. The control socket at /run/k3s-supervisor.sock accepts `status`, `restart`, `stop`, `graph` commands; every operator-facing k3s- sctl invocation dispatches through the socket rather than touching flag files. initKubeclient wires kubeclient.Default at WAIT_K3S_READY's transition (test-safe when the kubeconfig file is absent — transition-table tests exploit this). components.DeployAll takes a retryCtx that main.go supplies as d.rootCtx, so BestEffort background retries in the deploy graph survive per-invocation workCtx while still dying on daemon shutdown. Restart-reason is a typed field carried through the STOPPING_K3S → RUNNING_HOOKS → STARTING_K3S transition so restart hooks (/etc/k3s-supervisor/hooks.d) fire once for operator-driven restarts and never for crash recovery. The RT operator's PerformanceProfile-change flow depends on this — CPU isolation / RDT partitioning need to be reconfigured between stop and start, not on every crash. README.md sketches the architecture, package boundaries, and the future-work list (bootstrap-cert fingerprint pinning, config-package retirement, and so on). Two behaviours in the loop are worth calling out. A cluster transition that fails retries in place instead of falling through to a state that assumes it succeeded — falling through leaves a node half-transitioned, which is not recoverable by pushing new configuration. And images are imported on every k3s start rather than only on first boot, because a restart can find the content store missing entries that the running cluster still references. The operator breakpoints are plumbed through here: a wait item placed in the FSM's path lets a test or an engineer hold a node at a chosen point and release it, which is how the multi-node transitions are inspected on real hardware. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 9abf8cc)
One-shot tool that dials /run/k3s-supervisor.sock and prints the
reply. Reads until EOF so multi-line responses (graph) print
naturally alongside single-line ones (status / restart / stop).
K3S_SUPERVISOR_SOCKET overrides the path for tests.
Exit code follows the last line: non-zero when it starts with
ERR.
Subcommands:
- restart — graceful k3s restart (pre-restart hooks fire).
- status — one-line status report (state, k3s pid, uptime,
backoff, restart count, phase, transition-step).
- stop — stop the kube-init daemon.
- graph — resolved deploy-graph edges (one per line, each
tagged with its rule source).
Breakpoints are managed from the same CLI (break set / clear / list), so
holding a node at a chosen point does not require reaching into
/persist by hand.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 933cef7)
…helper pkg/kube/Dockerfile — the kube linuxkit container's entrypoint is /usr/bin/kube-init. Everything else the container needs to ship (k3s multi-call binary, longhorn nsmounter, CNI plugin staging, longhorn/kubevirt/CDI/multus tarball drops under /images/) is installed via the existing build.yml conventions. pkg/kube/k3s-control.sh — small wrapper that dispatches to k3s-sctl. Kept so external tooling (eden / eve exec kube …) that used to invoke this path continues to work; every subcommand it takes now goes over the control socket rather than touching flag files. pkg/kube/kubevip-delete.sh — minimal helper that KubeVIPDelete calls to remove leftover kube-vip state files after component deletion (leftover static-pod manifests, arp cache). Not part of the deploy graph; runs from the uninstall path. sqlite is included in the image because the datastore snapshot is taken with SQLite's VACUUM INTO, which is what makes an online-consistent copy possible without stopping k3s. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit 7487742)
Eleven files under pkg/kube/ implemented the previous cluster-init flow in POSIX shell + kubectl. Every operation they exposed lives in pkg/kube/kube-init/ now — no external caller depends on them, and keeping them alongside the Go daemon would leave two divergent implementations of the same paths that operators could accidentally invoke. Deleted: cluster-init.sh cluster-update.sh cluster-utils.sh descheduler-utils.sh kubevirt-utils.sh longhorn-utils.sh pubsub.sh registration-utils.sh tie-breaker-utils.sh utils.sh vnc-proxy.sh lib/config.sh Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit c0cfc4b)
…stDevices
The KubeVirt CRD requires spec.configuration.permittedHostDevices.
{pciHostDevices,mediatedDevices,usb} to be arrays. Bare keys
parse as `null` in YAML; server-side apply forwards `null` as an
explicit set-to-null, which the schema rejects with:
KubeVirt.kubevirt.io "kubevirt" is invalid:
spec.configuration.permittedHostDevices.mediatedDevices:
Invalid value: "null": ... must be of type array
Explicit `[]` satisfies the schema.
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
(cherry picked from commit 3b3e9aa)
The KubeVirt image catalog pinned the five virt-* images at v1.6.0 while every other declaration of the version says v1.7.3: the CR URL applied by InstallKubeVirt, the operator manifest that names the image and sets KUBEVIRT_VERSION, and expected_versions.yaml, which drives the update-component drift check. A node pre-loading v1.6.0 would never satisfy a v1.7.3 pod spec, so the operator's pods would pull from quay anyway and the pre-load would buy nothing. The mismatch is currently latent: no rule produces the /images/*.tar tarballs yet, and a missing tarball is a logged no-op that defers to kubelet's pull-on-first-use. Correcting it now keeps the catalog honest for whenever the tarballs start shipping. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 84b63ae)
Component versions were spread across five files: the Longhorn config path baked the version into a filename, the KubeVirt CR URL inlined it, CDI and k3s each had their own const, and the pre-loaded image catalog repeated all of them once per image. Nothing tied those copies together, which is how the KubeVirt tags came to disagree with the operator manifest. Introduce a leaf versions package holding one const per component and derive every other site from it. All const expressions, so this is a compile-time rearrangement with no runtime change. The k3s const stays a default rather than an authority: the controller's k3s.version config item still overrides it at runtime. Two declarations remain outside the Go tree and cannot reference the package — expected_versions.yaml and the vendored kubevirt-operator manifest — so the package doc names them as sites a bump must also touch. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit a0dd81e)
A device converted to native Kubernetes under an older EVE image records that in a marker whose name has since changed. The rename ran from main(), before the prereqs pass bind-mounts /var/lib, so it inspected the container's own empty directory, found nothing, and returned successfully having done nothing. The real marker was still under its legacy name once the mount appeared, so the daemon read the device as un-converted and reinstalled KubeVirt and CDI on a node that had deliberately had them removed. Run the migration inside the init work step instead, right after the prereqs pass returns and well before the marker is read. A failure now returns rather than calling log.Fatalf: at this point the daemon has a state machine that can back off and retry, which it did not when the call ran from main(). Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6f72bdd)
zedagent holds the device in BaseOsUpdating for as long as the cluster update status names an unfinished component. A k3s update reports its own step complete and then reboots; on the way back up the component pass finds everything already converged, returns early, and publishes nothing. The status stays parked on the k3s step and the device reports BaseOsUpdating indefinitely, which the shell avoided by advancing the status itself on that path. Advance it the same way: on the converged path, a status sitting on a completed k3s step is retroactively finished. Both halves of that condition matter — a status naming another component, or naming k3s at any stage short of completed, belongs to a pass still in flight and is left alone. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 259f9de)
The multus manifest declares the NetworkAttachmentDefinition CRD and an instance of that kind in the same file, so applying it depends on ApplyFile waiting for establishment between documents. Nothing checked that the manifest still ships both halves: drop the instance or rename the CRD and the ordering silently stops mattering, or silently starts failing. Read the manifest and assert both. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit ec4c168)
kubeclient set no rest.Config.Timeout, because that timeout also cuts open watch streams. Without it, an API call whose context has no deadline waits forever on an apiserver that accepts the connection and never answers. Give the request clients a 90s Timeout. This is longer than the apiserver's own 60s request timeout, so it only stops a call that gets no answer. Add watch clients without the timeout. Use them for the informer factories, the Longhorn wait informers and the kubectlx waits. The discovery client keeps the 32s default that client-go applies when the config has no Timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit a32ead2)
podExec runs from the monitor tick, whose context has no deadline. kubeclient's requestTimeout does not cover exec, so a stuck exec can stop the health checks. Give each podExec a 60s deadline. The SPDY executor ignores the context while it waits for the upgrade response, so try the WebSocket executor first. It applies the deadline to the handshake too. SPDY stays as the fallback for an apiserver that refuses the WebSocket upgrade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit ed0b52e)
On a single-node EVE-k device, k3s stores its state in sqlite through kine. kine keeps lease expiry timers in memory and, at every k3s start, sets each key's expiry to the start time plus its lease. With event-ttl at 72h, a device that reboots more often than every three days never expires an event. On sc-sm-e300-10, which reboots several times a day, 26 days of events accumulated: 295600 of 298249 rows and 236 MiB of the 644 MB datastore. kine lists scan the whole name index, so every list and every compaction gets slower as the table grows, and listing the events no longer finishes within two minutes. A 1h TTL is shorter than a typical uptime, so events expire within an hour of each k3s start, including those carried over from before a reboot. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj (cherry picked from commit 01073b8)
Lowering event-ttl only affects events written afterwards: kine stores each event's lease in its row and re-arms expiry from that lease at every k3s start. On a device that reboots more often than every three days, events written under the former 72h TTL therefore never expire, and an upgraded device keeps its whole backlog. Before starting k3s, kube-init now deletes the events that carry the 72h lease and vacuums the datastore if any were removed. Events from a user override with a different TTL are left alone. On sc-sm-e300-10 the same deletion, done by hand, removed 297835 events in 42 s and shrank the datastore from 644 MB to 81 MB. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj (cherry picked from commit 6babf29)
17.0-stable added two shell-era helpers that master never had, because master already ran kube-init when the matching fixes landed: - cluster-status-probe, the client half of the authenticated cluster status server (lf-edge#6753). kube-init's k3s/statusauth.go derives the same credentials in-process. - cni-state-utils.sh, which clears the retired pod subnet when a node joins a cluster (lf-edge#6727), with its host-side test and the `make -C pkg/kube test` target that ran it. kube-init's clustermode does the same cleanup. Nothing in the kube container runs either one any more; remove them, the Dockerfile stage that built the probe, and the Makefile and README references. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Three earlier 17.0 backports carried master commits whose pillar half names kube-init as the other side of a contract, and reworded those comments to name cluster-init.sh or cluster-status-probe, which ran on this branch then: - lf-edge#6584 (master 5327945): mgmtproxy's cni0 token and TLS cert, read by kube-init's mgmtproxy.PatchCDIProxyConfig. - lf-edge#6753 (master 042622f): the cluster status server credentials, derived again in kube-init's k3s/statusauth.go. - lf-edge#6917 (master a407274): the controller-restart marker, written by kube-init's clustermode.ControllerRestartPending. kube-init now owns all three, so restore master's wording. Comments, docs and one test failure message only; no code change. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's go.mod says go 1.25.12, the toolchain master's eve-alpine ships. 17.0-stable's eve-alpine has Go 1.25.11, so the kube image build stops with "go.mod requires go >= 1.25.12 (running go 1.25.11; GOTOOLCHAIN=local)". Nothing needs 1.25.12: the go directive is the minimum language version, every other module in the tree declares go 1.25.0 on both master and 17.0-stable, and no kube-init dependency asks for more than 1.25.0. Declare the same. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init came over from master with pillar pinned at a master commit, which is not on 17.0-stable. Pin pillar at the current 17.0-stable head instead, using go get, go mod tidy and go mod vendor only. kube-init uses nothing from pillar that 17.0-stable lacks, so no pillar change has to land first. go get also moves eve-api, otel, logr and genproto to the versions this pillar requires. tidy marks golang.org/x/sys as a direct dependency, since kube-init imports it itself. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The instance-manager gate is satisfied as soon as its CR reports Running, often seconds after the pod starts -- before Longhorn's disk monitor has ever run for that node. For a v1 (filesystem) disk, that monitor itself requires a running instance-manager, so a fresh one can pass the existing gate while Status.DiskStatus stays nil indefinitely, since nothing retriggers the monitor once the window passes. Observed live: a descheduler-driven VMI move landed a volume on a node 4 seconds after its instance-manager started, and the replica controller failed forever with "cannot find disk name for replica" -- even though longhorn-disk.cfg on disk held the correct UUID throughout. Add a gate requiring every disk in Spec.Disks to have a Status.DiskStatus entry with a recorded DiskUUID, and at least one of them to have a True Ready condition, the exact condition that stayed unmet. Only one ready disk is required so that a recorded but degraded disk does not strand a node that can still host replicas, matching the any-disk semantics of checkLonghornSchedulable. The gate lives in checkLonghornReady, so it applies to all of its callers, not only the descheduler: ClusterStorageReadyForVolumes (volume creation), WaitForLonghornReady (the CSI attach wait) and the components-ready poll. All of them need the node's disks resolved before replicas can be placed or attached. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 638874a) Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube linuxkit package now ships a Go daemon in place of the shell path. Adds a kube-init build rule that DOCKER_GO-invokes `go build ./cmd/... .` inside pkg/kube/kube-init/, staged into pkg/kube/dist/ before the linuxkit pkg build. pillar-* targets are unchanged. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit a7cdcba) Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
kube-init's vendor tree is upstream Go modules with well-known licenses; the SPDX generator's per-file scan blows out on the tree size without adding value. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> (cherry picked from commit d1b6f47) Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The kube-init daemon is its own Go module, and make test enumerates its modules explicitly, so none of kube-init's tests have ever been executed by the target — the package ships a test suite that no CI path runs. The container build only compiles the binary, and no workflow names the module either, so the gap is invisible short of noticing the absence in the log. Add it alongside the other per-module invocations. The suite needs ./... because kube-init's tests live in its subpackages rather than at the module root, unlike the newlog and edgeview entries above. Runs green under -race in about 75 seconds. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 3ef0d4a) Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
domainmgr retries WaitForKubernetes forever while the KubeVirt CR is not Available on single node and replicated storage clusters. KubeVirt is only installed on amd64, so on arm64 eve-k domainmgr never gets past startup, never subscribes to DomainConfig, and every app stays in Booting. Apps on those arches run as pods, so skip the KubeVirt readiness gate there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br> (cherry picked from commit e700961) Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A NOHYPER app on eve-k runs as a pod whose container image is loaded locally into containerd (PullNever). The pod does not read the app's PVC at all. After a reboot, contentTreeSatisfiedByPVCs sees the app's PVC already filled and accepts the content tree as REMOTELOADED, so the image is never downloaded and ingested again. The volume adopts the PVC, the pod is started and fails forever with ErrImageNeverPull. Never accept a content tree from PVCs when a native container volume references it, and don't take the PVC adoption fast path for such a volume either: it waits for the image like before, and the CSI handler reuses the existing, fully uploaded PVC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br> (cherry picked from commit 1e3232b) Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
| // https://github.com/lf-edge/eve/pull/5971. | ||
| httpsClient := &http.Client{ | ||
| Transport: &http.Transport{ | ||
| TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // lgtm[go/disabled-certificate-check] |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 17.0-stable #7058 +/- ##
===============================================
+ Coverage 23.53% 24.17% +0.63%
===============================================
Files 517 528 +11
Lines 95834 97450 +1616
===============================================
+ Hits 22559 23555 +996
- Misses 71358 72016 +658
+ Partials 1917 1879 -38 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
eriknordmark
left a comment
There was a problem hiding this comment.
Half of #6796 (pillar, not kube-init) never reached 17.0-stable since it depended on this go daemon. It makes sense to add those commits in this PR to avoid it getting forgotten (since it looks like it has already been backported).
When a node goes down, the virt-launcher pods of its VM apps stay in Terminating and KubeVirt often does not create a replacement. After two minutes checkAppsFailover() then scales the VMIRS to 0 and back to 1 to get one scheduled, but only on the node for which isDecisionNode() is true, which followed the eve-kube-stats-leader lease. That lease runs with a 300s duration. If the node that died was the stats leader, no survivor can acquire it for up to five minutes, and no node is allowed to do the reset in the meantime. This was hit on a 3-node cluster where the powered-off node had been the stats leader for two days: the pods went Terminating at 08:47:57 and the reset was due from about 08:50:50, but dell-5810-2 only took the lease at 08:52:17, reset the VMIRS at 08:53:04, and the replacement pod was created at 08:53:17, almost six minutes after the node was lost. The eve-app-op election was added for exactly this: deciding the one node allowed to act for an app whose node is down, with a 60s lease so the handover never dominates the outage, and with the tie-breaker excluded. Make isDecisionNode() follow it. The stats lease remains in use for the stats collector, the node-prune sweep and the other existing readers, none of which need a fast handover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br> (cherry picked from commit 902a643)
checkAppsFailover() enters its "failover start" branch for every app scheduled on this node that is not yet running, whether or not an old copy of its pod is Terminating anywhere. For an app that is just booting (pod in Init, for example) it then calls DetachOldWorkload() with an empty node and app domain name, which returns at once with "a node name is required", and logs "failover start/complete for appDomainName: " with an empty name every 10 seconds. It also keeps checkStuckPendingVMI() suppressed for that app although no failover is landing on this node. These spurious lines showed up on the failover node while debugging a slow failover, and made the real failover activity harder to find. Require a terminating pod before taking this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br> (cherry picked from commit 007a6ed)
@eriknordmark good catch! I added missing commits to the PR |
Description
Backport kube-init, the Go daemon that replaced
cluster-init.shon master (#5971), to 17.0-stable. 17.0-stable still runs the shell version. Every kube fix on master since #5971 lands in kube-init, so each one has had to be rewritten in shell to reach 17.0: #6584, #6727, #6753, #6832, #6886, #6887 and #6917 all were. 17.0 also misses the kube-init fixes nobody rewrote. With kube-init on 17.0-stable, master's kube fixes cherry-pick as they are.Backport of #6116, #5971, #6309, #6449, #6456, #6499, #6507, #6542, #6590, #6599, #6614, #6652, #6665, #6702, #6704, #6755, #6771, #6794, #6796, #6860, #6861, #6897, #6934
This is a single PR. The earlier plan (#7055 and a re-vendoring PR first) is dropped together with #6697, which is not being backported: kube-init does not use cnirpc or anything #6697 added.
What it contains, in order:
make testruns kube-init's unit tests,.spdxignoreskips its vendor directory, andmake rungives the QEMU VM 8 vCPUs instead of 4.eve-app-oplease instead of the 300s stats lease, so failover is not held back for up to five minutes when the dead node was the stats leader.cluster-status-probeandcni-state-utils.sh, the shell-era helpers that [17.0-stable] evetest: port scans; fix the exposure they found on EVE and EVE-k #6753 and [17.0-stable] kube: clear the retired pod subnet when a node joins #6727 added on 17.0 only; kube-init does both jobs in-process.go 1.25.0in kube-init's go.mod, like every other module: master'sgo 1.25.12does not build with 17.0-stable's Go 1.25.11.go get,go mod tidyandgo mod vendor. It came over pinned at a master commit.Not included:
tools/bump-and-commit.shfinds no package hash to update.Backported commits
Cherry-picked with
-xfrom master, grouped by master PR:#6116
#5971
#6309
#6449
#6456
#6499
#6507
#6590
#6542
#6599
#6614
#6652
#6665
#6704
#6755
#6771
#6794
#6796 (zedrouter commit already on 17.0-stable via #6797)
#6861
#6702
#6860
#6897
#6934
17.0-only commits, not cherry-picks:
How to test and validate this PR
Automated, in CI:
make testnow runs kube-init's unit tests, and the eve-k build compiles the daemon.Done locally:
make pkg/kubeandmake pkg/pillarbuild.-race(the commandmake testruns). The pillar unit tests for kubeapi, volumemgr, domainmgr and zedkube pass, including the new Longhorn disk-status and native-container tests and eve-k: fix VM failover held back by the stats lease and a false portmap overlap #6796'sTestIsDecisionNodeFollowsAppOpLease.go vet -tags k ./...on pillar is clean.tools/bump-and-commit.shfinds no package hash to update.Still to do on eve-k hardware (amd64 and arm64):
cluster-init.sh) to this build: kube-init takes over the existing k3s state without reinstalling, and the apps keep running.Changelog notes
eve-k: cluster bring-up and update now run in the Go kube-init daemon instead of
cluster-init.sh, as on master. It brings the fixes made to kube-init on master since, plus Longhorn readiness and arm64 app deployment fixes.Fixed VM app failover on eve-k clusters being delayed by up to five minutes when the failed node was the cluster's stats leader.
Checklist