Repository navigation
2-Node HA, part 1: witness as the third etcd voting member - #6933
Conversation
bf4eedb to
d5a7b50
Compare
d5a7b50 to
aba9c7c
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6933 +/- ##
==========================================
+ Coverage 29.11% 29.53% +0.42%
==========================================
Files 548 560 +12
Lines 99033 101017 +1984
==========================================
+ Hits 28830 29834 +1004
- Misses 67934 68950 +1016
+ Partials 2269 2233 -36 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4e26206 to
8fd3d67
Compare
|
@andrewd-zededa All the requested changes have been addressed. Thanks for the review. |
andrewd-zededa
left a comment
There was a problem hiding this comment.
A couple more items, thanks for the other fixes.
|
@milan-zededa This PR conflicts with master and needs a rebase. Its last Fuzz Tests run was cancelled at the 6h job limit while building |
8fd3d67 to
89560ae
Compare
89560ae to
6435d1d
Compare
|
@eriknordmark @andrewd-zededa Rebased and addressed the latest review comments. |
6435d1d to
e2900da
Compare
This comment has been minimized.
This comment has been minimized.
1ba82bb to
9569e13
Compare
|
@andrewd-zededa If you have no further objections, please approve this PR so we can move forward with running tests and merging it. |
9569e13 to
4acc1e1
Compare
taking another look |
andrewd-zededa
left a comment
There was a problem hiding this comment.
@milan-zededa I think this is my last finding
| case EvError: | ||
| d.lastError = ev.Err | ||
| if role.IsWitness() { | ||
| witnessstatus.SetError(ev.Err, d.witnessIP) |
There was a problem hiding this comment.
SetError only covers a readiness failure. A crash after the witness has joined takes the EvK3sExited path and never reaches this, so the status is not updated for the common failure.
There was a problem hiding this comment.
Good catch, thanks. Fixed in d78e5e0: an unexpected k3s exit in STARTING_K3S, WAIT_K3S_READY or RUNNING now publishes the witness error via a new reportWitnessK3sExit helper. The error clears when the next start publishes the joining state. Also rebased on the latest master.
NIM only created a Linux bridge for a port that needed DHCP or a static IP, was a VLAN parent, and so on. A port designated purely as the kubernetes cluster interface has no L3 config of its own, so it was never bridged, and nothing could be attached to it. Mark that port with Adapter.IsClusterPort and have isAdapterBridgedByNIM honour it. On top of that, add a Veth item to the DPC reconciler. One peer is enslaved to an adapter bridge, the other is moved into the network namespace of a named containerd system-service container and given an address. NIM uses it to interconnect the witness container's own netns with the cluster interface bridge on the node owning JoinServerIP, assigning WitnessIP from EdgeNodeClusterStatus. The intent follows BootstrapNode, so the interconnect moves with the witness and needs no separate teardown. Firewall rules are installed through the reconciler alongside it: INPUT accepts etcd traffic to and from WitnessIP on 2379:2380, FORWARD accepts it across the cluster bridge before the L3 forward deny, and a connmark distinguishes the witness's contribution to etcd traffic from the nodes' own. zedrouter subscribes to EdgeNodeClusterStatus to learn the resolved cluster interface name, so a single-port Switch NI on that port is recognised as NIM-bridged, and the cluster port is rejected from multi-port Switch NIs, which would otherwise take ownership of the bridge the witness veth depends on. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A two-node-HA cluster has only two physical nodes able to host a
Longhorn replica, so the default 'longhorn' StorageClass's
numberOfReplicas=3 can never be satisfied: every volume sits
permanently "degraded" instead of ever reaching "healthy", even
when both replicas are fully in sync.
GetSupportedReplicaCountForCluster already reduces to 2 replicas
for the older tie-breaker-node label case; extend it to also do so
whenever EdgeNodeClusterConfig.WitnessIP is set, i.e. a
2-physical-node witness cluster. Plumb this from volumemgr, which
already needs EdgeNodeClusterConfig at startup to gate the Longhorn
readiness wait: add a VolumeMgr.IsTwoNodeHACluster() accessor,
implemented by reading the (now persistent, not one-shot) subscription
on demand via Get("global"), so csihandler.go's CreateVolume can pass
the right flag into GetSupportedReplicaCountForCluster per volume.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The witness is this same daemon running in a second container: an etcd-only k3s that gives a two-node cluster its third quorum vote, and nothing else. No kubelet, no apiserver, no scheduler, no workloads. Both containers bind-mount the host /run and /persist, so every fixed runtime path had to become role-dependent: the pubsub agent name, which also fixes /run/<agent>/ and the agent socket; the control socket; the supervisor pid file; and the k3s and install logs. The control socket mattered most, since listenSocket removes a stale socket before binding, so whichever process started last would have silently answered k3s-sctl for both. k3s-sctl picks its default the same way, so a shell in the witness container reaches the witness. The role comes from /etc/kube-init-role, which the witness image carries and the kube image does not, in the shape of /etc/eve-hv-type. Not a flag: these paths live in package-level vars initialised before main can parse anything. Reading it from the image also reaches the tools beside the daemon, which never get a command line of their own to carry the role, and k3s-sctl above all: /run is shared, so both supervisor sockets are reachable from either container, and a wrong guess would answer for the other daemon rather than fail. It is the only mechanism, so there is nothing for it to disagree with. The orphan sweep is a bug in its own right, independent of any witness. It walked all of /proc, matched cmdline only and SIGKILLed every hit, and this container runs with pid: host. PID namespaces are hierarchical, so a k3s in another container is fully visible here and a signal from this ancestor namespace reaches it. It fires at the end of every stop, not just during recovery, so an ordinary restart would have killed the witness and silently cost a quorum vote. Candidates are now confined to our own PID namespace, and an unreadable namespace counts as foreign: leaving a stray orphan is recoverable, killing the witness is not. The witness gets a reduced state set. It skips STARTING_CTRD, since with --disable-agent there is no kubelet to dial a CRI socket; IMPORTING, since it runs no workloads; and DEPLOYING and SNAPSHOT, since it installs no components. Its prerequisites shrink to logging, a route to the seed, and its own /var/lib behind an unsealed vault, and it does not wait for device identity, its k3s node name being fixed. RUNNING keeps only the exit watcher, so a crashing k3s still reaches BACKOFF. Cluster membership stays the kube role's business: the join watchdog and cluster-config monitor do not run for the witness, which also makes the global cluster-to-single handler unreachable in that role by removing its only producer rather than by adding a second guard. One k3s binary serves both roles. Shipping a second copy in the witness image would be paid for in the rootfs of every eve-k device, and it cannot fetch its own, having no route off the cluster network. Reading the kube role's copy in place only works on ext4, where /var/lib is a bind of /persist/vault/kube; on ZFS that is the etcd-storage zvol, mounted in the kube container's namespace alone. So the kube role publishes to /persist/vault/k3s-shared, which both containers see either way, and inside the vault because everything k3s execs already lives on vault-encrypted storage. It prefers a hard link, which costs nothing on ext4 and is safe against an in-place upgrade because update installs through a rename. The witness renders its own k3s drop-ins, including the static parts the kube role takes from an image-supplied config.yaml. That is what lets a test hold its etcd settings against pkg/kube's, and they have to agree: k3s treats etcd configuration as cluster-critical, and a witness is a full voting member storing the same data as everyone else, not a partial replica. A witness given a smaller quota-backend-bytes than the nodes would reach its quota first and raise a NOSPACE alarm, which turns the whole cluster read-only. election-timeout is the one setting deliberately not matched: the witness has the slowest storage of the three, so a witness that wins an election makes every write wait on it. It runs only where it is wanted, which is the device that owns join_server_ip, since the witness co-locates with the seed and the API has no separate placement field. Everywhere else it parks in WITNESS_IDLE with k3s stopped, which is a destination rather than an error: most devices spend their whole life there. Leaving k3s running when idle would keep a vote in a cluster that no longer expects one. Being up is not the same as voting, so it reaches RUNNING only once its own member appears in the etcd membership as started and not a learner. A witness with a wrong token, an unreachable seed or a stale cluster ID keeps running and never joins, and etcd admits new members as learners that replicate without voting, so quorum does not improve until promotion. That check reuses WAIT_K3S_READY with a role-aware notion of ready, which also required not gating it on a kubeconfig that a witness with no apiserver never gets. Its membership marker is keyed on the cluster UUID and the recovery generation together, because a recovery resets etcd while keeping the UUID: keying on the UUID alone is a blind spot that would leave an operator to wipe and rejoin by hand. On a mismatch it wipes before k3s can start against state etcd will refuse. When it stops being wanted it removes its own etcd member first, while k3s is still up and the local etcd still answers, since a member left behind holds the witness address that the next witness will claim. The etcd package drives etcdctl, as clustermode already does. The official Go client was tried and rejected: for these few calls it vendored 246 files and added 3MB to the binary. Member.IDHex exists because member list reports IDs in decimal while member remove expects hex. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MountKubeRoot picks storage from /run/eve.persist_type, and on ZFS mounts the etcd-storage zvol at /var/lib. That zvol belongs to the node's own k3s. Left as-is, a witness on a ZFS device would mount the same zvol at its own /var/lib and the two k3s instances would run their etcd on one filesystem, overwriting each other's server/db. Add a witness-storage zvol, reserved at install time behind the new eve_install_k3s_witness option and sized from eve_install_k3s_etcd_sizeGB. Same shape as the etcd zvol: 4 KiB blocks to match etcd's write granularity, compression off so it adds no latency to the fsync behind every raft append, and encrypted with the vault key on TPM devices. unlockVault loads its key alongside the etcd zvol's, since pillar creates but never mounts either. Sizing it like etcd's is deliberate. A witness is a full etcd voting member: etcd has no partial-replica member type, so it replicates the entire keyspace and raft log and its database grows to the same size as any other member's. "Lightweight" describes the witness's compute footprint, which has no kubelet, apiserver, scheduler or workloads, not its storage. The reservation has to happen at install, before CreateZvolVault claims what is left: these zvols are thick-provisioned and the vault takes the whole remaining pool, so a device already in the field has no free space to carve one out of. Those devices instead bind a directory inside the vault, which is the upgrade path rather than a stopgap, and is why the witness never consults the persist type: its storage is chosen by what was reserved, not by the filesystem in use. Which of the two applies is decided by a marker pillar writes when it creates the zvol, not by looking for the device node. A missing node is ambiguous, meaning either that no zvol was ever reserved or that its node has not appeared yet, and those need opposite responses. So the marker selects, and a reserved-but-absent zvol waits and then fails rather than falling back: a witness that does not start costs one vote, whereas one that starts on empty vault-backed storage while its etcd state sits on the zvol would rejoin as a fresh member. The vault-backed path has a real cost worth knowing: it has no capacity reservation, so app volumes filling /persist/vault can starve the witness's etcd and drop the third vote. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4acc1e1 to
d09bbce
Compare
d09bbce to
5278e60
Compare
Adds the linuxkit package for the etcd-only member that gives a two-node HA cluster its third quorum vote, and puts it in every HV=k image alongside pkg/kube. Whether it runs is a runtime decision: it starts k3s only on the device that owns join_server_ip, and only when the controller has configured a witness address. A container of its own rather than a second process inside pkg/kube, so that a routine kube restart on the seed costs one vote instead of two. The node's own etcd and the witness would otherwise go down together, which is the whole cluster's quorum. It takes the kube-init binary from the pkg/kube image rather than building its own. The two roles share a k3s version, a config vocabulary and a set of pubsub types, and a skew between them would show up as a cluster that half agrees with itself. The pinned tag has to be bumped whenever pkg/kube changes; check-docker-hashes-consistency is what catches a stale one. The image carries no k3s binary and no k3s config. The witness reuses the k3s that pkg/kube publishes to the vault, so the rootfs does not carry a second copy of it, and it renders every drop-in it needs from Go, which is what lets a test hold its etcd settings against the node's. net: new gives it the network namespace NIM plugs into the cluster bridge, without which its etcd would collide with the node's on the same ports. It keeps its own PID namespace, unlike pkg/kube: nothing here needs to see the node's processes. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The device hosting the witness stamps the witness's status on its own Node object, forwarded from the topic the witness publishes. The elected cluster-info reporter reads it back and zedagent sends it on. The annotation exists because the reporter builds its message from the kube API, so anything not already there has to be put there by the node that knows it, and the witness has no Node object of its own to carry its status, having no kubelet. This is the route node-uuid already takes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
5278e60 to
f95ec1c
Compare
|
Evetest: Claude chose to skip 17 test/suite name(s) as unrelated to or redundant for this PR. Which ones and why: run summary. Comment |
Description
This is part 1 of 2 of the 2-Node HA feature for eve-k. A two-node cluster cannot
keep etcd quorum when one node fails, so a lightweight third voting member, the
witness, is added. It is an etcd-only k3s (no kubelet, apiserver, scheduler or
workloads) that co-locates with the seed node (the device owning
join_server_ip).With it, losing the non-seed node leaves two of three votes, so the cluster stays
quorate and apps fail over on their own.
This PR delivers the witness itself and the plumbing it needs:
pkg/witness): its own linuxkit package, included in everyHV=kimage. It is a separate container so that a routine kube restart on theseed costs one quorum vote instead of two. It reuses the
kube-initbinary andthe k3s binary from
pkg/kube.role-dependent runtime paths (selected via
/etc/kube-init-role). The witnessruns only on the device owning
join_server_ip, otherwise it parks inWITNESS_IDLE. It reaches RUNNING only once it is a started, non-learner etcdmember, and removes its own member when it is no longer wanted. The orphan sweep
is now confined to kube-init's own PID namespace so it cannot kill the witness.
witness-storagezvol on ZFS devices (reservedat install via
eve_install_k3s_witness), or a directory in the vault on devicesalready in the field.
bridged by NIM (
Adapter.IsClusterPort). A newVethitem in the DPC reconcilerconnects the witness container netns to the cluster bridge and assigns
WitnessIP.Firewall rules for etcd traffic to and from the witness are added, and zedrouter
rejects the cluster port in multi-port Switch NIs.
volumes would stay degraded forever with only two nodes able to host replicas.
the cluster info.
TestTwoNodeHAClusterin evetest, with the harness additions it needs.What comes in part 2
Part 2 builds on this and covers the recovery side of the feature:
generation, and the recovery status is reported to the controller
How to test and validate this PR
Covered by the automated evetest
TestTwoNodeHACluster(two eve-k devices pluswitness):
preferred affinity for the non-seed node, using a Longhorn volume.
seed and a disk marker written earlier is still readable. Power the node back on
and the app fails back.
responding while the app keeps running (boot time unchanged).
was never restarted.
Manual checks:
k3s-sctlinside the witness container reports the witness state;the controller shows the witness status in the cluster info.
Changelog notes
Adds support for a witness, a lightweight third etcd voting member, so that an
eve-k cluster of two nodes can stay available when one node fails. This is the
first part of the 2-Node HA feature; it is not usable end to end until part 2.
PR Backports
Checklist
check them.
🤖 Generated with Claude Code