Skip to content

2-Node HA, part 1: witness as the third etcd voting member - #6933

Merged
milan-zededa merged 7 commits into
lf-edge:masterfrom
milan-zededa:two-node-ha-pr1
Nov 9, 2026
Merged

milan-zededa merged 7 commits into
lf-edge:masterfrom
milan-zededa:two-node-ha-pr1

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Description

This is part 1 of 2 of the 2-Node HA feature for eve-k. A two-node cluster cannot
keep etcd quorum when one node fails, so a lightweight third voting member, the
witness, is added. It is an etcd-only k3s (no kubelet, apiserver, scheduler or
workloads) that co-locates with the seed node (the device owning join_server_ip).
With it, losing the non-seed node leaves two of three votes, so the cluster stays
quorate and apps fail over on their own.

This PR delivers the witness itself and the plumbing it needs:

  • Witness container (pkg/witness): its own linuxkit package, included in every
    HV=k image. It is a separate container so that a routine kube restart on the
    seed costs one quorum vote instead of two. It reuses the kube-init binary and
    the k3s binary from pkg/kube.
  • kube-init witness role: the same daemon runs with a reduced state set and
    role-dependent runtime paths (selected via /etc/kube-init-role). The witness
    runs only on the device owning join_server_ip, otherwise it parks in
    WITNESS_IDLE. It reaches RUNNING only once it is a started, non-learner etcd
    member, and removes its own member when it is no longer wanted. The orphan sweep
    is now confined to kube-init's own PID namespace so it cannot kill the witness.
  • Witness storage: a dedicated witness-storage zvol on ZFS devices (reserved
    at install via eve_install_k3s_witness), or a directory in the vault on devices
    already in the field.
  • Networking (pillar/NIM): a port designated only as the cluster interface is now
    bridged by NIM (Adapter.IsClusterPort). A new Veth item in the DPC reconciler
    connects the witness container netns to the cluster bridge and assigns WitnessIP.
    Firewall rules for etcd traffic to and from the witness are added, and zedrouter
    rejects the cluster port in multi-port Switch NIs.
  • Longhorn: replica count is capped at 2 when a witness is configured, otherwise
    volumes would stay degraded forever with only two nodes able to host replicas.
  • Controller reporting: the witness status is reported to the controller through
    the cluster info.
  • Test: TestTwoNodeHACluster in evetest, with the harness additions it needs.

What comes in part 2

Part 2 builds on this and covers the recovery side of the feature:

  • quorum-recovery: kube-init converges to the controller's quorum-recovery
    generation, and the recovery status is reported to the controller
  • zedkube skips the Kubernetes collectors while the apiserver is down
  • moving an app onto its designated node when that node joins (failback to new non-seed)
  • extending the evetest scenario to cover quorum-loss recovery
  • complete 2-Node HA documentation

How to test and validate this PR

Covered by the automated evetest TestTwoNodeHACluster (two eve-k devices plus
witness):

  1. Bring up a two-node eve-k cluster with the witness IP configured and an app with
    preferred affinity for the non-seed node, using a Longhorn volume.
  2. Power off the non-seed node. The cluster stays quorate, the app fails over to the
    seed and a disk marker written earlier is still readable. Power the node back on
    and the app fails back.
  3. Power off the seed. Quorum is lost: the surviving node's kube API stops
    responding while the app keeps running (boot time unchanged).
  4. Power the seed back on. All nodes become ready, the witness rejoins and the app
    was never restarted.

Manual checks: k3s-sctl inside the witness container reports the witness state;
the controller shows the witness status in the cluster info.

Changelog notes

Adds support for a witness, a lightweight third etcd voting member, so that an
eve-k cluster of two nodes can stay available when one node fails. This is the
first part of the 2-Node HA feature; it is not usable end to end until part 2.

PR Backports

  • 17.0-stable: No, new feature.
  • 16.0-stable: No, new feature.
  • 14.5-stable: No, new feature.
  • 13.4-stable: No, new feature.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

Comment thread pkg/kube/kube-init/k3s/paths.go Resolved
Comment thread pkg/kube/kube-init/main.go Resolved
Comment thread pkg/pillar/dpcreconciler/linux.go Resolved
Comment thread pkg/pillar/dpcreconciler/linux.go Resolved
Comment thread pkg/pillar/vault/handler_zfs.go Resolved
Comment thread evetest/edgedevice.go Resolved
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 15.98174% with 368 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.53%. Comparing base (9e884c7) to head (f95ec1c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/dpcreconciler/linuxitems/veth.go 15.00% 170 Missing ⚠️
pkg/pillar/dpcreconciler/linux.go 29.34% 61 Missing and 4 partials ⚠️
pkg/pillar/cmd/volumemgr/volumemgr.go 0.00% 33 Missing ⚠️
pkg/pillar/vault/handler_zfs.go 0.00% 33 Missing ⚠️
pkg/pillar/cmd/zedrouter/pubsubhandlers.go 0.00% 28 Missing ⚠️
pkg/pillar/cmd/zedrouter/zedrouter.go 0.00% 17 Missing ⚠️
pkg/pillar/cmd/zedrouter/networkinstance.go 0.00% 8 Missing ⚠️
pkg/pillar/cmd/zedkube/kubestatscollect.go 71.42% 3 Missing and 1 partial ⚠️
pkg/pillar/kubeapi/kubeapi.go 0.00% 3 Missing ⚠️
pkg/pillar/dpcreconciler/linuxitems/adapter.go 33.33% 2 Missing ⚠️
... and 4 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6933      +/-   ##
==========================================
+ Coverage   29.11%   29.53%   +0.42%     
==========================================
  Files         548      560      +12     
  Lines       99033   101017    +1984     
==========================================
+ Hits        28830    29834    +1004     
- Misses      67934    68950    +1016     
+ Partials     2269     2233      -36     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@milan-zededa
milan-zededa force-pushed the two-node-ha-pr1 branch 4 times, most recently from 4e26206 to 8fd3d67 Compare October 27, 2026 19:18
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@andrewd-zededa All the requested changes have been addressed. Thanks for the review.

@andrewd-zededa andrewd-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more items, thanks for the other fixes.

Comment thread pkg/kube/kube-init/main.go Resolved
@eriknordmark

Copy link
Copy Markdown
Contributor

@milan-zededa This PR conflicts with master and needs a rebase. Its last Fuzz Tests run was cancelled at the 6h job limit while building eve-cross-compilers:2dad5fbf from source, because that tag had no amd64 image on Docker Hub at the time. Rebasing onto current master picks up the published be9bca50, so the job pulls it instead of rebuilding gcc.

@milan-zededa

Copy link
Copy Markdown
Contributor Author

@eriknordmark @andrewd-zededa Rebased and addressed the latest review comments.

Comment thread pkg/kube/kube-init/main.go Resolved
@milan-zededa

This comment has been minimized.

@milan-zededa
milan-zededa force-pushed the two-node-ha-pr1 branch 2 times, most recently from 1ba82bb to 9569e13 Compare November 4, 2026 18:28
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@andrewd-zededa If you have no further objections, please approve this PR so we can move forward with running tests and merging it.

@lf-edge lf-edge deleted a comment from github-actions Bot Nov 5, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Nov 5, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Nov 5, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Nov 5, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Nov 5, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Nov 5, 2026

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run tests

@andrewd-zededa

Copy link
Copy Markdown
Contributor

@andrewd-zededa If you have no further objections, please approve this PR so we can move forward with running tests and merging it.

taking another look

@andrewd-zededa andrewd-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@milan-zededa I think this is my last finding

case EvError:
d.lastError = ev.Err
if role.IsWitness() {
witnessstatus.SetError(ev.Err, d.witnessIP)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SetError only covers a readiness failure. A crash after the witness has joined takes the EvK3sExited path and never reaches this, so the status is not updated for the common failure.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in d78e5e0: an unexpected k3s exit in STARTING_K3S, WAIT_K3S_READY or RUNNING now publishes the witness error via a new reportWitnessK3sExit helper. The error clears when the next start publishes the joining state. Also rebased on the latest master.

milan-zededa and others added 4 commits November 9, 2026 10:35
NIM only created a Linux bridge for a port that needed DHCP or a static
IP, was a VLAN parent, and so on. A port designated purely as the
kubernetes cluster interface has no L3 config of its own, so it was
never bridged, and nothing could be attached to it. Mark that port with
Adapter.IsClusterPort and have isAdapterBridgedByNIM honour it.

On top of that, add a Veth item to the DPC reconciler. One peer is
enslaved to an adapter bridge, the other is moved into the network
namespace of a named containerd system-service container and given an
address. NIM uses it to interconnect the witness container's own netns
with the cluster interface bridge on the node owning JoinServerIP,
assigning WitnessIP from EdgeNodeClusterStatus. The intent follows
BootstrapNode, so the interconnect moves with the witness and needs no
separate teardown.

Firewall rules are installed through the reconciler alongside it:
INPUT accepts etcd traffic to and from WitnessIP on 2379:2380, FORWARD
accepts it across the cluster bridge before the L3 forward deny, and a
connmark distinguishes the witness's contribution to etcd traffic from
the nodes' own.

zedrouter subscribes to EdgeNodeClusterStatus to learn the resolved
cluster interface name, so a single-port Switch NI on that port is
recognised as NIM-bridged, and the cluster port is rejected from
multi-port Switch NIs, which would otherwise take ownership of the
bridge the witness veth depends on.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A two-node-HA cluster has only two physical nodes able to host a
Longhorn replica, so the default 'longhorn' StorageClass's
numberOfReplicas=3 can never be satisfied: every volume sits
permanently "degraded" instead of ever reaching "healthy", even
when both replicas are fully in sync.

GetSupportedReplicaCountForCluster already reduces to 2 replicas
for the older tie-breaker-node label case; extend it to also do so
whenever EdgeNodeClusterConfig.WitnessIP is set, i.e. a
2-physical-node witness cluster. Plumb this from volumemgr, which
already needs EdgeNodeClusterConfig at startup to gate the Longhorn
readiness wait: add a VolumeMgr.IsTwoNodeHACluster() accessor,
implemented by reading the (now persistent, not one-shot) subscription
on demand via Get("global"), so csihandler.go's CreateVolume can pass
the right flag into GetSupportedReplicaCountForCluster per volume.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The witness is this same daemon running in a second container: an
etcd-only k3s that gives a two-node cluster its third quorum vote, and
nothing else. No kubelet, no apiserver, no scheduler, no workloads.

Both containers bind-mount the host /run and /persist, so every fixed
runtime path had to become role-dependent: the pubsub agent name, which
also fixes /run/<agent>/ and the agent socket; the control socket; the
supervisor pid file; and the k3s and install logs. The control socket
mattered most, since listenSocket removes a stale socket before binding,
so whichever process started last would have silently answered k3s-sctl
for both. k3s-sctl picks its default the same way, so a shell in the
witness container reaches the witness.

The role comes from /etc/kube-init-role, which the witness image carries
and the kube image does not, in the shape of /etc/eve-hv-type. Not a
flag: these paths live in package-level vars initialised before main can
parse anything. Reading it from the image also reaches the tools beside
the daemon, which never get a command line of their own to carry the
role, and k3s-sctl above all: /run is shared, so both supervisor sockets
are reachable from either container, and a wrong guess would answer for
the other daemon rather than fail. It is the only mechanism, so there is
nothing for it to disagree with.

The orphan sweep is a bug in its own right, independent of any witness.
It walked all of /proc, matched cmdline only and SIGKILLed every hit,
and this container runs with pid: host. PID namespaces are hierarchical,
so a k3s in another container is fully visible here and a signal from
this ancestor namespace reaches it. It fires at the end of every stop,
not just during recovery, so an ordinary restart would have killed the
witness and silently cost a quorum vote. Candidates are now confined to
our own PID namespace, and an unreadable namespace counts as foreign:
leaving a stray orphan is recoverable, killing the witness is not.

The witness gets a reduced state set. It skips STARTING_CTRD, since with
--disable-agent there is no kubelet to dial a CRI socket; IMPORTING,
since it runs no workloads; and DEPLOYING and SNAPSHOT, since it
installs no components. Its prerequisites shrink to logging, a route to
the seed, and its own /var/lib behind an unsealed vault, and it does not
wait for device identity, its k3s node name being fixed. RUNNING keeps
only the exit watcher, so a crashing k3s still reaches BACKOFF.

Cluster membership stays the kube role's business: the join watchdog and
cluster-config monitor do not run for the witness, which also makes the
global cluster-to-single handler unreachable in that role by removing
its only producer rather than by adding a second guard.

One k3s binary serves both roles. Shipping a second copy in the witness
image would be paid for in the rootfs of every eve-k device, and it
cannot fetch its own, having no route off the cluster network. Reading
the kube role's copy in place only works on ext4, where /var/lib is a
bind of /persist/vault/kube; on ZFS that is the etcd-storage zvol,
mounted in the kube container's namespace alone. So the kube role
publishes to /persist/vault/k3s-shared, which both containers see either
way, and inside the vault because everything k3s execs already lives on
vault-encrypted storage. It prefers a hard link, which costs nothing on
ext4 and is safe against an in-place upgrade because update installs
through a rename.

The witness renders its own k3s drop-ins, including the static parts the
kube role takes from an image-supplied config.yaml. That is what lets a
test hold its etcd settings against pkg/kube's, and they have to agree:
k3s treats etcd configuration as cluster-critical, and a witness is a
full voting member storing the same data as everyone else, not a partial
replica. A witness given a smaller quota-backend-bytes than the nodes
would reach its quota first and raise a NOSPACE alarm, which turns the
whole cluster read-only. election-timeout is the one setting
deliberately not matched: the witness has the slowest storage of the
three, so a witness that wins an election makes every write wait on it.

It runs only where it is wanted, which is the device that owns
join_server_ip, since the witness co-locates with the seed and the API
has no separate placement field. Everywhere else it parks in
WITNESS_IDLE with k3s stopped, which is a destination rather than an
error: most devices spend their whole life there. Leaving k3s running
when idle would keep a vote in a cluster that no longer expects one.

Being up is not the same as voting, so it reaches RUNNING only once its
own member appears in the etcd membership as started and not a learner.
A witness with a wrong token, an unreachable seed or a stale cluster ID
keeps running and never joins, and etcd admits new members as learners
that replicate without voting, so quorum does not improve until
promotion. That check reuses WAIT_K3S_READY with a role-aware notion of
ready, which also required not gating it on a kubeconfig that a witness
with no apiserver never gets.

Its membership marker is keyed on the cluster UUID and the recovery
generation together, because a recovery resets etcd while keeping the
UUID: keying on the UUID alone is a blind spot that would leave an
operator to wipe and rejoin by hand. On a mismatch it wipes before k3s
can start against state etcd will refuse. When it stops being wanted it
removes its own etcd member first, while k3s is still up and the local
etcd still answers, since a member left behind holds the witness address
that the next witness will claim.

The etcd package drives etcdctl, as clustermode already does. The
official Go client was tried and rejected: for these few calls it
vendored 246 files and added 3MB to the binary. Member.IDHex exists
because member list reports IDs in decimal while member remove expects
hex.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MountKubeRoot picks storage from /run/eve.persist_type, and on ZFS
mounts the etcd-storage zvol at /var/lib. That zvol belongs to the
node's own k3s. Left as-is, a witness on a ZFS device would mount the
same zvol at its own /var/lib and the two k3s instances would run their
etcd on one filesystem, overwriting each other's server/db.

Add a witness-storage zvol, reserved at install time behind the new
eve_install_k3s_witness option and sized from
eve_install_k3s_etcd_sizeGB. Same shape as the etcd zvol: 4 KiB blocks
to match etcd's write granularity, compression off so it adds no
latency to the fsync behind every raft append, and encrypted with the
vault key on TPM devices. unlockVault loads its key alongside the etcd
zvol's, since pillar creates but never mounts either.

Sizing it like etcd's is deliberate. A witness is a full etcd voting
member: etcd has no partial-replica member type, so it replicates the
entire keyspace and raft log and its database grows to the same size as
any other member's. "Lightweight" describes the witness's compute
footprint, which has no kubelet, apiserver, scheduler or workloads, not
its storage.

The reservation has to happen at install, before CreateZvolVault claims
what is left: these zvols are thick-provisioned and the vault takes the
whole remaining pool, so a device already in the field has no free
space to carve one out of. Those devices instead bind a directory
inside the vault, which is the upgrade path rather than a stopgap, and
is why the witness never consults the persist type: its storage is
chosen by what was reserved, not by the filesystem in use.

Which of the two applies is decided by a marker pillar writes when it
creates the zvol, not by looking for the device node. A missing node is
ambiguous, meaning either that no zvol was ever reserved or that its
node has not appeared yet, and those need opposite responses. So the
marker selects, and a reserved-but-absent zvol waits and then fails
rather than falling back: a witness that does not start costs one vote,
whereas one that starts on empty vault-backed storage while its etcd
state sits on the zvol would rejoin as a fresh member.

The vault-backed path has a real cost worth knowing: it has no capacity
reservation, so app volumes filling /persist/vault can starve the
witness's etcd and drop the third vote.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa and others added 3 commits November 9, 2026 10:51
Adds the linuxkit package for the etcd-only member that gives a two-node
HA cluster its third quorum vote, and puts it in every HV=k image
alongside pkg/kube. Whether it runs is a runtime decision: it starts k3s
only on the device that owns join_server_ip, and only when the
controller has configured a witness address.

A container of its own rather than a second process inside pkg/kube, so
that a routine kube restart on the seed costs one vote instead of two.
The node's own etcd and the witness would otherwise go down together,
which is the whole cluster's quorum.

It takes the kube-init binary from the pkg/kube image rather than
building its own. The two roles share a k3s version, a config vocabulary
and a set of pubsub types, and a skew between them would show up as a
cluster that half agrees with itself. The pinned tag has to be bumped
whenever pkg/kube changes; check-docker-hashes-consistency is what
catches a stale one.

The image carries no k3s binary and no k3s config. The witness reuses
the k3s that pkg/kube publishes to the vault, so the rootfs does not
carry a second copy of it, and it renders every drop-in it needs from
Go, which is what lets a test hold its etcd settings against the
node's.

net: new gives it the network namespace NIM plugs into the cluster
bridge, without which its etcd would collide with the node's on the same
ports. It keeps its own PID namespace, unlike pkg/kube: nothing here
needs to see the node's processes.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The device hosting the witness stamps the witness's status on its own
Node object, forwarded from the topic the witness publishes. The elected
cluster-info reporter reads it back and zedagent sends it on.

The annotation exists because the reporter builds its message from the
kube API, so anything not already there has to be put there by the node
that knows it, and the witness has no Node object of its own to carry its
status, having no kubelet. This is the route node-uuid already takes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Evetest: Claude chose to skip 17 test/suite name(s) as unrelated to or redundant for this PR. Which ones and why: run summary.

Comment /evetest all to run the full suite, or /evetest <name> <name> ... to run only specific tests/suites.

@milan-zededa
milan-zededa merged commit ddc4baa into lf-edge:master Nov 9, 2026
76 of 81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants