Skip to content

Report a cluster member's own connection to its cluster - #161

Merged
milan-zededa merged 2 commits into
lf-edge:mainfrom
milan-zededa:two-node-ha
Nov 4, 2026
Merged

milan-zededa merged 2 commits into
lf-edge:mainfrom
milan-zededa:two-node-ha

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

A controller needs to know when a cluster has lost etcd quorum. Today it cannot.

ZInfoKubeCluster is the only cluster status the device sends, and it is built and sent by a single elected reporter: the node holding the stats lease. Building it needs a working Kubernetes API and a working leader election, and both depend on etcd quorum. When quorum is lost, nobody can produce the report and no message is sent at all. The controller keeps showing the last report it received, typically "healthy", while the cluster is down.

Quorum loss is a normal failure mode of a 2-node HA cluster (a 2-node cluster plus a witness keeps a majority of the etcd voters only while two of the three are alive and in reach of each other):

  • The node hosting the witness dies. The remaining node alone has 1 of 3 voters.
  • Both nodes stay up and reach the controller, but lose sight of each other. The node hosting the witness still has a majority (itself plus the witness) and can keep reporting. The other node is left with 1 of 3 voters and loses quorum, so it can no longer report anything, though it is up and reaches the controller.

It can also happen in clusters with 3 or more nodes.

Reported cluster state also cannot be derived from the absence of a message: a silent reporter is indistinguishable from a healthy cluster that has not changed, because ZInfoKubeCluster is sent only on change.

Why ZInfoKubeCluster cannot be extended for this

  • It has one author per cluster. Only the elected reporter sends it. The node that matters most in a quorum loss is exactly the one that cannot win the election or read the API.
  • It is a view of the cluster, not of the sender. It carries the nodes, storage, apps and namespaces as seen through the Kubernetes API. Quorum loss is a statement about whether this member can reach a majority, which that view cannot express.
  • It is sent on change and implies liveness of the reporter. A "no quorum" value has to come from a node that is not the leader, with no API to build the rest of the message from.

So the report has to come from every member, be independent of the API and the election, and be small.

What this changes

ZInfoClusterNode already exists in ZInfoMsg (field 22) but nothing populates it. This PR extends it instead of adding a new message:

  • ZInfoClusterNode.cluster_id: the cluster the member belongs to.
  • ZInfoClusterNode.quorum (new ZInfoClusterNodeQuorum: UNSPECIFIED, ESTABLISHED, LOST): whether this member can reach a majority of the cluster datastore's voters (in EVE-k, whether its etcd reports a leader).
  • ZInfoClusterNode.error: the reason, for example etcdserver: no leader.
  • ZInfoClusterNode.quorum_change_time: when quorum last changed to its current value, as observed by the member. It is not kept across a device restart, so it is the time the member first determined the value after start. It stays correct when delivery to the controller is delayed.
  • ZInfoTypes.ZiClusterNode = 19: the info type for the message.

The device sends it on change, and once after start so the controller has a baseline. There is no periodic resend, in line with how ZInfoKubeCluster is sent. A controller can treat any member reporting LOST as a degraded cluster, and use the per-member message to tell a lost quorum from a lost reporter.

The change is wire-compatible: it only adds fields and an enum value. No device capability flag is added; a controller that does not know the message ignores it.

Device side

The implementation in EVE is validated on a 2-node cluster with a witness: with both nodes up each reports ESTABLISHED. About 30s after powering off the node hosting the witness, the other one reports LOST with etcdserver: no leader.

The generated Go and Python code is in a separate commit.

🤖 Generated with Claude Code

Comment thread proto/info/info.proto Outdated
Z_INFO_CLUSTER_NODE_STATUS_DOWN = 3; // cluster API server can not be reached
}

// ZInfoClusterNodeQuorum is a node's own view of whether the cluster's etcd

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this specific to using etcd, or does it apply independently of the DB being used?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the meaning does not depend on etcd: it is whether the node can reach a majority of the voters of the cluster datastore. etcd is just what EVE-k uses for HA (the embedded etcd of k3s); a single-node k3s uses SQLite and has no quorum. I reworded the enum and message docs to talk about the datastore and to say that in EVE-k it is etcd.

Comment thread proto/info/info.proto Outdated
ZInfoClusterNodeQuorum quorum = 3;

// Why the node considers the cluster unreachable, when it does, such as
// the failure of the last probe.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last probe time is useful, but would it also be useful to report the time when the last status transition occurred (e.g., from ESTABLISHED to LOST)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, added quorum_change_time: when quorum last changed to its current value (to LOST, or back to ESTABLISHED). It is deliberately not persisted: after a device restart quorum is probed again and the time is when the node first determined the value, which is documented on the field. Also fixed the error comment, which wrongly said it was about the last probe: error.timestamp is when the failure was first seen, so for LOST it is when quorum was lost.

@naiming-zededa naiming-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

milan-zededa and others added 2 commits November 2, 2026 10:40
ZInfoClusterNode already exists but nothing populates it. Extend it so that
every cluster member can report whether its etcd has quorum, independent of
the elected cluster-info reporter and of the cluster API.

ZInfoKubeCluster is sent only by the stats-lease holder and needs a working
API to be built. When a 2-node HA cluster loses quorum, or its members lose
sight of each other, no such report can be produced, and the controller keeps
showing the last one it received. A per-member message that does not rely on
either closes that gap.

Add cluster_id, a quorum state and an error to ZInfoClusterNode, and a
ZiClusterNode info type for the message.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa merged commit 7570235 into lf-edge:main Nov 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants