Repository navigation
Report a cluster member's own connection to its cluster - #161
Conversation
f0f9af4 to
3f37803
Compare
| Z_INFO_CLUSTER_NODE_STATUS_DOWN = 3; // cluster API server can not be reached | ||
| } | ||
|
|
||
| // ZInfoClusterNodeQuorum is a node's own view of whether the cluster's etcd |
There was a problem hiding this comment.
Is this specific to using etcd, or does it apply independently of the DB being used?
There was a problem hiding this comment.
No, the meaning does not depend on etcd: it is whether the node can reach a majority of the voters of the cluster datastore. etcd is just what EVE-k uses for HA (the embedded etcd of k3s); a single-node k3s uses SQLite and has no quorum. I reworded the enum and message docs to talk about the datastore and to say that in EVE-k it is etcd.
| ZInfoClusterNodeQuorum quorum = 3; | ||
|
|
||
| // Why the node considers the cluster unreachable, when it does, such as | ||
| // the failure of the last probe. |
There was a problem hiding this comment.
Last probe time is useful, but would it also be useful to report the time when the last status transition occurred (e.g., from ESTABLISHED to LOST)?
There was a problem hiding this comment.
Yes, added quorum_change_time: when quorum last changed to its current value (to LOST, or back to ESTABLISHED). It is deliberately not persisted: after a device restart quorum is probed again and the time is when the node first determined the value, which is documented on the field. Also fixed the error comment, which wrongly said it was about the last probe: error.timestamp is when the failure was first seen, so for LOST it is when quorum was lost.
ZInfoClusterNode already exists but nothing populates it. Extend it so that every cluster member can report whether its etcd has quorum, independent of the elected cluster-info reporter and of the cluster API. ZInfoKubeCluster is sent only by the stats-lease holder and needs a working API to be built. When a 2-node HA cluster loses quorum, or its members lose sight of each other, no such report can be produced, and the controller keeps showing the last one it received. A per-member message that does not rely on either closes that gap. Add cluster_id, a quorum state and an error to ZInfoClusterNode, and a ZiClusterNode info type for the message. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
3f37803 to
d518342
Compare
Why
A controller needs to know when a cluster has lost etcd quorum. Today it cannot.
ZInfoKubeClusteris the only cluster status the device sends, and it is built and sent by a single elected reporter: the node holding the stats lease. Building it needs a working Kubernetes API and a working leader election, and both depend on etcd quorum. When quorum is lost, nobody can produce the report and no message is sent at all. The controller keeps showing the last report it received, typically "healthy", while the cluster is down.Quorum loss is a normal failure mode of a 2-node HA cluster (a 2-node cluster plus a witness keeps a majority of the etcd voters only while two of the three are alive and in reach of each other):
It can also happen in clusters with 3 or more nodes.
Reported cluster state also cannot be derived from the absence of a message: a silent reporter is indistinguishable from a healthy cluster that has not changed, because
ZInfoKubeClusteris sent only on change.Why
ZInfoKubeClustercannot be extended for thisSo the report has to come from every member, be independent of the API and the election, and be small.
What this changes
ZInfoClusterNodealready exists inZInfoMsg(field 22) but nothing populates it. This PR extends it instead of adding a new message:ZInfoClusterNode.cluster_id: the cluster the member belongs to.ZInfoClusterNode.quorum(newZInfoClusterNodeQuorum:UNSPECIFIED,ESTABLISHED,LOST): whether this member can reach a majority of the cluster datastore's voters (in EVE-k, whether its etcd reports a leader).ZInfoClusterNode.error: the reason, for exampleetcdserver: no leader.ZInfoClusterNode.quorum_change_time: when quorum last changed to its current value, as observed by the member. It is not kept across a device restart, so it is the time the member first determined the value after start. It stays correct when delivery to the controller is delayed.ZInfoTypes.ZiClusterNode = 19: the info type for the message.The device sends it on change, and once after start so the controller has a baseline. There is no periodic resend, in line with how
ZInfoKubeClusteris sent. A controller can treat any member reportingLOSTas a degraded cluster, and use the per-member message to tell a lost quorum from a lost reporter.The change is wire-compatible: it only adds fields and an enum value. No device capability flag is added; a controller that does not know the message ignores it.
Device side
The implementation in EVE is validated on a 2-node cluster with a witness: with both nodes up each reports
ESTABLISHED. About 30s after powering off the node hosting the witness, the other one reportsLOSTwithetcdserver: no leader.The generated Go and Python code is in a separate commit.
🤖 Generated with Claude Code