Repository navigation
Bump eve-api - #160
Merged
Merged
Bump eve-api#160
Conversation
This EVE API update brings in new fields added for 2-Node HA solution, see: lf-edge/eve-api#156 Signed-off-by: Milan Lenco <milan@zededa.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Sep 29, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
controller-side operations the scenario is built on.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Sep 29, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
controller-side operations the scenario is built on.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 6, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
controller-side operations the scenario is built on.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 19, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
controller-side operations the scenario is built on.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 19, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
controller-side operations the scenario is built on.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 19, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
5 of 7 tasks
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 26, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 26, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 26, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 30, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 30, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 30, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Oct 31, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 4, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 4, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to milan-zededa/eve
that referenced
this pull request
Nov 5, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
added a commit
to lf-edge/eve
that referenced
this pull request
Nov 9, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.
The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.
Harness additions the scenario needed:
- EdgeClusterConfig.SetWitness, to configure the witness IP.
- EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
WaitUntilNodesAreReady that takes an arbitrary predicate over
ZInfoKubeCluster. It watches every device in parallel, because the
node that reports cluster info can change when the seed moves.
- netmodels.SeparateClusterPort becomes a function of the device
names, so a two-device topology can be expressed alongside the
existing three-device one. Existing callers pass their own names.
- Setup now fails when the network model names a device the test did
not request, which is what keeps the two-device and three-device
topologies from silently overlapping.
- ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
total rather than stating what the total should be, and the
framework's own reboot-driven flows have already contributed to it.
Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.
EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This EVE API update brings in new fields added for the 2-Node HA solution, see: lf-edge/eve-api#156