Skip to content

Bump eve-api - #160

Merged
eriknordmark merged 1 commit into
lf-edge:masterfrom
milan-zededa:update-eve-api
Sep 4, 2026
Merged

eriknordmark merged 1 commit into
lf-edge:masterfrom
milan-zededa:update-eve-api

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

This EVE API update brings in new fields added for the 2-Node HA solution, see: lf-edge/eve-api#156

This EVE API update brings in new fields added for 2-Node HA solution,
see: lf-edge/eve-api#156

Signed-off-by: Milan Lenco <milan@zededa.com>

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eriknordmark
eriknordmark merged commit 6168c52 into lf-edge:master Sep 4, 2026
2 checks passed
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Sep 29, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
    controller-side operations the scenario is built on.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Sep 29, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
    controller-side operations the scenario is built on.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 6, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
    controller-side operations the scenario is built on.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 19, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
    controller-side operations the scenario is built on.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 19, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. Powering off the seed takes two of the three
votes with it, so quorum is lost; the test asserts this by requiring the
surviving node's kube API to stop responding while the app keeps
running. Recovery is then driven entirely from the controller, by
repointing join_server_ip at the survivor and incrementing
quorum_recovery_generation in a single config change. The test waits for
the survivor to report convergence to that generation and for the
witness to rejoin at the same address, and confirms the app was never
restarted by comparing its boot time across the whole sequence.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness and TriggerQuorumRecovery, the two
    controller-side operations the scenario is built on.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

The two recovery waits are ten minutes because each spans far more than
the operation it is named after, down to a fixed reporting latency of
about a minute: status is stamped on a health tick and collected on
another before it can reach the controller at all. They are reasoned
estimates, not measurements, and want revisiting once this has run on
hardware.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 19, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 26, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 26, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 26, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 27, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 30, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 30, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 30, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Oct 31, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 2, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 4, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 4, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to milan-zededa/eve that referenced this pull request Nov 5, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa added a commit to lf-edge/eve that referenced this pull request Nov 9, 2026
TestTwoNodeHACluster covers a two-node eve-k cluster kept quorate by a
witness, a lightweight etcd-only third voting member that co-locates
with the seed. The app is configured with preferred affinity for the
non-seed node.

The test exercises both failure directions. Powering off the non-seed
leaves the seed and the witness holding two of three votes, so the
cluster stays quorate, the app fails over on its own, and it fails back
once the node returns. A disk marker written to the app's Longhorn
volume before the failover is re-read after each move, proving the node
the app lands on already had a synced replica. Powering off the seed
takes two of the three votes with it, so quorum is lost; the test
asserts this by requiring the surviving node's kube API to stop
responding while the app keeps running, confirmed by comparing its boot
time across the outage. Finally the seed is powered back on, and the test
waits for all nodes to be ready and the witness to rejoin, and checks
once more that the app was never restarted.

Harness additions the scenario needed:

  - EdgeClusterConfig.SetWitness, to configure the witness IP.
  - EdgeCluster.WaitUntilClusterInfoSatisfies, a generalisation of
    WaitUntilNodesAreReady that takes an arbitrary predicate over
    ZInfoKubeCluster. It watches every device in parallel, because the
    node that reports cluster info can change when the seed moves.
  - netmodels.SeparateClusterPort becomes a function of the device
    names, so a two-device topology can be expressed alongside the
    existing three-device one. Existing callers pass their own names.
  - Setup now fails when the network model names a device the test did
    not request, which is what keeps the two-device and three-device
    topologies from silently overlapping.
  - ExpectReboots becomes ExpectAdditionalReboots: it adds to a running
    total rather than stating what the total should be, and the
    framework's own reboot-driven flows have already contributed to it.

Also record in the README why building the test container against a
locally built Adam image fails under a docker-container buildx driver,
which cannot see the host's image store, and how to work around it.

EVETEST_ADAM_VERSION's default is bumped to 0.0.84: 0.0.82 predates
lf-edge/adam#160, which bumped Adam's own eve-api vendor to include the
witness and quorum-recovery types. Without that, Adam silently drops
the witness field when relaying EdgeNodeCluster config (its own
generated types have no such field), so the device never receives a
witness IP and TestTwoNodeHACluster times out waiting for it to join.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants