Repository navigation
[17.0-stable] eve-k: let a healthy peer act for a downed node's designated app (backup DNID) - #6867
Conversation
Info() reported HALTED as soon as the VMIRS Get returned NotFound. The three objects form an ownership chain: the VMIRS owns the VMI, and the VMI owns the virt-launcher pod. EVE deletes only the VMIRS, so it is the first to go, and Kubernetes then collects its dependents in the background. The pod holds the RWO disk until it goes away. A domain was therefore reported as halted while its workload still ran. A peer that stands in for a downed node reads that state to decide if an app is really down, and a controller reads it to decide the app can move. Both can act too early. Info() now takes the state from the existence of the VMIRS, the VMI and the pod, each found by its App-Domain-Name label. An absent VMIRS with a surviving dependent gives HALTING and keeps a non-zero DomainId, because a zero DomainId is the confirmed-absent token. HALTED with a zero DomainId needs all three to be absent. The dependent lookup runs only on the absent-VMIRS branch, so a steady-state poll makes no more API calls than it did before. A config apply must not wait for this. A kubevirt deactivate now publishes HALTING, deletes the VMIRS, and returns. verifyStatus settles the state and releases the resources when Info reports all three objects absent. Kubernetes owns how long that takes, and on a node that is off it has no end, so no budget on the config path is the correct one. The paths that must finish before they return keep their behavior, because impatient is already true for them: handleDelete unpublishes DomainStatus as soon as it returns, and a purge or a restart advances its counter only after the teardown clears status.Activated. Two other reads of HALTING had to be separated from this one. waitForDomainGone took it for a guest that powered off while its hypervisor process stayed parked and held resources, a qemu -no-shutdown condition, so it stopped waiting and let the caller reap the domain. verifyStatus took it for a teardown to finish now. Under kubevirt HALTING is the reverse of both: nothing is parked, and Kubernetes is still at work. Both short circuits now apply to the other hypervisors only. Start() already waits for a stale generation to disappear, object and pods, before it creates the next one, because generations share the disk and the veth names. That is what keeps an asynchronous deactivate safe beside a create. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 1a29cd5)
requestConfigByURL arms configGetStatus = ConfigGetFail before it sends, so every success path has to clear it. The unchanged-hash path did not: when the controller answers 200 with a config hash matching the one we already hold, the code jumps straight to cfgReceived, skipping the ConfigGetSuccess assignment further down. configTimerTask publishes ZedAgentStatus after every poll, so a healthy device whose config was not changing reported ConfigGetFail to every subscriber for as long as it held still. nodeagent escapes it only because it ignores a ConfigGetStatus that has not changed value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit f64f661)
…ures handleControllerStatusChange reacted to every ZedAgentStatus publish and stopped the leader election on anything that was not Success or ReadSaved. Two problems follow from that. ZedAgentStatus is published for many reasons, and zedagent arms ConfigGetFail before each request, so a publish driven by an unrelated field can carry a transient failure and stop the election. Guard on a changed ConfigGetStatus, as nodeagent already does for the same field. A single failed poll should not give up the lease either. A fall to a failure value now schedules the stop 90s out, longer than ConfigInterval's 60s maximum, and recovery inside that window cancels it. The rising edge still acts at once. ConfigGetTemporaryFail no longer stops the election at all. It is set only while an image update is in progress, and giving up the lease during a baseos update is not what a temporary failure asks for. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 180e5f8)
Every piece of election state was a scalar field on zedkube, and handleLeaderElection hardcoded the lease name and the stats-only callbacks inline, so a second election could not be added without copying the whole handler. Move all of it into a leaderElection type: lease name, lease timings, the two flags that gate contending, the state the callbacks write, the cancel func, the acquire watchdog and the re-entry backoff. zedkube holds one instance and a slice of every election it runs, and handleLeaderElection takes one as a parameter. The stats election keeps the behavior it has always had: 300s/180s/15s timings, a flat five-minute re-entry, and eligibility always true. Readers of the stats lease go through an isStatsLeader accessor. Two defects are fixed rather than carried into the new type. leaderIdentity was a plain string written from the client-go callback goroutine and the election goroutine, and read by publishLeaderElectionChange from those and from the handler loop, while its three siblings were atomics. It is now atomic too. publishLeaderElectionChange stamped LatestChange on every call, which defeated the dedupe pubsub already does for an unchanged item, so the topic republished whether or not any state moved. It now compares the four real fields against what is published and stamps LatestChange only when one of them moved. Its only reader uses that field for display, so it now means the time of the last real change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 62bedeb)
The backport of 62bedeb moves the election atomics out of zedkube.go and drops its sync/atomic import. On 17.0-stable, ad2cd56 (the backport of 042622f) already uses atomic.Pointer for clusterStatusCreds in that file, so zedkube stops compiling. On master 62bedeb landed first and 042622f re-added the import; on 17.0-stable the order was reversed. Restore the import. The import block now matches master. This is a 17.0-stable-only fixup; the cherry-pick before it is unedited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
eve-app-op elects the one node allowed to act on an app whose designated node is down. Nothing consumes it yet; this commit only elects and publishes a holder, as IsAppOpLeader and AppOpLeaderIdentity on KubeLeaderElectInfo. Its timings are its own, not the stats election's. At a 300s lease a peer could not acquire for five minutes after the holder died, which would dominate any outage threshold worth configuring, so the lease runs 60s/45s/10s. Re-entry after the election exits backs off from one retry period to a minute instead of sitting out a flat five minutes: when the designated node is down, the only eligible survivor staying out of the election is a feature outage rather than a delay. The tie-breaker is excluded. It exists for quorum and runs no app workloads, so it must never become the backup owner. Eligibility is decided rather than latched, because both of its inputs arrive late and in either order: IsTieBreakerNode reports false for an empty UUID, which reads as "not the tie-breaker", so eligibility is left false until this node's UUID is known, then decided there and again on every cluster config change. Every decision is logged, not only the changes, since a wrong first decision that is never revisited is otherwise invisible. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 904d528)
IsCurrentlyBackupDNID answers whether this node may act on an app in place of its designated node: this node holds the eve-app-op lease, and that node has been unhealthy for longer than cluster.dnid.backupnode.threshold. Nothing calls it yet. The threshold defaults to ten minutes. Its minimum is a minute rather than zero, because at zero a peer would take over as soon as the node-monitor grace period elapsed, which defeats requiring a sustained outage, and anything below the eve-app-op lease duration is bounded by lease handover regardless. Two properties of the health read are load-bearing. Healthy means the Ready condition is True and nothing else: a hard power-off leaves Ready at Unknown rather than False, since Kubernetes cannot know the node failed, only that it stopped reporting, so a check written against False would never fire for the case this exists for. And a zero or future LastTransitionTime is treated as unusable rather than as an outage, because time.Since of the zero time is centuries and would clear any threshold instantly for a node whose health was never established. The read uses a three-second timeout rather than kubeAPITimeout's thirty. It runs on an agent's main loop, where one thirty-second stall would eat most of the watchdog's warning budget. Failing fast costs nothing here: a read that cannot complete means "cannot confirm", which already resolves to not currently backup. Required-affinity apps are excluded, since Kubernetes would refuse to schedule one anywhere but its designated node. The exclusion lives in IsCurrentlyBackupDNID, on top of the affinity-blind backupDNIDEligible: a caller that places nothing, such as deleting a volume, has no such hazard to guard against. The UUID-to-node-name lookup is lifted out of zedkube's getLocalNode rather than written again, and drain.go now calls the lifted version. Only the name mapping is cached -- it is stable and re-stamped by kube-init's monitor -- so the label-selector List stays off the health path and a stale name is dropped and resolved again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 92ac44e)
getKubeAppActivateStatus let a non-designated node act only when Kubernetes had already placed the app there. Now it also acts when this node holds the eve-app-op lease and the app's designated node has been unhealthy past cluster.dnid.backupnode.threshold, which is what makes activate and deactivate work while that node is down. The designated node's own branch is unchanged: an owner never stands in for itself, so it still only stands down when the pod is running elsewhere. AppInstanceConfig gains DesignatedNodeUUID. It carried only IsDesignatedNodeID, a bool saying whether the owner is us, and zedagent discarded the UUID after computing it -- but asking whether a peer may act for that node means asking about that node's health, which needs its identity. The cluster status is now read by key instead of scanning GetAll and comparing AppUUID. zedkube publishes that topic under the app UUID, which is what Key() returns, so the scan was searching for the item a keyed Get returns directly: O(1) per app rather than O(n). The health question sits behind a closure, so placement decides the common cases without spending a live Kubernetes read, and everything cheap -- lease holder, cluster build, whether the app has a designated node other than this one -- is checked before the read is reached. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 21bb347)
A backup-DNID decision turns on how long a node has been unhealthy, which is a wall-clock event with no pubsub carrier. zedmanager had no periodic re-evaluation to notice it: of its four timer cases, freeResourceChecker needs MissingMemory, delayedStartTicker only touches START_DELAYED, priorityStartTimer is one-shot, and stillRunning has an empty body. Nothing else generates traffic either -- with an app halted and its designated node off, ENClusterAppStatus recomputes identically and publishes on diff only, so it fires once on the transition and never again. The consequence was that an operation issued during an outage but before the threshold elapsed evaluated the gate once, got false, and was never looked at again. That window is the node-monitor grace period plus the threshold, about eleven minutes at the default, which covers most operator reactions to a node going down. A jittered 15-25s ticker now re-drives the apps a backup decision could change. The scan that finds them is pure state -- designated node, affinity, current placement -- so a tick with no candidates costs no API call, and only a candidate reaches the live health read. Level-triggered rather than edge-triggered: it re-drives whenever the desired state is unmet, because an edge is consumed even when the resulting doUpdate cannot finish the job, and would then never fire again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 5d8f747)
canReport in publishAppInstanceStatus only recognized two reasons a node may report an app's status to the controller: it is where the pod is scheduled, or it is the DNID and the cluster shows the app nowhere at all. A backup-DNID node satisfies neither the instant Kubernetes finishes tearing the workload down: ScheduledOnThisNode goes false and IsDNidNode was already false, so the app's final state (e.g. HALTED) is silently dropped instead of reaching the controller. Fall back to isCurrentlyBackupDNIDForApp only when the existing two conditions are false, so the common case pays no extra cost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit fd4b2f2)
Delete's "not scheduled on me, but is scheduled elsewhere" skip assumed elsewhere always means a live pillar instance will handle its own teardown. That breaks deleting a Required-affinity app while its designated node is down: the VMIRS/pod object persists in Kubernetes (a dead node's bindings don't vanish), so scheduledOnNone reads false, the skip fires, and nothing ever issues the real delete -- the object sits there, permanently retried by Kubernetes' own ReplicaSet controller, since the designated node, once it reboots, has no config left telling it the app was deleted either. Add elsewhereReady, threaded through scheduledOnMe and its VMI/pod implementations, backed by a new kubeapi.IsNodeReady (reusing backupdnid.go's existing node-Get and Ready-condition helpers). Delete now only defers to elsewhere when that node is actually able to act; otherwise it does the delete itself. Info() and Stop() discard the new value -- their existing scheduledOnNone-based behavior is unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit df9492c)
Only rewrite a VMIRS's affinity back to this (DNID) node once the node currently named in it is confirmed gone from the cluster, not merely unreachable. A temporarily-down peer keeps its Node object, so this no longer fires the instant it reconnects, only once it's actually been replaced. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit f24a95a)
Creation always baked the creating node's own name into a new VMI or pod's affinity, never the app's actual designated node. During a backup-DNID failover the backup node runs the creation locally, so it stamped its own name -- meaning the live object's affinity never named the true home to begin with, no matter what zedkube's reconciler later did to the VMIRS template. Resolve AppInstanceConfig's DesignatedNodeUUID (now carried through DomainConfig) to its current node name and use that for the affinity, falling back to the local node if there's no UUID to resolve or the lookup fails. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit a40ce7f)
A replicated volume's own doUpdateVol short-circuit (status.IsReplicated -> CREATED_VOLUME, no real creation work) fires for every volume on a node that isn't its app's designated node -- confirmed live: a purge on a backup node left its new volume's PVC never even requested, so the VMI sat Pending on FailedPvcNotFound forever. The volume-generation policy driving the purge made no difference; the check is a static designated-node identity compare, unrelated to the volume's UUID or replica history. Bypass it once this node is confirmed eligible to act as backup DNID for that volume (holds the eve-app-op lease, the designated node has been unhealthy past the configured threshold, and the app is not Required-affinity) -- the same decision zedmanager already makes for app activation, reused here via kubeapi.IsCurrentlyBackupDNID. Needs its own DesignatedNodeUUID/AffinityType on VolumeConfig (cross-referenced from the owning app's VolumeRefList in parseVolumeConfig, mirroring the existing IsNativeContainer walk), since IsReplicated only ever encoded "not me", never "who". And needs its own re-drive: backup DNID changes nothing in the volume's own config, so nothing else notices a volume newly eligible for takeover -- added to volumemgr's existing gc-ticker batch of re-drive passes, alongside a new eve-app-op lease subscription mirroring zedmanager's. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 6b51afc)
An image-backed volume's content tree hits the same designated-node identity gate (IsLocal) as the volume itself and parks at LOADED with zero blobs, so doUpdateVol waits forever on casClient.GetImageHash -- keyed by the content tree's own UUID, which nothing dedupes across apps referencing identical content, so a sibling's download never helps. doUpdateContentTree never revisits an already-LOADED status, so this re-drives via the same delete-and-recreate path handleContentTreeModify already uses for a real DNID reassignment, passing the recreate a locally-overridden copy (IsLocal forced true). ContentTreeConfig gets its own DesignatedNodeUUID (carried directly on the wire, unlike volumes). Affinity is read off whichever volume(s) reference the content tree instead, since it has none of its own. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 6537841)
…ache isCurrentlyBackupDNIDWithClient did a live Nodes().Get() on every call, once per app in zedmanager and once per volume and content tree in volumemgr on every relevant pass -- flagged as expensive in review. zedkube now republishes every node's Ready status and its last transition time, keyed by device UUID, on every node's own appStatusTimer tick rather than only on the stats-lease holder. kubeapi.IsCurrentlyBackupDNID takes a NodeHealthLookup function instead of building a client itself; zedmanager and volumemgr each subscribe to the new publication and implement the lookup against their own local cache. The eligibility logic itself is unchanged: not-found, a zero/future transition time, and the ready/threshold checks all carry over exactly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 128a023)
doInstall deliberately skips its "volumes have reached CREATED_VOLUME" gate while a purge is in BringDown, because that phase is tearing the old domain down rather than standing a new one up. doUpdate then fell through to doActivate anyway, bringing the app up on volumes that are still being recreated -- a PVC binds and attaches from the moment it exists, so a domain can boot on one with nothing written into it. updateAIStatusUUID already routes BringDown to removeAIStatus and never reaches doUpdate, but its other callers do. Give all of them the same answer, in doUpdate itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit 0857877)
HandleCreated reported every CSI volume created as soon as the create worker returned, and that report is what lets a volume reach CREATED_VOLUME -- the state zedmanager gates app activation on. A PVC binds and attaches whether or not CDI ever wrote the image into it, so a volume whose upload did not finish would hand an app an empty disk to boot from. Confirm the upload actually completed for a volume that carries content, and report an error rather than success when it did not, or when it cannot be confirmed. A volume with no ReferenceName took the CreatePVC path and has no content to confirm. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit deafc1e)
ConfigGetTemporaryFail is set only while the new image is on trial, after the reboot into it, and this branch means to keep contending through that window. It did not cancel a stop already armed, though: zedagent arms ConfigGetFail before every request, so Fail -> TemporaryFail is the ordinary sequence there, and scheduleElectionStopAfter is a no-op while a timer exists. The armed stop fired ninety seconds in and released the lease mid-validation, the outcome the branch exists to prevent. The existing test only covered Success -> TemporaryFail, where nothing was armed, so the gap went unnoticed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com> (cherry picked from commit cbb17f4)
|
Claude finished @eriknordmark's task in 3m 29s —— View job Review: backport of #6505 to 17.0-stable
Verdict: looks sound. No blocking issues found. What I checked
Caveats on the fidelity verificationI was not able to do an exact I also could not compile-check ( Minor (non-blocking, pre-existing from master — not worth a change)A few doc comments on the volume/app gates still say "the one live API call" / "costs no API call" (e.g. If a maintainer wants a mechanical master-vs-backport diff, that needs a checkout with #6505's master commits fetched. |
|
/rerun red |
|
I have a local run of evetest against this PR running, I'll report back with results when it completes |
|
evetest run passes with PR #6889 |
Description
Backport of #6505
Cherry-picks #6505's 18 pillar commits from master with
-x, in masterorder. Every pick applies cleanly and none is edited.
The 5 evetest commits of #6505 are not picked: their content is already on
17.0-stable. 9f52a5d ("Copy evetest from master") brought in
evetest/tests/cluster/dnidbackup_test.go, identical to master, withTestDNIDandBackupDNIDregistered in the cluster suite. This PR adds thepillar side that test exercises.
One 17.0-only fixup commit, placed right after the pick it repairs:
62bedeb. On master, 62bedeb dropped zedkube.go's
sync/atomicimport,and evetest: port scans; fix the exposure they found on EVE and EVE-k #6665 (042622f), merged later, added it back. On 17.0-stable the
order is reversed: [17.0-stable] evetest: port scans; fix the exposure they found on EVE and EVE-k #6753, the backport of evetest: port scans; fix the exposure they found on EVE and EVE-k #6665, merged first and kept the
import that was already there. So the unedited 62bedeb pick removes an
import that
clusterStatusCredsstill uses, and zedkube doesn't compile.The fixup restores the one line, and the import block then matches master.
The net diff of this PR is #6505's diff outside
evetest/(the same 42files), minus that one import removal.
Three picks differ from their master commit only in surrounding context,
where 17.0-stable's code differs: 180e5f8, 21bb347 and 5d8f747. The
added and removed lines are identical.
PR dependencies
The four 17.0-stable pillar prerequisites are merged:
How to test and validate this PR
Same as #6505:
TestDNIDandBackupDNID, already on 17.0-stable, on athree-node EVE-k cluster:
Build checks:
make ZARCH=arm64 HV=k pkg/pillar(includesgo vet -tags kand gofmt) andmake -C pkg/pillar test(1918 tests,0 failures) pass.
Changelog notes
EVE-K clusters: when an app instance's designated node has been unhealthy
for longer than
cluster.dnid.backupnode.threshold(default 10 minutes),another cluster node can now activate, deactivate, purge, or delete that
app on its behalf, and the app automatically moves back once its
designated node recovers (requires
kubernetes.vmi.deschedule.eventstoinclude
boot). Applies only to apps configured withPreferrednodeaffinity; apps configured with
Requiredaffinity are unaffected andcontinue to wait for their designated node to come back.
Checklist
🤖 Generated with Claude Code