Repository navigation
Redfish: Host-HA fails to mark a powered-off KVM host as Down #13376
Description
TL;DR When Host-HA tries to fence an already shut down host via Redfish BMC driver - host status never moves the host into the Down state --> VM-HA never kick (VMs never get started on other hosts)
Redfish: Host-HA never marks a powered-off KVM host Down because the fence (Redfish OOBM power-off) can't succeed against an already-off chassis — VM-HA only triggers once the dead host is powered back on
ISSUE TYPE
- Bug Report
COMPONENT NAME
HA (host-HA framework), Out-of-band Management (Redfish/IPMI), KVM
CLOUDSTACK VERSION
Confirmed present with identical (or functionally identical) logic on:
- tag 4.22.1.0 (analyzed in detail)
- branch 4.22 (origin/4.22 @ 21b2025c) — all key files byte-identical to 4.22.1.0
- branch main (origin/main @ 6bc83a3c) — all key files byte-identical to 4.22.1.0
- branch 4.20 (origin/4.20 @ a3970bb1) — same logic; differences are cosmetic only
(method rename getHostStatus() -> getHostStatusFromHAConfig(); logger formatting)
The host-HA + OOBM fence design predates 4.20, so earlier 4.x releases are very likely affected too.
Per-branch verification of the relevant elements:
- KVMHAProvider.fence() = OOBM PowerOperation.OFF, returns resp.getSuccess(): same on 4.20 / 4.22 / main
- FenceTask: only transitions to Fenced on success; retries Fencing otherwise: byte-identical on 4.20 / 4.22 / main
- HAManagerImpl host-status mapping (Fenced->Down, Fencing->Disconnected): same on 4.20 (getHostStatus) and 4.22/main (getHostStatusFromHAConfig)
- RedfishWrapper: PowerOperation.OFF -> RedfishResetCmd.GracefulShutdown: byte-identical on 4.20 / 4.22 / main
- RedfishClient: throws unless HTTP status in 2XX (SC_OK..SC_MULTIPLE_CHOICES): byte-identical on 4.20 / 4.22 / main
CONFIGURATION
- KVM cluster with host-HA enabled on the hosts.
- Out-of-band Management enabled per host (reproduced with the Redfish driver against Dell iDRAC; the same logic applies to the ipmitool driver - but ipmi is not affected by the bug).
- VM-HA enabled (
VmHaEnabled). - Primary storage: Linstor (not material —
isStorageSupportHA() == true, so the legacy investigator is not the bottleneck here).
OS / ENVIRONMENT
- Management servers: Ubuntu 24.04, OpenJDK 21.
- Hypervisors: KVM.
- BMC: Dell iDRAC via Redfish (
/redfish/v1/Systems/System.Embedded.1).
SUMMARY
When a KVM host that has host-HA + OOBM enabled is hard powered off (e.g. forced chassis-off from the BMC console, or a real power/cable failure), CloudStack never transitions the host to Down and therefore never restarts its VMs on other hosts. The host stays in Alert/Disconnected indefinitely.
Root cause: the host-HA state machine only declares a host dead (HAState.Fenced → investigator Status.Down) after a successful fence, and the fence is implemented as an active OOBM power-off. Against an already-off chassis that power-off cannot succeed (the BMC rejects it), so the host is pinned in the Fencing state and retried forever. The investigator maps Fencing to Status.Disconnected, not Status.Down, so VM-HA is never invoked.
The perverse result: the VMs are only recovered once the original (dead) host is powered back on - even during BIOS booting stage — at which point the pending power-off finally succeeds, the host transitions to Fenced/Down, and HA restarts the VMs elsewhere. This defeats the purpose of HA.
All three current branches are affected by the identical issue: the relevant code is byte-identical on 4.22 and main, and functionally identical on 4.20 (only a method rename and logger formatting differ). There is no 4.21 branch upstream. Per-element diff verification is in the CLOUDSTACK VERSION section below.
STEPS TO REPRODUCE
- KVM cluster, host-HA enabled, OOBM (Redfish) configured and enabled on the hosts, VM-HA enabled. Place some HA-enabled VMs (incl. system VMs) on
hostA. - Forcefully power off
hostAat the BMC (chassis power off / simulate power loss). The BMC itself stays reachable. - Observe
hostAin CloudStack over the next 20+ minutes.
EXPECTED RESULTS
- Health check fails → activity check fails → host is fenced → host marked
Down→ VM-HA restartshostA's VMs on other hosts within a few minutes.
ACTUAL RESULTS
hostAremains inAlert(host status) with the host-HA state stuck inFencing.- The OOBM STATUS poll correctly reports the chassis as
Offthe entire time, but that knowledge is never used to declare the host down. - The agent investigator repeatedly reports the host as
Up(while HA state isSuspect) and thenDisconnected(while HA state isFencing) — neverDown. - VMs are not restarted; the scheduler keeps preferring the VM's last host (the dead
hostA). - The instant
hostAis powered back on, the fence power-off finally succeeds → host goesDown→ VM-HA restarts the VMs on other hosts.
ROOT CAUSE ANALYSIS
Decision chain (only Fenced yields Down)
- For an HA-eligible KVM host, the legacy investigator delegates to the host-HA framework:
KVMInvestigator.getHostAgentStatus()→haManager.getHostStatusFromHAConfig(host)
(plugins/hypervisors/kvm/src/main/java/com/cloud/ha/KVMInvestigator.java:81)
HAManagerImpl.getHostStatusFromHAConfig()maps HA state → host status
(server/src/main/java/org/apache/cloudstack/ha/HAManagerImpl.java:315):Fenced→Status.DownDegraded/Recovering/Fencing→Status.Disconnected- everything else (
Available/Suspect/Checking/Recovered) →Status.Up
AgentManagerImplonly fires theHostDownevent andscheduleRestartForVmsOnHost(...)when the investigator returnsStatus.**Down**
(engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:1147,:1200).
So VM-HA for an HA-eligible KVM host requires the host-HA state machine to reach Fenced.
Reaching Fenced requires a successful power-off
- The state machine only goes
Fencing → FencedonEvent.Fenced
(api/src/main/java/org/apache/cloudstack/ha/HAConfig.java:139). FenceTask.processResult()only firesEvent.Fencedwhen the fence returnedtrue; otherwise it does nothing and the poll loop retriesFencingforever viaRetryFencing
(server/src/main/java/org/apache/cloudstack/ha/task/FenceTask.java:45; retry atserver/src/main/java/org/apache/cloudstack/ha/HAManagerImpl.java:724).- The fence is an active OOBM power-off:
KVMHAProvider.fence()→outOfBandManagementService.executePowerOperation(host, PowerOperation.OFF, null)and returnsresp.getSuccess()
(plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java:87). executePowerOperation()throwsCloudRuntimeExceptionwhenever the driver response is not successful — it never returnssuccess=false
(server/src/main/java/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java:432).
Why the power-off fails against an already-off host (Redfish)
- The Redfish driver maps
PowerOperation.OFF→RedfishResetCmd.GracefulShutdown
(plugins/outofbandmanagement-drivers/redfish/src/main/java/org/apache/cloudstack/outofbandmanagement/driver/redfish/RedfishWrapper.java:34). RedfishClient.executeComputerSystemReset()POSTs to.../Actions/ComputerSystem.Resetand throwsRedfishExceptionif the HTTP status is not 2XX
(utils/src/main/java/org/apache/cloudstack/utils/redfish/RedfishClient.java:300-312).- An already-off system returns HTTP 409 (Conflict) — a
GracefulShutdownis invalid because there is no running OS to shut down. 409 ∉ 2XX →RedfishException→CloudRuntimeException→HAFenceException→FenceTaskseesresult=false→ noFencedtransition → stuck inFencing. - (The ipmitool driver has the analogous failure mode:
chassis power offagainst an already-off / unreachable BMC returns a non-zero exit code, judged purely by process exit status with no "already in target state" handling —IpmitoolWrapper.executeCommands()→result.isSuccess().)
Net effect
The fence requires confirming an active power-off transition, but a host that is already off (precisely the case where restarting its VMs is safe) cannot be "powered off successfully." The safety mechanism deadlocks in exactly the scenario it exists to handle. VMs recover only when the dead host returns.
LOG EVIDENCE (two-MS cluster; host kvm-host01, id:1, uuid aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee; hostnames/IPs/VM names below are anonymised examples)
OOBM STATUS poll knew the chassis was off the whole time (MS #1 log):
15:10:47 OutOfBandManagementServiceImpl Transitioned out-of-band management power state from On to Off
due to event: Off(Chassis Power is Off) for Host {id:1, kvm-host01}
Investigator never returns Down — Up while Suspect, then Disconnected while Fencing (MS #1 log):
15:07:51 KVMInvestigator was able to determine host {id:1} is in Up
... is considered Up (...). State: Suspect, Most recent health check failed.
15:14:51 HAManagerImpl HA: Agent [{id:1}] is disconnected. State: Fencing, The resource is undergoing fence operation.
The fence itself, on the MS node that owns the HA config (MS #2 log) — repeated every ~4s for ~20 min:
15:14:20 (first) ... it got Redfish httpmresponce '409'
15:14:28 KVMHAProvider OOBM service is not configured or enabled for this host {id:1} error is
Failed to execute System power command ... 'POST' ...
'.../Actions/ComputerSystem.Reset' ... The expected HTTP status code is '2XX' but it got '409'.
15:14:28 FenceTask Exception occurred while running FenceTask ...
org.apache.cloudstack.ha.provider.HAFenceException ... at KVMHAProvider.fence(KVMHAProvider.java:99)
Counts over the outage: ~618 × 409, ~308 × HAFenceException, 930 × Fencing state lines, Starting HA on ... = 1 (only at the very end).
VM-HA only fires after the host is powered back on (MS #2 log):
15:35:03 HighAvailabilityManagerExtImpl Scheduling restart for VMs on host {id:1, kvm-host01}
15:35:03 Host [kvm-host01 (id:1) ...] is down. Starting HA on the following VMs: vm-app01 vm-app02
(chassis Off→On detected ~15:35:05 in MS #1 log.)
SECONDARY BUGS surfaced by this incident
-
Misleading error message. Every fence failure logs
OOBM service is not configured or enabled for this host ..., but OOBM is configured and working. The catch-all inKVMHAProvider.fence()(plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java:97-100) assumes any exception means "OOBM not configured," hiding the real cause (HTTP 409 / already off). This actively misdirects troubleshooting. -
Misleading "fencing performed" alerts. Each failed fence attempt emits
alertType=30 — "HA Fencing of host id=1 ... performed"becauseFenceTask.processResult()callssendAlert(resource, HAState.Fencing)unconditionally regardless ofresult(server/src/main/java/org/apache/cloudstack/ha/task/FenceTask.java:54). Admins receive a flood of "fencing performed" alerts while fencing is in fact failing continuously.
SUGGESTED FIX (direction)
Make fencing treat "host is already off" as a successful fence, and stop hiding the real error:
- In
KVMHAProvider.fence(), query OOBM power STATUS first; if the chassis is alreadyOff, returntrue(host is effectively fenced) instead of issuing a power-off that 409s. (A confirmed-off host is safe to declare fenced.) - Redfish driver: treat an idempotent power-off (target state already reached, HTTP 409 on
GracefulShutdown/ForceOffwhen already off) as success; and/or preferForceOffoverGracefulShutdownfor the HA fence path. - Fix the
fence()catch block to surface the actual driver error rather than "OOBM not configured." - Make
FenceTaskalerts reflect actual success/failure of the fence.
NOTES
- Analysed against git tag
4.22.1.0. - Storage (Linstor) is not the bottleneck:
LinstorPrimaryDataStoreDriverImpl.isStorageSupportHA()returnstrue, so the legacy KVM investigator does not short-circuit; the host-HA framework path (above) is in effect.
Activity
andrijapanicsb commentedon Jun 8, 2026
Testing on CloudStack provider/driver (nested labs) never exposes this bug, as sending the force stop VM to parent ACS lab, doesn't return error, so nested hypervisor is always marked as Down - and VM HA would begin - i.e. we could never catch this bug in virtual/ACS env/testing.
So when it comes to "ok we should fence the host" - first ask "are you dead/off" - if yes, fine continue, if not, do ipmi/redfish/cloudstack STONITH, then check to make sure host is dead - do NOT relly on status code/error returned from IPMI/Redfish/ParentACS - do explicit check:
Update 11/06: for ipmi 2.0 -sending POWER OFF to a host which is alredy OFF (shutdown or previously executed ipmi.....power off command) - does NOT report errors - exit code ($?) = 0 always
[-]Host-HA never marks a powered-off KVM host Down because the OOBM fence (power-off) fails against an already-off chassis — VMs only recover after the dead host is powered back on[/-][+]Redfish: Host-HA never marks a powered-off KVM host Down because the OOBM fence (power-off) fails against an already-off chassis — VMs only recover after the dead host is powered back on[/+][-]Redfish: Host-HA never marks a powered-off KVM host Down because the OOBM fence (power-off) fails against an already-off chassis — VMs only recover after the dead host is powered back on[/-][+]Redfish: Host-HA fails to mark a powered-off KVM host as Down[/+]KVM HA: fence by confirming host power state, not the power-off result
KVM HA: fence by confirming host power state, not the power-off result
nikolauseppinger commentedon Jul 14, 2026
Independent reproduction on a second site with the
ipmitooldriver (this issue and #12921 are both Redfish — this confirms the failure mode is driver-agnostic, as the issue text anticipated).Environment
isStorageSupportHA() == true, so the legacy investigator is not the bottleneck — same as your Linstor note).What we observed — two distinct failure modes
Mode 1 — real power loss (both PSUs pulled), BMC therefore unreachable → OOBM fence can never complete → stuck in
Fencing:This repeated every ~4 s for the whole outage. As you note in "Secondary Bug #1", the message is misleading — OOBM is configured; the real cause is that the
ipmitoolpower-off against a powerless BMC throws (the BMC/iDRAC itself had no power), andKVMHAProvider.fence()'s catch-all masks it. The host only reachedFencedonce the iDRAC/BMC regained power and finished booting — at that point the still-retryingipmitoolpower-off finally succeeded — after which:(repeating every ~4 s — this is the spam addressed by #13588). VMs were only restarted after the host came back — exactly the "perverse result" described here.
Mode 2 — BMC reachable (graceful iDRAC shutdown, and separately a kernel panic
echo c > /proc/sysrq-trigger): host down, storage heartbeat stale, no failover for a full 20 minutes with Host-HA enabled. (We did not have DEBUG onorg.apache.cloudstack.haenabled, so we cannot yet pin the exact FSM state for this mode — will re-run with DEBUG.)A/B control — the clearest single piece of evidence: same kernel-panic failure, with Host-HA disabled, only legacy VM-HA active:
VMs back within ~5–6 minutes. So on this hardware the legacy path works fine; enabling Host-HA is what suppresses it — while the host-HA state is not
Fenced,KVMInvestigator.isVmAlive→HAManagerImpl.isVMAliveOnHost/getHostStatusFromHAConfigreportsUp/Disconnected, so VM-HA defers indefinitely.Takeaways
ipmitoolpath fails the same way when the BMC is unreachable. IPMI's "already-off returns exit 0" only helps when the BMC still has power; in a real power loss (BMC down with the host) the power-off simply throws.Fenced → IneligibleNoTransition spam and the misleadingfence()catch-all ("Secondary Bug 4.2 #1") both reproduce here verbatim.kvm.ha.fence.on.storage.heartbeat) uses as a fence fallback when OOBM fails. Between them they cover both "off but BMC alive" (KVM HA: fence by confirming host power state (fix host stuck in Fencing when already powered off) #13377) and "BMC dead with the host" (KVM: add storage-heartbeat fencing fallback when OOBM fence fails (host stuck in Fencing on total power loss) #13589).We're glad to help validate the reworked #13377 on a setup it hasn't been exercised on yet:
ipmitooldriver + SharedMountPoint (GFS2) zone-wide storage, incl. the real power-loss case. Happy to share full logs.EduFrazao commentedon Aug 4, 2026
Same problem here with Cloudstack 4.22.1.0, OpenBMC RedFish.
HA is never kicked, all vms on host are seen as started, but they are down, server HA status transits between degraded and suspect.
kiranchavala commentedon Aug 5, 2026
@EduFrazao Please test with the following global setting
kiranchavala commentedon Aug 5, 2026
@EduFrazao i have tested with a mockup redfish simulator and the host ha works
we have fixed vm ha issue with host ha and will be fixed in the upcoming 4.22.2 or 4.23
#13371
#13373
DaanHoogland commentedon Aug 5, 2026
@andrijapanicsb @EduFrazao , can you test 4.22-HEAD with real redfish hard-/firmware?
EduFrazao commentedon Aug 5, 2026
Tested. Same behavior: Host transits from degraded to suspect, VMs on it never gets started on another host (shows as running).
EduFrazao commentedon Aug 5, 2026
Very nice!!! Thank you very mutch.
EduFrazao commentedon Aug 5, 2026
Yes! There is pre-built packages of this testing version?
DaanHoogland commentedon Aug 5, 2026
No, the nightly builds should contain these. But I don’t think anyone in the community provides these publicly.
edit 🤦 : yes there are : http://download.cloudstack.org/testing/nightly/2026-08-05/