You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
vmcluster/vlcluster/vtcluster: maintenance-excluded storage pods can still block RollingUpdate-strategy rollouts #2751
#2717 adds maintenanceInsertNodeIDs/maintenanceSelectNodeIDs-based skipping of the storage pod readiness wait during reconcile, so a storage node under planned maintenance (e.g. a cordoned/drained Kubernetes node) no longer blocks unrelated reconciles.
This works correctly under the default rollingUpdateStrategy: OnDelete, where the operator drives pod updates itself via performRollingUpdateOnSts, which already defers updates for maintenance-excluded ordinals.
Problem
Under rollingUpdateStrategy: RollingUpdate, the operator delegates entirely to Kubernetes' native StatefulSet controller. That controller:
uses PodManagementPolicy: OrderedReady (set by the operator whenever RollingUpdate is selected), which blocks pod creation/deletion during scaling until each lower-ordinal pod is Ready, and
always rolls out template/revision changes ordinal-by-ordinal, waiting for each updated pod to become Ready before proceeding — regardless of PodManagementPolicy.
If a storage pod is under maintenance and therefore not expected to become Ready, either a replica-count change or a plain spec/image update can leave the StatefulSet stuck: status.updatedReplicas/status.replicas never reach the target, and the operator's reconcile wait (waitForStatefulSetReady) times out repeatedly until the maintenance pod recovers.
This is a Kubernetes-level constraint (native OrderedReady/RollingUpdate semantics), not a bug in the readiness-wait logic itself — PodManagementPolicy is immutable after StatefulSet creation, so it can't be toggled dynamically based on maintenance state, and even a Parallel-by-default change would only help new StatefulSets and would only fix the scaling half of the problem, not revision rollouts.
Proposed fix
Stop delegating to Kubernetes' native RollingUpdate controller for storage components, and instead route RollingUpdate-strategy storage StatefulSets through the same maintenance-aware, operator-driven update path already used for OnDelete (performRollingUpdateOnSts), so the operator — not Kubernetes — drives the rollout and can defer maintenance ordinals consistently regardless of the configured rollingUpdateStrategy.
Activity