Skip to content

vmcluster/vlcluster/vtcluster: maintenance-excluded storage pods can still block RollingUpdate-strategy rollouts #2751

Closed

Description

@AndrewChubatiuk

Background

#2717 adds maintenanceInsertNodeIDs/maintenanceSelectNodeIDs-based skipping of the storage pod readiness wait during reconcile, so a storage node under planned maintenance (e.g. a cordoned/drained Kubernetes node) no longer blocks unrelated reconciles.

This works correctly under the default rollingUpdateStrategy: OnDelete, where the operator drives pod updates itself via performRollingUpdateOnSts, which already defers updates for maintenance-excluded ordinals.

Problem

Under rollingUpdateStrategy: RollingUpdate, the operator delegates entirely to Kubernetes' native StatefulSet controller. That controller:

  • uses PodManagementPolicy: OrderedReady (set by the operator whenever RollingUpdate is selected), which blocks pod creation/deletion during scaling until each lower-ordinal pod is Ready, and
  • always rolls out template/revision changes ordinal-by-ordinal, waiting for each updated pod to become Ready before proceeding — regardless of PodManagementPolicy.

If a storage pod is under maintenance and therefore not expected to become Ready, either a replica-count change or a plain spec/image update can leave the StatefulSet stuck: status.updatedReplicas/status.replicas never reach the target, and the operator's reconcile wait (waitForStatefulSetReady) times out repeatedly until the maintenance pod recovers.

This is a Kubernetes-level constraint (native OrderedReady/RollingUpdate semantics), not a bug in the readiness-wait logic itself — PodManagementPolicy is immutable after StatefulSet creation, so it can't be toggled dynamically based on maintenance state, and even a Parallel-by-default change would only help new StatefulSets and would only fix the scaling half of the problem, not revision rollouts.

Proposed fix

Stop delegating to Kubernetes' native RollingUpdate controller for storage components, and instead route RollingUpdate-strategy storage StatefulSets through the same maintenance-aware, operator-driven update path already used for OnDelete (performRollingUpdateOnSts), so the operator — not Kubernetes — drives the rollout and can defer maintenance ordinals consistently regardless of the configured rollingUpdateStrategy.

Affected components: vmstorage (VMCluster), vlstorage (VLCluster), storage (VTCluster).

See discussion in #2717 (review thread on internal/controller/operator/factory/reconcile/statefulset.go, comment from @vrutkovs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions