Skip to content

fix(pulumi): stop the no-destroy guard refusing every vm/k3s re-provision - #570

Merged
smecsia merged 1 commit into
mainfrom
fix/destructive-replace-remote-command
Nov 8, 2026
Merged

smecsia merged 1 commit into
mainfrom
fix/destructive-replace-remote-command

Conversation

@smecsia

@smecsia smecsia commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

The bug

Since the destructive-replace guard landed (#556), the second sc provision of any vm or k3s parent stack aborts before touching anything.

SC nonces its remote commands on purpose: every install, readiness gate and remote read carries the run timestamp in triggers via pulumiVm.TriggersForGuardedChange / TriggersForRead, so a host whose k3s was stopped out of band is repaired rather than reported as "no changes" (the scripts are idempotent and carry their own args-marker guard — k3s's installer answers an unchanged config with already running as k3s with the requested configuration). A replace on every run is therefore the normal shape of a vm/k3s provision, and isDestructiveReplace flags any OpReplace on a resource with a live id unless its type is allowlisted.

Live-caught 2026-11-08 on a 3-node k3s stand:

Error executing command: failed to create stack "fleet": input-driven destructive replace:
preview would destroy 9 live resource(s) by re-pointing them at new objects. ...
  - urn:...::command:remote:Command::smoke-cluster-install--staging (command:remote:Command): keys=[]
  - urn:...::command:remote:Command::smoke-vm-host-0-ready--staging (command:remote:Command): keys=[]
  ... 7 more

Six k3s commands and three per-host readiness gates, every one a nonce-only triggers diff. The engine reported keys=[] — so the guard's own explanation ("a changed REPLACE-forcing input (zone id, region, immutable name, …)") did not even apply to what it was refusing, which is its own small bug and is also fixed here.

Why not the safe-to-replace allowlist

api.safeToReplaceTypes is per-TYPE, and command:remote:Command is not uniformly safe. A command carrying a delete script runs that script against a live machine when it is replaced, because pulumi-command replaces as create-then-delete. SC has exactly that shape in two places — the k3s uninstall (k3s/cluster.go's uninstallOnDestroy) and the vm ingress teardown (vm/caddy.go) — and both are deliberately registered with no triggers so that nothing but a destroy runs them. An input-driven replace of one of those (a changed connection address, a changed script) is a real teardown and is what the guard is for.

So the question is per-RESOURCE, and the answer is in the inputs: isStatelessRemoteCommand exempts a remote command only when neither the old nor the new state declares a non-empty delete.

Verified live, both directions

Against the same 3-node k3s stack, no SC_ALLOW_DESTRUCTIVE_REPLACE:

  • re-provision now completes — exit=0, {"replace":9,"same":6}, zero guard hits;
  • a probe build confirmed the engine does populate StepEventStateMetadata.Inputs on these events (inputKeys=6 hasDelete=false keys=[]), so the exemption is precise rather than degenerate;
  • the stack's three *-k3s-uninstall resources store delete as a plain 230-char string in their inputs, so hasCommandDelete sees it and they stay guarded.

The chosen failure direction is documented in the code: if inputs ever become unreadable, a remote command becomes exempt rather than wedging every provision — its id names no cloud object (it is a random suffix on the resource name), so the guard was never protecting state here, only the delete script.

Tests

go test ./pkg/clouds/pulumi/... -count=1 green. Added:

  • two cases to TestIsDestructiveReplace_OpCoverage — the nonce-triggered command (safe) and the delete-carrying uninstall (guarded);
  • TestIsStatelessRemoteCommand — wrong type, no delete key, empty/whitespace delete, delete on the new side only, non-string (secret placeholder) delete, nil states;
  • TestDestructiveReplaceError_NamesTriggersWhenNoKeys — no more keys=[], and the grep-stable message prefix is unchanged.

…sion

SC nonces its remote commands on purpose: every install, readiness gate
and remote read carries the run timestamp in `triggers` via
TriggersForGuardedChange / TriggersForRead, so a host whose k3s was
stopped out of band is repaired instead of being reported as "no
changes". A replace on every run is therefore the normal shape of a vm
or k3s provision — and since the destructive-replace guard landed, it is
also an outright refusal: the second `sc provision` of a stack aborts
before touching anything.

Live-caught 2026-11-08 on a 3-node k3s stand. Nine resources flagged
(six k3s commands, three per-host readiness gates), every one of them a
nonce-only trigger diff, and the engine reported `keys=[]` — so the
guard's own explanation ("a changed REPLACE-forcing input") did not even
apply to what it was refusing.

The exemption is per-RESOURCE, not per-type, so it cannot go on the
safe-to-replace allowlist: a remote command carrying a `delete` script
runs that script against a live machine when it is replaced
(pulumi-command replaces as create-then-delete), and the k3s uninstall
and the vm ingress teardown are exactly that shape. The question is
whether this command has a delete, and the answer is in its inputs.

Also stop rendering `keys=[]` under a sentence blaming a changed input;
point at `triggers` instead. The grep-stable message prefix is unchanged.

Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

Semgrep Scan Results

Repository: api | Commit: e923306

Check Status Details
⚠️ Semgrep Warning 11 warning(s), 15 total

Scanned at 2026-11-08 13:51 UTC

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

Security Scan Results

Repository: api | Commit: e923306

Check Status Details
✅ Secret Scan Pass No secrets detected
✅ Dependencies (Trivy) Pass 1 total (no critical/high)
✅ Dependencies (Grype) Pass 1 total (no critical/high)
📦 SBOM Generated 517 components (CycloneDX)

Scanned at 2026-11-08 13:52 UTC

@smecsia
smecsia enabled auto-merge (squash) November 8, 2026 15:36
@smecsia
smecsia merged commit 566307d into main Nov 8, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants