Repository navigation
feat: multi-node k3s clusters and multi-host vm pools (Phase 1a–1c) - #563
Merged
Merged
Conversation
Lifts the two config-tier refusals that kept a `vm` resource single-host: byo refused `len(hosts) > 1` and cloud refused `len(zones) > 1`. Everything below the config tier was already plural — `Driver.Hosts` returns a slice, the plugin wire protocol carries `repeated VmHostFacts`, and pulumi/vm/vm.go already loops every host with per-host readiness, per-host ingress and indexed exports. The refusals were the whole blocker. `VmHost.Role` lets a byo pool say which machine runs what (`compose`, `k3s-server`, `k3s-agent`), validated all-or-none: a half-declared pool is a pool whose remaining hosts are a guess, and the guess decides whether a machine joins a cluster. Mixing `compose` with a k3s role in one resource is refused — one machine, one port 443. An even server count above one is refused with the etcd-quorum reason rather than rounded. `VmConfig.Nodes` asks a cloud driver for N machines, defaulting to the zone count and then to 1, with zones assigned round-robin by index so host *i* never moves zone on a later apply. Pure config tier: the plugin that honours it is a separate slice. Roadmap: sc-vm-multi-node-clusters. Design: forge/docs/design/2026-11-07/multi-node-clusters/README.md Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
… etcd Lifts the third refusal: `k3s-cluster` accepted a `vm` resource with exactly one host. It now installs the whole pool. `servers:` is the topology, defaulting to 1 — one server and N-1 agents, which schedules pods across the fleet and survives the loss of any agent, and is the only shape a 2-machine pool can have. Set it to 3 (or 5) for an embedded-etcd control plane that survives losing a control-plane node. Even counts above one are refused: etcd elects by majority, so 2 members tolerate zero failures and are strictly worse than 1. On a byo pool that names its own `hosts[].role` the roles win and a contradicting `servers:` is refused rather than silently losing. How a node joins: - The join token is read off host 0 (`node-token`) rather than generated by SC. Both work for a cluster SC creates; only this works for the cluster SC finds, which is the normal case on byo hosts — a machine that already has a k3s datastore keeps its own token and would reject one SC invented. - The token travels as `K3S_TOKEN`, never in the argument vector: it would land in the on-host args marker and the Pulumi checkpoint, and would reinstall a healthy control plane the next time it was rotated. Its stdout carries both `LoggingNone` and `AdditionalSecretOutputs` — pulumi-command does not mark stdout secret, which is how a Yandex SA key reached a public log on 2026-09-30. - Servers after the first are chained, one at a time: etcd serialises member changes, so two joins racing means one gets a rejected proposal and fails a provision on a cluster that is otherwise fine. Agents then join in parallel, ordered after the last server. - `--tls-san` covers the endpoint and every SERVER address, deliberately not the agents'. Agents serve no API, and including them would change every server's args marker when a worker is added — scaling workers must not reinstall the control plane. - The readiness gate now counts the whole fleet. Without the count it passes the instant the first server registers, so a deploy ordered after it schedules onto a cluster of one machine; "no node is NotReady" does not catch it either, because an unregistered node is absent, not NotReady. Host 0's install keeps its un-suffixed resource name, so an existing single-node stack sees an update rather than a replaced control plane, and a single-host cluster gains no new resources at all — not even the token read. Also folds in the two filed single-node bugs that a fleet amplifies: - sc-k3s-cluster-resource-no-delete: k3s is now uninstalled on destroy. On one machine a leftover k3s is a nuisance; on a fleet it is N nodes holding ports 80/443 against whatever is deployed there next. The Delete lives on its own no-trigger resource — the installs are replaced every deploy by their nonce and pulumi-command replaces create-then-delete, so a Delete on an install would tear down what it just built (live on the vm ingress until 2026-10-18). - sc-vm-k3s-port80-race-not-validated: an explicit `disable:` that leaves traefik or servicelb enabled on an ingressed host is now refused, naming the components and both remedies. The default set already avoided the race; the escape hatch walked straight into it and provisioned green. Roadmap: sc-vm-multi-node-clusters, sc-k3s-cluster-resource-no-delete, sc-vm-k3s-port80-race-not-validated. Design: forge/docs/design/2026-11-07/multi-node-clusters/README.md Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
smecsia
requested review from
Cre-eD,
Laboratory and
universe-ops
as code owners
November 7, 2026 19:21
Semgrep Scan ResultsRepository:
Scanned at 2026-11-08 15:37 UTC |
Security Scan ResultsRepository:
Scanned at 2026-11-08 15:38 UTC |
`compose_stack.go` read the vm resource's host count, warned "this slice deploys onto the first one only", and then read index 0 seven times. A vm resource can hold N machines since the config tier was unblocked, so this makes the deploy a fleet deploy: one full container set per machine, rolled one machine at a time. Three properties carry the slice, and none of them would show up in a passing deploy — hence compose_fleet_test.go: - Host 0's resource names are byte-identical to a single-host deploy's (`hostResSuffix(0)` is empty). A renamed remote.Command is a NEW resource whose Create runs while its predecessor's Delete tears down the containers it just started, so growing a fleet has to APPEND machines, not replace the one already serving. Asserted by running the single-host fixture and the three-host fixture and requiring every step name of the first to survive in the second. - The machines are ordered one behind the next, on the previous host's `finalize` and not its `up`. A fleet exists so a bad release does not take every replica with it, and Pulumi would happily deploy all three at once — the dependency edge is the only thing that stops the roll at the machine that failed. `finalize` because between `up` and `finalize` a machine is serving a slot its state file does not yet name. - An image is built ONCE and shipped once per machine (`buildImages` no longer takes a host; `shipImagesTo` does). Registering the build per host would be N identical docker builds under N resource names. Blue-green stays per machine: two hosts of one fleet can legitimately be on different slots, and forcing them into step would mean reading one machine's state and acting on another's. `scale.max > 1` stops being refused in `convert.go`. The ask is satisfiable now — one replica per machine, so three machines serve three — but the machines are declared in the PARENT stack, which a client stack's config tier cannot see. So that tier warns where the replica count actually comes from, and `VmComposeStack` compares it against the real host count, which is the first place both numbers are in hand. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
…rivate addresses SC sets `--node-ip`, `--node-external-ip`, `--advertise-address` and `--flannel-iface` nowhere in pkg/, so k3s falls back to its own default: the address of the machine's default-route interface. On a cloud VM reached through a floating or NAT IP that is already the private address and everything is correct. On a single-NIC VPS with a directly attached public IP it is the PUBLIC address — and then etcd peer traffic (the cluster's whole state, every Secret) and flannel's VXLAN overlay, which pkg/clouds/k3s/cluster.go runs unencrypted, cross the public internet on a `byo` host that has no firewall to stop them. That deploy SUCCEEDS. `kubectl get nodes` is green and nothing in the output says the overlay is exposed, which is why the config tier refuses the shape rather than warning about it. `VmHost` gains `privateAddress` — the address a machine's SIBLINGS dial it on, as opposed to the `address` SC dials from outside — and `validateHostRoles` requires it on every entry of a multi-host k3s pool, all-or-none, duplicates refused. The check runs AFTER the role and server-count checks: a pool that is the wrong shape should say so rather than complain about addresses it would never have used. Exempt, and both have to be: a single-node cluster, which has no node-to-node traffic at all, so every k3s config written before this field keeps validating; and a compose fleet, whose machines never talk to each other (the only shared state is the ACME store, reached outbound over HTTPS). `HostFacts` gains the matching Output, and the four k3s consumers read it through `EffectivePrivateAddress()` — the SAN list, the join URL, and the node-ip handed to each server and agent. The accessor is not a convenience: every cloud VM driver is an out-of-tree plugin compiled against an older HostFacts, so the field arrives as the ZERO sdk.StringOutput, which carries no OutputState and panics inside pulumi's own gatherJoinSet the moment it reaches sdk.All rather than resolving to "". One accessor fixes that for every present and future consumer. node-ip is left empty on a single-host pool on purpose, so the args marker stays byte-identical and no existing control plane reinstalls for no behaviour change. On a server both `--node-ip` and `--advertise-address` are set, to the same value: without the second the API server still advertises its default-route address to its peers, which is the value this exists to avoid. The agent gets `--node-ip` only; `--advertise-address` is server-only and an agent handed it exits on an unknown flag. Hosters without private networking have nothing to put in the field, and that is the case the WireGuard mesh of roadmap item sc-vm-multi-node-clusters exists for. Until it lands the refusal points those operators at a compose fleet. See forge design 2026-11-07/multi-node-clusters §5.8. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
…ocument the fleet shapes The `nodes` field rides the marshalled VmConfig to the plugin, so a vm plugin built before the field existed unmarshals it into nothing and creates ONE machine for a resource that asked for three. That is a green provision, a `<vm>-vm-host-count` of 1, and a k3s pool or compose fleet that is silently a single machine — the same invisible-failure class as a multi-node pool with no `privateAddress`, and refused for the same reason. `byo` satisfies the check by definition, so it guards the cloud drivers, every one of which is out-of-tree. The error names the driver, both counts and the likely cause, because otherwise an operator reads it as their own server.yaml being wrong. Docs: the example `server.yaml` files still described the single-host shape that the three lifted refusals made stale. Each now carries the multi-node block it applies to — the k3s `hosts:` pool with `privateAddress:` and why that is refused rather than defaulted, `servers: 1` vs the odd-count etcd opt-in, the compose fleet and the two things it needs (`ingress.acmeStorage` and multi-A, with round-robin stated as round-robin and not failover), and `nodes:`/`zones:` as the fleet's size. The cloud examples say plainly that no driver creates N machines yet and that SC refuses the mismatch, so none of this reads as a fleet an operator can stand up today. On Evolution specifically, zone-spread k3s stays unsafe even after the driver lands: a VM create takes no subnet_id, so the node-to-node rules do not cover siblings in other zones. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
…rar that collapses the records A vm-compose stack on a multi-host vm resource now creates one A record per machine, so the stack's domain resolves to the whole fleet instead of host 0. The record is produced by the CHILD stack, not the vm resource: a vm resource describes machines and does not know which stacks will claim which domains. That is the same split kubernetes/kube_run.go and gcp/gke_autopilot_stack.go already use. The guard is the substance of the commit. NewDnsRecordRequest carries no resource-name hint, so only the registrar plugin can keep two same-name A records apart — and three of the four do not: sc-plugin-cloudru and sc-plugin-gcp derive the resource name from name+type alone, sc-plugin-yandex from the name alone, and only sc-plugin-cloudflare hashes the value in. On the other three the second record REPLACES the first: a green deploy, a resolvable name, and two thirds of the fleet never receiving a request. SC does not need to know which plugins are fixed to catch that. NewRecord hands back the resource's reference, so if two calls return the same reference the registrar collapsed them, and the provision is refused naming both hosts, the domain and the fleet's real size. Registrar-agnostic by construction, which is what lets the three plugins be fixed in any order. Not configured is NOT a failure: a parent stack with no registrar is supported (the operator points the name at the fleet by hand, as the docs for every cloud without a registrar say), so that warns and continues. Reaching that required moving NotConfiguredRegistrarError to pApi — the sentinel lived in package pulumi, which imports pulumi/vm, and `params.Registrar == nil` does not detect the case because the empty registrar type is registered to a non-nil no-op. Original name kept as an alias. DNS round-robin is load spreading and not failover — a dead machine costs roughly 1/N of requests, retried client-side. Said so in the doc comment and the log line rather than letting the docs imply HA. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
Phase 2d of the multi-node work. **Stacked**: based on `feat/vm-multi-host-roles` (#563) and carries #565's commits too (merge `f1823a50`) — it needs the provision tier from #563 and the `VmIngress` + Caddy render surface from #565. **Land #563 and #565 first.** The 2d diff itself is the single commit `dff11efc`. ## Why Of the four HA front doors a compose fleet can have, this is the only one needing neither a shared certificate store nor a new Caddy module. Cloudflare (or a load balancer) terminates TLS at the edge; Caddy on every machine serves plain HTTP on `:80` and runs no ACME at all. That makes it the working fallback on every cloud with no bucket resource — Cloud.ru, MWS, byo — where the shared-store path (#565) has nowhere to put the store. Background, including why a shared store is otherwise *mandatory* for a fleet (SC's Caddy image carries no `dns.*` provider, so HTTP-01 is the only challenge, and a name with N A records sends the CA's validator to an arbitrary machine that 404s a token a sibling minted): the multi-node design doc in `forge/docs/design/2026-11-07/multi-node-clusters/`. ## What - `VmIngress.OriginTls *bool`, unset = true, so every resource predating the field keeps terminating TLS. `OriginTlsEnabled()` beside `AcmeStorageEnabled()`. - Setting it false **refuses** every ACME setting rather than ignoring it — `email`, `acmeCa`, both EAB fields, `acmeStorage`. This file's standing rule, and it earns its keep: an operator who leaves `acmeStorage` behind believes the fleet still holds a certificate and would not find out until whatever is in front of the machine went away. The `acmeStorage` message says explicitly that 2b and 2d are alternative front doors, not a pair. - `email` stops being *required* on this path — there is no ACME account. - The parent exports the **negation** (`<vm>-vm-ingress-no-origin-tls`). Deliberate asymmetry against the positive config field: `pApi.GetValueFromStack` returns the zero value for an output that is not there, so the zero has to mean the old behaviour. Exporting the positive sense would silently flip every not-yet-re-provisioned fleet to plain HTTP on its next deploy. - Two places render the decision, because they have different owners and different provision cadences: the global `auto_https off` by the **resource's** provision, the site's `http://<domain>` prefix by the **stack's** deploy. - `import hsts` is kept. With an edge, the header reaches the browser over the edge's TLS and stops a downgrade; with no edge at all, browsers ignore HSTS received over cleartext, so it is inert. - `domainProxied` stops being refused outright and becomes legal on a machine with no origin TLS — which is what it is for. The refusal **moved tiers**: `validateStackConfig` runs in the convert tier with no parent reference, and whether a machine terminates TLS is a property of the parent's resource, so it now lives in `VmComposeStack` after `hostsFromParent` has read the exports. Cost: the operator sees it at `sc deploy` rather than at config load. Benefit: the arrangement is reachable at all. `domainProxied: false` is now a legal no-op (it used to be refused too). ## Edge-to-origin is cleartext Stated in the generated Caddyfile and in the field doc. `originTls: false` does not secure that hop — the machine's firewall (`ingressPorts`) is what has to admit only the edge's addresses. ## Tests `go test ./... -count=1` green; `gofumpt` + `gci` (api's own six sections) on all eleven files. - `TestValidateOriginTls` — the happy path with no `email`, plus a 5-case refusal table including `imageDelivery` still firing (the universal check is hoisted above the new branch, so a resource wrong in two ways still hears about the one that is wrong on every path). - `TestOriginTlsDefaultsToOn` — four backwards-compat assertions: unset, explicit true, nil ingress, and `enable: false` with `originTls: false` (a caller must not read "false" as "an edge terminates" on a machine that serves no sites). - `TestRenderCaddyfileNoOriginTls` — `auto_https off`, no ACME directives, no `storage s3`, all four snippets still inlined, conf.d import still present. - `TestRenderCaddySiteSchemeFollowsOriginTls` — both halves off the same input, so a regression rendering one scheme unconditionally fails one of them. - `TestVmCompose_DomainProxiedNeedsNoOriginTls` — refused / allowed / `false` no-op. - `TestVmCompose_NoOriginTlsRendersAnHttpSite` — the exported URL *and* the rendered site, decoded back out of the route script's base64. - `TestVmCompose_OldParentStackStillServesHttps` — asserts the export key is **absent** from the mocked outputs, then that the URL is `https://`. This is the backwards-compatibility test at the seam that actually carries the risk. --------- Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
baasforger
previously approved these changes
Nov 8, 2026
universe-ops
previously approved these changes
Nov 8, 2026
Signed-off-by: Ilya Sadykov <smecsia@gmail.com> # Conflicts: # pkg/clouds/vm/vm.go # pkg/clouds/vm/vm_test.go
universe-ops
dismissed stale reviews from baasforger and themself
via
November 8, 2026 12:51
ec94e8c
universe-ops
previously approved these changes
Nov 8, 2026
baasforger
previously approved these changes
Nov 8, 2026
`sc stack outputs -s infra` against a stack whose state storage is
plugin-owned fails before it reads anything:
Error: diy URL has an illegal prefix; expected one of: azblob, file, gs, postgres, s3
The double space is the whole diagnosis — the URL is empty. forge's infra
parent declares `state-storage: {type: s3-bucket, config: {bucketName:
sc-simple-forge-state}}`, and `s3-bucket` has no in-tree decoder: the AWS
plugin assembles `s3://<bucket>` in InitStateStore and publishes it via
PULUMI_BACKEND_URL, which login()'s diy branch reads only when the sidecar
has already run. Registering those sidecars is loadPreEnginePlugins' job,
and OutputsStack and CancelStack were the only two api.Provisioner entry
points that never called it — the other six (Provision, Deploy, Preview,
PreviewChild, DestroyParent, DestroyChild) all do. So `*api.PluginFieldConfig`
returned "" for a `url:` key the operator never writes, and diy.Login
rejected it with an error naming no stack, no plugin and no field.
Both now load from the parent descriptor before the toChildStack rewrite,
matching DestroyChildStack's ordering — `plugins:` is declared on the parent
server.yaml. Cancel is fixed alongside outputs rather than after its own
incident: it is what an operator reaches for when a deploy is already wedged,
so it is the worst possible place to surface this.
The guard is structural rather than a hand-written list. The test derives the
expected set from api.Provisioner by reflection — a method qualifies when it
takes a context and an api.Stack, which is exactly the shape of a call that
resolves a stack against the backend — so entry point #9 fails the test
instead of an operator's terminal. SetPublicKey and SetConfigReader are plain
setters and drop out on their own. Each case drives the real method with a
plugin spec that cannot be spawned and asserts the `pre-engine plugin load`
wrap, so passing requires actually attempting the load; verified it fails on
exactly OutputsStack and CancelStack with the fix reverted, and fails closed
when a covered method is removed from the table.
Known cost, documented at the call site: this is the full load, so an
`outputs` call on forge's six-plugin infra goes from ~200ms to a couple of
seconds. Correct and slow beats fast and broken. A state-store-only variant
cannot help by itself — which plugin owns the state store is advertised in a
Schema that only exists after the spawn — so the way out is to trust the
on-disk schema cache the pre-engine already writes and let read-only paths
load cache-only, falling back to a spawn on a miss. Left as a follow-up.
Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
universe-ops
dismissed stale reviews from baasforger and themself
via
November 8, 2026 13:26
f306d4c
A k3s cluster has no load balancer in front of it: klipper-lb runs a svclb DaemonSet, so a `type: LoadBalancer` Service binds its ports on EVERY node and any node serves any workload. The ingress export carried one address — host 0's, deliberately, because a single A record can hold exactly one — so a `domain:` on a k3s child stack resolved to one machine while the other N-1 nodes served nothing, and a reboot of that one node read as an outage of the whole cluster. The compose fleet already does this properly (one A record per host, plus a guard that refuses a registrar which collapses them). This gives the k3s path the same thing, reusing the same implementation rather than a second copy: - `fleetDnsRecords`/`manualRecordAdvice` move from pulumi/vm to pulumi/api as exported `FleetDnsRecords`/`ManualRecordAdvice`. pulumi/kubernetes cannot import pulumi/vm; both already import pulumi/api, which owns the Registrar interface and NotConfiguredRegistrarError. The tests stay in pulumi/vm — half of them assert the wiring through VmComposeStack and all of them run on that package's compose fixtures. - `ToIngressIpsExport` is a NEW output next to `ToIngressIpExport`, holding every node address comma-joined. Separate rather than a changed value: the old output is written straight into an A record's value by a child stack that may be deployed from an older checkout, where a joined list is not a resolvable address but is a successful deploy. - `KubeRunOpts.IngressIPsExport` is optional, and every way it can be absent degrades to today's single record: unset (every flavour but k3s), set but missing from the parent (any k3s parent provisioned before this), or `clusterIPAddress:` named on the client stack, which wins outright. So the fleet records appear on the first child deploy after the PARENT has been re-provisioned, and nothing in between changes behaviour. Found while smoking a 3-node k3s pool on three VPSs: all three nodes served :80 and load-balanced across pods on the others, but the parent exported `smoke-cluster--staging-ingress-ip: <host-0>` and nothing else. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
…t it Two problems, both visible the moment outputs started spawning plugins in f306d4c. `sc stack outputs` silences its logger (cmd_outputs.go) precisely because SC's logger writes to STDOUT and this command's stdout is a JSON document someone pipes into jq. Two channels escaped that silence: 1. The plugin LogStream consumers. SC's log level travels in the context as a value, and newPreEnginePluginBundle built its logCtx from context.Background(), so the level reset to the Info default and a plugin's LOG_SEVERITY_INFO events printed anyway. On forge infra that put the aws plugin's `configuring s3-bucket backend` line inside the payload, which is what actually breaks `| jq`. The bundle now derives logCtx from the caller's context. Close() still cancels it, and the caller always outlives the bundle because every entry point defers closePreEnginePlugins. 2. hashicorp/go-plugin's own hclog, which ResolveAndSchema was building at a hardcoded Debug. That is ~12 lines per plugin — spawn, RPC address, version, teardown — so five plugins wrapped the document in sixty lines of handshake. It goes to stderr so it did not corrupt a pipe, but "silent" that prints sixty lines is not silent. Debug stays the DEFAULT rather than being lowered across the board: it is what surfaces a crashed plugin's stderr, and a quieter default turns a panic into a bare `exit status 2` (live-caught 2026-10-30, forge-baas). The level now follows SC's own, so only a command that asked to be quiet gets Off, and provision/deploy are byte-for-byte unchanged. Reading the level needs an accessor the Logger interface cannot provide — the consumer is a third-party logger that writes on its own — hence logger.LevelFromContext, which returns the same default as the Logger methods so a bare context cannot accidentally read as silent. The JSON is now indented. These payloads are read by a person at a terminal at least as often as by a script, forge's infra returns about twenty keys that wrap into an unreadable block on one line, and indenting is what `gh api`, `aws` and `pulumi stack output --json` all do. Since this changes the output format of a machine-readable command, --compact restores the single line for anything that depended on it. Tests pin all three. The logCtx guard fails on the exact symptom if the derivation regresses (verified by reverting it), and also pins that Close still cancels. The hclog table pins Off for silent AND Debug for both a bare and an explicitly-Info context, so a future tidy-up cannot quietly cost us panic traces. The marshal test asserts the two shapes parse to an identical document, since indenting must be presentation-only. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
`domainProxied` lives on StackConfigCompose, which is where all three targets read it from (vm-compose, gke-autopilot, kubernetes). A `type: single-image` stack has its own config struct that had no such field, so the YAML key parsed and was then dropped on the floor by ToVmComposeSingleImageConfig: no error, no warning, and an operator who asked for a proxied record got an unproxied one. On a fleet that declares `ingress.originTls: false` that is not cosmetic. Proxying is the entire front door there -- Cloudflare terminates TLS and the machines answer plain HTTP on :80 with no TLS listener at all -- so an unproxied A record hands clients a cleartext origin. Live-caught on the 3-VPS compose smoke: `domainProxied: true` in the client.yaml produced three `proxied=false` A records. With this change the same stack produces three proxied records, HTTPS through Cloudflare serves a CF certificate, and port 443 stays closed on every origin. The unset case stays nil rather than becoming false: every reader uses lo.FromPtr, but keeping the pointer is what lets a future default change behave. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
smecsia
enabled auto-merge (squash)
November 8, 2026 15:34
universe-ops
approved these changes
Nov 8, 2026
baasforger
approved these changes
Nov 8, 2026
This was referenced Nov 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phases 1a–1c of the multi-node work. Design doc:
forge/docs/design/2026-11-07/multi-node-clusters/README.md; roadmap itemsc-vm-multi-node-clusters.What this removes
Three config-tier refusals were the whole blocker. Everything below them was already plural —
Driver.Hostsreturns a slice, the plugin wire protocol carriesrepeated VmHostFacts hostswith arole, andpulumi/vm/vm.goalready loops every host with per-host readiness, per-host ingress and indexed exports.pkg/clouds/vm/vm.go(byo)len(hosts) > 1rolepkg/clouds/vm/vm.go(cloud)len(zones) > 1nodes:machines, zones round-robin by indexpkg/clouds/pulumi/k3s/cluster.goTopology
servers:defaults to 1 — one server, N-1 agents: it schedules pods across the fleet, survives the loss of any agent, and is the only shape a 2-machine pool can have.servers: 3(or 5) gets an embedded-etcd control plane that survives losing a control-plane node. Even counts above one are refused, not rounded: etcd elects by majority, so 2 members tolerate zero failures and are strictly worse than 1.On a byo pool that names its own
hosts[].role, the roles win and a contradictingservers:is refused — the operator knows which box has the disk and the stable address, and SC cannot learn that. Host 0 must be a server (it is where SC installs first, reads the join token and reads the kubeconfig).The things that were easy to get wrong
K3S_TOKEN, never in the argument vector. In the args it would land in the on-host marker file and the Pulumi checkpoint, and a rotation would reinstall a healthy control plane. Its stdout carriesLoggingNoneandAdditionalSecretOutputs— pulumi-command does not mark stdout secret.--tls-sancovers servers only. Agents serve no API, and including them would change every server's args marker when a worker is added — scaling workers must not reinstall the control plane.Two filed bugs folded in, because a fleet amplifies both
sc-k3s-cluster-resource-no-delete— k3s is now uninstalled on destroy. The Delete lives on its own no-trigger resource: the installs are replaced every deploy by their per-run nonce and pulumi-command replaces create-then-delete, so a Delete on an install would tear down what it had just built. That exact fault was live on the vm ingress until 2026-10-18.sc-vm-k3s-port80-race-not-validated— an explicitdisable:leaving traefik or servicelb enabled on an ingressed host is refused, naming the components and both remedies. The default set already avoided the race; the documented escape hatch walked into it and provisioned green.Tests
Table tests for the role-set validator,
nodes×zonesdeterminism,serversodd/even/negative,EffectiveServers,rolesOf(declared vs derived, contradictions, agent-first),ServerArgs/AgentArgsper topology, the token staying out of the args, the role-aware uninstaller, the empty-token refusal, the fleet-count gate, and the port-80 refusal.go build ./...+go test ./pkg/clouds/... -count=1green;gofumptandgciclean against this repo's own.golangci.ymlsections.Not in this PR
Phase 1d (the compose fleet loop), Phase 2 (the HA front door — the Caddy S3 storage module, shared ACME store, registrar multi-A, conditional
domainProxied, cloud LB), and Phase 3 (plugins creating N machines). Live smokes on 3 BYO VPSs are pending the operator's go-ahead — a slice isshippedonly after the matching live run.