feat: multi-node k3s clusters and multi-host vm pools (Phase 1a–1c)

docs/docs/examples/k3s/byo-host/README.md

@@ -194,10 +194,17 @@ Stated plainly, because `scale:` will happily let you believe otherwise:
 
 - **One failure domain.** Replicas on one node survive a pod crash, not a machine
   reboot. `scale: {min: 1, max: 1}` in `client.yaml` is honesty, not a technical limit.
-- **No multi-node yet.** `vmResource` resolves to exactly one host today; more than one
-  is refused rather than silently using the first.
+- **Multi-node needs a private network.** More than one `hosts:` entry is accepted — see
+  the commented block in `server.yaml` — but every entry then needs `privateAddress:`,
+  and that is refused rather than defaulted. k3s otherwise advertises whichever address
+  the machine's default route carries, which on a single-NIC VPS is the public one, and
+  etcd plus flannel's unencrypted VXLAN overlay would cross the internet on a host where
+  SC has no firewall to write a rule in. If your machines share no private network,
+  run a [compose fleet](../../vm-compose/byo-host/README.md) instead — it needs no
+  node-to-node traffic at all.
 - **`local-storage` is the default storage class.** A PersistentVolume is a directory
-  on that one machine. It does not survive the machine.
+  on that one machine. It does not survive the machine, and nothing replicates it
+  across a pool.
 - **The control plane is pinned**, not tracking a release channel — k3s's installer
   resolves a channel at install time, which would otherwise make the deployed version
   depend on the day the deploy ran. Move it deliberately with `version:`.

docs/docs/examples/k3s/byo-host/client.yaml

@@ -34,11 +34,26 @@ stacks:
 
       # Reachability on a bare k3s stand.
       #
-      # `domain:` here behaves exactly as it does on GKE: SC creates an A record
-      # pointing at the cluster's ingress address, which for a single-node k3s cluster
-      # is the node's own address. It needs a registrar configured in the parent stack,
-      # and — because the node answers the ACME challenge itself — a record that is NOT
-      # proxied. Cloudflare's orange cloud breaks it.
+      # `domain:` here behaves as it does on GKE with one difference that matters on a
+      # pool: SC creates an A record per NODE, all sharing the name, because a k3s
+      # cluster has no load balancer in front of it. klipper-lb binds the Service's
+      # ports on every node, so every node is the edge; pointing the name at one of
+      # them would send every request to that node and lose the cluster the moment it
+      # reboots. On a single-node cluster that is one record, exactly as before.
+      #
+      # It needs a registrar configured in the parent stack, and — because the node
+      # answers the ACME challenge itself — a record that is NOT proxied. Cloudflare's
+      # orange cloud breaks it.
+      #
+      # What N records buy, stated plainly because DNS round-robin is routinely sold as
+      # high availability: load spreading, and nothing else. A resolver hands clients
+      # the addresses in some order and nothing checks whether a node is alive, so a
+      # dead node costs roughly 1/N of requests until it is replaced or removed from the
+      # record set. It is not failover.
+      #
+      # On a parent stack provisioned before the fleet export existed, this falls back
+      # to the single record on its own — re-run `sc provision` for the parent to get
+      # the whole pool into DNS.
       #
       # While smoking, leave this out and reach the Service directly instead. There is
       # no edge in the parent stack above (see its comments for why), so routing by host

docs/docs/examples/k3s/byo-host/server.yaml

@@ -61,6 +61,46 @@ resources:
                 port: 22
                 privateKey: ${secret:CLUSTER_VM_DEPLOY_KEY}
                 hostKey: "203.0.113.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPLE00"
+            # One machine, so no `role:` and no `privateAddress:` — both are inferred,
+            # and a single-node cluster has no node-to-node traffic to address. For
+            # more than one machine, both become required. The multi-node shape:
+            #
+            # hosts:
+            #   - address: 203.0.113.10        # what YOU and SC reach it on
+            #     privateAddress: 10.0.0.10    # what its SIBLINGS reach it on
+            #     role: k3s-server
+            #     user: ubuntu
+            #     privateKey: ${secret:CLUSTER_VM_DEPLOY_KEY}
+            #   - address: 203.0.113.11
+            #     privateAddress: 10.0.0.11
+            #     role: k3s-agent
+            #     user: ubuntu
+            #     privateKey: ${secret:CLUSTER_VM_DEPLOY_KEY}
+            #   - address: 203.0.113.12
+            #     privateAddress: 10.0.0.12
+            #     role: k3s-agent
+            #     user: ubuntu
+            #     privateKey: ${secret:CLUSTER_VM_DEPLOY_KEY}
+            #
+            # `role:` is all-or-none across the pool — SC will not guess which of
+            # several machines runs etcd, and the guess decides whether a machine
+            # joins the cluster at all. Mixing `compose` with a `k3s-*` role in one vm
+            # resource is refused: one machine has one port 443.
+            #
+            # `privateAddress:` is REQUIRED on every host of a multi-node pool, and
+            # the refusal is deliberate rather than a convenience. SC passes it to
+            # k3s as `--node-ip`; without it k3s advertises whichever address the
+            # machine's default route carries, which on a single-NIC VPS is the PUBLIC
+            # one — and then etcd peer traffic (the cluster's whole state, every
+            # Secret) and flannel's unencrypted VXLAN overlay cross the public
+            # internet, on a `byo` host where SC has no firewall to write a rule in.
+            # That deploy SUCCEEDS: three nodes `Ready`, nothing in the output saying
+            # the overlay is exposed. Hence a refusal and not a warning.
+            #
+            # If your machines share no private network at all — a lot of cheap VPS
+            # hosting has none — there is nothing honest to put here yet. Run a
+            # compose fleet instead (../../vm-compose/byo-host/), which needs no
+            # node-to-node traffic; SC-managed WireGuard is the planned answer.
             # Deliberately absent here: `ingress`.
             #
             # On the vm-compose path, `ingress.enable: true` runs a Caddy container that
@@ -88,6 +128,21 @@ resources:
             # after the vm resource it names, and reads the host facts from its output.
             vmResource: cluster-vm
 
+            # How many of the pool's machines run the control plane. Default 1.
+            #
+            # `servers: 1` + N agents uses **sqlite** and has no etcd at all, which is
+            # why it is the default: it is the simplest thing that works, and losing
+            # the server loses the control plane but not the running workloads.
+            #
+            # `servers: 3` opts into embedded etcd and real HA. It must be ODD and no
+            # larger than the host count — two members tolerate zero failures, so they
+            # are strictly worse than one, and four tolerate the same one as three
+            # while costing a machine. etcd is also latency-sensitive: across
+            # availability zones in one region (~1ms) is the intended topology;
+            # across regions or hosters (10-80ms) flaps raft heartbeats and triggers
+            # repeated leader elections.
+            # servers: 3
+
             # Pinned on purpose. k3s's installer resolves a RELEASE CHANNEL at install
             # time, so leaving this to the default would mean the control-plane version
             # depends on the day the deploy ran. SC pins a known version instead; set

docs/docs/examples/k3s/cloudru-evolution/README.md

@@ -275,7 +275,7 @@ Refused rather than accepted and ignored:
 | `hosts` | that is `provider: byo` |
 | `network` | the create call rejects `subnet_id`/`networks`/`nics` — the machine lands on the zone's default |
 | `extraSshKeys` | extra keys are authorised through user-data, which this cloud has none of; the create call takes exactly one key |
-| more than one zone | this is a single machine |
+| ~~more than one zone~~ | no longer refused at the config tier — but see the k3s caveat below before you use it |
 | `publicIP: false` | every delivery step is an outbound SSH connection, and SC has no bastion |
 
 To place a machine on your own network or security group, create it in the console and
@@ -288,11 +288,18 @@ Stated plainly, because `scale:` will happily let you believe otherwise:
 
 - **One failure domain.** Replicas on one node survive a pod crash, not a machine
   reboot. `scale: {min: 1, max: 1}` is honesty, not a technical limit.
-- **No multi-node yet.** `vmResource` resolves to exactly one host today; more than one
-  is refused rather than silently using the first.
+- **Multi-node is not usable on this cloud yet, for two separate reasons.** The config
+  tier accepts it — `nodes:`, several `zones:`, `servers:` on the cluster resource — but
+  (1) no cloud driver creates more than one machine yet, and SC refuses the mismatch
+  rather than handing you a one-machine "fleet"; and (2) even once it does, an Evolution
+  VM create takes no `subnet_id`, so each machine lands on its own zone's default subnet
+  and the node-to-node firewall rules SC writes (flannel, kubelet, etcd) are scoped to
+  the subnet the machine landed on — which does not cover its siblings in other zones.
+  Keep a k3s pool in one zone here. A compose fleet is unaffected: those machines never
+  talk to each other.
 - **`local-storage` is the default storage class.** A PersistentVolume is a directory on
   that one machine, and it does not survive the machine. Cloud.ru's own block storage is
-  not wired to a CSI driver here.
+  not wired to a CSI driver here, and nothing replicates a volume across a pool.
 - **A destroy/provision cycle moves the address.** This driver allocates a new floating
   IP, so every saved kubeconfig and every DNS record points at the old one. Set
   `endpoint:` to a DNS name if that matters.

docs/docs/examples/k3s/cloudru-evolution/server.yaml

@@ -79,6 +79,29 @@ resources:
             # Placement is the availability zone.
             zones: [ru.AZ-1]
 
+            # How many machines to create. Defaults to `len(zones)`, then 1 — so this
+            # file makes one. Listing three zones makes three machines, one per zone;
+            # `nodes: 3` with one zone makes three there; `nodes: 3` with two zones
+            # round-robins (0→a, 1→b, 2→a) and the assignment is index-stable, so a
+            # machine never changes zone on a later apply.
+            # nodes: 3
+            #
+            # ⚠ ACCEPTED HERE, NOT YET HONOURED BY THIS DRIVER. Creating N machines
+            # is per-plugin work and `sc-plugin-cloudru` has not had it. SC refuses
+            # the mismatch rather than quietly giving you one machine: a resource
+            # that asks for three and gets one fails the provision, naming the
+            # driver and both counts.
+            #
+            # ⚠ On **Evolution specifically**, spreading k3s nodes across zones does
+            # not work yet. An Evolution VM create takes no `subnet_id`, so each
+            # machine lands on its own zone's default subnet — and the node-to-node
+            # firewall rules SC writes (flannel, kubelet, etcd) are scoped to "the
+            # subnet this machine landed on", which does not cover its siblings in
+            # other zones. Keep a k3s pool in ONE zone here until the SC-managed
+            # WireGuard mesh lands, which makes the rules one mesh CIDR regardless of
+            # zone. A compose fleet is unaffected: those machines never talk to each
+            # other.
+
             # Required on every cloud driver. The kubeconfig SC exports points at this
             # address, and `sc deploy` connects to it from wherever it runs.
             publicIP: true

docs/docs/examples/vm-compose/byo-host/server.yaml

@@ -98,6 +98,41 @@ resources:
                 # before you trust it — reading it off the network is what it protects
                 # you from.
                 hostKey: "203.0.113.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPLE00"
+            # A FLEET is the same list with more entries. Add a second machine and every
+            # stack on this resource deploys a full container set onto each one, rolled
+            # one host at a time — the next host starts only after the previous host's
+            # new slot is live, so a bad image stops the roll instead of taking the
+            # fleet down. Each machine runs its own Caddy and its own blue-green pair;
+            # blue-green is WITHIN a host and is not, and will not be, across machines.
+            #
+            # hosts:
+            #   - address: 203.0.113.10
+            #     user: ubuntu
+            #     privateKey: ${secret:PORTAL_VM_DEPLOY_KEY}
+            #     hostKey: "…"
+            #   - address: 203.0.113.11
+            #     user: ubuntu
+            #     privateKey: ${secret:PORTAL_VM_DEPLOY_KEY}
+            #     hostKey: "…"
+            #
+            # No `role:` needed — a pool with no roles declared is a compose fleet, and
+            # no `privateAddress:` either, because these machines never talk to each
+            # other. That is the useful difference from a multi-node k3s pool (see
+            # ../../k3s/byo-host/server.yaml), and it is why a compose fleet works on
+            # cheap VPS hosting that offers no private network at all.
+            #
+            # Two things a fleet needs that a single machine does not:
+            #
+            #  * `ingress.acmeStorage` — otherwise each machine runs its own ACME
+            #    client against the same name and you get N certificates, N sets of
+            #    rate-limit consumption, and HTTP-01 challenges answered by whichever
+            #    node DNS happened to pick (which 404s a token another node minted).
+            #    One shared store makes it one certificate. See `ingress` below.
+            #  * More than one A record for the name, so requests reach more than one
+            #    machine. Note what this does and does not give you: a dead node means
+            #    roughly 1/N of requests fail and are retried by the client. It is
+            #    round-robin, not failover — a cloud load balancer with health checks
+            #    is strictly better where the cloud has one.
             # The front door: a Caddy container in its own compose project, terminating
             # TLS for every stack on this machine. It belongs to the resource and not to
             # a template because one machine has one port 443 — and because a stack's

docs/docs/examples/vm-compose/cloudru-evolution/README.md

@@ -181,9 +181,19 @@ stack plus the Caddy ingress without the build cache pushing it into swap.
 names are not the same across projects, and a guess here would be a guess about your
 quota.
 
-`zones` takes at most one entry (`ru.AZ-1` … `ru.AZ-3`) — this is a single machine, and a
-list of zones would describe a cluster. Left out, SC takes the first zone
-`GET /availability-zones` reports.
+`zones` (`ru.AZ-1` … `ru.AZ-3`) is also the fleet's size: SC creates one machine per
+zone, and `nodes:` sets the count independently (`nodes: 3` with one zone puts three
+there; with two it round-robins, index-stably, so a machine never changes zone on a
+later apply). Left out, SC takes the first zone `GET /availability-zones` reports and
+creates one machine.
+
+**But no cloud driver creates more than one machine yet** — that is per-plugin work and
+`sc-plugin-cloudru` has not had it. SC refuses the mismatch rather than quietly giving
+you a one-machine fleet: asking for three and receiving one fails the provision, naming
+the driver and the count. So `nodes:`/multi-zone here is config you can write ahead of
+the driver, not a fleet you can stand up today. On machines that already exist,
+`provider: byo` with several `hosts:` entries works now — see
+[that example](../byo-host/README.md).
 
 ## Which ports are open
 
@@ -256,7 +266,7 @@ in which the operator is certain the setting applied.
 | `hosts` | SC creates the machine; a host list describes one you brought (`provider: byo`) |
 | `network` | the create call accepts no subnet or security group at all (measured), so the machine always joins the zone's defaults |
 | `extraSshKeys` | extra keys are authorised through user-data, which this cloud does not run; the create takes exactly one public key — the deploy key. Add one by hand, or keep console access for incidents |
-| `zones` with more than one entry | one machine is in one zone |
+| ~~`zones` with more than one entry~~ | no longer refused — it is the fleet's size, but see the driver caveat above |
 | `size.memory` that is not whole GiB | memory resolves a flavour by exact match; rounding would hand you a different machine |
 
 The `vm-compose` template's own refusals (`scale` other than 1, `domainProxied`,

docs/docs/examples/vm-compose/cloudru-evolution/server.yaml

@@ -76,8 +76,28 @@ resources:
             # request. Placement is the availability zone. A `region:` SC silently
             # dropped would read as placement you controlled.
             #
-            # One entry at most: this is a single machine. Left out, SC takes the first
-            # zone `GET /availability-zones` reports.
+            # One entry here, because this file deploys one machine. Left out, SC takes
+            # the first zone `GET /availability-zones` reports.
+            #
+            # Several entries make several machines — a compose fleet, one zone each.
+            # `nodes:` sets the count independently: it defaults to `len(zones)`, then
+            # 1, so `nodes: 3` with one zone puts three machines there and `nodes: 3`
+            # with two zones round-robins (0→a, 1→b, 2→a). The assignment is
+            # index-stable, so a machine never moves zone on a later apply.
+            #
+            # A compose fleet spreads across zones safely: each machine runs a full
+            # container set and its own Caddy, and they never talk to each other. Give
+            # them `ingress.acmeStorage` so they share one certificate instead of
+            # racing for N, and point more than one A record at the name. (A multi-node
+            # k3s pool is NOT safe across zones on this cloud yet — see
+            # ../../k3s/cloudru-evolution/server.yaml for why.)
+            #
+            # ⚠ ACCEPTED HERE, NOT YET HONOURED BY THIS DRIVER. Creating N machines is
+            # per-plugin work and `sc-plugin-cloudru` has not had it. SC refuses the
+            # mismatch rather than quietly giving you one machine: a resource that asks
+            # for three and gets one fails the provision, naming the driver and both
+            # counts. Several machines work today on `provider: byo`, where they
+            # already exist.
             zones: [ru.AZ-1]
 
             # Required on every cloud driver, and refused if false: every delivery step