Repository navigation
vm: let an edge terminate TLS (ingress.originTls: false) - #566
Conversation
`ingress.acmeStorage` points every machine of a vm resource at one S3-compatible
bucket, rendered as Caddy's global `storage s3` block.
On a fleet this is not a tuning knob, it is the thing that makes origin HTTPS
possible. SC's Caddy image carries no `dns.*` provider, so HTTP-01 is the only
ACME challenge available, and a name with N A records sends the CA's validator
to an arbitrary machine — which 404s a token a sibling minted. certmagic's
distributed solver shares that challenge state through the storage module, so
one store turns N machines into one ACME client holding one certificate instead
of N, and keeps the fleet out of a rate-limit lockout.
The module is techknowlogick/certmagic-s3, baked into simplecontainer/caddy by
the preceding image change. Backends covered by one code path: Yandex
`yc-bucket`, GCP `gcp-bucket` over the GCS interop endpoint, real AWS S3, and
minio on host 0 where the cloud has no bucket resource.
Two rendering rules, both forced by that module's own parser:
- every boolean carries an explicit value. `UnmarshalCaddyfile` reads each
subdirective with `if !d.Args(&value) { continue }`, so a bare
`use_path_style` is neither an error nor true — it is silently skipped, and
a fleet on minio would then address buckets virtual-host style and fail
every request without saying why. A test asserts the property directly.
- credentials reach Caddy as `{$VAR}` placeholders resolved from the 0600 env
file, never as literals in a Caddyfile that `caddy adapt` echoes.
`use_path_style` defaults to true whenever a custom endpoint is set, because the
module's own default is right for AWS and wrong for everything else. Six
refusals in `validateAcmeStorage` catch the shapes that otherwise fail late and
quietly, with the only evidence in `docker logs sc-caddy` on whichever machine
the CA happened to dial: a missing bucket, half a credential pair (the module
uses static credentials only when it has both and otherwise falls through to the
AWS default chain, which on a VPS finds nothing), neither endpoint nor region,
an endpoint without a scheme, an encryption key that is not exactly 32 bytes,
and `insecure: true` against a plaintext endpoint.
Wiring a bucket resource through `uses:` and standing minio up on host 0 are
follow-ons; this is the config surface and the rendering they both target.
Phase 2b of roadmap item sc-vm-multi-node-clusters; design notes in
forge/docs/design/2026-11-07/multi-node-clusters §5.2.
Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
Signed-off-by: Ilya Sadykov <smecsia@gmail.com> # Conflicts: # pkg/clouds/vm/vm_test.go
The one HA front-door arrangement a compose fleet can have without a shared certificate store or a new Caddy module: Cloudflare (or a load balancer) terminates TLS at the edge, and Caddy on every machine serves plain HTTP on :80 and runs no ACME at all. That makes it the working fallback on every cloud with no bucket resource — Cloud.ru, MWS, byo — where the shared-store path has nowhere to put the store. `ingress.originTls: *bool` on the vm resource, unset meaning true, so every resource that predates the field keeps terminating TLS. Setting it false refuses every ACME setting rather than ignoring it (`email`, `acmeCa`, the two EAB fields, `acmeStorage`) — an operator who leaves `acmeStorage` behind believes the fleet still holds a certificate and would not find out until whatever is in front of the machine went away. `acmeStorage` says explicitly that the two are alternative front doors, not a pair. The parent stack exports the NEGATION (`<vm>-vm-ingress-no-origin-tls`), and that asymmetry against the positive config field is deliberate: a child deployed against a parent provisioned before the export existed reads a missing output as the zero value, so the zero has to mean the old behaviour. Exporting the positive sense would silently flip every fleet that has not been re-provisioned to plain HTTP on its next deploy. Two places render the decision, because they have different owners and different cadences: the global `auto_https off` is written by the RESOURCE's provision, and the site's `http://<domain>` prefix by the STACK's deploy. A machine not re-provisioned since the flag was set would otherwise go asking a CA for a certificate on behalf of a stack that cannot answer the challenge. `domainProxied` stops being refused outright and becomes legal on a machine with no origin TLS — which is the whole point of it. The refusal moved from validateStackConfig to VmComposeStack because whether a machine terminates TLS is a property of the PARENT's resource and the convert tier holds no parent reference; the cost is a refusal at `sc deploy` rather than at config load, and the alternative was leaving the arrangement unreachable. Edge-to-origin is CLEARTEXT on this path. Said in the generated Caddyfile and in the field doc: `ingressPorts` is what has to admit only the edge. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
|
CI note for reviewers: this PR's base is
The normal PR checks (CodeQL, DCO, semgrep, govulncheck) will fire automatically once #563 merges and GitHub retargets this PR to |
…reachable #566 made `domainProxied: true` legal on a fleet that declares `ingress.originTls: false`, and left three things behind it that still described the world where the convert tier refused the setting outright. - The comment above the `fleetDnsRecords` call still said `DomainProxied` is nil on this path because `validateStackConfig` refuses it. That refusal moved into this file and is now conditional; the comment now says which fleet reads which value and why each is right. - Both no-registrar warnings told the operator to point the name at the fleet "unproxied — the machine answers the ACME challenge itself". Correct for a TLS-terminating fleet, and actively harmful for a no-origin-TLS one: following it bypasses the edge that was supposed to terminate TLS and puts clients on a cleartext origin over the public internet. Extracted as manualRecordAdvice(proxied), which states which way each half fails. - Nothing asserted that `domainProxied` actually reaches `api.DnsRecord.Proxied`. A flag read correctly and then dropped on the way down would have been a green deploy with a resolvable name, an edge never in the request path, and no signal anywhere. Now covered end to end through VmComposeStack for a 3-host proxied fleet, with the unproxied default asserted beside it so the test cannot pass by hardcoding either answer. No behaviour change for a fleet that terminates its own TLS. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
|
Pushed Making
CI re-dispatched by hand, since a stacked PR matches no workflow's |
Phase 2d of the multi-node work. Stacked: based on
feat/vm-multi-host-roles(#563) andcarries #565's commits too (merge
f1823a50) — it needs the provision tier from #563 and theVmIngress+ Caddy render surface from #565. Land #563 and #565 first. The 2d diff itself isthe single commit
dff11efc.Why
Of the four HA front doors a compose fleet can have, this is the only one needing neither a shared
certificate store nor a new Caddy module. Cloudflare (or a load balancer) terminates TLS at the
edge; Caddy on every machine serves plain HTTP on
:80and runs no ACME at all. That makes it theworking fallback on every cloud with no bucket resource — Cloud.ru, MWS, byo — where the
shared-store path (#565) has nowhere to put the store.
Background, including why a shared store is otherwise mandatory for a fleet (SC's Caddy image
carries no
dns.*provider, so HTTP-01 is the only challenge, and a name with N A records sendsthe CA's validator to an arbitrary machine that 404s a token a sibling minted): the multi-node
design doc in
forge/docs/design/2026-11-07/multi-node-clusters/.What
VmIngress.OriginTls *bool, unset = true, so every resource predating the field keepsterminating TLS.
OriginTlsEnabled()besideAcmeStorageEnabled().email,acmeCa,both EAB fields,
acmeStorage. This file's standing rule, and it earns its keep: an operatorwho leaves
acmeStoragebehind believes the fleet still holds a certificate and would not findout until whatever is in front of the machine went away. The
acmeStoragemessage saysexplicitly that 2b and 2d are alternative front doors, not a pair.
emailstops being required on this path — there is no ACME account.<vm>-vm-ingress-no-origin-tls). Deliberate asymmetryagainst the positive config field:
pApi.GetValueFromStackreturns the zero value for an outputthat is not there, so the zero has to mean the old behaviour. Exporting the positive sense would
silently flip every not-yet-re-provisioned fleet to plain HTTP on its next deploy.
cadences: the global
auto_https offby the resource's provision, the site'shttp://<domain>prefix by the stack's deploy.import hstsis kept. With an edge, the header reaches the browser over the edge's TLS and stopsa downgrade; with no edge at all, browsers ignore HSTS received over cleartext, so it is inert.
domainProxiedstops being refused outright and becomes legal on a machine with no origin TLS —which is what it is for. The refusal moved tiers:
validateStackConfigruns in the converttier with no parent reference, and whether a machine terminates TLS is a property of the parent's
resource, so it now lives in
VmComposeStackafterhostsFromParenthas read the exports. Cost:the operator sees it at
sc deployrather than at config load. Benefit: the arrangement isreachable at all.
domainProxied: falseis now a legal no-op (it used to be refused too).Edge-to-origin is cleartext
Stated in the generated Caddyfile and in the field doc.
originTls: falsedoes not secure thathop — the machine's firewall (
ingressPorts) is what has to admit only the edge's addresses.Tests
go test ./... -count=1green;gofumpt+gci(api's own six sections) on all eleven files.TestValidateOriginTls— the happy path with noemail, plus a 5-case refusal table includingimageDeliverystill firing (the universal check is hoisted above the new branch, so a resourcewrong in two ways still hears about the one that is wrong on every path).
TestOriginTlsDefaultsToOn— four backwards-compat assertions: unset, explicit true, nilingress, and
enable: falsewithoriginTls: false(a caller must not read "false" as "an edgeterminates" on a machine that serves no sites).
TestRenderCaddyfileNoOriginTls—auto_https off, no ACME directives, nostorage s3, allfour snippets still inlined, conf.d import still present.
TestRenderCaddySiteSchemeFollowsOriginTls— both halves off the same input, so a regressionrendering one scheme unconditionally fails one of them.
TestVmCompose_DomainProxiedNeedsNoOriginTls— refused / allowed /falseno-op.TestVmCompose_NoOriginTlsRendersAnHttpSite— the exported URL and the rendered site, decodedback out of the route script's base64.
TestVmCompose_OldParentStackStillServesHttps— asserts the export key is absent from themocked outputs, then that the URL is
https://. This is the backwards-compatibility test at theseam that actually carries the risk.