Skip to content

vm: let an edge terminate TLS (ingress.originTls: false) - #566

Merged
smecsia merged 4 commits into
feat/vm-multi-host-rolesfrom
feat/vm-no-origin-tls
Nov 8, 2026
Merged

smecsia merged 4 commits into
feat/vm-multi-host-rolesfrom
feat/vm-no-origin-tls

Conversation

@smecsia

@smecsia smecsia commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Phase 2d of the multi-node work. Stacked: based on feat/vm-multi-host-roles (#563) and
carries #565's commits too (merge f1823a50) — it needs the provision tier from #563 and the
VmIngress + Caddy render surface from #565. Land #563 and #565 first. The 2d diff itself is
the single commit dff11efc.

Why

Of the four HA front doors a compose fleet can have, this is the only one needing neither a shared
certificate store nor a new Caddy module. Cloudflare (or a load balancer) terminates TLS at the
edge; Caddy on every machine serves plain HTTP on :80 and runs no ACME at all. That makes it the
working fallback on every cloud with no bucket resource — Cloud.ru, MWS, byo — where the
shared-store path (#565) has nowhere to put the store.

Background, including why a shared store is otherwise mandatory for a fleet (SC's Caddy image
carries no dns.* provider, so HTTP-01 is the only challenge, and a name with N A records sends
the CA's validator to an arbitrary machine that 404s a token a sibling minted): the multi-node
design doc in forge/docs/design/2026-11-07/multi-node-clusters/.

What

  • VmIngress.OriginTls *bool, unset = true, so every resource predating the field keeps
    terminating TLS. OriginTlsEnabled() beside AcmeStorageEnabled().
  • Setting it false refuses every ACME setting rather than ignoring it — email, acmeCa,
    both EAB fields, acmeStorage. This file's standing rule, and it earns its keep: an operator
    who leaves acmeStorage behind believes the fleet still holds a certificate and would not find
    out until whatever is in front of the machine went away. The acmeStorage message says
    explicitly that 2b and 2d are alternative front doors, not a pair.
  • email stops being required on this path — there is no ACME account.
  • The parent exports the negation (<vm>-vm-ingress-no-origin-tls). Deliberate asymmetry
    against the positive config field: pApi.GetValueFromStack returns the zero value for an output
    that is not there, so the zero has to mean the old behaviour. Exporting the positive sense would
    silently flip every not-yet-re-provisioned fleet to plain HTTP on its next deploy.
  • Two places render the decision, because they have different owners and different provision
    cadences: the global auto_https off by the resource's provision, the site's
    http://<domain> prefix by the stack's deploy.
  • import hsts is kept. With an edge, the header reaches the browser over the edge's TLS and stops
    a downgrade; with no edge at all, browsers ignore HSTS received over cleartext, so it is inert.
  • domainProxied stops being refused outright and becomes legal on a machine with no origin TLS —
    which is what it is for. The refusal moved tiers: validateStackConfig runs in the convert
    tier with no parent reference, and whether a machine terminates TLS is a property of the parent's
    resource, so it now lives in VmComposeStack after hostsFromParent has read the exports. Cost:
    the operator sees it at sc deploy rather than at config load. Benefit: the arrangement is
    reachable at all. domainProxied: false is now a legal no-op (it used to be refused too).

Edge-to-origin is cleartext

Stated in the generated Caddyfile and in the field doc. originTls: false does not secure that
hop — the machine's firewall (ingressPorts) is what has to admit only the edge's addresses.

Tests

go test ./... -count=1 green; gofumpt + gci (api's own six sections) on all eleven files.

  • TestValidateOriginTls — the happy path with no email, plus a 5-case refusal table including
    imageDelivery still firing (the universal check is hoisted above the new branch, so a resource
    wrong in two ways still hears about the one that is wrong on every path).
  • TestOriginTlsDefaultsToOn — four backwards-compat assertions: unset, explicit true, nil
    ingress, and enable: false with originTls: false (a caller must not read "false" as "an edge
    terminates" on a machine that serves no sites).
  • TestRenderCaddyfileNoOriginTls — auto_https off, no ACME directives, no storage s3, all
    four snippets still inlined, conf.d import still present.
  • TestRenderCaddySiteSchemeFollowsOriginTls — both halves off the same input, so a regression
    rendering one scheme unconditionally fails one of them.
  • TestVmCompose_DomainProxiedNeedsNoOriginTls — refused / allowed / false no-op.
  • TestVmCompose_NoOriginTlsRendersAnHttpSite — the exported URL and the rendered site, decoded
    back out of the route script's base64.
  • TestVmCompose_OldParentStackStillServesHttps — asserts the export key is absent from the
    mocked outputs, then that the URL is https://. This is the backwards-compatibility test at the
    seam that actually carries the risk.

`ingress.acmeStorage` points every machine of a vm resource at one S3-compatible
bucket, rendered as Caddy's global `storage s3` block.

On a fleet this is not a tuning knob, it is the thing that makes origin HTTPS
possible. SC's Caddy image carries no `dns.*` provider, so HTTP-01 is the only
ACME challenge available, and a name with N A records sends the CA's validator
to an arbitrary machine — which 404s a token a sibling minted. certmagic's
distributed solver shares that challenge state through the storage module, so
one store turns N machines into one ACME client holding one certificate instead
of N, and keeps the fleet out of a rate-limit lockout.

The module is techknowlogick/certmagic-s3, baked into simplecontainer/caddy by
the preceding image change. Backends covered by one code path: Yandex
`yc-bucket`, GCP `gcp-bucket` over the GCS interop endpoint, real AWS S3, and
minio on host 0 where the cloud has no bucket resource.

Two rendering rules, both forced by that module's own parser:

  - every boolean carries an explicit value. `UnmarshalCaddyfile` reads each
    subdirective with `if !d.Args(&value) { continue }`, so a bare
    `use_path_style` is neither an error nor true — it is silently skipped, and
    a fleet on minio would then address buckets virtual-host style and fail
    every request without saying why. A test asserts the property directly.
  - credentials reach Caddy as `{$VAR}` placeholders resolved from the 0600 env
    file, never as literals in a Caddyfile that `caddy adapt` echoes.

`use_path_style` defaults to true whenever a custom endpoint is set, because the
module's own default is right for AWS and wrong for everything else. Six
refusals in `validateAcmeStorage` catch the shapes that otherwise fail late and
quietly, with the only evidence in `docker logs sc-caddy` on whichever machine
the CA happened to dial: a missing bucket, half a credential pair (the module
uses static credentials only when it has both and otherwise falls through to the
AWS default chain, which on a VPS finds nothing), neither endpoint nor region,
an endpoint without a scheme, an encryption key that is not exactly 32 bytes,
and `insecure: true` against a plaintext endpoint.

Wiring a bucket resource through `uses:` and standing minio up on host 0 are
follow-ons; this is the config surface and the rendering they both target.

Phase 2b of roadmap item sc-vm-multi-node-clusters; design notes in
forge/docs/design/2026-11-07/multi-node-clusters §5.2.

Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
Signed-off-by: Ilya Sadykov <smecsia@gmail.com>

# Conflicts:
#	pkg/clouds/vm/vm_test.go
The one HA front-door arrangement a compose fleet can have without a shared
certificate store or a new Caddy module: Cloudflare (or a load balancer)
terminates TLS at the edge, and Caddy on every machine serves plain HTTP on
:80 and runs no ACME at all. That makes it the working fallback on every
cloud with no bucket resource — Cloud.ru, MWS, byo — where the shared-store
path has nowhere to put the store.

`ingress.originTls: *bool` on the vm resource, unset meaning true, so every
resource that predates the field keeps terminating TLS. Setting it false
refuses every ACME setting rather than ignoring it (`email`, `acmeCa`, the
two EAB fields, `acmeStorage`) — an operator who leaves `acmeStorage` behind
believes the fleet still holds a certificate and would not find out until
whatever is in front of the machine went away. `acmeStorage` says explicitly
that the two are alternative front doors, not a pair.

The parent stack exports the NEGATION (`<vm>-vm-ingress-no-origin-tls`), and
that asymmetry against the positive config field is deliberate: a child
deployed against a parent provisioned before the export existed reads a
missing output as the zero value, so the zero has to mean the old behaviour.
Exporting the positive sense would silently flip every fleet that has not
been re-provisioned to plain HTTP on its next deploy.

Two places render the decision, because they have different owners and
different cadences: the global `auto_https off` is written by the RESOURCE's
provision, and the site's `http://<domain>` prefix by the STACK's deploy. A
machine not re-provisioned since the flag was set would otherwise go asking
a CA for a certificate on behalf of a stack that cannot answer the challenge.

`domainProxied` stops being refused outright and becomes legal on a machine
with no origin TLS — which is the whole point of it. The refusal moved from
validateStackConfig to VmComposeStack because whether a machine terminates
TLS is a property of the PARENT's resource and the convert tier holds no
parent reference; the cost is a refusal at `sc deploy` rather than at config
load, and the alternative was leaving the arrangement unreachable.

Edge-to-origin is CLEARTEXT on this path. Said in the generated Caddyfile and
in the field doc: `ingressPorts` is what has to admit only the edge.

Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
@smecsia

smecsia commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

CI note for reviewers: this PR's base is feat/vm-multi-host-roles, and every workflow in .github/workflows filters pull_request: branches: [main, staging] — so GitHub reports no checks on it. That is the branch filter, not a skipped build.

branch.yaml was dispatched on the branch directly and is green on all nine jobs, including unit and integration tests: https://github.com/simple-container-com/api/actions/runs/42417747314

The normal PR checks (CodeQL, DCO, semgrep, govulncheck) will fire automatically once #563 merges and GitHub retargets this PR to main.

…reachable

#566 made `domainProxied: true` legal on a fleet that declares
`ingress.originTls: false`, and left three things behind it that still
described the world where the convert tier refused the setting outright.

- The comment above the `fleetDnsRecords` call still said `DomainProxied`
  is nil on this path because `validateStackConfig` refuses it. That
  refusal moved into this file and is now conditional; the comment now
  says which fleet reads which value and why each is right.

- Both no-registrar warnings told the operator to point the name at the
  fleet "unproxied — the machine answers the ACME challenge itself".
  Correct for a TLS-terminating fleet, and actively harmful for a
  no-origin-TLS one: following it bypasses the edge that was supposed to
  terminate TLS and puts clients on a cleartext origin over the public
  internet. Extracted as manualRecordAdvice(proxied), which states which
  way each half fails.

- Nothing asserted that `domainProxied` actually reaches
  `api.DnsRecord.Proxied`. A flag read correctly and then dropped on the
  way down would have been a green deploy with a resolvable name, an edge
  never in the request path, and no signal anywhere. Now covered end to
  end through VmComposeStack for a 3-host proxied fleet, with the
  unproxied default asserted beside it so the test cannot pass by
  hardcoding either answer.

No behaviour change for a fleet that terminates its own TLS.

Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
@smecsia

smecsia commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 7e3a7f6c — the downstream cleanup this PR's own change made necessary, no behaviour change for a fleet that terminates its own TLS.

Making domainProxied: true reachable left three places describing the world where the convert tier refused it outright:

  • The comment above the fleetDnsRecords call still said DomainProxied is nil here because validateStackConfig refuses it. That refusal moved into this file and is now conditional.
  • Both no-registrar warnings told the operator to point the name at the fleet "unproxied — the machine answers the ACME challenge itself". Correct for a TLS-terminating fleet; on a ingress.originTls: false fleet it is advice to bypass the edge that was supposed to terminate TLS and put clients on a cleartext origin across the public internet. Now manualRecordAdvice(proxied), with both halves tested and a comment saying which way each fails.
  • Nothing asserted that domainProxied reaches api.DnsRecord.Proxied. A flag read correctly and dropped on the way down would have been a green deploy, a resolvable name, an edge never in the request path, and no signal anywhere. Now covered end to end through VmComposeStack for a 3-host proxied fleet, with the unproxied default asserted beside it so the test cannot pass by hardcoding either answer — and mutation-checked by pinning the call site to false, which fails it.

CI re-dispatched by hand, since a stacked PR matches no workflow's branches: [main, staging] filter: run 42419719557 — green on all nine jobs, unit and integration tests included.

@smecsia
smecsia merged commit e86a1c2 into feat/vm-multi-host-roles Nov 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant