Pre-RFC: running cluster mode on Kubernetes without hostNetwork and TLS shenanigans

[BrandonArp]

Hi all. We'd like feedback on a direction before writing any code. If the maintainers are on board, we'd like to build it ourselves as a series of small PRs.

Summary

We want to run a 3-node Iggy cluster on Kubernetes as a StatefulSet, without hostNetwork. As far as we can tell, the one thing in the way is how the replica plane uses cluster.nodes[*].ip. The PSK handshake, replica TLS and VSR look fine as they are. Below are three opt-in server changes and a chart mode, plus some open questions we'd value your view on.

Where we are

We run Iggy 0.9.0 on Kubernetes (Kubernetes 1.36 with Calico) with the Rust SDK and the connectors runtime. Today that's a single node, and we'd like to move to cluster mode now that it has shipped. The chart's cluster mode (#4035) explains why it needs hostNetwork plus one release pinned to each node: the consensus listener binds the roster ip verbatim, pods don't have stable addresses, and a Service ClusterIP can't be bound.

For us that costs the following:

• The server's ports bind on the node's own network, outside NetworkPolicy.
• Each node is its own Helm release, pinned with a nodeSelector and rolled out with Recreate.
• Staging and production share one Kubernetes cluster, so they'd need separate nodes or separate ports.
• Replica TLS uses the roster ip as the server name, so node certificates need IP SANs. cert-manager can issue those, but per-pod Kubernetes certificates are normally issued for DNS names.
• Follower-to-primary HTTP forwarding pins the node's own leaf certificate (feat(server-ng): forward follower HTTP requests to VSR primary #3744), so every node must serve the same HTTP certificate. When that certificate is renewed, nodes that have restarted onto the new one and nodes still on the old one can't forward to each other until every node has restarted. With cert-manager's default 90-day certificates, that happens every 60 days.

What already works in our favour

This is from reading the code; we haven't tested it yet.

• The acceptor ignores the connection's source address. A peer identifies itself in ReplicaHello and, with auth on, proves PSK membership. Replica traffic should therefore work through a Service VIP or NAT.
• fix(server-ng): bind cluster TCP to [tcp].address, not the roster ip #3783 already moved the client listeners off the roster ip. They bind their own address and take only the port from the roster, and the chart README notes that the replica listener is the exception.
• advertised_address accepts hostnames (feat(server-ng): add cluster.nodes.advertised_address for clients #3757), and feat(server-ng): add per-client-network advertised address selectors #3818 added advertised-address selectors for clients behind NAT and in Kubernetes. Only the replica side needs work.

Proposal

Each item below would be its own issue and PR, in this order. All of them are opt-in, so existing configs keep their current behaviour.

1. A bind address for the replica listener

Today the replica listener binds node.ip:tcp_replica. We'd add an optional bind IP for it, for example cluster.replica_bind_address (name open).

• When it's set, the listener binds <that ip>:<roster tcp_replica port>. When it's unset, nothing changes.
• The roster ip remains the address peers dial and the TLS server name.
• We'd rather not reuse [tcp].address the way fix(server-ng): bind cluster TCP to [tcp].address, not the roster ip #3783 did. It defaults to 0.0.0.0, so a multi-homed host would silently start accepting replica traffic on every interface. That would break the trusted-L2 deployments the handshake docs describe.

This change alone removes the need for hostNetwork:

• Each replica gets a ClusterIP Service with a fixed spec.clusterIP.
• The roster lists those IPs.
• Each pod binds 0.0.0.0.

That works, but operators have to choose Service IPs in advance, which leads to the next item.

2. DNS names for the replica plane

Allow a hostname as a node's replica address. It could go in ip or in a new field so that ip keeps its meaning; we'd follow your preference.

• The dialer resolves the name on every connect and reconnect, so a pod that comes back with a new IP is found again.
• The TLS server name becomes the hostname, so certificates use DNS SANs.
• HTTP forwarding and the fallback client address would use the same name.
• It depends on (1), since a hostname can't be bound.

With a headless Service, the roster becomes iggy-0.iggy-headless.<ns>.svc, iggy-1.…, which fits a StatefulSet and per-pod cert-manager certificates.

This partly reverses #3923, which made a non-IP roster ip a startup error. As far as we can tell, that check exists because the replica listener binds the value, which (1) makes unnecessary. If it protects something else, we'd like to know.

Open points:

• Resolution: use compio-net's ToSocketAddrsAsync (getaddrinfo on the blocking pool) on every dial, or cache results?
• Names that don't resolve yet, such as StatefulSet peers still starting: we'd treat them like a refused connection and retry on the existing backoff, rather than failing boot.
• Names with several addresses: try each in order?

3. CA verification for HTTP forwarding

As an option, verify the primary's HTTP certificate against a CA (for example a new http.tls.ca_file) and the target node's name, instead of the exact-DER pin. The pin stays the default. We agree it's the stronger check when the roster is fixed. The CA option is there so nodes can hold their own certificates and renew them independently.

4. A StatefulSet cluster mode in the chart

• One release with replicas: 3, a headless Service, and a roster of DNS names generated from the replica count.
• --replica-id comes from the apps.kubernetes.io/pod-index label via the downward API (Kubernetes 1.28+), so identity needs no server change.
• Updates roll one pod at a time, and a PodDisruptionBudget with maxUnavailable: 1 protects quorum.
• Open point: is there an endpoint that reports "rejoined and caught up"? A readiness probe on it would stop a rolling update from moving on before quorum is back.
• The existing one-release-per-node mode stays for bare metal.

Out of scope

No changes to the PSK handshake, TLS channel binding, wire protocol or VSR.

Questions for the maintainers

1. Does this fit where you want cluster mode to go? Is there a reason for binding the roster ip verbatim that we've missed?
2. Does this need a full RFC under the new process? Our guess is that (1) and (3) can go through an issue and PR, while (2) may count as substantial because it changes replication config. We're happy to write the RFC.
3. Do you have preferences on config names, or on reusing ip versus adding a new field?
4. Is anyone already working on any of this?

If you're on board, we'll open the issues and send PRs one at a time. Each will include tests and be run locally and checked on our own cluster.

Disclosure: I used Claude Code to help trace the code paths and PR history behind this post.

[hubcio]

hello,
i will check this proposal this week.

[hubcio]

overall, i like this direction. separating the replica listener’s bind address from the address peers dial would address the restriction behind the current hostNetwork deployment. the proposal itself needs somerefinement.

an optional replica bind IP, independent of [tcp].address, looks like a reasonable first change. existing literal-IP configurations should keep their behavior when it’s unset. for DNS-based roster entries, i’d initially require an explicit bind IP so listener startup doesn’t depend on resolving the node’s own hostname.

DNS support needs to cover stored peer endpoints, validation, reconnects, internal HTTP forwarding, and the fallback address advertised to clients. the connector currently stores numeric SocketAddr values. we should retain the configured hostname for reconnects and certificate verification. each per-node name must identify that replica. multiple answers can describe its IPv4/IPv6 addresses, but shouldn’t distribute connections across different replicas.

i’d start with compio’s existing resolver on each connection attempt, without adding another DNS cache. temporary resolution failures should follow the reconnect path. we also need to define how timeouts work across multiple answers so a blackholed first result doesn’t consume every attempt before another address gets tried.

the HTTP certificate concern is valid. forwarding pins the local leaf certificate, so distinct per-node certificates and mixed old/new certificates fail verification on new forwarding connections. an opt-in CA mode with normal verification of the configured peer name makes sense, retaining pinning when absent.

once every node uses that mode, valid certificates under the trusted CA could be renewed through rolling restarts. the initial migration needs its own compatibility plan: nodes still using pins reject new leaves, and existing shared certificates may lack the required DNS names. live certificate reload is separate.

readiness needs a clear definition before we can call the rolling update safe. /ping checks responsiveness, and cluster metadata marks every configured node Healthy. that misleading status is tracked in #4498. fixing peer status reporting is related work, but it won’t by itself establish that a replica has rejoined and caught up.

#4438, with a fix proposed in #4483, gives a concrete example: it reports a replica serving a stale prefix while state transfer remains stuck after rolling crash-restarts. the readiness definition should also align with #4266, which asks what evidence a recovered replica needs before it can resume participation under Replicated durability. the check needs to support healthy followers and account for metadata and relevant partition recovery. we should define what evidence allows the rollout to remove the next replica, while keeping liveness independent of temporary quorum loss.

if readiness requires quorum, default OrderedReady creation can block initial bootstrap. a separate headless peer-discovery service with publishNotReadyAddresses: true allows peers to discover pods before they’re ready. client-facing services should retain readiness filtering. kubernetes DNS documentation

Parallel pod management would let the initial pods start without waiting for each other’s readiness. rolling replacement should remain one pod at a time. scaling needs separate treatment because Parallel also changes scale-down behavior. a PDB helps with eviction-based maintenance, but it doesn’t gate statefulset rolling updates. pod management, PDB limitations

i’d suggest a short RFC, following the process discussed in #4410, covering the address model, compatibility, TLS migration, and readiness behavior, followed by the small implementation PRs you proposed. we can settle field naming there, keeping one authoritative replica dial address. tests should cover cold bootstrap, temporary DNS failure, changed pod IPs, multiple DNS answers, certificate migration and renewal, and a recovering replica holding up the next planned replacement.

[hubcio]

one more thing that i remembered, interaction between #4055 and this proposal also needs to be analyzed. right now i dont have time to do that.