feat: run Gitaly as a three-node cluster behind Praefect - #26
Merged
Merged
Conversation
GitLab's minimal Gitaly Cluster puts three Praefect nodes between the Gitaly clients and three Gitaly nodes, which Praefect keeps as replicas of every repository. It replaces the single Gitaly node, so the stack is nine nodes: two Rails, three Gitaly, three Praefect and one Redis, on the same RDS instance and S3 bucket. terraform/aws.tfvars declares it for the pinned framework: - tcnaw-gitaly01 to 03 (t3.medium) and tcnaw-praefect01 to 03 (t3.small), on the SSM-only org profile, spread over the two Rails zones: 01 and 03 in us-east-1c, 02 in us-east-1a; - the load balancer gains listener 2305 and a praefect target group, attached by Function tag. Client IP preservation is off on it, unlike 80 and 22, and its health check is a TCP connect (interval 10, timeout 5, thresholds 2), as the reference architecture's HAProxy checks Praefect, which serves no HTTP; - a Praefect node admits 2305 from the load balancer alone and reaches only the database and the Gitaly nodes. It carries gitlab-db-client, the group the database admits; - a Rails node reaches Praefect through the load balancer on 2305 and has no route to Gitaly at all, so a Gitaly node's 8075 serves only Praefect and its peers, which replicate from one another; - a Gitaly node reaches GitLab's internal API on 80 and Praefect on 2305 through the load balancer, the route GitLab's Gitaly Cluster firewall table requires; nothing here exercises it yet. Only tcnaw-gitaly01, which runs the proofs, keeps SSH to the load balancer. No rule lets a Gitaly node reach a Praefect node directly: with client addresses off on 2305, every caller and every health check arrives from the load balancer. No rule lets Praefect reach GitLab: GitLab's firewall table lists Praefect to the API on 80, but Praefect 19.4.1 imports no GitLab client. No node is both a client and a target of one listener: a Rails node is a target of 80 and 22 and a client of 2305, a Gitaly node a client and never a target, and a Praefect node a target of 2305 and a client of none. A mock plan against 9b5e6ae plans 9 instances, 3 target groups, 3 listeners and 7 attachments, the praefect group on TCP 2305 with client addresses off and a TCP check. No IAM or estate change is needed: the runner's grants already cover the instance sizes, a third target group and listener, and the run-scoped rules. In the gitlab role: - praefect is a node role. It names no role, since omnibus has none for it, so every service the default role would start is switched off by name. It listens on praefect.port for clients presenting praefect.token, and reaches every node in praefect.nodes with the internal token at a stated replication factor; - the virtual storage is 'default' in the template, the name GitLab requires and the Rails storage uses, not an input that could differ; - a Rails node's one storage is that virtual storage through the load balancer, and carries Praefect's token itself. The global gitlab_rails['gitaly_token'] is not set, so nothing falls back to it; - a Gitaly node's storage takes its name from gitaly.storage, the playbook gives it the host's name, and it accepts only the internal token; - the database certificate bundle and the database bootstrap run on any node whose service connects to the database, Rails or Praefect, and the bootstrap is titled for either service; - END on a Praefect node runs praefect sql-ping as git and requires its OK line before it waits for port 2305: Praefect opens its database before it registers a listener, so a port wait alone could only say that nothing listens; - validate.yml checks the database inputs for Rails and Praefect alike, and holds the Praefect inputs to their shapes, with the two tokens different. Praefect itself refuses an empty node list, a repeated storage and a replication factor beyond the node count. Praefect connects to its own database, praefect_production, as its own unprivileged role, praefect, directly and with no PgBouncer, over verify-full against the pinned RDS bundle. Praefect opens few connections, and the ones it holds for LISTEN need sessions of their own. tcnaw-praefect01, the first Praefect node by name, is the only other node given the master credentials, and on every converge it reads Praefect's role, database and extensions as the master user and creates whatever is absent, before its first reconfigure migrates the database. Its only extension is pgaudit; plpgsql, which Praefect's triggers use, is in every new database already. The run secrets replace the one Gitaly token with Praefect's two, the external token Gitaly clients present to Praefect and the internal token Praefect presents to Gitaly, so no client can reach a Gitaly node past Praefect. Praefect's database password joins them. All are generated once into a root-only file on the Redis node, read through the controller under no_log, and written on each node that needs one into gitlab.rb, which holds credentials and is never diffed, and from there by omnibus into the service's own configuration. The external token reaches the Rails and Praefect nodes, the internal one the Praefect and Gitaly nodes. The bootstrap gives PostgreSQL only a SCRAM verifier of the database password, computed on the host from the password on standard input. Praefect's connect errors cannot print the password: pgx v5.10.0's pgconn/errors.go has ConnectError name the user and the database only, and ParseConfigError pass the connection string through redactPW. The playbook runs the cluster in the only order that works: Redis, the three Gitaly nodes, the first Praefect node, the other two, then a new play on all three Praefect nodes that runs praefect check as git and requires its exit status and "All checks passed.", since a check that fails without being fatal still exits 0. It runs before any Rails node converges, so a broken path from Praefect to the Gitaly nodes is named first, and it is read-only, so a second converge still changes nothing. The Rails nodes follow. The contract requires three gitlab_gitaly and three gitlab_praefect hosts and the praefect target group's ARN, and nftables opens 2305 on a Praefect node with no rate limit. A Rails node's readiness with all=1 proves its database and Redis, and that a Praefect answers through the load balancer, but not the Gitaly nodes: Praefect answers that health check itself. praefect check proves the Gitaly nodes, and the proof's push proves the external token. The proof otherwise runs as before, from the first Gitaly node, a client of the HTTP and SSH listeners and a target of none, and first waits for the praefect target group to report all three Praefect nodes healthy. The deploy workflow reads the praefect target group's ARN from Terraform's outputs with the database and the other two target groups, and refuses to go on without it. The role README gains the praefect node role and a Gitaly Cluster section; the role metadata, the Terraform README, the inventory README and the dependencies README's load balancer specifics describe the nine nodes, the 2305 listener and the rules. MANIFEST.sha256 is regenerated. The all-in-one node still renders byte for byte what the single node was proven with.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Region 3.1 of the GitLab HA layout replaces the single Gitaly node with Gitaly Cluster: three Praefect nodes in
front of three Gitaly nodes, which Praefect keeps as replicas of every repository. The stack becomes 9 nodes:
It keeps the same RDS instance and S3 bucket.
praefecttarget group, with client IP preservation off and a TCPhealth check. Rails reaches Praefect only through it, and Praefect alone reaches Gitaly on 8075.
praefect_productionbelongs to its own unprivilegedpraefectrole on the existing RDSinstance. The master user bootstraps it on
tcnaw-praefect01with the existing bootstrap code. Praefect connectsdirectly, with no PgBouncer, over verify-full TLS, and the database's only extension is pgaudit.
run secrets, carried as R2's are. pgx v5.10.0 never prints the password:
pgconn/errors.goConnectErrornamesonly the user and database, and
ParseConfigErrorpasses the DSN throughredactPW.praefect checkon all three Praefect nodes before any Rails node converges, so abroken Praefect→Gitaly path is named first. Rails'
?all=1readiness cannot see that path: Praefect answers thehealth check itself.
R2's whole proof runs unchanged through Praefect: the degenerate-case regression gate. Region 3.2 adds the cluster
proof (stop the primary Gitaly mid-push; the push still succeeds; the node rejoins and reconciles), developed on this
change's held stack.
Review
connection, and no IAM change.
Proof
Local (no AWS):
security-group rule checks and two mutations that fail;
ruby -c, and the all-in-one render is byte-identical to R0's;default;nft -con every role;Live: this merge is followed by a 240-minute hold dispatch, the region's first live run. R3.2's proof is built on
that held stack.
Prove live:
praefect checkbefore any Rails node exists;