feat(playbook): prove the Gitaly Cluster survives losing a primary - #27
Merged
Merged
Conversation
The proof already ran its checks through Praefect, but did not show that Praefect itself works as declared, or that the cluster survives losing a Gitaly node. Two sections now follow the Rails failover. Praefect's database: from the first Praefect node, connecting as Praefect's own role with its password from the run secrets on psql's standard input, the proof reads every backend of that role and the addresses that hold its LISTEN connections. Each Praefect node opens two, for its read cache and its repository locks. Every backend must be on TLSv1.3, and every Praefect node's private address must hold a LISTEN connection, with a short wait for a listener caught reconnecting. The role's and the instance's connection counts and the database logs' size and read time are reported, not asserted. The master user's read of the login roles now requires three SCRAM verifiers: GitLab's role, Praefect's and its own. Three copies through a lost primary: Praefect's metadata for the proof project's repository, found by GitLab's hashed path of its id, must show three replicas that are assigned, fully up to date, healthy and valid primaries, and dataloss must then find every repository fully available. The primary is read from that metadata and its Gitaly is stopped. Once the metadata reports it Healthy: false, a commit pushed over HTTP must succeed and HTTP must serve it; the metadata must name a new primary, with the stopped replica unhealthy and "behind by" the push and the other two fully up to date; and dataloss must find every repository available. Gitaly is started again whatever happened, in the block's always section, and a named cleanup assert requires the start to have run and Gitaly to listen again. The proof then waits until the rejoined replica is reconciled and all three are again assigned, fully up to date, healthy and valid primaries, so the second converge's praefect check finds nothing outdated, and dataloss must once more find every repository fully available. Last, gitlab:praefect:replicas must show three equal checksums, one on each Gitaly node and one marked primary. Its table is what is judged, since it prints "Something went wrong" and still exits 0 when it fails. Every expected line is the format of the 19.4.1 source that prints it. Lines dataloss indents are matched in its output, not among its lines. The Terraform README no longer calls the public address at launch in us-east-1a unproven: every run since the first to place a node there has reached it that way. The workflow's proof step comment describes the added checks and states its 30-minute budget: the 11 minutes the proof took on run 37119822220, and the 3 to 5 the Gitaly Cluster checks are estimated to add. The role README says what the proof shows of the Gitaly Cluster.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
R3.1 ran R2's whole proof through Praefect but did not show that Praefect works as declared, or that the cluster
survives losing a Gitaly node. This adds both checks to the opt-in proof, after the Rails failover.
TLSv1.3, and a LISTEN connection from each of the three Praefect nodes. Each node opens two: the read cache and
the repository locks. The master user's read now requires three SCRAM verifiers: GitLab's, Praefect's and its
own. The connection counts, and the database logs' size and read time, are reported.
praefect metadataand stop its Gitaly.Healthy: false.stopped replica "behind by" and the other two current.
alwayssection judged by a named cleanup assert.gitlab:praefect:replicasmust show three equal checksums.Every expected line is the 19.4.1 source's own format. The parsers were checked against real
metadata,datalossand rake output, captured from the held stack of run 37119822220:Primary: "tcnaw-gitaly01";fully up to date,Healthy: trueandValid Primary: true;(primary)header.Review
valid_primariesclause. Both metadata waits nowrequire
Valid Primary: true, so the read after them cannot lag.Proof
Local:
0 secrets in any log;
Budget: the proof step stays at 30 minutes. That covers the 11 minutes measured through Praefect (run 37119822220)
plus an estimated 3–5 for these checks; about 26.5 minutes is the realistic worst case. The first green run's measured
duration is recorded on this PR. Above 20 minutes, a follow-up raises the step to 45.
Live: this merge is followed by a 240-minute hold dispatch. That run is the cluster proof's first live run, and the
region's acceptance run.