Skip to content

feat(playbook): prove the Gitaly Cluster survives losing a primary - #27

Merged
NWarila merged 1 commit into
mainfrom
feat/gitaly-cluster-proof
Oct 3, 2026
Merged

NWarila merged 1 commit into
mainfrom
feat/gitaly-cluster-proof

Conversation

@NWarila

@NWarila NWarila commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

What

R3.1 ran R2's whole proof through Praefect but did not show that Praefect works as declared, or that the cluster
survives losing a Gitaly node. This adds both checks to the opt-in proof, after the Rails failover.

  1. Praefect's database. Connecting as Praefect's own role, the proof requires every backend of that role on
    TLSv1.3, and a LISTEN connection from each of the three Praefect nodes. Each node opens two: the read cache and
    the repository locks. The master user's read now requires three SCRAM verifiers: GitLab's, Praefect's and its
    own. The connection counts, and the database logs' size and read time, are reported.
  2. Losing the primary. The steps, in order:
    1. Wait for three replicas that are assigned, fully up to date, healthy and valid primaries.
    2. Dataloss must find every repository fully available.
    3. Read the primary from praefect metadata and stop its Gitaly.
    4. Wait for the primary to read Healthy: false.
    5. Push over HTTP. The push must succeed and HTTP must serve it. Praefect must name a new primary, with the
      stopped replica "behind by" and the other two current.
    6. Gitaly restarts in an always section judged by a named cleanup assert.
    7. Wait until the rejoined replica is reconciled, so all three are valid primaries again.
    8. Dataloss must find every repository fully available again.
    9. gitlab:praefect:replicas must show three equal checksums.

Every expected line is the 19.4.1 source's own format. The parsers were checked against real metadata,
dataloss and rake output, captured from the held stack of run 37119822220:

  • Primary: "tcnaw-gitaly01";
  • three replicas, each fully up to date, Healthy: true and Valid Primary: true;
  • both dataloss endings, with their two-space indent;
  • three equal checksums under a (primary) header.

Review

  • Audits: four domain audits, then a confirmation round; zero open findings.
  • Gates: gate r1 (notes c, d, e, i), and Revision 3 with gate r3 (A–D).
  • Gate r3's source check: dataloss runs metadata's query plus a valid_primaries clause. Both metadata waits now
    require Valid Primary: true, so the read after them cannot lag.

Proof

Local:

  • 47 parser cases, from the extracted expressions and the real output formats;
  • an end-to-end stub flow: green; no-election and push-failure cases each fail by name and still restart Gitaly;
    0 secrets in any log;
  • the trimmed SQL run on PostgreSQL 17;
  • the conditional scan: 437 items, all strings;
  • lint, syntax-check and actionlint.

Budget: the proof step stays at 30 minutes. That covers the 11 minutes measured through Praefect (run 37119822220)
plus an estimated 3–5 for these checks; about 26.5 minutes is the realistic worst case. The first green run's measured
duration is recorded on this PR. Above 20 minutes, a follow-up raises the step to 45.

Live: this merge is followed by a 240-minute hold dispatch. That run is the cluster proof's first live run, and the
region's acceptance run.

The proof already ran its checks through Praefect, but did not show
that Praefect itself works as declared, or that the cluster survives
losing a Gitaly node. Two sections now follow the Rails failover.

Praefect's database: from the first Praefect node, connecting as
Praefect's own role with its password from the run secrets on psql's
standard input, the proof reads every backend of that role and the
addresses that hold its LISTEN connections. Each Praefect node opens
two, for its read cache and its repository locks. Every backend must
be on TLSv1.3, and every Praefect node's private address must hold a
LISTEN connection, with a short wait for a listener caught
reconnecting. The role's and the instance's connection counts and the
database logs' size and read time are reported, not asserted. The
master user's read of the login roles now requires three SCRAM
verifiers: GitLab's role, Praefect's and its own.

Three copies through a lost primary: Praefect's metadata for the
proof project's repository, found by GitLab's hashed path of its id,
must show three replicas that are assigned, fully up to date, healthy
and valid primaries, and dataloss must then find every repository
fully available. The primary is read from that metadata and its
Gitaly is stopped. Once the metadata reports it Healthy: false, a
commit pushed over HTTP must succeed and HTTP must serve it; the
metadata must name a new primary, with the stopped replica unhealthy
and "behind by" the push and the other two fully up to date; and
dataloss must find every repository available.

Gitaly is started again whatever happened, in the block's always
section, and a named cleanup assert requires the start to have run
and Gitaly to listen again. The proof then waits until the rejoined
replica is reconciled and all three are again assigned, fully up to
date, healthy and valid primaries, so the second converge's praefect
check finds nothing outdated, and dataloss must once more find every
repository fully available. Last, gitlab:praefect:replicas must show
three equal checksums, one on each Gitaly node and one marked
primary. Its table is what is judged, since it prints "Something went
wrong" and still exits 0 when it fails.

Every expected line is the format of the 19.4.1 source that prints
it. Lines dataloss indents are matched in its output, not among its
lines.

The Terraform README no longer calls the public address at launch in
us-east-1a unproven: every run since the first to place a node there
has reached it that way. The workflow's proof step comment describes
the added checks and states its 30-minute budget: the 11 minutes the
proof took on run 37119822220, and the 3 to 5 the Gitaly Cluster
checks are estimated to add. The role README says what the proof
shows of the Gitaly Cluster.
@NWarila
NWarila merged commit 49df30d into main Oct 3, 2026
1 check passed
@NWarila
NWarila deleted the feat/gitaly-cluster-proof branch October 3, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant