Repository navigation
Allowlist DB_HOST_OVERRIDE in set-ec2-env-var.yml for the PostgreSQL 18 rollback - #3782
Merged
Merged
Conversation
4 tasks
…rollback can point DB_HOST back at the old instance (BS#3776) The fast rollback for the 14.22 -> 18.6 Blue/Green upgrade moves the host instead of renaming instances: it upserts DB_HOST in the host's .env and recreates backend, auth and enrichment-worker, one workflow run each. The workflow reads only allowlisted secrets, so DB_HOST_OVERRIDE goes in both the resolve step's env block and its SECRET_NAME case, with a comment saying it is temporary, set just before use, and removed once the old instance is deleted. The secret stays unset until then, and an unset secret stops the run in the resolve step before it touches the host. DB_HOST_OVERRIDE is added to ALLOWLISTED_KEYS in the allowlist test, next to the existing full-set equality check of the two blocks.
…ment what the secret names and when it goes (BS#3776) secret_name defaults to key, and every other allowlisted secret is used with the two equal, so a hand-typed `-f key=DB_HOST_OVERRIDE` would write a DB_HOST_OVERRIDE line that nothing reads, recreate the container against the unchanged DB_HOST, and go green. A check after the SECRET_NAME case now stops that run with an error naming the right flags. It sits outside the case arm, so the arm the allowlist test pins is unchanged; no test pins the step's other guard (the AWS reserved-name check) either, so none is added for this one. The comment now says the secret names whichever instance DB_HOST should point at during the rollback, not only the old one; that the rollback script sets it just before its runs and deletes it afterwards, and a run stops in this step while it is unset; and that the env line, the case arm, the guard and the ALLOWLISTED_KEYS test entry come out once the rollback window has closed, when the instance left over from the upgrade is deleted.
jakebromberg
force-pushed
the
ci/db-host-override-secret
branch
from
November 5, 2026 05:13
e8ea005 to
8ad7a40
Compare
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3776
What
set-ec2-env-var.ymlgains one allowlisted secret,DB_HOST_OVERRIDE, so the PostgreSQL 18 upgrade's fast rollback can point the host'sDB_HOSTat whichever instance it needs (the old 14.22 instance, a snapshot-restored one, or the original instance name) and recreatebackend,authandenrichment-workerwith it. Today that dispatch fails withUnsupported secret_name.env:block readsDB_HOST_OVERRIDE, with a comment like its neighbours': it is temporary, it names whichever instanceDB_HOSTshould point at during the rollback, the rollback script sets it just before its runs and deletes it afterwards, and it is removed once the rollback window has closed.case "$SECRET_NAME"block forwards it.DB_HOST_OVERRIDEunder any key butDB_HOST.secret_namedefaults tokey, so without it a hand-typed-f key=DB_HOST_OVERRIDEwould write a line nothing reads, recreate the container against the unchangedDB_HOST, and go green.ALLOWLISTED_KEYSintests/unit/scripts/set-ec2-env-var-allowlist.test.tslists it. The test's existing full-set equality check keeps the two workflow halves in step. The new check sits outside thecasearm the test pins, and no test pins the step's other guard (the AWS reserved-name check), so none is added for it.Nothing else changes, including the container-recreate flags.
How it is used
The rollback script sets the secret, then runs the workflow once per container, one run at a time, then deletes the secret:
gh workflow run set-ec2-env-var.yml --ref main -f key=DB_HOST -f secret_name=DB_HOST_OVERRIDE -f restart_target=backend # then auth, then enrichment-workerDB_HOSTpasses the key guard, which refuses only the four AWS SDK credential names, and nothing else in the workflow special-cases database keys. While the secret is unset, a run stops in the resolve step withSecret 'DB_HOST_OVERRIDE' is empty or not set, before the SSH step, so merging this changes nothing on the host. A run with any other key stops in the same step withDB_HOST_OVERRIDE is only for key=DB_HOST.The issue covers the rest of the path: the anchored upsert, masking, which processes read
DB_HOST, the healthcheck, and the recreate caveat this PR leaves alone.Timing
This has to merge before the upgrade window starts, because merging deploys and deploys are frozen during the window. The rollback dispatches with
--ref main.Checks run locally
--checkon both changed files: clean.origin/mainalready has. No new findings.yaml2.9.1: failed before the workflow change (2 of 13), passed after (33 env keys, 33 case arms).yamland its${{ }}expressions stubbed:bash -nclean. Run against stub env files,key=DB_HOST_OVERRIDEandkey=FOO secret_name=DB_HOST_OVERRIDEboth exit 1 with the new error,key=DB_HOST secret_name=DB_HOST_OVERRIDEexits 1 with "empty or not set" when the secret is empty and 0 when it is set, andkey=DB_HOSTalone still exits 1 withUnsupported secret_name.Follow-up
Once the rollback window has closed, which is when the instance left over from the upgrade is deleted, remove the
env:line and its comment, thecasearm, thekey=DB_HOSTcheck and theALLOWLISTED_KEYSentry, and confirm the repo secret is gone.