Skip to content

Allowlist DB_HOST_OVERRIDE in set-ec2-env-var.yml for the PostgreSQL 18 rollback - #3782

Merged
jakebromberg merged 2 commits into
mainfrom
ci/db-host-override-secret
Nov 5, 2026
Merged

jakebromberg merged 2 commits into
mainfrom
ci/db-host-override-secret

Conversation

@jakebromberg

@jakebromberg jakebromberg commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Closes #3776

What

set-ec2-env-var.yml gains one allowlisted secret, DB_HOST_OVERRIDE, so the PostgreSQL 18 upgrade's fast rollback can point the host's DB_HOST at whichever instance it needs (the old 14.22 instance, a snapshot-restored one, or the original instance name) and recreate backend, auth and enrichment-worker with it. Today that dispatch fails with Unsupported secret_name.

  • The resolve step's env: block reads DB_HOST_OVERRIDE, with a comment like its neighbours': it is temporary, it names whichever instance DB_HOST should point at during the rollback, the rollback script sets it just before its runs and deletes it afterwards, and it is removed once the rollback window has closed.
  • The case "$SECRET_NAME" block forwards it.
  • A check after that block refuses DB_HOST_OVERRIDE under any key but DB_HOST. secret_name defaults to key, so without it a hand-typed -f key=DB_HOST_OVERRIDE would write a line nothing reads, recreate the container against the unchanged DB_HOST, and go green.
  • ALLOWLISTED_KEYS in tests/unit/scripts/set-ec2-env-var-allowlist.test.ts lists it. The test's existing full-set equality check keeps the two workflow halves in step. The new check sits outside the case arm the test pins, and no test pins the step's other guard (the AWS reserved-name check), so none is added for it.

Nothing else changes, including the container-recreate flags.

How it is used

The rollback script sets the secret, then runs the workflow once per container, one run at a time, then deletes the secret:

gh workflow run set-ec2-env-var.yml --ref main -f key=DB_HOST -f secret_name=DB_HOST_OVERRIDE -f restart_target=backend
# then auth, then enrichment-worker

DB_HOST passes the key guard, which refuses only the four AWS SDK credential names, and nothing else in the workflow special-cases database keys. While the secret is unset, a run stops in the resolve step with Secret 'DB_HOST_OVERRIDE' is empty or not set, before the SSH step, so merging this changes nothing on the host. A run with any other key stops in the same step with DB_HOST_OVERRIDE is only for key=DB_HOST.

The issue covers the rest of the path: the anchored upsert, masking, which processes read DB_HOST, the healthcheck, and the recreate caveat this PR leaves alone.

Timing

This has to merge before the upgrade window starts, because merging deploys and deploys are frozen during the window. The rollback dispatches with --ref main.

Checks run locally

  • Prettier 3.9.9, the lockfile version, --check on both changed files: clean.
  • actionlint 1.7.12 on the workflow: one shellcheck style note, SC2129, which origin/main already has. No new findings.
  • The allowlist test's assertions, replicated outside jest with yaml 2.9.1: failed before the workflow change (2 of 13), passed after (33 env keys, 33 case arms).
  • The resolve step's script, extracted with yaml and its ${{ }} expressions stubbed: bash -n clean. Run against stub env files, key=DB_HOST_OVERRIDE and key=FOO secret_name=DB_HOST_OVERRIDE both exit 1 with the new error, key=DB_HOST secret_name=DB_HOST_OVERRIDE exits 1 with "empty or not set" when the secret is empty and 0 when it is set, and key=DB_HOST alone still exits 1 with Unsupported secret_name.

Follow-up

Once the rollback window has closed, which is when the instance left over from the upgrade is deleted, remove the env: line and its comment, the case arm, the key=DB_HOST check and the ALLOWLISTED_KEYS entry, and confirm the repo secret is gone.

…rollback can point DB_HOST back at the old instance (BS#3776)

The fast rollback for the 14.22 -> 18.6 Blue/Green upgrade moves the host instead of renaming instances: it upserts DB_HOST in the host's .env and recreates backend, auth and enrichment-worker, one workflow run each. The workflow reads only allowlisted secrets, so DB_HOST_OVERRIDE goes in both the resolve step's env block and its SECRET_NAME case, with a comment saying it is temporary, set just before use, and removed once the old instance is deleted. The secret stays unset until then, and an unset secret stops the run in the resolve step before it touches the host.

DB_HOST_OVERRIDE is added to ALLOWLISTED_KEYS in the allowlist test, next to the existing full-set equality check of the two blocks.
…ment what the secret names and when it goes (BS#3776)

secret_name defaults to key, and every other allowlisted secret is used with the two equal, so a hand-typed `-f key=DB_HOST_OVERRIDE` would write a DB_HOST_OVERRIDE line that nothing reads, recreate the container against the unchanged DB_HOST, and go green. A check after the SECRET_NAME case now stops that run with an error naming the right flags. It sits outside the case arm, so the arm the allowlist test pins is unchanged; no test pins the step's other guard (the AWS reserved-name check) either, so none is added for this one.

The comment now says the secret names whichever instance DB_HOST should point at during the rollback, not only the old one; that the rollback script sets it just before its runs and deletes it afterwards, and a run stops in this step while it is unset; and that the env line, the case arm, the guard and the ALLOWLISTED_KEYS test entry come out once the rollback window has closed, when the instance left over from the upgrade is deleted.
@jakebromberg
jakebromberg force-pushed the ci/db-host-override-secret branch from e8ea005 to 8ad7a40 Compare November 5, 2026 05:13
@jakebromberg
jakebromberg merged commit 29f465e into main Nov 5, 2026
6 checks passed
@jakebromberg
jakebromberg deleted the ci/db-host-override-secret branch November 5, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI/CD and testing infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

set-ec2-env-var.yml: allowlist DB_HOST_OVERRIDE so a PostgreSQL 18 rollback can point DB_HOST back at the old instance

1 participant