Skip to content

ci: encrypt the migration dry-run's connections to its restored snapshot - #3775

Merged
jakebromberg merged 2 commits into
mainfrom
ci/pg18-dryrun-ssl
Nov 5, 2026
Merged

jakebromberg merged 2 commits into
mainfrom
ci/pg18-dryrun-ssl

Conversation

@jakebromberg

Copy link
Copy Markdown
Member

Closes #3774

Keeps the required check Migration Dry-Run (prod-shaped data) able to connect after prod moves to PostgreSQL 18. The newest snapshot will then restore with default.postgres18, whose rds.force_ssl = 1 refuses the plaintext connections this job makes today. The fix is on the client side; the restore command is unchanged.

What changed

One file, .github/workflows/test.yml, in the migrate-dryrun job:

  • PGSSL: require on the two steps that connect to the sandbox, Probe pre-migration data shape and Run drizzle:migrate against sandbox. Both use postgres-js with no ssl option (scripts/schema-shape-report.mjs, scripts/dryrun-migrate.mjs). No other step connects to the sandbox, and none uses psql or libpq.
  • In the dry-run step, before node scripts/dryrun-migrate.mjs, a short postgres-js check with the same options and env. It fails the step unless SELECT ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid() returns true, and logs sandbox connection ssl: <value>. RDS 14 accepts TLS without forcing it, so this passes on today's 14.22 snapshots and proves PGSSL is honored before the upgrade makes it matter.

Why PGSSL is enough

The install step resolves postgres to 3.4.9 (root package.json range ^3.4.9; confirmed with npm install --dry-run using the step's flags).

  • postgres-js fills each unset option from env['PG' + k.toUpperCase()] (src/index.js L476 at v3.4.9).
  • 'require' sets rejectUnauthorized: false (src/connection.js L283-L284), so it encrypts without a CA bundle.
  • Only 'prefer' falls back to plaintext (L271).
  • postgres-js 4.0.0 behaves the same: src/index.js L460 and secureOptions in src/socket.js.

Verification

This PR's own CI will report the dry-run as skipped. The job is gated on the db-init paths filter, and test.yml is not in it. A workflow_dispatch run would skip it too, because dispatch computes the filter against the merge base. The first migration PR after this merges is the first real run.

So I verified the step locally. I extracted its run: script from the YAML and ran it under bash -eo pipefail against a local PostgreSQL 18.6 with TLS on and a self-signed certificate:

postgres-js Server PGSSL Result
3.4.9 TLS on require ssl: true, exit 0
3.4.9 TLS on unset ssl: false, ::error::, exit 1
4.0.0 TLS on require ssl: true, exit 0
4.0.0 TLS on unset ssl: false, ::error::, exit 1
3.4.9 plaintext rejected unset refused: no encryption, exit 1
3.4.9 plaintext rejected require ssl: true, exit 0

"Plaintext rejected" is a hostnossl ... reject rule in pg_hba.conf, which stands in for rds.force_ssl = 1.

Also checked:

  • prettier --check (3.9.9, the lockfile version) passes.
  • actionlint reports the same findings as on main, none on the changed steps.
  • scripts/check-sql-claim-docs.mjs passes.

…hot (BS#3774)

Once prod is on PostgreSQL 18, the newest snapshot restores with default.postgres18, which sets rds.force_ssl = 1 and refuses the plaintext connections both postgres-js clients in migrate-dryrun make today. Set PGSSL=require on the shape-probe and dry-run steps, and fail the dry-run step unless pg_stat_ssl reports its connection as encrypted, which proves PGSSL is honored while the snapshots are still 14.22. The restore command is unchanged.
… of dumping an uncaught trace (BS#3774)

A timeout, auth or TLS error in the inline check used to surface as a raw Node uncaught-exception trace with no `::error::` annotation. It now prints `::error::sandbox SSL check failed: <message>` and exits 1. Only `err.message` is printed, never the error object, which can carry the connection options.
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Schema constraint shape report

no schema or migration changes detected in this PR

@jakebromberg

Copy link
Copy Markdown
Member Author

Exercised the migration dry-run against the current prod snapshot (the newest automated snapshot, taken 2026-11-03 21:04 PST, engine 14.22). The PR's own CI skips that job because test.yml isn't in the db-init filter, so a temporary commit touched dev_env/format-pg-error.mjs to trip it. That commit has since been dropped.

Run: https://github.com/WXYC/Backend-Service/actions/runs/41965162921. Migration Dry-Run (prod-shaped data) passed. The new check logged sandbox connection ssl: true right before dryrun-migrate: ok, and the shape probe ran under the same PGSSL: require without a warning.

The head (d7e19ae) also wraps the check in try/catch. A connect or query failure now produces a ::error::sandbox SSL check failed: <message> annotation, printing only the error message, where it used to dump an uncaught Node trace.

@jakebromberg
jakebromberg merged commit aa7a54b into main Nov 5, 2026
6 checks passed
@jakebromberg
jakebromberg deleted the ci/pg18-dryrun-ssl branch November 5, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI/CD and testing infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migration dry-run: connect to the restored snapshot over TLS before prod moves to PostgreSQL 18

1 participant