Repository navigation
Conversation
- Add a Container Apps job spec that dumps each database with pg_dump -Fc and uploads to blob storage at <lane>/<database>/<timestamp>.dump - Run it on a 03:00 UTC schedule, deployed per lane by a standalone workflow - Render the spec with envsubst over an explicit variable list so the container commands' run-time references survive deploy-time substitution - Guard both container commands with set -e, so a dump that fails on any database but the last no longer reports Succeeded - Document the new lane variables, secret, and the required Container Apps Jobs Contributor grant
- JSON-encode both secrets before rendering the spec. envsubst is a byte substitution, so a value containing a backslash was silently altered by YAML double-quoted escape rules and one containing a quote broke the parse. A JSON string is valid YAML and survives either. - Install the containerapp CLI extension before the deploy, matching delete-container-app.yml. It is not preinstalled on the runner, and without it the first az containerapp call failed while reporting an unresolvable environment name.
Commit 9a74b1a dropped the encoding along with BACKUP_CONTAINER, while the job spec still reads LANE_PGPASSWORD_JSON and LANE_STORAGE_KEY_JSON. Both secret values rendered as those literal strings.
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0315-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0315-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-315.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0315-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0315 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0315 |
Jobs (2)
| job | image | postgres_db | api_url | auth_mode |
|---|---|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0315@sha256:086f5d6cf8f25868a9c14aa73ac39c17efeb936704b8b5f277e320aae7eaeeba |
pr_0315 |
||
| seed | ghcr.io/rmi/stitch-seed:pr-0315@sha256:209ba069df46b745fa26379e1157b749db50690a63e5af5d90cf12efb5d4d4a5 |
https://pr-0315-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 |
stitch-client-bearer-token |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-02T20:53:08Z | 2026-10-02T20:52:43Z | 8461310 | ghcr.io/rmi/stitch-api:pr-0315 |
ghcr.io/rmi/stitch-api:pr-0315@sha256:086f5d6cf8f25868a9c14aa73ac39c17efeb936704b8b5f277e320aae7eaeeba |
| 2026-10-02T20:53:09Z | 2026-10-02T20:52:43Z | 8461310 | ghcr.io/rmi/stitch-entity-linkage:pr-0315 |
ghcr.io/rmi/stitch-entity-linkage:pr-0315@sha256:1d787391930c21dd19c9d80f82a1cf9d7ae2c700be6ef099fcd5128da74a0080 |
| 2026-10-02T20:53:09Z | 2026-10-02T20:52:43Z | 8461310 | ghcr.io/rmi/stitch-seed:pr-0315 |
ghcr.io/rmi/stitch-seed:pr-0315@sha256:209ba069df46b745fa26379e1157b749db50690a63e5af5d90cf12efb5d4d4a5 |
| 2026-10-02T20:53:06Z | 2026-10-02T20:52:43Z | 8461310 | ghcr.io/rmi/stitch-stitch-llm:pr-0315 |
ghcr.io/rmi/stitch-stitch-llm:pr-0315@sha256:8efa7863a61dd540f16b64b9b97ce1e0dac92d98accf93136eadab684b094c7b |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Account-wide storage credentials and backup reliability limitations should be addressed before approval.
Review effort: Balanced
Findings: 1
Open (3)
What changed in this PR
Adds scheduled Azure Container Apps jobs for nightly PostgreSQL backups in staging and production.
Changes:
- Adds the scheduled dump-and-upload job specification.
- Adds a GitHub Actions deployment workflow and script.
- Documents configuration, permissions, retention, and operations.
| File | Description |
|---|---|
deployments/db/jobs/backup-job.yaml |
Defines database dump and Blob upload containers. |
.github/scripts/deploy_backup_job.sh |
Creates or updates the backup job. |
.github/workflows/deploy-backup-job.yml |
Deploys jobs for staging and production. |
deployments/CI_DEPLOYMENTS.md |
Documents backup deployment and configuration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - { name: BACKUP_ENV, value: "${LANE_BACKUP_ENV}" } | ||
| - { name: BACKUP_DATABASES, value: "${LANE_BACKUP_DATABASES}" } | ||
| - { name: ACCOUNT, value: "${LANE_STORAGE_ACCOUNT}" } | ||
| - { name: KEY, secretRef: storagekey } |
There was a problem hiding this comment.
@mbarlow12 I agree with copilot on this one. I'd rather give the blob writer role to the Managed identity that we're using for the ACA Job and handle blob permissions that way.
| configuration: | ||
| triggerType: Schedule | ||
| replicaTimeout: 1800 | ||
| replicaRetryLimit: 0 |
| - { name: PGPASSWORD, secretRef: pgpassword } | ||
| command: ["/bin/sh", "-c"] | ||
| args: | ||
| - 'set -e; for db in $BACKUP_DATABASES; do pg_dump -Fc -d "$db" -f "/scratch/$db.dump"; done' |
AlexAxthelm
left a comment
There was a problem hiding this comment.
overall good, but worth disscussing some of the details before we commit to it.
Like the blob design though. unblocks the "restore to staging" part of the picture
| - { name: BACKUP_ENV, value: "${LANE_BACKUP_ENV}" } | ||
| - { name: BACKUP_DATABASES, value: "${LANE_BACKUP_DATABASES}" } | ||
| - { name: ACCOUNT, value: "${LANE_STORAGE_ACCOUNT}" } | ||
| - { name: KEY, secretRef: storagekey } |
There was a problem hiding this comment.
@mbarlow12 I agree with copilot on this one. I'd rather give the blob writer role to the Managed identity that we're using for the ACA Job and handle blob permissions that way.
| --name backups \ | ||
| --output none | ||
|
|
||
| IFS=$'\t' read -r LANE_ENVIRONMENT_ID LANE_LOCATION <<<"$( |
There was a problem hiding this comment.
This feels like a really brittle way to parse this. How would you feel about --output json and parsing with jq? it's available on GH runners by default.
| - { name: KEY, secretRef: storagekey } | ||
| command: ["/bin/sh", "-c"] | ||
| args: | ||
| - 'set -e; ts=$(date -u +%Y%m%dT%H%M%SZ); for db in $BACKUP_DATABASES; do az storage blob upload --account-name "$ACCOUNT" --account-key "$KEY" -c backups -n "$BACKUP_ENV/$db/$ts.dump" -f "/scratch/$db.dump" -o none; done' |
There was a problem hiding this comment.
I'm thinking that this command could be a bit cleaner if we're only backing up PROD, since we could instead do something where we initiate a container to backup each DB of interest independently (as constructed, this could fail silently if a db doesn't dump nicely)
- drop BACKUP_STORAGE_KEY; the job uploads with a user-assigned identity - scope the blob role to the backups container, not the whole account - omit location from the job spec, which inherits it from its environment - document the one-time per-lane identity setup in CI_DEPLOYMENTS.md
- move the deploy script to scripts/deploy_backup_job.sh so a developer and the workflow run the same code instead of two paths that can drift - add `setup <lane>` to create the backup identity, the backups container and its container-scoped grant, so CI never needs identity or role-assignment permissions - find the identity by name, dropping the BACKUP_IDENTITY_ID and BACKUP_IDENTITY_CLIENT_ID lane variables - derive the job name and blob prefix from the lane argument, dropping JOB_NAME and LANE_BACKUP_ENV - ignore .env.* and scripts/data/, which became stageable once scripts/ held a tracked file
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0315-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0315 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0315 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0315@sha256:7e5d7802225f20784cb6f19e7880f0adc96bc89829ae39e52cef2160df2f8e48 |
pr_0315 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-08T02:39:49Z | 2026-10-08T02:39:35Z | 1451afa | ghcr.io/rmi/stitch-api:pr-0315 |
ghcr.io/rmi/stitch-api:pr-0315@sha256:7e5d7802225f20784cb6f19e7880f0adc96bc89829ae39e52cef2160df2f8e48 |
| 2026-10-08T02:39:54Z | 2026-10-08T02:39:35Z | 1451afa | ghcr.io/rmi/stitch-entity-linkage:pr-0315 |
ghcr.io/rmi/stitch-entity-linkage:pr-0315@sha256:bfd4c40caf7cb69cc47d595dc8088f51a00e57832c2ed6251104bc7070122666 |
| 2026-10-08T02:39:50Z | 2026-10-08T02:39:35Z | 1451afa | ghcr.io/rmi/stitch-seed:pr-0315 |
ghcr.io/rmi/stitch-seed:pr-0315@sha256:3cbed3c73f239332234b52f79a59f8f2775451305ea7b8d938b869e80ef8de5b |
| 2026-10-08T02:39:55Z | 2026-10-08T02:39:35Z | 1451afa | ghcr.io/rmi/stitch-stitch-llm:pr-0315 |
ghcr.io/rmi/stitch-stitch-llm:pr-0315@sha256:3eed74f9e5148a5119dc7f2c9252c3baf52675ef6e503aa7030bba4563c2ebcb |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0315-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0315-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-315.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0315-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0315 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0315 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0315@sha256:10a6884ab70202d9ef7e344523a9ecaaccbcb92567e71b4380cda1d9e298a569 |
pr_0315 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-08T02:47:05Z | 2026-10-08T02:41:39Z | 6b270e5 | ghcr.io/rmi/stitch-api:pr-0315 |
ghcr.io/rmi/stitch-api:pr-0315@sha256:10a6884ab70202d9ef7e344523a9ecaaccbcb92567e71b4380cda1d9e298a569 |
| 2026-10-08T02:47:09Z | 2026-10-08T02:41:39Z | 6b270e5 | ghcr.io/rmi/stitch-entity-linkage:pr-0315 |
ghcr.io/rmi/stitch-entity-linkage:pr-0315@sha256:05d8382d64de5291e3edc58d7dc8346dc7348d195914e1b81f4fc9f5052d25d7 |
| 2026-10-08T02:47:08Z | 2026-10-08T02:41:39Z | 6b270e5 | ghcr.io/rmi/stitch-seed:pr-0315 |
ghcr.io/rmi/stitch-seed:pr-0315@sha256:2e13504b11281541e370cc2d28a25380c56ed947b9c574696db0106413606d67 |
| 2026-10-08T02:47:09Z | 2026-10-08T02:41:39Z | 6b270e5 | ghcr.io/rmi/stitch-stitch-llm:pr-0315 |
ghcr.io/rmi/stitch-stitch-llm:pr-0315@sha256:15b299d7ef88a293ffab46db1d934ce402fde994e5ad0a3f0ab62e3f52be04ad |
- take no lane argument: the job name and blob prefix are now constants, so check_lane, the usage string and the argument count check all go - collapse the single-value workflow matrix, since it no longer varies - rewrite the backup docs for one lane, dropping the staging examples and the two-lane framing Staging is no longer deployed. Restoring it means reinstating the lane argument and the matrix.
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0315-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0315-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-315.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0315-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0315 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0315 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0315@sha256:97d4b885aec8461525083102678248a3ca725f82e14ac1297150f41cd79d5785 |
pr_0315 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-08T04:32:06Z | 2026-10-08T04:31:48Z | d1f6320 | ghcr.io/rmi/stitch-api:pr-0315 |
ghcr.io/rmi/stitch-api:pr-0315@sha256:97d4b885aec8461525083102678248a3ca725f82e14ac1297150f41cd79d5785 |
| 2026-10-08T04:32:04Z | 2026-10-08T04:31:48Z | d1f6320 | ghcr.io/rmi/stitch-entity-linkage:pr-0315 |
ghcr.io/rmi/stitch-entity-linkage:pr-0315@sha256:d81df87e28b6ecc2bd8b0b4d9e739677f579e1d8351c2e8bd88f9f31783b3020 |
| 2026-10-08T04:32:08Z | 2026-10-08T04:31:48Z | d1f6320 | ghcr.io/rmi/stitch-seed:pr-0315 |
ghcr.io/rmi/stitch-seed:pr-0315@sha256:56d4f0f784d48a6e6a24f295f4cbdb4f163185a1a1b325495b6c19aa797ca1db |
| 2026-10-08T04:32:05Z | 2026-10-08T04:31:48Z | d1f6320 | ghcr.io/rmi/stitch-stitch-llm:pr-0315 |
ghcr.io/rmi/stitch-stitch-llm:pr-0315@sha256:aad19bcccea6919cac5cf6c789a4bb75eaaa2f432b112a07d102987a5f16a718 |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0315-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0315-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-315.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0315-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0315 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0315 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0315@sha256:f4bf57225c1e90d780801223a943b562dfad6ce4a1bc5f21b836b864fed18441 |
pr_0315 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-10-08T20:23:17Z | 2026-10-08T20:22:57Z | 31c6bb1 | ghcr.io/rmi/stitch-api:pr-0315 |
ghcr.io/rmi/stitch-api:pr-0315@sha256:f4bf57225c1e90d780801223a943b562dfad6ce4a1bc5f21b836b864fed18441 |
| 2026-10-08T20:23:17Z | 2026-10-08T20:22:57Z | 31c6bb1 | ghcr.io/rmi/stitch-entity-linkage:pr-0315 |
ghcr.io/rmi/stitch-entity-linkage:pr-0315@sha256:d3e44e21aadd7be080f0308686a6a9b097253a7a8bec6642f558f9ab3617f606 |
| 2026-10-08T20:23:15Z | 2026-10-08T20:22:57Z | 31c6bb1 | ghcr.io/rmi/stitch-seed:pr-0315 |
ghcr.io/rmi/stitch-seed:pr-0315@sha256:69b9feaf22d28f8956d957f64972749929f54abe25c4b8eda19795057373fc9f |
| 2026-10-08T20:23:16Z | 2026-10-08T20:22:57Z | 31c6bb1 | ghcr.io/rmi/stitch-stitch-llm:pr-0315 |
ghcr.io/rmi/stitch-stitch-llm:pr-0315@sha256:01a5d791ea75534e7c9817764cf6d4abf22cac694264b557c7ca181b347a6835 |


Summary
Adds a github action that creates/updates a scheduled Container App Job to connect to a db instance, run
pg_dump, then upload the file to a blob storage container.Runs nightly on "production" and "dress rehearsal". No backups for pr branches.
*demo*New secrets and environment variables set on
productionandstagingGitHub environments:az storage account keys list --account-name {stitchstaging,rmistitchprod} -g STITCH-{PROD,DEV}-RG \ --query '[0].value' -o tsv | tr -d '\n' \ | gh secret set BACKUP_STORAGE_KEY --env {production,staging}Restoration
Not implemented here, but I've separately confirmed that
pg_restoreworks as expected. So the general process would be:azaz storage blob download ...targeting the desired backuppg_restorewith appropriate argsTip
We could use this restore workflow for all PR deployments prior to running any migrations. That'd give complete prod-like dbs without needing to run ETL or create merges. Or it could be configurable from special syntax in the PR.
Considered Alternatives
az storage blob upload ...favored the latter at the timepg_restore(which this PR supports)AI Assistance
Planned with Claude until I gave up as complexity continued to grow. Brainstorm with ChatGPT to develop a high-level outline of "CI/CD deploys scheduled ACA Job + ACA Job runs
pg_dumpand uploads to blob storage". Ran through a spike implementation based on the outline, then steered CC in translating the spiked script to the yaml job config, deploy script, and github workflow. CC drafted CI_DEPLOYMENTS.md changes.Related issues
Closes: STIT-787
Testing
Used
azto deploy a manually triggered job to backup an existing db. Confirmed blob container creation &pg_dumpexistence after the job completed. Downloaded and restored on a local db.Checklist
- [ ] Tests pass locally and in CIN/A