Repository navigation
v1.4.0 - #10
Merged
Merged
v1.4.0#10
Conversation
Build both majors from one Dockerfile and CI matrix so existing clusters stay on latest/dev, with pg15/dev-pg15 for new deployments and a startup guard against swapping image tags on incompatible data directories.
Two compounding defects could empty a live cluster: a node on fresh storage (e.g. a Flux reschedule) would initdb a new empty epoch, and the system-ID self-heal would then propagate that empty epoch by wiping the PostgreSQL data dir on every node whose system ID no longer matched. - AUTO_BOOTSTRAP_IF_FRESH now defaults to false: a fresh/empty node keeps retrying to JOIN the existing cluster instead of creating a blank one. Intentional first-time creation uses ALLOW_NEW_CLUSTER_BOOTSTRAP=true. - checkPatroniSystemID no longer wipes /var/lib/postgresql/data unless ALLOW_PG_DATA_WIPE=true (default false); it also now requires the live primary (not just any re-cloned replica) to confirm the authoritative system ID, and otherwise logs a loud, actionable error and leaves data intact. - etcd-start prints operator guidance when a fresh node refuses to bootstrap. - Document the new defaults/flag, first-time bootstrap procedure, and a backups-are-not-optional warning in the README.
Adds a `backup` subcommand (supervisord program, runs as postgres) that takes periodic logical backups of the cluster. Disabled by default. Safety properties: - Backs up only on the node Patroni reports as a healthy, running primary, so there is a single authoritative copy rather than one per node. - Dumps to a temp file and integrity-checks it (valid gzip + pg_dumpall completion marker) before replacing anything; an unhealthy DB/cluster or a truncated dump never overwrites the previous good backup. - Prunes after each successful backup: keeps the newest BACKUP_RETENTION_COUNT files (default 1), then drops oldest until total size is under BACKUP_MAX_TOTAL_BYTES (0 = unlimited); the just-made backup is never pruned, so the directory cannot grow unbounded. Configurable via BACKUP_ENABLED, BACKUP_INTERVAL_SECONDS, BACKUP_DIR, BACKUP_RETENTION_COUNT, BACKUP_MAX_TOTAL_BYTES. Documented in README, including a durability note and restore command.
Covers the data-safety fixes (no fresh-epoch auto-bootstrap, gated PG data wipe) and the new opt-in pg_dumpall backup agent.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR releases v1.4.0 of flux-pg-cluster by adding multi-version (PostgreSQL 14/15) Docker image support, introducing an opt-in built-in logical backup agent, and expanding operational documentation around bootstrap and data-loss prevention.
Changes:
- Refactors the Docker image build to support PostgreSQL 14/15 via
POSTGRES_MAJOR, and updates Patroni config generation to use the correct PostgreSQLbin_dir. - Adds a supervisord-managed
flux-agent backupprocess plus a new Go backup agent implementing periodicpg_dumpallwith integrity checks and pruning. - Expands README guidance and adds new safety/backup environment variables; bumps version to 1.4.0.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
VERSION |
Bumps project version to 1.4.0. |
supervisord.conf |
Adds a supervised backup program to run the built-in backup agent. |
README.md |
Documents new image tags, backup behavior/config, and safer bootstrap/data-loss controls. |
patroni.yml.tpl |
Switches Patroni bin_dir to a template placeholder for multi-PG support. |
internal/config/config.go |
Adds ALLOW_PG_DATA_WIPE and changes AUTO_BOOTSTRAP_IF_FRESH default to false. |
Dockerfile |
Installs PostgreSQL via PGDG with POSTGRES_MAJOR, adds OCI labels. |
docker-compose.pg15-smoke.yml |
Adds a local smoke-test override to build all nodes with PostgreSQL 15. |
cmd/flux-agent/main.go |
Adds the backup subcommand wiring. |
cmd/flux-agent/init.go |
Discovers Postgres bin_dir, validates existing data dir major version, passes bin_dir into Patroni template rendering. |
cmd/flux-agent/etcd_start.go |
Improves split-brain/data-loss prevention logging in bootstrap decision path. |
cmd/flux-agent/daemon.go |
Adds safety gating for PG data wipe based on primary confirmation + ALLOW_PG_DATA_WIPE. |
cmd/flux-agent/backup.go |
New backup agent implementing primary-only pg_dumpall backups with verification + pruning. |
.github/workflows/docker-image.yml |
Refactors Docker build/push to a matrix for PG 14/15 and passes build args/labels. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The PGDG repo line was echoed to stdout instead of being written to /etc/apt/sources.list.d/, so apt-get only saw Ubuntu's default repos. PostgreSQL 14 ships in jammy's default repos (so latest/dev built fine), but 15 only exists in PGDG, causing "Unable to locate package postgresql-15" for the pg15/dev-pg15 image builds. Redirect the echo into pgdg.list so the repo is actually added. Verified by building the PG15 image (installs PostgreSQL 15.18 from PGDG).
Revert AUTO_BOOTSTRAP_IF_FRESH back to true: disabling it broke first-boot cluster formation (and rescheduled-node rejoin), which the integration suite exposed. The load-bearing data-loss protection is ALLOW_PG_DATA_WIPE (default false) plus live-primary confirmation, not refusing to bootstrap. - config: AUTO_BOOTSTRAP_IF_FRESH default true again; clarify comments - etcd_start: revert now-inaccurate bootstrap-refusal operator hint - README: document the wipe-guard as the primary safety control - tests: enable ALLOW_PG_DATA_WIPE in test compose so EC4b exercises the guarded wipe/rejoin path; update EC4 docstring - tests: demux etcdctl stdout/stderr so transient client warnings no longer corrupt returned key values (fixes EC4a flake) Full integration suite green (12/12).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces support for both PostgreSQL 14 and 15 in the Docker image, adds a built-in logical backup agent, and significantly expands documentation around backup, data-loss prevention, and cluster bootstrap controls. The Docker build and deployment workflow is refactored to support multiple PostgreSQL versions from a single Dockerfile, and several new environment variables are documented for safer cluster management.
Multi-version PostgreSQL Docker support:
POSTGRES_MAJORbuild argument, supporting both versions 14 and 15 with a single image. The image is now labeled with its PostgreSQL and app version. [1] [2]latest,dev) and 15 (pg15,dev-pg15) using a build matrix. Image tags now reflect both branch and PostgreSQL version. [1] [2]README.md. [1] [2]Backups and data safety:
pg_dumpall) that runs only on the healthy primary, with configurable retention and storage limits. Environment variables for backup configuration are now documented. [1] [2]AUTO_BOOTSTRAP_IF_FRESH,DEAD_CLUSTER_RECOVERY,ALLOW_PG_DATA_WIPE) are introduced and explained, with strong recommendations for safe cluster creation and recovery practices. [1] [2]Versioning and documentation:
1.4.0in bothVERSIONand the version badge inREADME.md. [1] [2]README.md. [1] [2]These changes make it easier and safer to deploy, upgrade, and operate PostgreSQL clusters on Flux, with improved support for new PostgreSQL versions and robust backup/restore guidance.