Skip to content

build: mirror postgres_scanner instead of pinning a nightly - #1179

Merged
fuziontech merged 1 commit into
mainfrom
james/mirror-postgres-scanner
Sep 11, 2026
Merged

build: mirror postgres_scanner instead of pinning a nightly#1179
fuziontech merged 1 commit into
mainfrom
james/mirror-postgres-scanner

Conversation

@fuziontech

Copy link
Copy Markdown
Member

Summary

  • Mirror the DuckDB 1.5.5 postgres_scanner nightly to PostHog/duckdb-postgres
  • Fetch it by release tag, dropping the repository URL and the per-arch SHA-256 pins

Goals

  • A scanner fetch that does not break when upstream rebuilds its nightly
  • Keep the snapshot fix 71b85668 that made us take a nightly in the first place

Non-goals

  • Moving to the stable scanner. Stable 1.5.5 is revision 41223e5, which predates 71b85668, so it reintroduces the bug build: pin postgres scanner snapshot fix #1033 avoided.
  • Changing the DuckDB version, or the httpfs/ducklake extensions.

Why

The nightly URL is mutable, so a content pin against it is a treadmill. It has hard-failed the build five times since 2026-08-03 (afc9a0f77ff4913f574bd831fdeb8a14, and now again), each needing a human to re-pin. Warm Docker layer caches hide the break until some unrelated branch builds cold, which is how it surfaced this time.

The matrix comment already named the fix:

Mirroring the artifact we want, instead of pinning a hash against a URL whose contents change, is the real fix.

That is this PR. The mirror holds the nightly captured at duckdb-postgres a3516c0; gh api .../compare/71b85668...a3516c0 reports ahead_by 8, behind_by 0, so the required snapshot fix is present.

Because a release asset URL does not move, the repository URL and both checksums stop earning their keep. That makes postgres_scanner identical in shape to httpfs and ducklake, which already come from tagged PostHog releases: one tag per extension instead of three values duplicated across five files. The checksums are recorded in the release notes.

Production risk

Low, but this does change what the worker image bundles. The bundled scanner moves from revision 4053b61 to a3516c0; both descend from 71b85668. The build fails closed if the tag is missing (curl -fsSL 404 writes nothing, and the [ -s ... ] check rejects an empty file).

The mirror is a new dependency on a PostHog-owned repo. It is the same trust we already place in PostHog/duckdb-httpfs and PostHog/ducklake.

Start here

  1. Dockerfile — the scanner fetch now sits beside httpfs and ducklake and looks the same.
  2. .github/workflows/container-image-worker-cd.ymlpg_scanner_tag replaces three matrix values.

Dockerfile.worker and the other two workflows are the same substitution.

Test plan

  • docker build --target builder -f Dockerfile . on arm64: the extension step fetches all four extensions and passes the size check
  • The postgres_scanner.duckdb_extension in the built image reports a3516c0 / v1.5.5 / linux_arm64 and hashes to b2e464b1..., byte-identical to the mirrored asset
  • Both release assets re-downloaded from the tag and checksum-verified
  • CI on this PR exercises the worker image build

Note

Needs #1178 (MinIO moved to quay.io) for the container-based lanes to go green; the two are independent and can land in either order.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe

@github-actions

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 0 0 0
E2E/journey files 0 0 0
Workflow files 0 3 0

Signals

  • Test cases: +0 / -0
  • Assertions: +0 / -0
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

The stable DuckDB 1.5.5 postgres_scanner predates duckdb-postgres
71b85668, which fixes inconsistent snapshots across scan connections,
so we take a nightly build. The nightly URL is mutable, and pinning a
content hash against it has hard-failed the build five times between
2026-08-03 and 2026-09-11. Each break needed a human to re-pin, and
warm Docker layer caches hid it until an unrelated branch built cold.

Mirror the artifact instead, which the matrix comment already named as
the real fix. PostHog/duckdb-postgres now holds the nightly captured at
a3516c0, a descendant of 71b85668, under an immutable release tag.

This makes postgres_scanner identical in shape to httpfs and ducklake,
which already come from tagged PostHog releases: one tag per extension,
no repository URL and no per-arch checksums to carry through five
files. The checksums are recorded in the release notes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe
@fuziontech
fuziontech force-pushed the james/mirror-postgres-scanner branch from 5734ef4 to e5cf728 Compare September 11, 2026 22:52
@fuziontech
fuziontech merged commit 72f5f45 into main Sep 11, 2026
33 of 35 checks passed
@fuziontech
fuziontech deleted the james/mirror-postgres-scanner branch September 11, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant