Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 11 additions & 21 deletions .github/workflows/container-image-worker-cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,29 +80,21 @@ jobs:
ducklake: "v1.0-posthog.7"
# The stable 1.5.5 scanner predates duckdb-postgres
# 71b85668, which fixes inconsistent snapshots across
# scan connections, so we deliberately take the nightly
# and content-pin it.
# scan connections, so we need a nightly build. The
# nightly URL is mutable, and pinning a hash against it
# hard-failed the build five times between 2026-08-03
# and 2026-09-11. So we mirror the artifact instead:
# PostHog/duckdb-postgres holds that nightly captured at
# duckdb-postgres a3516c0, a descendant of 71b85668.
# A release asset URL does not move, so this tag needs
# no refresh — it changes only to adopt a new revision.
#
# WARNING: the nightly URL is a MOVING target — upstream
# rebuilds it and the pinned hash then stops matching,
# which hard-fails every build until someone re-pins
# (2026-08-04 -> 2026-08-08 was one such break). These
# hashes were re-pinned to the nightly served on
# 2026-09-01. Its extension footer identifies
# duckdb-postgres 4053b61, a descendant of 71b85668, so
# the required snapshot fix remains present.
# Mirroring the artifact we want, instead of pinning a
# hash against a URL whose contents change, is the real
# fix.
#
# These values are duplicated in e2e-mw-dev.yml,
# This value is duplicated in e2e-mw-dev.yml,
# scenario-dev.yml, Dockerfile and Dockerfile.worker.
# The workflow build-args OVERRIDE the Dockerfile ARG
# defaults, so updating only the Dockerfiles does
# nothing for CI — change all five together.
pg_scanner_repo: "https://nightly-extensions.duckdb.org"
pg_scanner_sha256_amd64: "fdeb8a146f79a91b4af0c00708f62496b5c40ae6e741786e9bc53cfc59b322c2"
pg_scanner_sha256_arm64: "d5663be7338af2cce5595c4c0ca170c7939d6979fec5adede284021aba405b1f"
pg_scanner_tag: "v1.5.5-a3516c0"
# Fleet default: also publishes the unsuffixed tags and
# dispatches Charts.
default: true
Expand Down Expand Up @@ -161,9 +153,7 @@ jobs:
DUCKDB_EXTENSION_VERSION=${{ matrix.duckdb.version }}
HTTPFS_EXTENSION_TAG=${{ matrix.duckdb.httpfs }}
DUCKLAKE_EXTENSION_TAG=${{ matrix.duckdb.ducklake }}
POSTGRES_SCANNER_REPOSITORY=${{ matrix.duckdb.pg_scanner_repo }}
POSTGRES_SCANNER_SHA256_AMD64=${{ matrix.duckdb.pg_scanner_sha256_amd64 }}
POSTGRES_SCANNER_SHA256_ARM64=${{ matrix.duckdb.pg_scanner_sha256_arm64 }}
POSTGRES_SCANNER_TAG=${{ matrix.duckdb.pg_scanner_tag }}
cache-from: type=gha,scope=worker-${{ matrix.duckdb.version }}-${{ matrix.platform.slug }}
cache-to: type=gha,mode=max,scope=worker-${{ matrix.duckdb.version }}-${{ matrix.platform.slug }}

Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/e2e-mw-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,7 @@ jobs:
HTTPFS_EXTENSION_TAG=v1.5.5-cred-refresh-write-retry
DUCKLAKE_EXTENSION_TAG=v1.0-posthog.7
DUCKDB_EXTENSION_REPOSITORY=https://extensions.duckdb.org
POSTGRES_SCANNER_REPOSITORY=https://nightly-extensions.duckdb.org
POSTGRES_SCANNER_SHA256_AMD64=fdeb8a146f79a91b4af0c00708f62496b5c40ae6e741786e9bc53cfc59b322c2
POSTGRES_SCANNER_SHA256_ARM64=d5663be7338af2cce5595c4c0ca170c7939d6979fec5adede284021aba405b1f
POSTGRES_SCANNER_TAG=v1.5.5-a3516c0
secrets:
ecr-role: ${{ vars.AWS_ECR_PRS_PUBLISH_IAM_ROLE }}

Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/scenario-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,7 @@ jobs:
HTTPFS_EXTENSION_TAG=v1.5.5-cred-refresh-write-retry
DUCKLAKE_EXTENSION_TAG=v1.0-posthog.7
DUCKDB_EXTENSION_REPOSITORY=https://extensions.duckdb.org
POSTGRES_SCANNER_REPOSITORY=https://nightly-extensions.duckdb.org
POSTGRES_SCANNER_SHA256_AMD64=fdeb8a146f79a91b4af0c00708f62496b5c40ae6e741786e9bc53cfc59b322c2
POSTGRES_SCANNER_SHA256_ARM64=d5663be7338af2cce5595c4c0ca170c7939d6979fec5adede284021aba405b1f
POSTGRES_SCANNER_TAG=v1.5.5-a3516c0
secrets:
ecr-role: ${{ vars.AWS_ECR_PRS_PUBLISH_IAM_ROLE }}

Expand Down
30 changes: 11 additions & 19 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,14 @@ ARG DUCKDB_EXTENSION_VERSION=1.5.5
ARG HTTPFS_EXTENSION_TAG=v1.5.5-cred-refresh-write-retry
ARG DUCKLAKE_EXTENSION_TAG=v1.0-posthog.7
ARG DUCKDB_EXTENSION_REPOSITORY=https://extensions.duckdb.org
# Repository for postgres_scanner specifically. The checksums content-pin the
# DuckDB 1.5.5 nightly artifact built from duckdb-postgres 4053b61; CI overrides
# all three values together for rollback rows.
ARG POSTGRES_SCANNER_REPOSITORY=https://nightly-extensions.duckdb.org
ARG POSTGRES_SCANNER_SHA256_AMD64=fdeb8a146f79a91b4af0c00708f62496b5c40ae6e741786e9bc53cfc59b322c2
ARG POSTGRES_SCANNER_SHA256_ARM64=d5663be7338af2cce5595c4c0ca170c7939d6979fec5adede284021aba405b1f
# postgres_scanner comes from a PostHog mirror, not from DuckDB's extension
# repositories. The stable 1.5.5 scanner predates duckdb-postgres 71b85668, which
# fixes inconsistent snapshots across scan connections, so we need a nightly build.
# The nightly URL is mutable: upstream rebuilds it and a content pin then stops
# matching, which hard-failed the build five times between 2026-08-03 and
# 2026-09-11. The mirror is that nightly artifact captured at a known-good
# revision behind an immutable URL, so there is nothing left to re-pin.
ARG POSTGRES_SCANNER_TAG=v1.5.5-a3516c0
# `: ${VAR:?msg}` asserts every required input is non-empty — catches a
# CI matrix row that forgets to pass a build-arg and would otherwise
# silently fall back to the ARG default, producing a cross-version
Expand All @@ -44,26 +46,16 @@ RUN : "${DUCKDB_EXTENSION_VERSION:?must be set}" \
&& : "${HTTPFS_EXTENSION_TAG:?must be set}" \
&& : "${DUCKLAKE_EXTENSION_TAG:?must be set}" \
&& : "${DUCKDB_EXTENSION_REPOSITORY:?must be set}" \
&& : "${POSTGRES_SCANNER_REPOSITORY:?must be set}" \
&& case "${TARGETARCH}" in \
amd64) postgres_scanner_sha256="${POSTGRES_SCANNER_SHA256_AMD64}" ;; \
arm64) postgres_scanner_sha256="${POSTGRES_SCANNER_SHA256_ARM64}" ;; \
*) echo "ERROR: unsupported TARGETARCH for postgres_scanner: ${TARGETARCH}" >&2; exit 1 ;; \
esac \
&& : "${postgres_scanner_sha256:?postgres_scanner checksum must be set}" \
&& : "${POSTGRES_SCANNER_TAG:?must be set}" \
&& mkdir -p "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}" \
&& curl -fsSL "https://github.com/PostHog/duckdb-httpfs/releases/download/${HTTPFS_EXTENSION_TAG}/httpfs-linux-${TARGETARCH}.duckdb_extension" \
-o "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/httpfs.duckdb_extension" \
&& curl -fsSL "https://github.com/PostHog/ducklake/releases/download/${DUCKLAKE_EXTENSION_TAG}/ducklake-linux-${TARGETARCH}.duckdb_extension" \
-o "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/ducklake.duckdb_extension" \
&& curl -fsSL "${DUCKDB_EXTENSION_REPOSITORY}/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/json.duckdb_extension.gz" \
| gunzip > "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/json.duckdb_extension" \
&& curl -fsSL "${POSTGRES_SCANNER_REPOSITORY}/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/postgres_scanner.duckdb_extension.gz" \
-o /tmp/postgres_scanner.duckdb_extension.gz \
&& echo "${postgres_scanner_sha256} /tmp/postgres_scanner.duckdb_extension.gz" | sha256sum -c - \
&& gunzip -c /tmp/postgres_scanner.duckdb_extension.gz \
> "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/postgres_scanner.duckdb_extension" \
&& rm /tmp/postgres_scanner.duckdb_extension.gz \
&& curl -fsSL "https://github.com/PostHog/duckdb-postgres/releases/download/${POSTGRES_SCANNER_TAG}/postgres_scanner-linux-${TARGETARCH}.duckdb_extension" \
-o "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/postgres_scanner.duckdb_extension" \
&& for f in httpfs ducklake json postgres_scanner; do \
[ -s "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/$f.duckdb_extension" ] \
|| { echo "ERROR: $f.duckdb_extension is empty after fetch" >&2; exit 1; }; \
Expand Down
30 changes: 11 additions & 19 deletions Dockerfile.worker
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,14 @@ ARG DUCKDB_EXTENSION_VERSION=1.5.5
ARG HTTPFS_EXTENSION_TAG=v1.5.5-cred-refresh-write-retry
ARG DUCKLAKE_EXTENSION_TAG=v1.0-posthog.7
ARG DUCKDB_EXTENSION_REPOSITORY=https://extensions.duckdb.org
# Repository for postgres_scanner specifically. The checksums content-pin the
# DuckDB 1.5.5 nightly artifact built from duckdb-postgres 4053b61; CI overrides
# all three values together for rollback rows.
ARG POSTGRES_SCANNER_REPOSITORY=https://nightly-extensions.duckdb.org
ARG POSTGRES_SCANNER_SHA256_AMD64=fdeb8a146f79a91b4af0c00708f62496b5c40ae6e741786e9bc53cfc59b322c2
ARG POSTGRES_SCANNER_SHA256_ARM64=d5663be7338af2cce5595c4c0ca170c7939d6979fec5adede284021aba405b1f
# postgres_scanner comes from a PostHog mirror, not from DuckDB's extension
# repositories. The stable 1.5.5 scanner predates duckdb-postgres 71b85668, which
# fixes inconsistent snapshots across scan connections, so we need a nightly build.
# The nightly URL is mutable: upstream rebuilds it and a content pin then stops
# matching, which hard-failed the build five times between 2026-08-03 and
# 2026-09-11. The mirror is that nightly artifact captured at a known-good
# revision behind an immutable URL, so there is nothing left to re-pin.
ARG POSTGRES_SCANNER_TAG=v1.5.5-a3516c0

# Cross-check that DUCKDB_EXTENSION_VERSION (which keys the bundled-extension
# directory layout) matches the DuckDB version implied by DUCKDB_BINDINGS_VERSION.
Expand Down Expand Up @@ -92,26 +94,16 @@ RUN : "${DUCKDB_EXTENSION_VERSION:?must be set}" \
&& : "${HTTPFS_EXTENSION_TAG:?must be set}" \
&& : "${DUCKLAKE_EXTENSION_TAG:?must be set}" \
&& : "${DUCKDB_EXTENSION_REPOSITORY:?must be set}" \
&& : "${POSTGRES_SCANNER_REPOSITORY:?must be set}" \
&& case "${TARGETARCH}" in \
amd64) postgres_scanner_sha256="${POSTGRES_SCANNER_SHA256_AMD64}" ;; \
arm64) postgres_scanner_sha256="${POSTGRES_SCANNER_SHA256_ARM64}" ;; \
*) echo "ERROR: unsupported TARGETARCH for postgres_scanner: ${TARGETARCH}" >&2; exit 1 ;; \
esac \
&& : "${postgres_scanner_sha256:?postgres_scanner checksum must be set}" \
&& : "${POSTGRES_SCANNER_TAG:?must be set}" \
&& mkdir -p "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}" \
&& curl -fsSL "https://github.com/PostHog/duckdb-httpfs/releases/download/${HTTPFS_EXTENSION_TAG}/httpfs-linux-${TARGETARCH}.duckdb_extension" \
-o "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/httpfs.duckdb_extension" \
&& curl -fsSL "https://github.com/PostHog/ducklake/releases/download/${DUCKLAKE_EXTENSION_TAG}/ducklake-linux-${TARGETARCH}.duckdb_extension" \
-o "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/ducklake.duckdb_extension" \
&& curl -fsSL "${DUCKDB_EXTENSION_REPOSITORY}/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/json.duckdb_extension.gz" \
| gunzip > "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/json.duckdb_extension" \
&& curl -fsSL "${POSTGRES_SCANNER_REPOSITORY}/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/postgres_scanner.duckdb_extension.gz" \
-o /tmp/postgres_scanner.duckdb_extension.gz \
&& echo "${postgres_scanner_sha256} /tmp/postgres_scanner.duckdb_extension.gz" | sha256sum -c - \
&& gunzip -c /tmp/postgres_scanner.duckdb_extension.gz \
> "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/postgres_scanner.duckdb_extension" \
&& rm /tmp/postgres_scanner.duckdb_extension.gz \
&& curl -fsSL "https://github.com/PostHog/duckdb-postgres/releases/download/${POSTGRES_SCANNER_TAG}/postgres_scanner-linux-${TARGETARCH}.duckdb_extension" \
-o "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/postgres_scanner.duckdb_extension" \
&& for f in httpfs ducklake json postgres_scanner; do \
[ -s "/build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/$f.duckdb_extension" ] \
|| { echo "ERROR: $f.duckdb_extension is empty after fetch" >&2; exit 1; }; \
Expand Down
Loading