Migrate local and CI object storage to pinned PGSTY Silo - #1175
Merged
Conversation
bill-ph
added this pull request to stack #1176
September 11, 2026 21:37
bill-ph
force-pushed
the
codex/migrate-silo
branch
from
September 11, 2026 21:47
d327760 to
1ba9817
Compare
bill-ph
removed this pull request from stack #1176
September 11, 2026 21:48
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: neutral or increased No coverage-reduction warnings detected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the local/CI MinIO service and client with maintained PGSTY Silo while preserving the existing S3 endpoints, bucket policies, credentials, service/container names, volumes, and credential-revocation assertions. Duckgres continues to use its existing generic S3 integration.
This PR targets
mainand includes upstream through72f5f45889301363cc407075cc7f07ccc0e8d0e6. Its diff contains only the Silo migration; no Hoglake performance or scanner-artifact changes are included. The upstream MinIO registry workaround is superseded by the pinned Silo server and bundled client.Changes and pins
docker.io/pgsty/silo:RELEASE.2026-09-03T13-18-01Z@sha256:b616a0cf8cb281e7e6bb3c9b1fb53875b4016a2878223925541c18f82d6c5ca3.mclireleaseRELEASE.2026-09-03T07-13-05Z. The image digest pins both binaries. No client is downloaded at runtime.just test-silo, which checks private-bucket access, user disable/enable/removal, and the Trino reader policy. Client/admin setup failures in the rotation test now fail instead of skipping.MINIO_*,/minio/*, and legacy service names). Add a short storage runbook with the image pin, defaults, initialization, migration, and rollback instructions. Real-S3 daily performance workflows are unchanged.Provenance and compatibility
Verified both linux/amd64 and linux/arm64 manifests, runtime versions, shell/tool availability, server entrypoint, and readiness. Server/client Linux archives match release checksum manifests verified with GitHub attestations; extracted binaries match those shipped in both images. The storage/admin contract passed on both architectures (amd64 under emulation). Root initialization preserves anonymous downloads; Kubernetes initialization preserves its private bucket.
Sources: Silo release, mcli release.
Validation
The main-based branch was rebuilt and checked with
just test-silo, focused Compose/credential-revocation integration tests, andjust trino-ducklake-smoke. The broader results below are from the same migration before unstacking; its storage configuration and runtime contract are unchanged.mcli, then passed against Silo.just test-silo: passed.just test-integration: passed at unchanged repository ports, including stock HTTPFS credential revocation and DuckLake concurrency. 149 top-level passes; existing COPY-to-stdout and unavailable patched-HTTPFS tests skipped, plus existing subtest skips. No assertions weakened.just test-controlplane: passed on an isolated fixture; the DuckLake boolean/MERGE test also passed with unchanged repository ports.just trino-ducklake-smoke: passed. Its opt-in performance tests were run separately.just perf-trino-ducklake: passed, 1,000,000 rows.TRINO_DUCKLAKE_REALISTIC_PERF_PROFILE=realistic-local just perf-trino-ducklake-realistic: passed, 1,000,000 events.git diff --check: passed.just test-siloandGOFLAGS='-run=TestDockerCompose -count=1' just test-integrationpassed.just lint: rerun on the main-based branch; the same six pre-existing SA4023 diagnostics remain atmain.go:335,cmd/duckgres-controlplane/main.go:184, andcontrolplane/control.go:528(including related locations). These production files are unchanged from main.Migration and rollback
Stop writers and back up object data plus metadata consistently before replacing persistent local storage. Reconcile the pinned Compose service, wait for health, rerun initialization, and run the storage contract. The integration harness alone does not upgrade an already-running old service. For rollback, restore the previous checkout/image digests and the paired pre-migration backups; do not assume an older server can safely reuse data written by Silo. Disposable test fixtures can be explicitly recreated and reseeded. Full commands and recovery guidance are in the runbook.