Skip to content

Migrate local and CI object storage to pinned PGSTY Silo - #1175

Merged
bill-ph merged 5 commits into
mainfrom
codex/migrate-silo
Sep 12, 2026
Merged

Migrate local and CI object storage to pinned PGSTY Silo#1175
bill-ph merged 5 commits into
mainfrom
codex/migrate-silo

Conversation

@bill-ph

@bill-ph bill-ph commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Replace the local/CI MinIO service and client with maintained PGSTY Silo while preserving the existing S3 endpoints, bucket policies, credentials, service/container names, volumes, and credential-revocation assertions. Duckgres continues to use its existing generic S3 integration.

This PR targets main and includes upstream through 72f5f45889301363cc407075cc7f07ccc0e8d0e6. Its diff contains only the Silo migration; no Hoglake performance or scanner-artifact changes are included. The upstream MinIO registry workaround is superseded by the pinned Silo server and bundled client.

Changes and pins

  • Root and integration Compose, local Kubernetes dependencies, and CI use docker.io/pgsty/silo:RELEASE.2026-09-03T13-18-01Z@sha256:b616a0cf8cb281e7e6bb3c9b1fb53875b4016a2878223925541c18f82d6c5ca3.
  • Initialization and admin operations use the same image's bundled mcli release RELEASE.2026-09-03T07-13-05Z. The image digest pins both binaries. No client is downloaded at runtime.
  • CI initializes the same Compose fixture and runs just test-silo, which checks private-bucket access, user disable/enable/removal, and the Trino reader policy. Client/admin setup failures in the rotation test now fail instead of skipping.
  • Preserve compatibility vocabulary (MINIO_*, /minio/*, and legacy service names). Add a short storage runbook with the image pin, defaults, initialization, migration, and rollback instructions. Real-S3 daily performance workflows are unchanged.
  • Keep test scaffolding limited to immutable-image consistency and the runtime storage contract. The contract uses the existing fixed fixture credentials; only its container name is configurable for isolated validation.

Provenance and compatibility

Verified both linux/amd64 and linux/arm64 manifests, runtime versions, shell/tool availability, server entrypoint, and readiness. Server/client Linux archives match release checksum manifests verified with GitHub attestations; extracted binaries match those shipped in both images. The storage/admin contract passed on both architectures (amd64 under emulation). Root initialization preserves anonymous downloads; Kubernetes initialization preserves its private bucket.

Sources: Silo release, mcli release.

Validation

The main-based branch was rebuilt and checked with just test-silo, focused Compose/credential-revocation integration tests, and just trino-ducklake-smoke. The broader results below are from the same migration before unstacking; its storage configuration and runtime contract are unchanged.

  • TDD: new Compose guards failed before migration and passed afterward; storage contract failed against the old image's missing mcli, then passed against Silo.
  • just test-silo: passed.
  • just test-integration: passed at unchanged repository ports, including stock HTTPFS credential revocation and DuckLake concurrency. 149 top-level passes; existing COPY-to-stdout and unavailable patched-HTTPFS tests skipped, plus existing subtest skips. No assertions weakened.
  • just test-controlplane: passed on an isolated fixture; the DuckLake boolean/MERGE test also passed with unchanged repository ports.
  • just trino-ducklake-smoke: passed. Its opt-in performance tests were run separately.
  • just perf-trino-ducklake: passed, 1,000,000 rows.
  • TRINO_DUCKLAKE_REALISTIC_PERF_PROFILE=realistic-local just perf-trino-ducklake-realistic: passed, 1,000,000 events.
  • Compose configuration validation, shell syntax, and git diff --check: passed.
  • Final test-scaffolding checks: just test-silo and GOFLAGS='-run=TestDockerCompose -count=1' just test-integration passed.
  • just lint: rerun on the main-based branch; the same six pre-existing SA4023 diagnostics remain at main.go:335, cmd/duckgres-controlplane/main.go:184, and controlplane/control.go:528 (including related locations). These production files are unchanged from main.

Migration and rollback

Stop writers and back up object data plus metadata consistently before replacing persistent local storage. Reconcile the pinned Compose service, wait for health, rerun initialization, and run the storage contract. The integration harness alone does not upgrade an already-running old service. For rollback, restore the previous checkout/image digests and the paired pre-migration backups; do not assume an older server can safely reuse data written by Silo. Disposable test fixtures can be explicitly recreated and reseeded. Full commands and recovery guidance are in the runbook.

@bill-ph
bill-ph added this pull request to stack #1176 September 11, 2026 21:37
@bill-ph
bill-ph removed this pull request from stack #1176 September 11, 2026 21:48
@bill-ph
bill-ph changed the base branch from gwyang/hoglake-full-perf to main September 11, 2026 21:48
@github-actions

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 0 7 0
E2E/journey files 0 0 0
Workflow files 0 1 0

Signals

  • Test cases: +1 / -0
  • Assertions: +5 / -0
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

@bill-ph
bill-ph merged commit 17bfdd3 into main Sep 12, 2026
35 checks passed
@bill-ph
bill-ph deleted the codex/migrate-silo branch September 12, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant