Fix DuckLake worker catalog loading without metadata password files - #38
Closed
bill-ph wants to merge 1 commit into
Closed
Fix DuckLake worker catalog loading without metadata password files#38bill-ph wants to merge 1 commit into
bill-ph wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A DuckLake catalog can be created successfully on the coordinator while a worker's Kubernetes Secret volume has not yet received the metadata password file. The first distributed query then fails with HTTP 500 during worker catalog initialization, even though workers do not access the metadata database.
Validate password-file existence only on coordinators. Preserve the missing-file error text used by the provisioner's pending-state classifier, password-source mutual exclusion, and file reads on every new metadata connection.
Additional context and related issues
Addresses the separate worker catalog-loading failure exposed by PostHog/duckgres#1181. Duckgres needs to adopt an image containing this change before its E2E lane can exercise the fix.
The new distributed regression reproduced the original HTTP 500 before the change. With the fix, real INSERT and SELECT statements succeed both on the initial worker and on a replacement worker, with coordinator task scheduling disabled and the worker password file absent throughout. Additional checks cover coordinator rejection, password-source mutual exclusion, and recovery after deleting and restoring the coordinator password file.
Validation:
TZ=UTC ./mvnw -pl plugin/trino-ducklake test: 197 checks passed, no failures or skips. UTC is needed by two existing native DuckDB timestamp-string assertions../mvnw -pl plugin/trino-ducklake -am validate: passed, including formatting and checkstyle across the connector and its dependencies.Release notes
(x) Release notes are required, with the following suggested text: