Repository navigation
perf: resolve seed placeholder and cast types once per load - #1722
Merged
Merged
Conversation
sd-db
added a commit
that referenced
this pull request
Oct 15, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
Coverage report
This PR does not seem to contain any modification to coverable code. |
jprakash-db
approved these changes
Oct 19, 2026
databricks__load_csv_rows rendered get_binding_char() for every cell and, for parquet seeds, called adapter.convert_type for every cell. Resolve the placeholder once per load and the cast types once per column. get_binding_char() still dispatches, so project overrides keep working. The generated SQL is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
sd-db
force-pushed
the
sd-db/perf/seed-load-csv-rows-resolve-once
branch
from
October 28, 2026 06:52
9795541 to
2deccf0
Compare
Co-authored-by: Isaac <no-reply@databricks.com>
…e-once" This reverts commit 734a7ac. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #
Description
databricks__load_csv_rowsrenderedget_binding_char()once per cell and, forfile_format: parquetseeds, called
adapter.convert_typeonce per cell. Neither value depends on the row, so a10,000-row × 80-column seed made 800,000 calls of each just to build the same SQL. This resolves the
placeholder once per load and the cast types (including
column_typesoverrides) once per column.get_binding_char()still goes through dispatch, so a project-level override keeps working. The SQLsent to Databricks is unchanged.
The same loop made dbt seeds hang in the Fusion engine (dbt-labs/dbt#16748); this is the v1 version
of that fix.
Measured on a serverless SQL warehouse with a 10,000 × 80 seed:
Query text size exceeds limitStock and patched delta loads produce identical tables: same row count, content hash, and column
types. The parquet error happens at the warehouse with or without this change: at the default
10,000-row batch size, a wide parquet seed's single
insertis too long. A 1,000-row batch loads thepatched parquet seed in 87.5s. Batch sizing is not changed here.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.dbt-databricks-pr-readyproject skill for this PR and addressed its merge-readiness feedbackTests: new
tests/unit/macros/materializations/test_seed_macros.pypins the generated SQL and failson the old macro's per-cell
convert_typecalls;tests/unit/macros(333) andtests/functional/adapter/simple_seedondatabricks_uc_sql_endpoint(19) pass.This pull request and its description were written by Isaac.