Skip to content

feat: accept use_materialization_v2 as a model config - #1718

Merged
sd-db merged 7 commits into
mainfrom
sd-db/use-materialization-v2-model-config
Nov 2, 2026
Merged

sd-db merged 7 commits into
mainfrom
sd-db/use-materialization-v2-model-config

Conversation

@sd-db

@sd-db sd-db commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Description

use_materialization_v2 can only be set project-wide today, as a behavior flag under flags:. For table and incremental models, it chooses between two materialization paths with a real tradeoff:

  • Default: a single atomic CREATE OR REPLACE TABLE … AS SELECT. Readers never see an empty table, but column masks can't be applied at create time.
  • V2: create the table, then insert. Column masks, comments, and constraints are applied at create time, but the table is briefly empty during a refresh.

A project that needs create-time masks on a few sensitive tables has to give up atomic refresh everywhere, or the other way round. This makes use_materialization_v2 a model config for table and incremental models, matching the dbt Fusion engine change in dbt-labs/dbt#16661:

  • DatabricksConfig.use_materialization_v2: Optional[bool]. It can be set in {{ config(...) }}, in properties YAML, or per folder with +use_materialization_v2 in dbt_project.yml, and inherits like other model configs.
  • A new use_materialization_v2() macro returns the model's setting when set. If the setting is unset or null, it returns the use_materialization_v2 behavior flag.
  • Quoted values such as "false" arrive in Jinja as strings, and 'false' is truthy. The macro compares them case-insensitively to true, as Fusion does when it parses the config.
  • The table and incremental materializations branch on the macro instead of reading the flag directly.

Views use one shared implementation in 1.13.0 and ignore this config. That unification is separate work, and this PR doesn't change view.sql. view_update_via_alter stays a separate opt-in setting. Seeds, snapshots, materialized views, streaming tables, metric views, and clones don't use the flag and are unchanged.

Projects that don't set the model config behave exactly as before.

Until dbt Core picks up the updated config schema from Fusion, dbt Core may warn that use_materialization_v2 is an unrecognized config key. The setting still takes effect.

Testing

  • Unit (tests/unit/macros/materializations/test_use_materialization_v2.py): an unset or null setting follows the flag both ways. true/false and the strings "true"/"False" override it. This is the regression coverage for the change. Both table and incremental paths are already covered by the existing V1 and V2 functional suites.
  • End to end, run locally on a UC SQL warehouse: with the flag off and +use_materialization_v2: true, table and incremental models were built by V2: DESCRIBE HISTORY showed a WRITE and no … AS SELECT. With the flag on and config(use_materialization_v2=false), they were built by the default path's CTAS. Models that didn't set the config followed the flag. These checks were not committed as functional tests.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

This pull request and its description were written by Isaac.

@sd-db
sd-db marked this pull request as ready for review October 13, 2026 22:00
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  dbt/adapters/databricks
  impl.py
Project Total  

This report was generated by python-coverage-comment-action

tauhid621 added a commit to dbt-labs/dbt that referenced this pull request Oct 26, 2026
…16879) (#16892)

Backport only the use_materialization_v2 entries from the dbt-fusion
schema sync in #16879: +use_materialization_v2 in ProjectModelConfig and
use_materialization_v2 in the model, seed, snapshot, source, data test,
unit test, and function configs. The blocks match #16879 byte for byte.

dbt-databricks is making use_materialization_v2 a per-model config
(databricks/dbt-databricks#1718). Without these entries, Databricks
projects that set it get a spurious CustomKeyInConfigDeprecation warning.

The rest of #16879 is left out. It does not cherry-pick cleanly here
because earlier syncs (#16421, #16525) were never backported, and the
newer schema also drops or changes existing keys, which could cause new
warnings in a patch release. This change only adds keys.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
tauhid621 added a commit to dbt-labs/dbt that referenced this pull request Oct 26, 2026
…16879) (#16893)

Backport only the use_materialization_v2 entries from the dbt-fusion
schema sync in #16879: +use_materialization_v2 in ProjectModelConfig and
use_materialization_v2 in the model, seed, snapshot, source, data test,
unit test, and function configs. The blocks match #16879 byte for byte.

dbt-databricks is making use_materialization_v2 a per-model config
(databricks/dbt-databricks#1718). Without these entries, Databricks
projects that set it get a spurious CustomKeyInConfigDeprecation warning.

The rest of #16879 is left out. It does not cherry-pick cleanly here
because earlier syncs (#16421, #16525) were never backported, and the
newer schema also drops or changes existing keys, which could cause new
warnings in a patch release. This change only adds keys.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from 1.13.latest to main October 28, 2026 05:11
sd-db and others added 3 commits October 28, 2026 11:59
Let a project or an individual table, incremental, or view model choose
its materialization path. A new use_materialization_v2() macro returns the
model's use_materialization_v2 config when set, coercing quoted string
values, and otherwise falls back to the behavior flag. The three
materializations branch on it instead of reading the flag directly.

Co-authored-by: Isaac <no-reply@databricks.com>
The path switch is a macro-only change, and both materialization paths
are already covered by the existing V1 and V2 functional suites, so drop
the dedicated functional tests and keep the resolver unit tests. Link the
changelog entry to its PR.

Co-authored-by: Isaac <no-reply@databricks.com>
…mental

View materialization is being unified into a single implementation for
1.13.0, so the model config no longer selects a View path. Restore
view.sql to read only the behavior flag, and scope the changelog and flow
docs to table and incremental models, describing the CTAS versus
create-then-insert tradeoff.

Co-authored-by: Isaac <no-reply@databricks.com>
@sd-db
sd-db force-pushed the sd-db/use-materialization-v2-model-config branch from f1452c3 to 1226cd4 Compare October 28, 2026 06:52
@sd-db
sd-db merged commit b0c5e8f into main Nov 2, 2026
9 checks passed
@sd-db
sd-db deleted the sd-db/use-materialization-v2-model-config branch November 2, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants