Repository navigation
materialized='table' causes brief service interruption (empty table window) #1269
Description
Describe the bug
When using the default table materialization in a dbt project on Databricks (Delta Lake), the process of replacing the table results in a brief, but critical, service interruption or "empty table" window for consumers.
This downtime occurs during the execution of the final SQL step, which is equivalent to a form of CREATE OR REPLACE TABLE AS SELECT ....
While Databricks/Delta Lake documentation suggests this operation is atomic and provides Snapshot Isolation (protecting long-running queries), we observe that new, short-lived queries (e.g., from dashboards, reports, or subsequent dbt models) that start immediately after the transaction commits can either:
-
Fail due to the table being in a transient state.
-
Return zero records, perceiving the table as momentarily empty before the new snapshot is fully available/indexed.
This behavior prevents true zero-downtime deployment for models that require a full refresh, which is a major constraint for production data pipelines.
Steps To Reproduce
In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.
Expected behavior
The table materialization should support a true zero-downtime swap, ensuring that consumers always have access to a consistent, non-empty snapshot of the data.
System information
dbt --version:*
[2025-11-17 00:59:31,575] INFO:root:00:59:31 Running with dbt=1.9.4
[2025-11-17 00:59:34,932] INFO:root:00:59:34 Registered adapter: databricks=1.10.2
[2025-11-17 00:59:35,093] INFO:root:00:59:35 Unable to do partial parsing because saved manifest not found. Starting full parse.
**The output of python --version:Python 3.11.11
Additional context
Add any other context about the problem here.
Activity
benc-db commentedon Nov 24, 2025
Are you using 'use_materialization_v2'? It not, I'm genuinely surprised, as you should be getting atomic create or replace. If you are setting use_materialization_v2 to true, there we are trading atomicity for correctness due to a hole in the Databricks SQL api that we are working to resolve internally. When set to true, you can use this model config
use_safer_relation_operations: trueto get close to zero downtime swapping, but at the cost of UC losing history (because it tracks history by object identifier, and the swapped in table is a new object). These various issues are why we are not flipping this flag to true by default, but instead or trying to get CREATE OR REPLACE to work with a broader range of config, to retain atomicity and correctness (i.e. masks set at create time, constraints enforced before the data enters the table).pandabrun commentedon Dec 30, 2025
Hi Ben,
Yep, we are using use_materialization_v2 = true.
We’ve looked at
use_safer_relation_operations, but we’re not going to use it because losing Unity Catalog history due to the object swap is a deal-breaker for us in prod.So the empty / failure window we’re seeing lines up with the v2 trade-offs you described. This basically confirms that true zero-downtime full refresh isn’t possible for us right now with v2.
jelstongreen commentedon Feb 20, 2026
IMHO this should be way more obvious when using V2. We were intermittently losing data in production dashboards and experiencing this and it took far longer than it should have done to track down the problem.
github-actions commentedon Aug 20, 2026
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.
sd-db commentedon Oct 24, 2026
Thanks for the report. This is the expected trade-off of
use_materialization_v2: truerather than a bug: V2 creates the table and inserts the rows as two separate commits, so column comments, constraints and masks can be set at create time, and readers can see an empty table in between. We re-checked on1.13.latest: DESCRIBE HISTORY still shows the two versions, while the default V1 path does a single atomicCREATE OR REPLACE TABLE ... AS SELECT. This is the lack of atomicity that theuse_materialization_v2docs call out. Closing in favor of #1102, which tracks restoring atomicity for V2; until then, keep models that readers query during rebuilds on V1, or useuse_safer_relation_operations: trueif losing the table's Unity Catalog history is acceptable.