Conversation
|
|
||
| override protected def run(): Seq[InternalRow] = { | ||
| catalog.loadTable(ident) match { | ||
| catalog |
singhpk234
left a comment
singhpk234
left a comment
There was a problem hiding this comment.
However, if the materialized view is stale, the method simply returns to allow SparkCatalog's loadView to run. In turn, loadView returns the metadata for the virtual view itself, triggering the usual Spark view logic that computes the result set based on the current state of the base tables.
1/ was wondering if auto-refresh of MV on staleness detection should be an opt-in feature ?
2/ Any ideas / plans for incremental refresh ?
These are very good questions. To me looks like if there is an external process that guarantees the freshness, then the current implementation still holds. Manual For (2): We have not discussed incremental refresh plans in the Iceberg community, but there is some relevant work here. You can review some of the test cases here. |
@wmoustafa, Read this today, was wondering if there is something we can utilize from CDC (considering iceberg has support for that) perspective ? how expensive the refreshes of a PB size tables are and what is the ideal frequency of updates in this model, if you can share some datapoints ? rewrite to get incremental refresh by computing deltas between the snapshots and then joining it with other deltas and having union of those does seems user-friendly though |
It really depends on the query and the size of the delta and whole table etc. There is an extension of that work that is currently taking place to get an idea about the cost of some basic queries (e.g., a few joins/aggregations + filters & projections), and coming up with a reasonable cost model (including choosing to not perform incremental at all if incremental is deemed more expensive). |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Pushed updated changes to the upstream branch. They may not be reflected here since the PR was closed. |
github-actions
Bot
added
API
Specification
|
@wmoustafa I reopened it. Can you rebase to resolve the conflicts? |
7c64e98 to
92d4ac6
Compare
|
|
||
| View view = loadIcebergView(); | ||
| // storage-table should be set on the view version, not as a property | ||
| assertThat(view.currentVersion().storageTable()).isNotNull(); |
There was a problem hiding this comment.
executing the following statement
sql("SHOW TABLES")
returns:
default.tabledefault.materialized_view__storage
What is the added value of seeing default.materialized_view__storage in the table listing ?
For reference the Trino MV do not store the MV storage table in the metastore - see trinodb/trino#18853
There was a problem hiding this comment.
There was a problem hiding this comment.
This is a direction that was aligned by the community (to express MVs as separate view and table objects). This implementation should be compliant with the spec from that perspective.
| tableState.name()); | ||
| try { | ||
| org.apache.iceberg.Table sourceTable = | ||
| ((org.apache.iceberg.catalog.Catalog) icebergCatalog()).loadTable(sourceId); |
There was a problem hiding this comment.
remove (org.apache.iceberg.catalog.Catalog)
Lightweight surgical patch extracted from wmoustafa's (apache/iceberg#9830) storage-table/refresh-state files applied on top of the stable apache-iceberg-1.11.0 release tag instead, published as org.apache.iceberg:*:1.11.0-findinpath-mv1 from https://github.com/findinpath/iceberg (branch materialized-views-1.11.0). This is a temporary showcase reference so this PR can build and run its test suite for real; it is not meant to land as-is.
Replaces the 1.12.0-SNAPSHOT dependency (a full rebase of wmoustafa's materialized-views branch onto origin/main, carrying a lot of unrelated upstream churn) with a much smaller, surgical patch of just the storage-table/refresh-state files (apache/iceberg#9830) applied on top of the stable apache-iceberg-1.11.0 release tag instead, published as org.apache.iceberg:*:1.11.0-findinpath-mv1 from https://github.com/findinpath/iceberg (branch materialized-views-1.11.0). Decouples testing/trino-product-tests' iceberg-spark-runtime dependency into its own dep.iceberg.spark-runtime.version property pinned to the real released 1.11.0, since that jar is only used for Spark interop verification and has nothing to do with this patch. Adds a temporary <repositories> entry serving the custom build from https://raw.githubusercontent.com/findinpath/iceberg/maven-repo/ so CI can resolve it. This is a temporary showcase reference so this PR can build and run its test suite for real; it is not meant to land as-is.
Replaces the 1.12.0-SNAPSHOT dependency (a full rebase of wmoustafa's materialized-views branch onto origin/main, carrying a lot of unrelated upstream churn) with a much smaller, surgical patch of just the storage-table/refresh-state files (apache/iceberg#9830) applied on top of the stable apache-iceberg-1.11.0 release tag instead, published as org.apache.iceberg:*:1.11.0-findinpath-mv1 from https://github.com/findinpath/iceberg (branch materialized-views-1.11.0). Decouples testing/trino-product-tests' iceberg-spark-runtime dependency into its own dep.iceberg.spark-runtime.version property pinned to the real released 1.11.0, since that jar is only used for Spark interop verification and has nothing to do with this patch. Adds a temporary <repositories> entry serving the custom build from https://raw.githubusercontent.com/findinpath/iceberg/maven-repo/ so CI can resolve it. This is a temporary showcase reference so this PR can build and run its test suite for real; it is not meant to land as-is.
- Replace block imports with individual imports - Remove empty line separating import groups - Break long lines to stay within 120 char limit - Applied to both v3.5 and v4.1 versions
- Use static imports for assertj Assertions in TestRefreshStateParser - Rename parameter to avoid hidden field in BaseMetastoreViewCatalog
- Use ExtensionsTestBase instead of SparkExtensionsTestBase - Replace JUnit 4 annotations with JUnit 5 (TestTemplate, BeforeEach, AfterEach) - Use ParameterizedTestExtension and Parameters instead of Parameterized - Remove JUnit 4 constructor-based parameter injection
Checkstyle requires assertThatThrownBy to include a .hasMessage() check. Applied to both v3.5 and v4.1 TestMaterializedViews.
Set up validationCatalog manually instead of calling super.before(), matching the v4.1 approach, to avoid IllegalArgumentException from the base class not recognizing InMemoryCatalogWithLocalFileIO.
REFRESH MATERIALIZED VIEW previously only recorded SourceTableState for base tables discovered via the analyzed plan's leaves, silently ignoring any nested views the MV's query depends on. isFresh() likewise never checked SourceViewState even though the type already existed. - RefreshMaterializedViewExec: also collect SubqueryAlias nodes across the whole analyzed plan (not just leaves) to discover every nested source view transitively, and record a SourceViewState for each. - SparkCatalog.isFresh: validate SourceViewState entries by comparing the source view's current version id against the recorded one. - Add TestMaterializedViews coverage for both REFRESH capturing nested view state and isFresh detecting staleness from a nested view change.
Materialized views are supported only on Spark 4.2, so remove the Spark 3.5 and Spark 4.1 implementations. Spark 4.2 introduces RelationCatalog.loadRelation, a unified table-or-view lookup that lets a catalog route a materialized view to its storage table explicitly. Spark 3.5 and 4.1 have no such entry point, so those versions have to redirect the engine by throwing from loadView, which leaks an unchecked exception to callers that resolve views directly. Supporting a single version keeps the feature on the supported routing path. Removes the per-version implementations under spark/v3.5 and spark/v4.1 and reverts the materialized view changes to the view analysis, planning, and catalog files there. The engine-independent support in api/ and core/ is unchanged and is shared by the Spark 4.2 implementation.
Add CREATE MATERIALIZED VIEW, REFRESH MATERIALIZED VIEW and DROP for materialized views, following the view spec's materialized view design: a materialized view is a view whose current version names a storage table, and the storage table's snapshot summary records the state of the sources the result was computed from. Reading a materialized view goes through RelationCatalog.loadRelation, which compares the recorded source state against the sources as they are now. When they agree the storage table is read, and when they do not the view's query is read instead, so a stale materialized view returns the same rows as the view it materializes rather than stale ones. A refresh runs the view's query and writes the result to the storage table. Each view column takes its values from the query column recorded for it when the view was created, matching how Spark reads the view, so column aliases and a reordered source table both keep returning the same values. When a recorded name is no longer in the query's output the refresh fails and names the columns it cannot read. Replacing a materialized view is rejected. The spec records the storage table per view version and leaves open what a new version does with the table the previous definition materialized, so the statement is rejected rather than settling that here. CREATE OR REPLACE VIEW over a materialized view is rejected for the same reason.
Replaces the 1.12.0-SNAPSHOT dependency (a full rebase of wmoustafa's materialized-views branch onto origin/main, carrying a lot of unrelated upstream churn) with a much smaller, surgical patch of just the storage-table/refresh-state files (apache/iceberg#9830) applied on top of the stable apache-iceberg-1.11.0 release tag instead, published as org.apache.iceberg:*:1.11.0-findinpath-mv1 from https://github.com/findinpath/iceberg (branch materialized-views-1.11.0). Decouples testing/trino-product-tests' iceberg-spark-runtime dependency into its own dep.iceberg.spark-runtime.version property pinned to the real released 1.11.0, since that jar is only used for Spark interop verification and has nothing to do with this patch. Adds a temporary <repositories> entry serving the custom build from https://raw.githubusercontent.com/findinpath/iceberg/maven-repo/ so CI can resolve it. This is a temporary showcase reference so this PR can build and run its test suite for real; it is not meant to land as-is.
Replaces the 1.12.0-SNAPSHOT dependency (a full rebase of wmoustafa's materialized-views branch onto origin/main, carrying a lot of unrelated upstream churn) with a much smaller, surgical patch of just the storage-table/refresh-state files (apache/iceberg#9830) applied on top of the stable apache-iceberg-1.11.0 release tag instead, published as org.apache.iceberg:*:1.11.0-findinpath-mv1 from https://github.com/findinpath/iceberg (branch materialized-views-1.11.0). Decouples testing/trino-product-tests' iceberg-spark-runtime dependency into its own dep.iceberg.spark-runtime.version property pinned to the real released 1.11.0, since that jar is only used for Spark interop verification and has nothing to do with this patch. Adds a temporary <repositories> entry serving the custom build from https://raw.githubusercontent.com/findinpath/iceberg/maven-repo/ so CI can resolve it. This is a temporary showcase reference so this PR can build and run its test suite for real; it is not meant to land as-is.
Add dependency to the apaceh/iceberg updated class model required for materialized views by making a surgical patch of just the storage-table/refresh-state files (apache/iceberg#9830) applied on top of the stable apache-iceberg-1.11.0 release tag, published as org.apache.iceberg:*:1.11.0-findinpath-mv1 from https://github.com/findinpath/iceberg (branch materialized-views-1.11.0). Decouples testing/trino-product-tests' iceberg-spark-runtime dependency into its own dep.iceberg.spark-runtime.version property pinned to the real released 1.11.0, since that jar is only used for Spark interop verification and has nothing to do with this patch. Adds a temporary <repositories> entry serving the custom build from https://raw.githubusercontent.com/findinpath/iceberg/maven-repo/ so CI can resolve it. This is a temporary showcase reference so this PR can build and run its test suite for real; it is not meant to land as-is.
08101bc to
4df4eb5
Compare
10 participants
Summary
This PR adds support for materialized views in Iceberg and integrates the implementation with Spark SQL.
Spec
Full Materialized View Spec can be found in #11041. A materialized view is an Iceberg view whose current version has a
storage-tablefield: a struct withnamespaceandnameidentifying an Iceberg table that holds the precomputed results. The storage table is used to return the precomputed results of the view as long as the results are "fresh".Freshness is tracked through a
refresh-stateJSON string stored in the storage table's snapshot summary. The refresh state captures:A materialized view is considered fresh when the view version ID and all source snapshot/version IDs in the refresh state match their current values.
Core
New model classes:
ViewVersion.storageTable()— nullableTableIdentifieron the view version; non-null indicates a materialized viewRefreshState/RefreshStateParser— model and JSON serialization for refresh state stored in snapshot summariesSourceState/SourceTableState/SourceViewState— polymorphic source state model discriminated by atypefield (tableorview)Spark SQL
This PR adds support for
CREATE MATERIALIZED VIEWand extendsDROP VIEWto handle materialized views:CREATE MATERIALIZED VIEWcreates the storage table first, then registers the view metadata with astorage-tablereference on the view version. The storage table identifier can be specified via aSTORED AS '<identifier>'clause; otherwise a default<name>__storageidentifier is used.DROP VIEWon a materialized view removes both the view metadata and its associated storage table.REFRESH MATERIALIZED VIEWis left as a future enhancement.Spark Catalog
The
SparkCatalogdetermines whether to serve precomputed data from the storage table or fall back to the view's SQL query:loadTable()checks if the requested identifier corresponds to a fresh materialized view. If so, it returns aSparkMaterializedViewbacked by the storagetable, allowing queries to read the precomputed data directly.
loadView()checks if the materialized view is fresh. If fresh, it defers toloadTable(). If stale, it returns aSparkView, triggering the usual Spark view logic that re-executes the query against the current state of the source tables.Notes
InMemoryCataloghas been extended with a testLocalFileIOto support data file operations required by the storage table.