Apache Iceberg version
main (development)
Query engine
Spark
Please describe the bug 馃悶
TestViews.readFromViewReferencingTempFunction failed on the Spark 4.2 extensions tests in a main push build:
https://github.com/apache/iceberg/actions/runs/34809521107/job/103867881304
TestViews > readFromViewReferencingTempFunction() > catalogName = spark_hive_with_views, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default, cache-enabled=false} FAILED
org.apache.spark.sql.catalyst.analysis.FunctionAlreadyExistsException: [ROUTINE_ALREADY_EXISTS] Cannot create the routine `test_avg919406` because a routine of that name already exists.
Choose a different name, drop or replace the existing routine, or add the IF NOT EXISTS clause to tolerate a pre-existing routine. SQLSTATE: 42723
at app//org.apache.spark.sql.errors.QueryCompilationErrors$.functionAlreadyExistsError(QueryCompilationErrors.scala:1309)
at app//org.apache.spark.sql.execution.command.CreateFunctionCommand.run(functions.scala:68)
...
at app//org.apache.iceberg.spark.TestBase.sql(TestBase.java:131)
at app//org.apache.iceberg.spark.extensions.TestViews.readFromViewReferencingTempFunction(TestViews.java:614)
Likely cause
The temp function name is generated with viewName("test_avg"), which appends new Random().nextInt(1000000). The temp function is never dropped.
TestViews runs each test once per catalog parameter (spark_with_views, spark_catalog, spark_hive_with_views) against the same SparkSession. The temp functions created by the earlier parameter runs therefore stay registered. When the random suffix repeats, CREATE TEMPORARY FUNCTION fails.
Spark 4.2's CreateFunctionCommand throws this error only when SessionCatalog.isRegisteredFunction finds the unqualified name in the session's temp function registry. It does not check the metastore, so the name must have been registered earlier in the same session. The collision is rare, which makes the test flaky.
The same pattern (random suffix, no DROP TEMPORARY FUNCTION) exists in readFromViewReferencingTempFunction, createViewReferencingTempFunction, and createViewReferencingQualifiedTempFunction. It is present in the Spark 3.5, 4.0, 4.1 and 4.2 versions of TestViews.
Possible fix
- Drop the temp function in a
finally block (DROP TEMPORARY FUNCTION IF EXISTS ...), as createViewReferencingTemporaryVariable already does for its temporary variable; and/or
- generate collision-free names in
viewName() (e.g. an incrementing counter or a UUID-based suffix) instead of new Random().nextInt(1000000).
Willingness to contribute
Apache Iceberg version
main (development)
Query engine
Spark
Please describe the bug 馃悶
TestViews.readFromViewReferencingTempFunctionfailed on the Spark 4.2 extensions tests in amainpush build:https://github.com/apache/iceberg/actions/runs/34809521107/job/103867881304
Likely cause
The temp function name is generated with
viewName("test_avg"), which appendsnew Random().nextInt(1000000). The temp function is never dropped.TestViewsruns each test once per catalog parameter (spark_with_views,spark_catalog,spark_hive_with_views) against the sameSparkSession. The temp functions created by the earlier parameter runs therefore stay registered. When the random suffix repeats,CREATE TEMPORARY FUNCTIONfails.Spark 4.2's
CreateFunctionCommandthrows this error only whenSessionCatalog.isRegisteredFunctionfinds the unqualified name in the session's temp function registry. It does not check the metastore, so the name must have been registered earlier in the same session. The collision is rare, which makes the test flaky.The same pattern (random suffix, no
DROP TEMPORARY FUNCTION) exists inreadFromViewReferencingTempFunction,createViewReferencingTempFunction, andcreateViewReferencingQualifiedTempFunction. It is present in the Spark 3.5, 4.0, 4.1 and 4.2 versions ofTestViews.Possible fix
finallyblock (DROP TEMPORARY FUNCTION IF EXISTS ...), ascreateViewReferencingTemporaryVariablealready does for its temporary variable; and/orviewName()(e.g. an incrementing counter or a UUID-based suffix) instead ofnew Random().nextInt(1000000).Willingness to contribute