API: Prevent intermediate overflow in integer truncation - #18112
Open
laserninja wants to merge 3 commits into
Open
laserninja wants to merge 3 commits into
laserninja wants to merge 3 commits into
Conversation
Use a nonnegative remainder without overflowing the width addition so matching partitions are not incorrectly pruned. Fixes apache#18111 Generated-by: Codex
Verify that SQL inserts remain visible to range filters across Parquet, Avro, and ORC. Generated-by: Codex
1 task
uros-b
approved these changes
Sep 15, 2026
Fokko
approved these changes
Sep 15, 2026
Fokko
left a comment
Fokko
left a comment
Contributor
There was a problem hiding this comment.
It is a bit of an edge case, but I believe it is a bug. Thanks @laserninja for fixing this. Can you follow up on my comment?
| } | ||
|
|
||
| @TestTemplate | ||
| void scanWithLargeIntegerTruncateWidth() { |
Contributor
There was a problem hiding this comment.
Can we add this one for the other Spark versions as well?
Contributor
Author
There was a problem hiding this comment.
Added the regression test for Spark 3.5, 4.0, and 4.2. All nine added cases passed across Parquet, Avro, etc. Updated validation notes too.
Add the scan regression to Spark 3.5, 4.0, and 4.2 so each supported engine version verifies matching rows survive partition pruning. Generated-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #18111.
Truncating
Integer.MAX_VALUE - 1with widthInteger.MAX_VALUEcurrently produces a negative partition key. A scan filtered byid >= 0then omits the matching file.Use
Math.floorModin the int and narrow-integer helpers to avoid overflowing the intermediate remainder-plus-width addition. This preserves public APIs and existing behavior when the final mathematical result is outside the target type's range.Validation on JDK 17:
:iceberg-api:testsuite and targeted core truncation and table scan regressions passed. The new regressions failed before the fix.:iceberg-api:spotlessCheck :iceberg-core:spotlessCheckpassed.:iceberg-api:revapi :iceberg-core:revapipassed.Spark SQL regression (Spark 4.1): create an integer table partitioned by
truncate(2147483647, id), insert2147483646, and queryWHERE id >= 0. On the original implementation the query returns no rows in Parquet, Avro, and ORC; with the fix it returns the inserted row in all three formats. The new Spark test, formatting, and Spark test checkstyle pass locally on JDK 17.Additional local validation at
28aff7bfb: the full CI all-module build command (./gradlew -DallModules build -x test -x javadoc -x integrationTest) passed on JDK 17, including formatting and style checks across the engine versions. Apache RAT (dev/check-license),./gradlew checkAllRuntimeDeps -q -DallModules=true, and./gradlew -Pquick=true javadocalso passed. This build excludes tests; the Spark regression results above are from a separate test run.Review follow-up: the same Spark SQL regression now covers Spark 3.5 (Scala 2.12), 4.0, and 4.2 (Scala 2.13), in addition to 4.1. All nine added version/format cases passed locally on JDK 17, together with
spotlessCheckandcheckstyleTestfor the three changed Spark modules.AI Disclosure