Repository navigation
feat: decimal → string/double/float coercions - #6
Closed
Baoyuan-Xing wants to merge 1 commit into
Closed
Baoyuan-Xing wants to merge 1 commit into
Baoyuan-Xing wants to merge 1 commit into
Conversation
BigQuery's Storage Read API surfaces NUMERIC as Decimal128(38, 9) and
BIGNUMERIC as Decimal256(76, 38), and money-typed warehouse columns
(GMV, order values, price aggregates) are NUMERIC by default. Both
decimal widths were asserted Incompatible with every proto kind, so any
table carrying a money column could not be transcoded at all: producers
had to pre-CAST in SQL. On a real table with 18 NUMERIC fields nested up
to three struct levels deep (including inside a repeated struct), that
pre-CAST means mechanically rebuilding every enclosing struct — ~160
lines of generated SQL that exist only because of this gap.
Protobuf has no decimal scalar kind, so every decimal mapping is a
representation choice. That is exactly what the coercion tier is for:
- Decimal128/256 → string: Lossless. Arrow's own scale-preserving
decimal formatting ("12.340000000" for Decimal(38, 9)), so the stored
value round-trips exactly. The right target for money.
- Decimal128/256 → double/float: PrecisionLoss. Unscaled integer
divided by 10^scale in f64 (i256 via ToPrimitive), rounding to the
nearest binary float. The right target for ML-feature columns.
- Decimal → integer kinds stay Incompatible: dropping the scale is a
data decision, not a type mapping, and belongs upstream.
Nothing becomes Compatible: an unannotated decimal field still fails at
plan time with the existing field-pathed incompatibility error, so the
strict-by-default posture is unchanged.
Implementation follows the established coercion pattern end to end
(compatibility gate → plan arm → ScalarKind kernel). num-traits is a new
direct dependency (already in the tree transitively via arrow-buffer)
for i256::to_f64.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
BigQuery's Storage Read API surfaces
NUMERICasDecimal128(38, 9)andBIGNUMERICasDecimal256(76, 38)— and money-typed warehouse columns (GMV, order values, price aggregates) areNUMERICby default. Both decimal widths are currently assertedIncompatiblewith every proto kind, so a table carrying one money column cannot be transcoded at all.The workaround is pre-
CASTing in SQL, and on nested schemas it is brutal: a real table with 18 NUMERIC fields nested up to three struct levels deep (one inside a repeated struct) requires mechanically rebuilding every enclosing struct — ~160 lines of generated SQL that exist only because of this gap.Design
Protobuf has no decimal scalar, so every decimal mapping is a representation choice — which is exactly what the coercion tier expresses. Nothing becomes
Compatible; all of this is opt-in via annotation or--coerce:Decimal128/256 → stringLossless"12.340000000"forDecimal(38,9)) — exact round-trip; the right target for moneyDecimal128/256 → double/floatPrecisionLossi256viaToPrimitive), nearest binary float; the right target for ML featuresDecimal → int kindsIncompatibleAn unannotated decimal field still fails at plan time with the existing field-pathed incompatibility error, so the strict-by-default posture is unchanged — the failure mode observed in the wild is a clear
arrow Decimal128(38, 9) ↔ proto double: incompatible typesnaming the full nested path, and that stays the behavior unless the caller opts in.Implementation
Follows the established coercion pattern end to end: compatibility gate → plan arm →
ScalarKindkernel (6 new kernels, both widths × 3 targets).num-traitsbecomes a direct dependency (already transitive viaarrow-buffer) fori256::to_f64.Tests
Decimal128/256removed from the unsupported-types fixture;Nullremains