Is your feature request related to a problem or challenge?
This issue follows up on #10909 and accompanies the fix for nullif on
RunEndEncoded and Union arrays.
Several compute kernels can introduce logical nulls, for example through null
take indices or a nullif condition.
For most Arrow arrays, a top-level validity bitmap represents these nulls.
RunEndEncoded and Union arrays are different:
- RunEndEncoded derives logical nulls from its
values child.
- Union has no top-level validity bitmap; a logical null must be represented by
a nullable child selected by its type id.
As a result, a kernel can currently produce physical nulls even when the
corresponding field metadata is marked as non-nullable.
Describe the solution you'd like
A kernel must not silently drop or fabricate requested nulls.
- For
RunEndEncoded, if the values field is non-nullable and an operation
needs to introduce a null, return a compute error.
- For
Union, a null may be represented using a nullable child field.
If no child field is nullable and an operation needs to introduce a null,
return a compute error.
- Kernels that preserve an existing null should continue to do so without
requiring a new nullable representation.
- This policy should be applied consistently to kernels that can introduce
nulls, including take, nullif, and similar compute kernels.
Describe alternatives you've considered
Together with #10909 and the nullif fix, this policy keeps the physical
result consistent with its field metadata.
In particular, a field marked as non-nullable must not produce output
containing nulls. Returning an error when a requested null cannot be
represented is preferable to producing a result whose data contradicts its
declared nullability.
Additional context
PR #10909 fixed handling of null take indices for RunEndEncoded and Union
arrays. This issue tracks the broader nullability contract and its consistent
application across compute kernels.
This issue tracks the broader nullability contract for REE and Union arrays and
the consistent application of that contract across compute kernels.
Is your feature request related to a problem or challenge?
This issue follows up on #10909 and accompanies the fix for
nullifonRunEndEncoded and Union arrays.
Several compute kernels can introduce logical nulls, for example through null
take indices or a
nullifcondition.For most Arrow arrays, a top-level validity bitmap represents these nulls.
RunEndEncoded and Union arrays are different:
valueschild.a nullable child selected by its type id.
As a result, a kernel can currently produce physical nulls even when the
corresponding field metadata is marked as non-nullable.
Describe the solution you'd like
A kernel must not silently drop or fabricate requested nulls.
RunEndEncoded, if thevaluesfield is non-nullable and an operationneeds to introduce a null, return a compute error.
Union, a null may be represented using a nullable child field.If no child field is nullable and an operation needs to introduce a null,
return a compute error.
requiring a new nullable representation.
nulls, including
take,nullif, and similar compute kernels.Describe alternatives you've considered
Together with #10909 and the
nulliffix, this policy keeps the physicalresult consistent with its field metadata.
In particular, a field marked as non-nullable must not produce output
containing nulls. Returning an error when a requested null cannot be
represented is preferable to producing a result whose data contradicts its
declared nullability.
Additional context
PR #10909 fixed handling of null take indices for RunEndEncoded and Union
arrays. This issue tracks the broader nullability contract and its consistent
application across compute kernels.
This issue tracks the broader nullability contract for REE and Union arrays and
the consistent application of that contract across compute kernels.