Skip to content

fix(plugins): handle unsigned integer columns in pandas/polars converters - #22

Open
belowzeroff wants to merge 1 commit into
RayforceDB:masterfrom
belowzeroff:fix/dataframe-unsigned-columns
Open

fix(plugins): handle unsigned integer columns in pandas/polars converters#22
belowzeroff wants to merge 1 commit into
RayforceDB:masterfrom
belowzeroff:fix/dataframe-unsigned-columns

Conversation

@belowzeroff

Copy link
Copy Markdown
Contributor

Problem

Unsigned integer columns (uint8/uint16/uint32/uint64) were absent from the pandas and polars dtype maps, so they fell through to Symbol and crashed the converter on otherwise-valid input:

>>> from_pandas(pd.DataFrame({"a": np.array([1, 2, 3], dtype="uint64")}))
TypeError: bad argument type for built-in operation

The same happened for polars UInt8/UInt16/UInt32/UInt64. The pyarrow plugin already handles unsigned columns (#18); pandas and polars were still broken.

Fix

Map unsigned columns to a signed type wide enough to hold every value without wrapping, mirroring the pyarrow plugin:

source dtype ray type
uint8 U8
uint16 I32
uint32 I64
uint64 I64

uint64 has no lossless target, so values >= 2**63 overflow — identical to pyarrow's behaviour. The numpy "u" dtype kind is also added to the pandas kind fallback.

Tests

Added test_from_pandas_unsigned_ints and test_from_polars_unsigned_ints, each covering all four widths with values above each signed type's midpoint (e.g. uint16 65535, uint32 4294967295) to prove they no longer wrap negative.

pytest tests/plugins/test_pandas.py tests/plugins/test_polars.py  ->  36 passed

🤖 Generated with Claude Code

…ters

Unsigned integer columns (uint8/uint16/uint32/uint64) were absent from the
pandas and polars dtype maps, so they fell through to Symbol and crashed the
converter with "TypeError: bad argument type for built-in operation".

Map them to a signed type wide enough to hold every value without wrapping,
mirroring the pyarrow plugin: uint8 -> U8, uint16 -> I32, uint32 -> I64,
uint64 -> I64 (no lossless target; values >= 2**63 overflow, same as pyarrow).
Also add the numpy "u" dtype kind to the pandas kind fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant