Repository navigation
Add pandas data loader with schema inference - #2
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| datetime_coerced = pd.to_datetime(non_null, errors="coerce") | ||
| valid_datetime_ratio = float( | ||
| datetime_coerced.notna().sum() / len(non_null) | ||
| ) | ||
| if valid_datetime_ratio > 0.9: | ||
| return "datetime" |
There was a problem hiding this comment.
In _detect_logical_type the heuristic tries to coerce object columns to datetimes before attempting numeric detection. Because pandas.to_datetime accepts plain integers as nanosecond timestamps, an object column that is mostly numeric values with a few string placeholders (common in Excel sheets) will satisfy the valid_datetime_ratio > 0.9 check and the loader will tag it as datetime. _apply_schema will then cast those numeric values to datetime64[ns], corrupting columns such as ages or counts. You can reproduce with pd.Series([1,2,…,100,'?'], dtype=object) which yields datetime from this logic, while the correct classification is numeric. Swap the order so numeric coercion is tested before datetime to avoid misclassifying integer data as timestamps.
Useful? React with 👍 / 👎.
Summary
DataLoaderclass that wraps pandas CSV and Excel readers with schema inference and chunked iterationmacrobase_pypackage exports and conversions to typed DataFramesTesting
https://chatgpt.com/codex/tasks/task_e_68fade0e60108320a333a43a54539ec5