Skip to content

Fix FileReader compression override forwarding - #1560

Open
ktz03 wants to merge 1 commit into
meta-pytorch:mainfrom
ktz03:fix/file-reader-compression-options
Open

ktz03 wants to merge 1 commit into
meta-pytorch:mainfrom
ktz03:fix/file-reader-compression-options

Conversation

@ktz03

@ktz03 ktz03 commented Oct 19, 2026 •

Copy link
Copy Markdown

Fixes #1559

Changes

  • Forward the documented transport_params["compression"] option to smart_open.open as its top-level compression keyword. This fixes forced decompression of extensionless gzip/bzip2 files and preserves raw bytes when decompression is disabled.
  • Copy the transport mapping before extracting the option, preserving caller options and other transport parameters. Add real-file regressions for text/binary reads, disabled decompression, default extension inference and checkpoint restoration; correct the existing mock assertion.

Validation

  • The complete FileReader module passes all 29 tests, including offline Moto S3 emulation. On the unchanged production code, the six affected real-file cases fail while the two inference controls pass.
  • All seven pinned full-repository pre-commit hooks and the documentation completeness check pass.
  • Mypy 1.8.0 reports no type errors in 35 source files. The existing quoted Python-version setting emits a configuration warning; an additional run explicitly targeting Python 3.13 also succeeds.
  • The complete test/nodes/ suite passes: 405 passed, 15 existing CUDA skips. Tested locally on Windows/Python 3.12.14 with Torch 2.14.0+cpu and smart_open 8.0.4; no live cloud-service test was run.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 19, 2026
@ktz03
ktz03 marked this pull request as ready for review October 19, 2026 15:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FileReader ignores documented compression overrides

1 participant