Fix panic on modules declaring their encoding as e.g. latin-1 - #322
Open
pylaterreur wants to merge 2 commits into
Open
pylaterreur wants to merge 2 commits into
pylaterreur wants to merge 2 commits into
Conversation
Modules can declare their encoding (PEP 263) using any name that Python
accepts, such as `latin-1`, `utf_8` or `euc_jp`. But encoding_rs only knows
the WHATWG labels (`latin1`, `utf-8`, `euc-jp`...), so these names weren't
recognized, and building the graph panicked with:
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err`
value: PyErr { type: <class 'TypeError'>, value: TypeError('function
takes exactly 5 arguments (1 given)'), traceback: None }
Normalize the declared name like Python does before looking it up.
Modules that really can't be decoded also caused this panic, as the error
was created as a UnicodeDecodeError, which can't be created from just a
message, and was then unwrapped. Raise it as a UnicodeError instead, which
keeps the message naming the file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN
Merging this PR will not alter performance
Comparing Footnotes
|
The previous commit stored file read errors in GrimpError as a PyErr.
GrimpError's Debug and Display implementations then call into Python, so
the Rust test binaries started linking libpython, and `just test-rust`
failed wherever libpython isn't on the loader path:
error while loading shared libraries: libpython3.13.so.1.0: cannot
open shared object file: No such file or directory
Make FileSystem::read return Rust errors instead (GrimpError::FileNotFound
and GrimpError::UndecodableFile), and convert them to FileNotFoundError
and UnicodeError where they reach Python, like the other GrimpError
variants. The exception types and messages are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #321.
Since 3.10,
build_graphpanics withTypeError('function takes exactly 5 arguments (1 given)')on:latin-1,latin_1,utf_8,utf-8-sig,euc_jporiso8859_15;Changes:
RealBasicFileSystem::readnormalizes the declared name like Python does (_get_normal_nameinLib/tokenize.py, and_vs-) when encoding_rs doesn't recognize it as is. Names that encoding_rs already knows are looked up as before.UnicodeError, with the existing messages that name the file, e.g.Failed to decode file /path/to/pkg/broken.py as UTF-8: invalid utf-8 sequence of 1 bytes from index 5. Before, they were created asUnicodeDecodeError, which can't be created from a message alone.FileSystem::readreturns Rust errors (newGrimpError::FileNotFoundandGrimpError::UndecodableFilevariants), which are converted toFileNotFoundErrorandUnicodeErrorwhere they reach Python, like the otherGrimpErrorvariants.scan_for_importspropagates them instead of unwrapping.The new functional tests use
tmp_path, as each case needs its own package: one test per encoding name above, and one per kind of decoding failure (invalid UTF-8, invalid for the declared encoding, unknown encoding).This isn't on a hot path: names that encoding_rs knows take the same route as before, and only the error handling changed otherwise.
Encoding names that Python accepts but that still aren't WHATWG labels after normalizing, such as
cp932ormac_roman, now raise a clearUnicodeErrorrather than panicking. Supporting them would need a fallback to Python's codecs.latestsection at the top ofCHANGELOG.rst. (If it's not there, add it.)AUTHORS.rst.just full-check. (I ran the lint (Python, plus Rust with the pinned 1.97.0 toolchain), the docs build,cargo test, and the Python tests on 3.10 to 3.14. I didn't run them on 3.14t, 3.15 or 3.15t.)🤖 Generated with Claude Code
https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN