Skip to content

fix(core): use real byte offsets when splitting header and key-value text - #50

Open
Benny93 wants to merge 1 commit into
raphaelmansuy:mainfrom
Benny93:fix/utf8-split-offsets
Open

Benny93 wants to merge 1 commit into
raphaelmansuy:mainfrom
Benny93:fix/utf8-split-offsets

Conversation

@Benny93

@Benny93 Benny93 commented Nov 3, 2026

Copy link
Copy Markdown

Problem

edgeparse.convert(path, format="markdown") (0.3.2, also main at 8d706e0) panics on a born-digital (text-layer) PDF we cannot share:

pyo3_runtime.PanicException: end byte index 118 is not a char boundary; it is inside '’' (bytes 116..119 of string)

In crates/edgeparse-core/src/pipeline/stages/cluster_table_detector.rs, split_compact_header_row and split_merged_key_value_text rebuild byte offsets as the sum of word.len() + 1 over split_whitespace() words:

cursor += words[split_word_idx - 1].len() + 1;
let (left, right) = trimmed.split_at(cursor.saturating_sub(1));

That assumes exactly one 1-byte space between words. A run of spaces, a tab or a U+00A0 makes cursor fall short of the real offset, so split_at can land inside a multi-byte char such as ’ and panic.

Change

  • New helper word_byte_offsets yields (byte_offset, word), with the offset taken from the split_whitespace slice itself, so it is always a char boundary.
  • Both functions split at the start of the word instead of at the reconstructed space. Both halves are trimmed afterwards, so the result for single-space text is unchanged.
  • Three unit tests: double space and U+00A0 before ’ for split_compact_header_row, double space for split_merged_key_value_text. On main all three panic with end byte index 10 is not a char boundary; it is inside '’' (bytes 8..11 of string); with this change they pass.

Testing

  • cargo test (workspace): all pass, edgeparse-core lib 554 passed, 0 failed.
  • cargo fmt --all --check and cargo clippy -- -A clippy::all -D clippy::correctness -D clippy::suspicious (the CI flags): clean.
  • Regression check on the 200 PDFs in benchmark/pdfs: converted to Markdown with release builds of main and of this branch. 198 outputs are byte-identical. The other two (01030000000071, 01030000000191) vary between runs on main as well; over 6 runs each, both builds produce the same two alternative outputs. So that is existing nondeterminism, not a change from this PR. I did not run ./scripts/bench.sh, since the outputs are identical.
  • The PDF that triggered the panic now converts without error.

Separately, a panic anywhere in the pipeline reaches Python as PanicException. Wrapping the edgeparse_core::convert(...) call in crates/edgeparse-python/src/lib.rs with std::panic::catch_unwind and mapping it to PyRuntimeError would let callers handle a bad document like any other error. I left that out to keep this PR small. I'm happy to add it here or in a follow-up.

🤖 Generated with Claude Code

…text

split_compact_header_row and split_merged_key_value_text rebuilt byte
offsets as the sum of `word.len() + 1` over `split_whitespace()` words.
That assumes exactly one 1-byte space between words. A run of spaces, a
tab or a U+00A0 makes the offset fall short, so `split_at` can land
inside a multi-byte char and panic ("end byte index ... is not a char
boundary"), which reaches Python as a PanicException.

Take each word's byte offset from the slice itself and split at the
start of the word. Both halves are trimmed afterwards, so output for
single-space text is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant