[dependabot]

Bumps sentencepiece from 0.2.2 to 0.2.3.

Release notes

Sourced from sentencepiece's releases.

v0.2.3

Release Notes v0.2.3

This release introduces SentencePiece Lite, a zero-dependency C++20 runtime, new vocabulary-selection algorithms for training (Auto-character Coverage and Sparse L1 Unigram pruning), a Bazel (Bzlmod) build, a large number of security and robustness fixes found by OSS-Fuzz and community reports, and a cleanup of long-deprecated APIs and flags.

1. SentencePiece Lite (New Zero-Dependency Runtime)

• New lite/ runtime: A compact (~45 KB stripped, < 2k LOC) C++20 tokenizer engine that runs directly on memory-mapped FlatBuffers model binaries (.spm.fb) with no runtime dependency on Protobuf or Abseil (#1330, Commit a2cb4d8). See lite/README.md.

2. Training: New Vocabulary Selection Algorithms

• Auto-character Coverage (--auto_character_coverage): New TrainerSpec.auto_character_coverage option for both BPE and Unigram that globally optimizes the allocation between single characters and multi-character subwords within the vocabulary budget, while keeping lossless round-trip via byte fallback (Commits e23ece8, df18339, d34a6ce). See doc/auto_character_coverage.md.
• Sparse L1 Unigram pruning (--use_sparse_pruning): Optional L1-regularized EM vocabulary selection with Post-Lasso debiased refit for the Unigram model, replacing the discrete heuristic pruning (#1295, Commit fd53b5c). Required when --auto_character_coverage is used with Unigram. See doc/sparse_l1_unigram.md.
• PruneUnreachableSentencePieces: Remove pieces that can never appear in a Viterbi path during Unigram training, and fix the discrete pruning loop order (Commits b292ce5, 59c3b80).
• Custom pretokenizer callback: Trainer now accepts a std::function<std::vector<std::string>(absl::string_view)> pretokenizer, exposed to Python via the pretokenizer= training argument; pretokenization_delimiter and callback are mutually exclusive (#1287, Commit d1b6696).
• unk_id = -1 with byte_fallback: Training with --unk_id=-1 is now allowed when --byte_fallback=true; null-byte pieces are treated as UNUSED when byte fallback is enabled (fixes #1308, Commit c2fbf86).
• spm_eval metrics: Added Bytes/Tok, Model PPL, Empirical PPL, and Coverage metrics (Commit 01305c9).

3. Python Wrapper

• Pickling: SentencePieceProcessor now preserves encode options and the full instance dict (including subclasses) when pickled (#1349, Commits 4e66c53, 52b1a47).
• Normalizer fixes:
  • Preserve add_dummy_prefix / escape_whitespaces / remove_extra_whitespaces flags loaded from a model or spec instead of silently forcing them to False (#1337, fixes #1334).
  • Bind SentencePieceNormalizer.LoadFromSerializedNormalizerSpec, fixing AttributeError with normalizer_spec= (#1333).
• Decode / Encode edge cases:
  • Accept the legacy out_type alias in DecodeIds / DecodePieces (#1336).
  • Fix Decode() of empty input for non-str return types (#1289).
  • Fix encode() with an empty batch and out_type='offset_mapping' (#1320).
  • Fix decoding a batch whose first sequence is empty (#1321).
• Buffer protocol & integer inputs: Accept 8/16/32/64-bit signed/unsigned integer buffers with saturation clamping and zero-copy lifetime safety; accept general sequences (including tuples) in decode APIs (Commit b3717da).
• Free-threaded Python: Fix a pretokenizer callback race on free-threaded builds (cp314t) and GIL-safety issues in _TrainFromMap (Commits ca0f388, ee1d3dd). Free-threaded wheels are now tested on 3.14t / 3.15t.
• Python version support: Requires Python >= 3.10; added Python 3.15 classifier and cibuildwheel 4.3.1 (Commits fa40c16, ab49d75).
• AIX: Fixed the Python build on AIX (#1359).

4. Performance Optimizations

• BPE training: Replaced the per-symbol position btree_set with a vector and parallelized initial indexing; removed the 65,535-character sentence length limit by replacing 16-bit packed positions with a Position struct (Commits a44ce92, b3717da).
• Sentence preprocessing: Parallelized TrainerInterface::LoadSentences and SplitSentencesByWhitespace, with RunBatch-based dynamic load balancing for normalization (Commits a44ce92, d1b6696).
• Decode: Avoid intermediate allocations and skip redundant StrReplaceAll in Decode() (Commit a44ce92).
• Unigram: Avoid unnecessary allocations in Lattice::NBest / Lattice::Sample, heap-free score precomputation in BoundedPriorityQueue::Gc, and deduplicated corpus byte counting (Commit ee1d3dd).
• ParallelEncode / Normalizer: Skip unused SentencePieceText allocations in ParallelEncode overloads and skip alignment bookkeeping when not requested (Commit ee1d3dd).

5. Security & Hardening

• OSS-Fuzz findings: Bounded hypothesis allocations, agenda shrink rounds, and periodic timeout checks in unigram::Lattice::NBest to fix timeouts/OOM; added piece-ID bounds checks in IdToPiece, GetScore, IsControl, IsUnknown, IsUnused, IsByte (fixes #1342, #1344, #1347, Commit 2e432ca).
• Lattice bounds: Reject invalid positions/lengths in Lattice::Insert, verify EOS/begin nodes before Viterbi backtrace, and guard against null prev pointers and cycles (Commit fa40c16).
• Empty-piece guards: Guard empty pieces in PrefixMatcher::PrefixMatch and Unigram PieceToId (#1326, #1325).
• Charsmap validation: Validate leaf node offsets and safely slice values in DecompileCharsMap; fix Builder::BuildNFC_CFMap and regenerate nfc_cf.tsv (#1329, #1327).
• Trainer guards: RET_CHECK on zero character/word counts, clamp thread pool size to 1024, fix swap-and-pop bug when removing empty normalized sentences, and fix NaN in Gumbel sampling / thread-local RNG re-seeding (Commits b3717da, 70ed5fc, a44ce92).
• Lite hardening: Alignment and bounds checks on FlatBuffers models, direct-mapping validation, and endian-swap overflow protection (Commit b3717da).
• General input validation: Additional edge-case hardening across C++, Lite, and Python (Commit 5d21631).

6. Normalization

• nmt_nfkc: Normalize U+061C (ARABIC LETTER MARK) to space (#1331).

... (truncated)

Commits

• eafd99c python: cap build requirement to pybind11<3.1
• e25c2b7 fix(unigram): restore SplitSentencesByWhitespace() before EM
• 8c77bef fix: enable OpenMP_C and set CMAKE_C_FLAGS for libsais.c in CMake build
• 4826c62 Merge pull request #1370 from google/dependabot/pip/dot-github/workflows/requ...
• 7f0a495 Merge pull request #1371 from google/dependabot/github_actions/github-actions...
• 48bec13 build(deps): bump the github-actions group with 3 updates
• ccfa6de build(deps): bump the build-time-deps group
• e4dda6e python: require pybind11>=2.13 and document load() thread-safety
• d7510ae fix: do not re-read the input list size in Python batch encode
• 7bde6cb fix: read batch size once in Python batch decode and reset cached ids on reload
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[amrit110]

Automated fix attempt failed

Timed out after 330 minutes: state=OPEN; pending: codecov/changes; 10 passed

Manual intervention may be required.

View PR history on dashboard

AI Engineering Maintenance Bot