Skip to content

Add a Julia Encoding type mirroring the Python one #16

Open

Description

@CarloLucibello

encode and encode_batch (src/Tokenizer.jl) keep only the tokens and ids of each Python Encoding, so some tokenizer features give incomplete results:

  • pair encoding (encode(tok, a, b)) loses type_ids, the segment ids BERT-style models need;
  • with padding enabled (tok.py_tokenizer.enable_padding()), the ids come back padded, without the attention_mask that marks the padding;
  • truncation with a stride loses the extra windows in overflowing.

Proposal: have them return a Julia Encoding with all the fields and methods of the Python Encoding. result.tokens, result.ids and (; tokens, ids) = result keep working; tokens, ids = result does not, so this needs a minor version bump. encode_batch_ids (#15) stays the fast path when only ids are needed.

Python Julia
ids, type_ids Vector{Int} fields
tokens Vector{String} field
offsets Vector field, see below
attention_mask, special_tokens_mask Vector{Int} fields
word_ids, sequence_ids Vector{Union{Int, Nothing}} fields
overflowing Vector{Encoding} field
n_sequences n_sequences(e)
len(e) length(e)
words deprecated alias of word_ids, skip
char_to_token(char_pos, sequence_index=0) char_to_token(e, char_pos, sequence_index)
char_to_word(char_pos, sequence_index=0) char_to_word(e, char_pos, sequence_index)
token_to_chars(token_index) token_to_chars(e, token_index)
token_to_sequence(token_index) token_to_sequence(e, token_index)
token_to_word(token_index) token_to_word(e, token_index)
word_to_chars(word_index, sequence_index=0) word_to_chars(e, word_index, sequence_index)
word_to_tokens(word_index, sequence_index=0) word_to_tokens(e, word_index, sequence_index)
pad(length, direction="right", pad_id=0, pad_type_id=0, pad_token="[PAD]") pad!(e, length; kwargs...)
truncate(max_length, stride=0, direction="right") truncate!(e, max_length; kwargs...)
set_sequence_id(sequence_id) set_sequence_id!(e, sequence_id)
Encoding.merge(encodings, growing_offsets=True) merge(encodings; growing_offsets=true)

The methods that return None when there is no match (e.g. char_to_token on a space) would return nothing.

Design questions:

  1. Plain struct or wrapper. Converting every field up front lets the Python Encoding be freed right away (its cost is what Fast ids-only batch encoding (encode_batch is 4x slower and 10x more memory) #13 measured), but pad!, truncate!, merge and Tokenizer.post_process (which takes an Encoding) then need Julia implementations, since the Python Encoding has no public constructor from its fields. Wrapping the Python object and converting fields on access gets all methods for free, but keeps each Python Encoding alive until Julia's GC runs.
  2. Indexing. Python's token, word, character and sequence indices are 0-based, and spans are (first, last + 1) tuples. Julia convention would be 1-based indices and UnitRanges. Token ids would stay 0-based, as encode returns them today.
  3. Offsets count Unicode characters ("café au lait" gives (5, 7) for "au"), while Julia strings are indexed by bytes, so text[r] needs byte ranges. Converting needs the input text, which encode has.

Activity

  1. changed the title [-]`encode` and `encode_batch` drop `type_ids` and `attention_mask`[/-] [+]Add a Julia `Encoding` type mirroring the Python one[/+] on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions