Repository navigation
Add a Julia Encoding type mirroring the Python one #16
Copy link
Copy link
Open
Description
encode and encode_batch (src/Tokenizer.jl) keep only the tokens and ids of each Python Encoding, so some tokenizer features give incomplete results:
- pair encoding (
encode(tok, a, b)) losestype_ids, the segment ids BERT-style models need; - with padding enabled (
tok.py_tokenizer.enable_padding()), the ids come back padded, without theattention_maskthat marks the padding; - truncation with a stride loses the extra windows in
overflowing.
Proposal: have them return a Julia Encoding with all the fields and methods of the Python Encoding. result.tokens, result.ids and (; tokens, ids) = result keep working; tokens, ids = result does not, so this needs a minor version bump. encode_batch_ids (#15) stays the fast path when only ids are needed.
| Python | Julia |
|---|---|
ids, type_ids |
Vector{Int} fields |
tokens |
Vector{String} field |
offsets |
Vector field, see below |
attention_mask, special_tokens_mask |
Vector{Int} fields |
word_ids, sequence_ids |
Vector{Union{Int, Nothing}} fields |
overflowing |
Vector{Encoding} field |
n_sequences |
n_sequences(e) |
len(e) |
length(e) |
words |
deprecated alias of word_ids, skip |
char_to_token(char_pos, sequence_index=0) |
char_to_token(e, char_pos, sequence_index) |
char_to_word(char_pos, sequence_index=0) |
char_to_word(e, char_pos, sequence_index) |
token_to_chars(token_index) |
token_to_chars(e, token_index) |
token_to_sequence(token_index) |
token_to_sequence(e, token_index) |
token_to_word(token_index) |
token_to_word(e, token_index) |
word_to_chars(word_index, sequence_index=0) |
word_to_chars(e, word_index, sequence_index) |
word_to_tokens(word_index, sequence_index=0) |
word_to_tokens(e, word_index, sequence_index) |
pad(length, direction="right", pad_id=0, pad_type_id=0, pad_token="[PAD]") |
pad!(e, length; kwargs...) |
truncate(max_length, stride=0, direction="right") |
truncate!(e, max_length; kwargs...) |
set_sequence_id(sequence_id) |
set_sequence_id!(e, sequence_id) |
Encoding.merge(encodings, growing_offsets=True) |
merge(encodings; growing_offsets=true) |
The methods that return None when there is no match (e.g. char_to_token on a space) would return nothing.
Design questions:
- Plain struct or wrapper. Converting every field up front lets the Python
Encodingbe freed right away (its cost is what Fast ids-only batch encoding (encode_batch is 4x slower and 10x more memory) #13 measured), butpad!,truncate!,mergeandTokenizer.post_process(which takes anEncoding) then need Julia implementations, since the PythonEncodinghas no public constructor from its fields. Wrapping the Python object and converting fields on access gets all methods for free, but keeps each PythonEncodingalive until Julia's GC runs. - Indexing. Python's token, word, character and sequence indices are 0-based, and spans are
(first, last + 1)tuples. Julia convention would be 1-based indices andUnitRanges. Token ids would stay 0-based, asencodereturns them today. - Offsets count Unicode characters (
"café au lait"gives(5, 7)for"au"), while Julia strings are indexed by bytes, sotext[r]needs byte ranges. Converting needs the input text, whichencodehas.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]`encode` and `encode_batch` drop `type_ids` and `attention_mask`[/-][+]Add a Julia `Encoding` type mirroring the Python one[/+]