Skip to content

fix(local_tokenizer): align TokensInfo roles with the texts actually tokenized - #2987

Open
Linxiushen wants to merge 1 commit into
googleapis:mainfrom
Linxiushen:fix-local-tokenizer-compute-tokens-roles
Open

fix(local_tokenizer): align TokensInfo roles with the texts actually tokenized#2987
Linxiushen wants to merge 1 commit into
googleapis:mainfrom
Linxiushen:fix-local-tokenizer-compute-tokens-roles

Commits

Commits on Sep 24, 2026