fix(local_tokenizer): align TokensInfo roles with the texts actually tokenized

[Linxiushen]

What

LocalTokenizer.compute_tokens() builds roles with one entry per Part:

for content in processed_contents:
  text_accumulator.add_content(content)
  if content.parts:
    for _ in content.parts:
      roles.append(content.role)

but _TextsAccumulator does not emit one text per part. A function_call / function_response part contributes the function name plus every key and every string value of its args/response as separate texts; a thought_signature-only part contributes none. The two lists are then combined with zip() at both tokenizer branches (lines 386 and 405), and zip silently truncates to the shorter one.

Effect

Model turn with function_call(get_weather, {location: "NYC"}), then a user text:

texts actually sent to the tokenizer:  'get_weather', 'location', 'NYC', "What's the weather?"
TokensInfo entries returned:           2
  role='model'  tokens=['get_weather']
  role='user'   tokens=['location']        <- model's own argument, labelled user

Two of the four texts are dropped from the result, and the surviving entries carry the wrong role. With a thought_signature-only part first (routine on Gemini 2.5/3 function-calling and thinking turns), the user's own text is labelled role='model'.

TokensInfo.role is documented as "the role from the corresponding Content" (types.py), which this violates. The output also matches neither "one entry per part" nor "one entry per text", so it isn't a defensible alternate semantics.

Reproduced on the real gemma3 SentencePiece model (_local_tokenizer_loader's pinned download) and, for the HuggingFace/gemma4 branch, with the same mock pattern test_local_tokenizer.py already uses for that branch.

Fix

Extend roles by the number of texts the accumulator actually added for each Content:

roles: list[Optional[str]] = []
...
for content in processed_contents:
  texts_before = len(text_accumulator)
  text_accumulator.add_content(content)
  roles.extend([content.role] * (len(text_accumulator) - texts_before))

with a two-line _TextsAccumulator.__len__. Both zip() sites are now aligned with get_texts() by construction. Plain-text contents add exactly one text per part, so their behaviour is unchanged.

After the fix the example above returns 4 entries with roles model, model, model, user.

Verification

• pytest google/genai/tests/local_tokenizer/: 35 passed (33 existing + 2 new). The 2 new tests fail on main.
• mypy google/genai/local_tokenizer.py with the repo's strict config: 0 errors in this file, before and after. (My environment reports 7 pre-existing errors in _api_client.py / _local_tokenizer_loader.py / errors.py that are identical with and without this change — dependency-version noise, not from this diff.)

+12 −4 in local_tokenizer.py, plus one test file covering both branches.

CLA

I'll complete the Google CLA when the bot prompts.

---

Investigated and fixed with AI assistance (Claude); I reviewed the change and ran the verification above myself.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Venkaiahbabuneelam]

Hi @Linxiushen,
Please resolve the conflicts.
Thanks

[Linxiushen]

Done, @Venkaiahbabuneelam — the branch is now up to date with main (merged upstream/main in via a merge commit, no force-push, so the review history is intact). GitHub had it as BEHIND rather than a content conflict, and the merge was clean: main's recent changes (voices / gaos / types) don't touch local_tokenizer.py, so this PR's diff is still just the two files — google/genai/local_tokenizer.py and the new tests/local_tokenizer/test_roles_alignment.py.

pytest google/genai/tests/local_tokenizer/test_roles_alignment.py — 2 passed on the merged branch; the branch's mypy, check-changes and conventionalcommits checks are green.

The only remaining red check is cla/google. That one needs the account owner to sign the Google CLA (at cla.developers.google.com) — it's an individual authorization that has to come from them, not something I can complete on their behalf. I've flagged it to them. Happy to make any further changes to the code in the meantime.

[Linxiushen]

@Venkaiahbabuneelam Rebased onto current main at c4c6ba2 — the branch is mergeable again. The two commits are unchanged in content; the tokenizer tests pass before and after (35/35). The cla/google check is a separate step and will be handled from the account side.