[Bug]: Streamed final text keeps the start of a multi-token stop word; Thai digit and tone-mark stops end in U+FFFD #21318

System Info

• TensorRT-LLM 1.4.0rc2 (nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc2, arm64, git b96ba434e). The code involved is unchanged on current main (7d8d94684): tensorrt_llm/executor/result.py and tensorrt_llm/tokenizer/tokenizer.py in the image are byte-identical to main.
• DGX Spark (GB10), PyTorch backend. The CPU repro below needs no GPU.
• Tokenizer: Qwen/Qwen3-0.6B. The ids of the 13 Thai characters below (๐-๙, ๊ ํ ๎) are the same with the typhoon-ai/typhoon2.5-qwen3-4b and typhoon-ai/typhoon-s-thaillm-8b-instruct-research-preview tokenizers (checked for all 13).

Who can help?

chenfeiz0326 (author of #20424, which added the final-response flush to the HF streaming detokenizer).

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

When the sampler matches a stop word by token id, _handle_sequence trims the stop ids from token_ids (_trim_stop_word_outputs, result.py around line 498). A non-streamed request then decodes the trimmed ids and the text is clean.

A streamed request (stream_interval=1) behaves differently. The stop ids that arrived in earlier responses were already passed to decode_incrementally by DetokenizedGenerationResultBase._handle_response (result.py around line 1026). After the trim, token_ids_diff is empty, so nothing removes them from text. The text-level stop check can't help either, because the text never contains the whole stop string.

Thai makes this deterministic. The Qwen3 tokenizer splits each Thai digit ๐-๙ and the marks ๊ ํ ๎ into token 8158 (the bytes E0 B9) plus one continuation-byte token. For example, "๒" is [8158, 240]. With stop="๒":

• 8158 arrives first, and the decoder holds it because it isn't a complete character.
• 240 arrives with FinishReason.STOP_WORDS. Both ids are trimmed, but the final-response flush added in [https://nvbugs/6713424][fix] Flush the HF streaming detokenizer on the final response; fix the AgentX aiperf install check #20424 still emits the held 8158 as '�'.
• The streamed text ends in U+FFFD. So does the last text_diff, which is what the OpenAI-compatible stream handlers send as the last delta.

Multi-token stop words that decode to text ("คำตอบ:", "</answer>") leave their leading tokens in the streamed .text.

CPU-only repro. It drives the real DetokenizedGenerationResultBase and TransformersTokenizer with the Qwen3-0.6B tokenizer, one response per token, and the last response finishes with STOP_WORDS, as the PyTorch executor sends it. It runs inside the release container without --gpus (symlink the CUDA stub as libcuda.so.1 so tensorrt_llm imports):

from types import SimpleNamespace
from tensorrt_llm.bindings import executor as tllm
from tensorrt_llm.executor.result import DetokenizedGenerationResultBase
from tensorrt_llm.sampling_params import SamplingParams
from tensorrt_llm.tokenizer import tokenizer_factory

tok = tokenizer_factory("Qwen/Qwen3-0.6B")

def response(ids, final):
    finish = tllm.FinishReason.STOP_WORDS if final else tllm.FinishReason.NOT_FINISHED
    result = SimpleNamespace(
        is_final=final, context_phase_params=None, decoding_iter=1,
        avg_decoded_tokens_per_iter=None, finish_reasons=[finish],
        output_token_ids=[list(ids)], sequence_index=0, cum_log_probs=None,
        log_probs=None, generation_logits=None, per_pos_drafted=None,
        per_pos_accepted=None, spec_dec_totals=None, time_breakdown_metrics=None,
        request_perf_metrics=None, additional_context_outputs=None,
        additional_generation_outputs=None, context_logits=None)
    return SimpleNamespace(has_error=lambda: False, result=result)

def run(text, stop, streaming):
    sp = SamplingParams(stop=stop)
    sp._setup(tok, None, None)
    sp._stream_interval = 1
    ids = tok.encode(text, add_special_tokens=False)  # text ends with the stop word
    res = DetokenizedGenerationResultBase(0, sp, tokenizer=tok, streaming=streaming)
    chunks = [[t] for t in ids] if streaming else [ids]
    for i, chunk in enumerate(chunks):
        res._handle_response(response(chunk, i == len(chunks) - 1))
    return res.outputs[0].text

for text, stop in [("ข้อ ๑ ถูก ข้อ ๒", "๒"), ("ราคา ๓", "๓"), ("วางบนโต๊", "๊"),
                   ("ราคา ๑๐ บาท คำตอบ:", "คำตอบ:"), ("ตอบว่า ก๊าซ", " ก๊าซ"),
                   ("<answer>42</answer>", "</answer>")]:
    print(repr(stop), "non-stream:", repr(run(text, stop, False)), "stream:", repr(run(text, stop, True)))

Expected behavior

The streamed final .text matches the non-streamed .text. finish_reason and stop_reason already match.

actual behavior

Output of the script above on 1.4.0rc2, one row per printed line. Both TLLM_INCREMENTAL_DETOKENIZATION_BACKEND=HF (the default) and =TRTLLM print the same six lines. The stop ids column is tok.encode(stop):

End to end on DGX Spark (GB10) with LLM.generate / LLM.generate_async(streaming=True), Qwen3-0.6B, greedy decoding, the release image as shipped. The engine matches the stop ids itself:

prompt='ตัวเลขไทย: ๐ ๑' stop='๒' stop_ids=[8158, 240]
   unstopped gen ids=[220, 8158, 240, 220, 8158, 241, ...] text=' ๒ ๓ ๔ ๕ ๖ ๗ ...'
   non-stream .text=' ' (stop,'๒') ids=[220]
   stream     .text=' �' (stop,'๒') ids=[220]
prompt='ตัวเลขไทย: ๐ ๑ ๒' stop='๓' stop_ids=[8158, 241]
   non-stream .text=' ' (stop,'๓') ids=[220]
   stream     .text=' �' (stop,'๓') ids=[220]

additional notes

• Where it shows. The LLM API (generate_async(..., streaming=True)) ends with this text; this was observed above, on CPU and on GB10. For the Thai cases, the last text_diff carries the '�', and that diff is what the OpenAI-compatible stream handlers send. From code reading, not run: the gRPC OpenEngine path (grpc/openengine/streaming.py:104-130) holds back stop-prefix text while generating, then sends everything up to len(output.text) once finish_reason is set, so it should send the leaked text as well.
• Before [https://nvbugs/6713424][fix] Flush the HF streaming detokenizer on the final response; fix the AgentX aiperf install check #20424. The HF backend didn't emit the '�': the held byte was dropped, which happened to be right here (checked by overlaying the tokenizer.py from 2f5bb049f^ on the same container). The leftover stop prefix (คำตอบ, </answer) predates [https://nvbugs/6713424][fix] Flush the HF streaming detokenizer on the final response; fix the AgentX aiperf install check #20424. With the same overlay, the TRTLLM backend (TLLM_INCREMENTAL_DETOKENIZATION_BACKEND=TRTLLM) still prints the '�' and the stop prefix, as it does on 1.4.0rc2: [https://nvbugs/6713424][fix] Flush the HF streaming detokenizer on the final response; fix the AgentX aiperf install check #20424 only changed the HF DecodeStream path.
• Out of scope. The deltas of the OpenAI-compatible stream handlers already sent the stop prefix (คำตอบ, </answer) before the stop matched, because those handlers have no holdback. The gRPC path does have one (_StopPrefixTracker). That is a separate change and is not part of this issue.
• Proposed fix (PR to follow). In DetokenizedGenerationResultBase._handle_response, when the token-level trim removed ids that earlier responses already streamed (len(token_ids) < _last_token_ids_len), decode the kept ids afresh, as the non-streaming path does, and drop the incremental state, which also drops the pending flush.

This issue was drafted with AI assistance (Claude Code). The repro and every output above were run and checked locally.