how to benchmark vllm(prefill) + decoder(rilert)? #53

Activity

1. WSEmma commented on Jul 29, 2026

   

   WSEmma

   on Jul 29, 2026

   Contributor

   More actions

   A couple of things need to debug this:

   • More detailed vLLM logs from the prefill worker (behind :40047). The benchmark output truncates the error and the router's 502 masks the real 400 from the worker, so I need the worker's own log with the full, untruncated 400 message.

   • The full launch commands for both the prefill (vLLM) and decode (TileRT) servers, with all flags — especially --max-model-len / --max-seq-len / tilert_max_seq_len and --kv-transfer-config.
2. yiminghub2024 commented on Jul 31, 2026

   

   yiminghub2024

   on Jul 31, 2026

   Author

   More actions

   vllm-prefill
   --served-model-name=glm5.1
   --tensor-parallel-size=8
   --enforce-eager
   --trust-remote-code
   --return-tokens-as-token-ids
   --gpu-memory-utilization=0.5
   --kv-cache-dtype=fp8_ds_mla
   --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
   --kv-transfer-config '{ "kv_connector": "TileRTConnector", "kv_connector_module_path": "tilert.pd_vllm.prefill_connector", "kv_role": "kv_producer", "kv_connector_extra_config":{ "tilert_host":"[10.91.0.187]", "tilert_ctrl_port":5556, "tilert_model":"glm5", "tilert_max_seq_len":202752 } }'

   tilert-decoder
   --engine=tilert
   --model=glm5
   --with-mtp
   --max-seq-len=202752
   --kv-cache-dtype=fp8
   --ctrl-port=5556
   --http-port=5557

   tilert router:
   --vllm-url http://10.91.0.187:40046
   --decode 10.91.0.187:5556:5557

   log:

   glm5-ewem3_2026-07-31_23-24-30.txt
   glm5.1-prefill-upvqa_2026-07-31_23-24-28.txt
   glm5.1-cv5ee_2026-07-31_23-24-32.txt

   i know the reason. because rilert router now only support stream:false , error:"{"error":"streaming is supported on /v1/chat/completions"}"

   but vllm bench serve only support stream=true benchmark ,so i change the test tool to 'evalscope[perf]' to test ,it support stream:false test

   64K/1K*10

   evalscope perf
   --url http://10.91.0.187:40040/v1/completions
   --api openai
   --model glm5.1
   --dataset random
   --tokenizer-path /var/lib/gpustack/cache/model_scope/ZhipuAI/GLM-5.1-FP8
   --min-prompt-length 64000 --max-prompt-length 64000
   --max-tokens 3000
   --number 10 --parallel 1
   --no-stream

   

   same with https://vllm.ai/blog/2026-07-14-vllm-tilert-pd

   

   but why not and when tilert router can support stream request ? @WSEmma
3. WSEmma commented on Aug 6, 2026

   

   WSEmma

   on Aug 6, 2026

   Contributor

   More actions

   Hi,
   we support streaming requests now — it's available in the latest release (v0.1.5.post2).
   Please give it a try. Thanks for your feedback!
4. yiminghub2024 commented on Aug 6, 2026

   

   yiminghub2024

   on Aug 6, 2026 via email

   Author

   More actions

   OK i will try asap thanks 发自我的iPhone

   …

   ------------------ Original ------------------ From: Shuang Wu ***@***.***> Date: Thu,Aug 6,2026 4:16 PM To: tile-ai/TileRT ***@***.***> Cc: yiminghub2024 ***@***.***>, Author ***@***.***> Subject: Re: [tile-ai/TileRT] how to benchmark vllm(prefill) + decoder(rilert)? (Issue #53) WSEmma left a comment (tile-ai/TileRT#53) Hi, we support streaming requests now — it's available in the latest release (v0.1.5.post2). Please give it a try. Thanks for your feedback! — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>
5. yiminghub2024 commented on Aug 6, 2026

   

   yiminghub2024

   on Aug 6, 2026

   Author

   More actions

   vllm bench serve \ --backend openai-chat \ --base-url http://10.91.0.187:23333 \ --model /var/lib/gpustack/cache/model_scope/ZhipuAI/GLM-5.1-FP8 \ --served-model-name glm5.1 \ --dataset-name random \ --random-input-len 64000 \ --random-output-len 3000 \ --random-range-ratio 0.0 \ --num-prompts 10 \ --request-rate inf \ --max-concurrency 1 yes, its works fine ,i will start a pr to note how to flow benchmark tilert @WSEmma
6. yiminghub2024 commented on Aug 6, 2026

   

   yiminghub2024

   on Aug 6, 2026

   Author

   More actions

   #58 @WSEmma this is pr for benchmark for tilert ,please take a look thanks
7. yiminghub2024

   closed this as completedon Oct 10, 2026