[JadenMathias]

model: amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4

Serve:

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16 \
vllm serve /data/models/Muse-Spark-Open-816B-A42B-Quark-MXFP4 \
  --served-model-name musespark \
  --tokenizer-mode hf --config-format auto --load-format safetensors --generation-config vllm \
  --language-model-only --dtype bfloat16 \
  --tensor-parallel-size 4 --distributed-executor-backend mp \
  --moe-backend aiter_triton_mxfp4_mxfp4 \
  --attention-backend TRITON_ATTN \
  --max-model-len 32768 --max-num-seqs 64 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser musespark1x --tool-call-parser musespark1x --enable-auto-tool-choice \
  --port 8000

Accuracy with limit 100

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.96	±	0.0197
		strict-match	5	exact_match	↑	0.96	±	0.0197

1k 1k 64conc bench serve

============ Serving Benchmark Result ============
Successful requests: 640
Failed requests: 0
Maximum request concurrency: 64
Benchmark duration (s): 351.79
Total input tokens: 655360
Total generated tokens: 655360
Request throughput (req/s): 1.82
Output token throughput (tok/s): 1862.91
Peak output token throughput (tok/s): 2432.00
Peak concurrent requests: 89.00
Total token throughput (tok/s): 3725.81
---------------Time to First Token----------------
Mean TTFT (ms): 1975.01
Median TTFT (ms): 1245.37
P99 TTFT (ms): 19602.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.45
Median TPOT (ms): 31.44
P99 TPOT (ms): 49.12
---------------Inter-token Latency----------------
Mean ITL (ms): 32.45
Median ITL (ms): 28.77
P99 ITL (ms): 30.68