[Bug]: Marlin NVFP4 MoE applies the activation to the up projection (Qwen3.5-MoE NVFP4 returns garbage on Hopper) #20965

System Info

• 1x H200, nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc1.dev202610201007
• ornith-ai/Ornith-1.5-35B-A3B-NVFP4 (Qwen3.5-MoE, NVFP4 experts)
• marlin/base.py in this image is identical to main; line numbers below are from main fe180a952d.

Who can help?

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

cat > cfg.yaml <<'EOF'
max_batch_size: 256
kv_cache_config:
  host_cache_size: 0  # works around a separate host-memory startup failure on the test host
EOF
trtllm-serve ornith-ai/Ornith-1.5-35B-A3B-NVFP4 \
  --reasoning_parser qwen3_5 --tool_parser qwen3 --config cfg.yaml

Then send one /v1/chat/completions request with the user message "What is the capital of France? Answer in one word.", temperature: 0 and max_tokens: 1024. With moe_config unset, Qwen3.5-MoE NVFP4 checkpoints default to the Marlin MoE backend on SM89-SM99 (modeling_qwen3_5.py:64-83, added in #17293), so this runs the Marlin path.

Expected behavior

content is Paris. With moe_config: {backend: CUTLASS} added to cfg.yaml, the same request does return Paris.

actual behavior

content is empty and reasoning_content is random tokens:

``` """ARSEizzo警觉 ,
"\""", etcr",uuk"l"tv"ea",cedr"tlinesFG立马sWithizzo.com vener虎ARSEa. "","s"b """, ...

On the first 20 GSM8K test questions (zero-shot chat, temperature 0, scored by a small client script), the default Marlin backend gets 0/20 and moe_config: {backend: CUTLASS} gets 19/20.

additional notes

• The loader packs FC1 as [w3; w1], up first (fused_moe/quantization.py:3516); w1 is gate_proj. The Marlin method NVFP4MarlinFusedMoEMethod inherits this loader (quantization.py:3593).
• CUTLASS reads the first half as the linear value and the second half as the gate (moe_kernels.cu:2261-2263).
• Marlin takes the first half as the gate (fused_moe/marlin/base.py:240-241), so the experts compute silu(up) * gate instead of silu(gate) * up.
• The Marlin test reference has the same swap (tests/unittest/_torch/moe/quantize_utils.py:972-973), so the Marlin unit tests agree with the kernel. The Marlin accuracy tests in CI use NemotronH, whose relu2 experts do not split FC1.
• nvidia/Qwen3.6-35B-A3B-NVFP4 has the same architecture (Qwen3_5MoeForConditionalGeneration) and gets the same default, so it takes this path too. Only the checkpoint above was run.
• Workaround: moe_config: {backend: CUTLASS}.

A fix PR will follow.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.