Question about Qwen-Image-Bench evaluation results

[hongsexiaotanhua]

Hi LLaDA-Image team,

Thanks for releasing such a great work! It provides us with some new ideas and insights for training image generation models.

I recently evaluated LLaDA-Image and Z-Image-Turbo on Qwen-Image-Bench (zh) and noticed some differences from the results reported in the paper:

Model	My Evaluation	Reported in the Paper
Z-Image-Turbo	49.52	52.71
LLaDA-Image	51.25	53.38

For comparison, I also evaluated three other open-source models using the same pipeline, and their scores were all within around 0.5 points of the official Qwen-Image-Bench results.

Could you please clarify the evaluation setup used for the reported results, particularly:

• the evaluation code used for Qwen-Image-Bench;

• whether there were any modifications to the prompts or preprocessing;

• the inference configuration (e.g., resolution, sampling steps, seed, etc.).

I would like to make sure my evaluation setup is aligned with yours and better understand the difference in scores. If possible, sharing the relevant evaluation configuration or command would be very helpful.

Thanks!

[StormyXJ]

Hi,

Thanks for your interest in our work!

• We did not rewrite or modify the raw prompts.
• Our inference configuration was: (1024 $\times$ 1024) resolution, 50 sampling steps, seed 42, and CFG 5.0.
• For the evaluation code, we followed the official Qwen-Image-Bench implementation, with max_new_tokens set to 16384. We increased this from the default value to avoid truncation for responses that may exceed 4096 tokens.

[hongsexiaotanhua]

Hi,

Thanks for your interest in our work!

• We did not rewrite or modify the raw prompts.
• Our inference configuration was: (1024
  ×
  1024) resolution, 50 sampling steps, seed 42, and CFG 5.0.
• For the evaluation code, we followed the official Qwen-Image-Bench implementation, with max_new_tokens set to 16384. We increased this from the default value to avoid truncation for responses that may exceed 4096 tokens.

Thanks for the clarification!

I updated my evaluation accordingly, using 1024×1024 resolution, 50 sampling steps, seed 42, CFG 5.0, and also set max_new_tokens=16384 in the Qwen-Image-Bench evaluation code. I used the official judge.py script for evaluation without any other modifications.

However, after re-evaluating, my score is still nearly 3 points lower than the result reported in the paper. Since my evaluations of several other open-source models are generally consistent with the official Qwen-Image-Bench results (within ~0.5 points), I am still a bit confused about where this gap comes from.

Could you please provide any further details about your inference and evaluation setup that might explain the difference? If possible, it would be especially helpful if you could share the exact evaluation script/configuration used to reproduce the reported results.

Thanks again for your help!