Yicheng Ji*, Jun Zhang*, Jinpeng Chen, Cong Wang, Lidan Shou, Gang Chen, Huan Li (* equal contribution)
- 2025.08: Our paper "SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning" has been accepted to EMNLP 2025 Main.
- 2026.02: Our paper "ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding" has been accepted to CVPR 2026.
- 2026.04: Our paper "See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs" has been accepted to ACL 2026 Main with oral nomination.
Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency due to autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec is high-fidelity and rapid: it preserves
Install the required dependencies:
conda create -n lvspec python==3.10 -y
conda activate lvspec
pip install torch torchvision
pip install -r requirements.txt- For LLaVA-OneVision models
- For Qwen2.5-VL models
- For VideoDetailCaption dataset
- For lmm-eval evaluation
Run the demo script to quickly evaluate LVSpec:
# Qwen2.5-VL
sh run_qwenvl.sh
# LLaVA-OneVision
sh run_llava.shWe support autoregressive, naive speculative decoding, SpecVLM, and LVSpec inference.
Please also modify the model path, data path, pruning ratio, and frame number in the .sh file.
After running the script, the output and efficiency metric will be stored in the save_path.
We use lmm-eval to evaluate the performance metric on VideoDetailCaption, VideoDetailDescription, MovieChat, and Video-MME. You may take the following steps:
# Clone lmm-eval and install the dependancies.
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git-
Put the file
evaluation/llava_onevision_lvspec.pyandevaluation/qwen2_5_vl_lvspec.pyunderlmms-eval/lmms_eval/models/and modifylmms-eval/lmms_eval/models/__init__.pyto register. Also put the two.shfile underlmms-eval/examples/models. -
To make sure the LVSpec repository is accessible in lmm-eval, you can clone the LVSpec repo to
lmms-eval/lmms_eval. -
Modify the corresponding configs in
/lmms-eval/lmms_eval/tasks. Then launch the LLM judge as inLVSpec/evaluation/sglang.sh.
# In lmm-eval
# llava-onevision
bash examples/models/llava_onevision_specvlm.sh
# qwen2.5-vl
bash examples/models/qwen25vl_specvlm.shAfter running, both the performance metric and efficiency metric will be recorded.
We recommand using quick inference for quick testing, and using benchmark evaluation to reproduce the results in the paper. Because running within lmm-eval enables support for vLLM, standard benchmark configurations, and qwen-vl-utils.
- Explore vision-aware drafting structures.
- Adapt to complex visual reasoning scenarios.
- Explore design in the training phase.
- Our method primarily targets resource-constrained long-video scenarios, where GPU memory bandwidth constitutes the main bottleneck during inference. Hence, the speedup ratio is sensitive to hardware settings. Users are advised to set the input length according to GPU capacity. Theoretically, as frame number grows, SpecVLM and LVSpec achieve higher acceleration ratios.
- Our LVSpec is built upon the repository of SpecVLM. To focus on studying the effect of vision-aware verification, we apply simple uniform visual token pruning.
If you find our work useful or relevant to your research, please kindly cite our papers:
@inproceedings{ji2025specvlm,
title={Specvlm: Enhancing speculative decoding of video llms via verifier-guided token pruning},
author={Ji, Yicheng and Zhang, Jun and Xia, Heming and Chen, Jinpeng and Shou, Lidan and Chen, Gang and Li, Huan},
booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
pages={7216--7230},
year={2025}
}
@misc{ji2026foresttreeslooselyspeculative,
title={See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs},
author={Yicheng Ji and Jun Zhang and Jinpeng Chen and Cong Wang and Lidan Shou and Gang Chen and Huan Li},
year={2026},
eprint={2604.05650},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2604.05650}
}
@article{kong2026parallelvlm,
title={ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding},
author={Kong, Quan and Shen, Yuhao and Ji, Yicheng and Li, Huan and Wang, Cong},
journal={arXiv preprint arXiv:2603.19610},
year={2026},
url={https://arxiv.org/abs/2603.19610}
}
@misc{zhang2026efficientinferencelargevisionlanguage,
title={Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects},
author={Jun Zhang and Yicheng Ji and Feiyang Ren and Yihang Li and Bowen Zeng and Zonghao Chen and Ke Chen and Lidan Shou and Gang Chen and Huan Li},
year={2026},
eprint={2604.05546},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2604.05546},
}