- Model and Dataset: Used the Qwen2-VL-2B-Instruct model (from GitHub and Hugging Face) and tested it on the MathVista dataset (Hugging Face).
- Accelerated Inference: Integrated the vllm toolkit to speed up inference processing.
- Performance Evaluation:
- Assessed model performance on multimodal reasoning tasks.
- Compared inference time with and without vllm.
- Analyzed model predictions and identified bad cases.
- VQGAN: Reviewed the VQGAN paper and code (GitHub), focusing on the image tokenization process.
- Model Execution: Ran VQGAN on the DIV2K dataset (DIV2K) using a subset of 200 images in Google Colab.
- Visualizations:
- Visualized quantized token ID frequency distributions.
- Performed dimensionality reduction (PCA and t-SNE) on token ID embeddings to study uniformity.
- Compared quantized token differences between two similar images.