MiniCLIP — CLIP-style Model on Flickr30k

A minimal CLIP (Contrastive Language–Image Pretraining) implementation trained on the Flickr30k dataset.

Architecture

Component	Model	Embedding Dim
Image Encoder	ConvNeXt-Small (timm)	768
Text Encoder	DistilBERT	768
Projection Head	Linear → 512-d	512

Contrastive loss with learnable temperature parameter.

Project Structure

├── config.py              # Hyperparameters and CLI argument parsing
├── dataset.py             # Flickr30k dataset and DataLoader utilities
├── models/
│   ├── image_encoder.py   # Image encoder (ConvNeXt via timm)
│   ├── text_encoder.py    # Text encoder (DistilBERT)
│   └── clip_model.py      # MiniCLIP (combines encoders + projection)
├── train.py               # Training script
├── evaluate.py            # Recall@K evaluation script
├── requirements.txt       # Dependencies
└── notebooks/             # Original Kaggle notebook

Setup

pip install -r requirements.txt

Data

Download the Flickr30k dataset and place it under data/:

data/
├── flickr30k_images/      # Image files
└── results.csv            # Captions

Training

python train.py --data_path data/flickr30k_images --caption_path data/results.csv

CLI options:

Flag	Default	Description
--data_path	data/flickr30k_images	Path to images
--caption_path	data/results.csv	Path to captions CSV
--batch_size	32	Batch size
--epochs	5	Number of epochs
--lr	1e-4	Learning rate
--save_path	model_weights.pth	Output weights path

Evaluation

python evaluate.py --weights model_weights.pth --data_path data/flickr30k_images --caption_path data/results.csv

Results using finetuned convnext_small and distilbert CLIP Style (Flickr30k validation set)

Metric	I2T	T2I	Mean
R@1	27.69%	25.24%	26.47%
R@5	53.99%	51.08%	52.53%
R@10	65.85%	63.13%	64.49%

Results using Locked image Tuning (LiT) with text tower finetuned (Flickr30k validation set)

Metric	I2T	T2I	Mean
R@1	13.70%	11.03%	12.36%
R@5	33.21%	28.46%	30.84%
R@10	44.35%	39.42%	41.89%

Acknowledgements

• Flickr30k Dataset
• CLIP paper (Radford et al., 2021)
• ALIGN paper (Jia et al., 2021)
• LiT paper (Zhai et al., 2022)
• timm / HuggingFace Transformers