RadGazeIntent

Interpreting why a radiologist looks at each location in a chest X-ray

🏆 Outstanding Paper in the Content Theme at ACM Multimedia 2025

Paper · arXiv · Dataset · Quick start · Citation

RadGazeIntent is a transformer framework for fixation-level intention prediction. Given a chest X-ray and a radiologist's gaze sequence, it predicts which of 13 radiological findings the radiologist is investigating at every fixation. Unlike gaze heatmap or scanpath-prediction systems, RadGazeIntent models the diagnostic purpose behind observed eye movements.

The project also introduces the GazeIntent benchmark, which derives three complementary intention paradigms from EGD and REFLACX:

• RadSeq: systematic, one-finding-at-a-time search.
• RadExplore: uncertainty-driven exploration of multiple possible findings.
• RadHybrid: an initial overview followed by focused sequential search.

Quick start

1. Create the reference environment

The reference environment is Linux, Python 3.8, PyTorch 1.10, CUDA 11.3, and Detectron2 0.6. The paper used one NVIDIA A6000 GPU with 48 GB memory.

git clone https://github.com/UARK-AICV/RadGazeIntent.git
cd RadGazeIntent
conda env create -f environment.yml
conda activate radgazeintent

The code includes a portable PyTorch implementation of deformable attention. For faster CUDA execution, the custom extension can be built optionally:

bash radgazeintent/pixel_decoder/ops/make.sh

2. Authenticate and download the benchmark

GazeIntent is gated because its chest X-rays derive from credentialed medical datasets. Request access on the dataset page, confirm that you satisfy the underlying data-use requirements, then authenticate:

huggingface-cli login
python scripts/download_data.py \
  --output-dir data/gazeintent \
  --include-images \
  --acknowledge-data-license

To download only the six small annotation files, omit --include-images and --acknowledge-data-license.

3. Download feature-extractor weights

The initialization weights are hosted with the gated GazeSearch dataset. After access is approved, run:

python scripts/download_assets.py

The downloader verifies both files against published SHA-256 checksums.

4. Validate everything before using a GPU

python scripts/validate_setup.py \
  --hparams configs/train_egd_radseq.json \
  --dataset-root data/gazeintent

The validator checks annotations, images, model configuration, backbone weights, and pixel-decoder weights and reports every missing path together.

5. Train

python train.py \
  --hparams configs/train_egd_radseq.json \
  --dataset-root data/gazeintent \
  --device cuda \
  --gpu-id 0

Checkpoints, TensorBoard events, and the resolved configuration are written to outputs/train/egd_radseq/. If the configured batch does not fit in GPU memory, use the optional low-memory path:

python train.py \
  --hparams configs/train_egd_radseq.json \
  --dataset-root data/gazeintent \
  --device cuda \
  --batch-size 4 \
  --gradient-accumulation-steps 8 \
  --amp

This updates the optimizer after eight micro-batches for an effective batch of 32. Mixed precision can introduce small numerical differences. The public JSON provides one train/test split; the five fold definitions used for the paper's confidence intervals are not included.

6. Infer

python infer_all.py \
  --hparams configs/train_egd_radseq.json \
  --dataset-root data/gazeintent \
  --checkpoint outputs/train/egd_radseq/ckp_04000.pt \
  --output-dir outputs/inference/egd_radseq

Inference produces a compact image → fixation intentions JSON file and a second file shaped like the source annotations.

Benchmark configurations

Source	RadSeq	RadExplore	RadHybrid
EGD	train_egd_radseq.json	train_egd_radexplore.json	train_egd_radhybrid.json
REFLACX	train_reflacx_radseq.json	train_reflacx_radexplore.json	train_reflacx_radhybrid.json

Hosted setting 1 maps to RadSeq, setting 2 to RadExplore, and setting 3 to RadHybrid. Each experiment file inherits the shared defaults in configs/base.json.

Expected data layout:

data/gazeintent/
├── images/
│   └── <study-id>.jpg
├── egd_nolabel_coco_224_setting_1.json
├── egd_nolabel_coco_224_setting_2.json
├── egd_nolabel_coco_224_setting_3.json
├── reflacx_nolabel_coco_224_setting_1.json
├── reflacx_nolabel_coco_224_setting_2.json
└── reflacx_nolabel_coco_224_setting_3.json

Repository structure

radgazeintent/   transformer, metrics, paths, and validation
common/          dataset, loss, configuration, and fixation utilities
configs/         paper defaults and six experiment overrides
scripts/         dataset/weight downloads and setup validation
train.py         model training and checkpoint creation
infer_all.py     fixation-level JSON prediction export

Responsible use and limitations

• This is research software, not a medical device. It must not be used for diagnosis, treatment, triage, or other clinical decisions.
• The benchmark inherits population, acquisition, annotation, and access limitations from EGD, REFLACX, and MIMIC-CXR.
• Never publish credentialed images, patient information, or access tokens in issues, logs, or derived artifacts.

Citation

@inproceedings{pham2025radgazeintent,
  author    = {Trong-Thang Pham and Anh Nguyen and Zhigang Deng and
               Carol C. Wu and Hien Nguyen and Ngan Le},
  title     = {Interpreting Radiologist's Intention from Eye Movements in
               Chest X-ray Diagnosis},
  booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia},
  pages     = {257--266},
  year      = {2025},
  publisher = {ACM},
  doi       = {10.1145/3746027.3755039}
}

License

This repository is distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The underlying medical datasets remain subject to their own credentialed-access agreements and usage terms.

Acknowledgments

This material is based upon work supported by NSF OIA-1946391, NSF 2223793 EFRI BRAID, and NIH 1R01CA277739-01. The benchmark builds on EGD and REFLACX, and the implementation builds on Detectron2, Mask2Former, and Deformable DETR.

Questions and collaboration inquiries: Trong-Thang Pham (tp030@uark.edu). If the project supports your work, please cite the paper and consider starring the repository.