Confirming Data Preprocessing for MAE Feature Extraction Inference vs. Pretraining

[ChingYuanYu]

I would like to verify whether the data preprocessing steps for MAE (Masked Autoencoder) feature extraction during inference are identical to those used during MAE pretraining. Could you clarify if the same preprocessing pipeline should be applied, or are there specific differences to consider for inference?

[mathpluscode]

I would like to verify whether the data preprocessing steps for MAE (Masked Autoencoder) feature extraction during inference are identical to those used during MAE pretraining. Could you clarify if the same preprocessing pipeline should be applied, or are there specific differences to consider for inference?

Hi, if I understand your question correctly, the preprocessing for MAE pretraining was mainly 1) resample to 1mm x 1mm x10 mm resolution, 2) cropping at the center of LV ventricle or LAX/SAX intersactions to reduce input sizes, example code is https://github.com/mathpluscode/CineMA/blob/main/cinema/examples/dicom_to_nifti.py, then the image went through some transformation/data augmentation steps, as listed in https://github.com/mathpluscode/CineMA/blob/main/cinema/mae/pretrain.py#L157. For feature extraction, similar transformations were performed as shown in https://github.com/mathpluscode/CineMA/blob/main/cinema/examples/inference/mae_feature_extraction.py#L21.

Thus, if you ask whether we should resample the image resolutions and crop them, I may suggest yes. But the only hard constraint is that the input has to be of size 192x192x16, if your image is small, it requires padding; if it's larger, then multiple patches need to be put for inference. Alternatively, you can resize the image directly. Here is an example script used for segmentation https://github.com/mathpluscode/CineMA/blob/main/cinema/segmentation/train.py#L183.

At last, for feature extraction, depending on the tasks you do, if you have fine-tuning steps on top, as long as all your images have gone through the same preprocessing, it would be fine. But indeed, the results should be more reliable if the steps are closer to the pre-training steps.

Hope this answers your question!

[ChingYuanYu]

Thank you for the clarification. It makes things clear now.