Calibration-Aware Prompt Learning for Medical Vision-Language Models


Abhishek Basu (Mohamed bin Zayed University of Artificial Intelligence), Fahad Shamshad (Mohamed bin Zayed University of Artificial Intelligence), Ashshak Sharifdeen (Mohamed bin Zayed University of Artificial Intelligence), Karthik Nandakumar (Michigan State University), Muhammad Haris Khan (Mohamed Bin Zayed University of Artificial Intelligence)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Medical Vision-Language Models (Med-VLMs) have demonstrated remarkable performance across diverse medical imaging tasks by leveraging large-scale image-text pretraining. However, their confidence calibration is largely unexplored, and so remains a significant challenge. As such, miscalibrated predictions can lead to overconfident errors, undermining clinical trust and decision-making reliability. To address this, we introduce \texttt{CalibPrompt}, the first framework to calibrate Med-VLMs during prompt tuning. \texttt{CalibPrompt} optimizes a small set of learnable prompts with carefully designed calibration objectives under scarce labeled data regime. First, we study a regularizer that attempts to align the smoothed accuracy with the predicted model confidences. Second, we introduce an angular separation loss to maximize textual feature proximity toward improving the reliability in confidence estimates of multimodal Med-VLMs. Extensive experiments on four publicly available Med-VLMs and five diverse medical imaging datasets reveal that \texttt{CalibPrompt} consistently improves calibration without drastically affecting clean accuracy. Our code is available at \url{https://github.com/iabh1shekbasu/CalibPrompt}.

Citation

@inproceedings{Basu_2025_BMVC,
author    = {Abhishek Basu and Fahad Shamshad and Ashshak Sharifdeen and Karthik Nandakumar and Muhammad Haris Khan},
title     = {Calibration-Aware Prompt Learning for Medical Vision-Language Models},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_1062/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection