Intra-Modal Divergence-Weighted Distillation for Vision-Language Models


Addad Youva (Université de Caen Basse Normandie), Alexis Lechervy (Université de Caen Basse Normandie), Frédéric Jurie (Université de Caen Normandie)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Large vision-language models like CLIP offer strong zero-shot capabilities but are computationally demanding. Knowledge distillation is crucial for creating efficient student models; however, effectively transferring the teacher's nuanced understanding of within-modality relationships, especially among negative examples, remains challenging. We introduce a novel distillation method focused on capturing the teacher's intra-modal relational knowledge. Our approach employs Kullback-Leibler divergence to measure the disagreement between student and teacher pairwise similarity distributions within each modality. This disagreement score then dynamically weights the distillation loss, compelling the student to prioritize learning from samples exhibiting the most significant relational discrepancies. This strategy encourages closer alignment of the student's internal representation space with the teacher's. Experiments demonstrate our method produces performant and efficient student models by effectively transferring this vital relational information. The source code will be made publicly available.

Citation

@inproceedings{Youva_2025_BMVC,
author    = {Addad Youva and Alexis Lechervy and Frédéric Jurie},
title     = {Intra-Modal Divergence-Weighted Distillation for Vision-Language Models},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_458/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection