Dual Polarity Prompts with Stochastic Entropy Perturbation for Label Noise


Changhui Hu (Universitat de Barcelona), Bhalaji Nagarajan (Barcelona Supercomputing Center), Ricardo Marques (Universitat Pompeu Fabra), Petia Radeva Ivanova (Universitat de Barcelona)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Pre-trained Vision-Language Models (VLMs) have been widely adopted across various applications through few-shot prompt tuning. However, their potential to effectively address label noise, a common challenge in real-world data, remains largely unexplored. While recent prompt tuning methods exhibit robustness under low to moderate noise, they struggle in handling higher noise ratios or more complex noise patterns. To bridge this gap, we introduce “DP-SEP”, a novel framework designed to enhance the adaptation of VLMs under challenging label noise. DP-SEP consists of two key components: (1) Dual Polarity (DP) Prompts, which dynamically adjust text-image alignments by bringing the correct description closer, while pushing away incorrect ones. DP includes a novel class reordering function to generate negative prompts and a Polarity Alignment Reweighting (PAR) factor to effectively balance the dual prompts. (2) Stochastic Entropy Perturbation (SEP), which injects controlled randomness to reduce the VLM’s overconfidence in noisy labels. Extensive experiments on 7 synthetic datasets and two real-world noisy benchmarks demonstrate that DP-SEP significantly outperforms existing methods. In particular, we achieve an impressive average gain of 11.17% and 16.98% under 65% and 80% sym. noise. Significant improvements of DP-SEP establishes as a new direction for handling label noise in adopting VLMs.

Citation

@inproceedings{Hu_2025_BMVC,
author    = {Changhui Hu and Bhalaji Nagarajan and Ricardo Marques and Petia Radeva Ivanova},
title     = {Dual Polarity Prompts with Stochastic Entropy Perturbation for Label Noise},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_992/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection