Catching the Unknown with Limited Data: Bi-Directional Prompt Tuning in CLIP for Few-Shot Open-Set Adaptation


Moloud Abdar (The University of Queensland), Md Mehedi Hasan (Deakin University), Biplab Banerjee (Indian Institute of Technology Bombay), Abbas Khosravi (Deakin University), Pietro Lio (University of Cambridge)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Domain-adaptive few-shot open-set (DAFSOS) learning aims to transfer knowledge from a label-rich source domain to a target domain where only a few labeled samples are available. Both domains contain disjoint sets of classes, and at inference time, models must accurately classify unlabeled samples from novel target-domain classes using limited support examples, while also recognizing outliers. Existing approaches often rely on meta-learning with sophisticated loss functions to obtain domain-invariant, discriminative features, yet overlook semantic information. We propose a novel prompt-learning framework for CLIP, named BIMAP-CLIP (Bi-directional Multi-modal Attribute-enriched Prompting for CLIP), tailored to the DAFSOS setting. Its core innovation is a bi-directional, multi-modal prompt-learning mechanism that shares learnable tokens across both image and text encoders, enhancing visual–semantic alignment. We also introduce a hybrid prompt initialization strategy that integrates manual and learnable prompts to make the prompts resilient to domains and incorporates fine-grained class attributes to tackle the scarcity of training samples. In parallel, a thresholding-based method on prompt-image similarity scores is employed for outlier detection. Across multiple benchmarks, BIMAP-CLIP consistently outperforms existing counterparts.

Citation

@inproceedings{Abdar_2025_BMVC,
author    = {Moloud Abdar and Md Mehedi Hasan and Biplab Banerjee and Abbas Khosravi and Pietro Lio},
title     = {Catching the Unknown with Limited Data: Bi-Directional Prompt Tuning in CLIP for Few-Shot Open-Set Adaptation},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_517/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection