SALT: Parameter-Efficient Fine-Tuning via Singular Value Adaptation with Low-Rank Transformation


Abdelrahman Elsayed (Mohamed bin Zayed University of Artificial Intelligence), Sarim Hashmi (Mohamed bin Zayed University of Artificial Intelligence), Mohammed Elseiagy (Mohamed bin Zayed University of Artificial Intelligence), Hu Wang (Mohamed bin Zayed University of Artificial Intelligence), Mohammad Yaqub (Mohamed bin Zayed University of Artificial Intelligence), Ibrahim Almakky (Mohamed bin Zayed University of Artificial Intelligence)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)

Abstract

The complex nature of medical image segmentation calls for models that are specifically designed to capture detailed, domain-specific features. Large foundation models offer considerable flexibility, yet the cost of fine-tuning these models remains a significant barrier. Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model weights with low-rank matrices but may suffer from underfitting when the chosen rank is insufficient to capture domain-specific nuances. Conversely, full-rank Singular Value Decomposition (SVD) based methods provide comprehensive updates by modifying all singular values, yet they often lack flexibility and exhibit variable performance across datasets. We propose SALT (Singular Value Adaptation with Low-Rank Transformation), a method that selectively adapts the most influential singular values using trainable scale and shift parameters while complementing this with a low-rank update for the remaining subspace. This hybrid approach harnesses the advantages of both LoRA and SVD, enabling effective adaptation without relying on increasing the size or depth of the model. Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2\% to 5\% in Dice with only 3.9\% trainable parameters, demonstrating robust adaptation even in low-resource settings. The code for this paper is available at : https://github.com/BioMedIA-MBZUAI/SALT

Citation

@inproceedings{Elsayed_2025_BMVC,
author    = {Abdelrahman Elsayed and Sarim Hashmi and Mohammed Elseiagy and Hu Wang and Mohammad Yaqub and Ibrahim Almakky},
title     = {SALT: Parameter-Efficient Fine-Tuning via Singular Value Adaptation with Low-Rank Transformation},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_1013/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection