Audio-Visual Separation with Hierarchical Fusion and Representation Alignment


Han Hu (University of Birmingham), Dongheng Lin (University of Birmingham), Qiming Huang (University of Birmingham), Yuqi Hou (University of Birmingham), Hyung Jin Chang (University of Birmingham), Jianbo Jiao (University of Birmingham)
The 35th British Machine Vision Conference
PDFPosterSupplementary

Abstract

Self-supervised audio-visual source separation leverages natural correlations between audio and vision modalities to separate mixed audio signals. In this work, we first systematically analyse the performance of existing multimodal fusion methods for audio-visual separation task, demonstrating that the performance of different fusion strategies is closely linked to the characteristics of the sound—middle fusion is better suited for handling short, transient sounds, while late fusion is more effective for capturing sustained and harmonically rich sounds. We thus propose a hierarchical fusion strategy that effectively integrates both fusion stages. In addition, training can be made easier by incorporating high-quality external audio representations, rather than relying solely on the audio branch to learn them independently. To explore this, we propose a representation alignment approach that aligns the latent features of the audio encoder with embeddings extracted from pre-trained audio models. Extensive experiments on MUSIC, MUSIC-21 and VGGSound datasets demonstrate that our approach achieves state-of-the-art results, surpassing existing methods under the self-supervised setting. We further analyse the impact of representation alignment on audio features, showing that it reduces modality gap between the audio and visual modalities. The project page is at: https://happy-new-bears.github.io/hfra-audiosep/.

Citation

@inproceedings{Hu_2025_BMVC,
author    = {Han Hu and Dongheng Lin and Qiming Huang and Yuqi Hou and Hyung Jin Chang and Jianbo Jiao},
title     = {Audio-Visual Separation with Hierarchical Fusion and Representation Alignment},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_493/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection