DEAD: Data-Efficient Audiovisual Dubbing using Neural Rendering Priors


Jack Saunders (University of Bath), Vinay Namboodiri (University of Bath)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Visual dubbing is the process of generating lip motions of an actor in a video to synchronize with given audio. Visual dubbing allows video-based media to reach global audiences. Recent advances have made progress towards realizing this goal. However, existing person-specific models see only one frame of the actor and, therefore, lack the ability to capture identity in the form of characteristic motion and related idiosyncracies, or they are expensive methods requiring off-putting large data requirements and costly model training. Our key insight is to train a large, multi-person prior network, which can then be adapted to new users. This method allows for $\textbf{high-quality visual dubbing with just a few seconds of data}$, that enables video dubbing for any actor - from A-list celebrities to background actors. We show that we achieve state-of-the-art in terms of $\textbf{visual quality}$ and $\textbf{recognizability}$ both quantitatively and qualitatively through two user studies. Our prior learning and adaptation method $\textbf{is able to adapt to small datasets better than baselines}$. Our experiments on real-world, limited data scenarios find that our model is preferred over baseline models.

Citation

@inproceedings{Saunders_2025_BMVC,
author    = {Jack Saunders and Vinay Namboodiri},
title     = {DEAD: Data-Efficient Audiovisual Dubbing using Neural Rendering Priors},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_811/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection