Robust Human Registration with Body Part Segmentation on Noisy Point Clouds


Kai Lascheit (ETH Zurich), Francis Engelmann (Stanford University), Daniel Barath (ETH Zurich), Marc Pollefeys (Microsoft), Leonidas Guibas (Stanford University)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Registering human meshes to 3D point clouds is essential for applications such as augmented reality and human-robot interaction but often yields imprecise results due to noise and background clutter in real-world data. We introduce a hybrid approach, called SegFit, that incorporates body-part segmentation into the mesh fitting process, enhancing both human pose estimation and segmentation accuracy. SegFit first assigns body part labels to individual points, which then guide a two-step SMPL-X mesh fitting: initial pose and orientation estimation using body part centroids, followed by global refinement of the point cloud alignment. Additionally, we demonstrate that the fitted human mesh can refine body part labels, leading to improved segmentation. Evaluations on the cluttered and noisy real-world datasets InterCap, EgoBody, and BEHAVE show that our approach significantly outperforms prior methods in both pose estimation and segmentation accuracy. The code and results are available on our project website: https://segfit.github.io.

Citation

@inproceedings{Lascheit_2025_BMVC,
author    = {Kai Lascheit and Francis Engelmann and Daniel Barath and Marc Pollefeys and Leonidas Guibas},
title     = {Robust Human Registration with Body Part Segmentation on Noisy Point Clouds},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_1110/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection