EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models


Yue Hu (AgiBot, Harbin Institute of Technology), Siyuan Huang (Shanghai Jiao Tong University), Yue Liao (National University of Singapore), Shengcong Chen (Agibot), Pengfei Zhou (Agibot), Liliang Chen (AgiBot), Guanghui Ren (AgiBot), Maoqing Yao (Agibot)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Recent advances in creative AI have enabled the synthesis of high-fidelity images and videos conditioned on language instructions. Building on these developments, text-to-video diffusion models have evolved into embodied world models (EWMs) capable of generating physically plausible scenes from language commands, effectively bridging vision and action in embodied AI applications. This work addresses the critical challenge of evaluating EWMs beyond general perceptual metrics to ensure the generation of physically grounded and action-consistent behaviors. We propose the Embodied World Model Benchmark (\Ours), a dedicated framework designed to evaluate EWMs based on three key aspects: visual scene consistency, motion correctness, and semantic alignment. Our approach leverages a meticulously curated dataset encompassing diverse scenes and motion patterns, alongside a comprehensive multi-dimensional evaluation toolkit, to assess and compare candidate models. The proposed benchmark not only identifies the limitations of existing video generation models in meeting the unique requirements of embodied tasks but also provides valuable insights to guide future advancements in the field. The dataset, evaluation tools will be open-sourced at https://github.com/AgibotTech/EWMBench.

Citation

@inproceedings{Hu_2025_BMVC,
author    = {Yue Hu and Siyuan Huang and Yue Liao and Shengcong Chen and Pengfei Zhou and Liliang Chen and Guanghui Ren and Maoqing Yao},
title     = {EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models},
booktitle = {36th British Machine Vision Conference 2025, {BMVC} 2025, Sheffield, UK, November 24-27, 2025},
publisher = {BMVA},
year      = {2025},
url       = {https://bmva-archive.org.uk/bmvc/2025/assets/papers/Paper_736/paper.pdf}
}


Copyright © 2025 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection