Expressive Body Capture:
3D Hands, Face, and Body from a Single Image
G. Pavlakos*, V. Choutas*, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas and M. J. Black (*authors contributed equally)
Computer Vision and Pattern Recognition (CVPR) 2019, Long Beach, CA
Title
Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
Abstract
To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.
Video:
News
17 June 2021
Unity SMPL-X integration is now available in the Downloads section.
5 May 2021
Blender SMPL-X add-on is now available in the Downloads section.
11 January 2021
We provide a SMPL-X model with the FLAME 2020 expression blendshapes.
3 November 2020
We now support conversion between all the models in the SMPL family, i.e. SMPL, SMPL+H, SMPL-X.
23 September 2020
The UV map of SMPL-X is now available - see the Downloads section.
20 August 2020
The full shape and expression space of SMPL-X are now available.
Referencing SMPL-X and SMPLify-X
@inproceedings{SMPL-X:2019,
title = {Expressive Body Capture: {3D} Hands, Face, and Body from a Single Image},
author = {Pavlakos, Georgios and Choutas, Vasileios and Ghorbani, Nima and Bolkart, Timo and Osman, Ahmed A. A. and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
pages = {10975--10985},
year = {2019}
}
Contact
For questions, please contact smplx@tue.mpg.de.
For commercial licensing, please contact sales@meshcapade.com.