We develop a neural technique for articulating 3D characters using enveloping with a pre-defined skeletal structure, which is essential for animating a character with motion capture (mocap) data. Furthermore, we propose neural blend shapes — a set of corrective pose-dependent shapes that are used to address notorious artifacts caused by standard rigging and skinning technique in the joint regions.
Starting with a character model in T-pose, and the joint rotations on the desired skeleton hierarchy, our envelope branch predicts the corresponding skinning and rigging parameters and deforms the input character using a differential enveloping. In parallel, a residual deformation branch uses the input mesh to predict blend shapes and uses the joint rotations to predict the corresponding blending coefficients. This design also enables the network to be trained by only observing deformed shapes using indirect supervision, with no assumption on the underlying deformation model.
Our network is built upon the MeshCNN operators of Hanocka et al. [2019] and the Skeleton-Aware operators of Aberman et al. [2020], which enables generation of high-quality deformations for arbitrary mesh connectivities.
Our method can predict accurate skinning weights and skeleton rigs for unseen characters:
Specifically, our model outputs a skeleton complying with the target skeletal hierarchy, which is important for exploiting existing mocap data. RigNet [2020] only provides limited user control and is not able to reach the desired skeletal hierarchy.
Since our model is trained using high-quality deformations, the envelope branch outperforms the baseline LBS technique. Further boosted by the neural blend shapes, our method can generate favorable deformations in the joint region:
And even capture subtle soft tissue movements:
@article{li2021learning,
author = {Li, Peizhuo and Aberman, Kfir and Hanocka, Rana and Liu, Libin and Sorkine-Hornung, Olga and Chen, Baoquan},
title = {Learning Skeletal Articulations with Neural Blend Shapes},
journal = {ACM Transactions on Graphics (TOG)},
volume = {40},
number = {4},
pages = {1},
year = {2021},
publisher = {ACM}
}