Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation

  title={Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation},
  author={Shanyan Guan and Jingwei Xu and Michelle Z. He and Yunbo Wang and Bingbing Ni and Xiaokang Yang},
  journal={IEEE transactions on pattern analysis and machine intelligence},
We consider a new problem of adapting a human mesh reconstruction model to out-of-domain streaming videos, where the performance of existing SMPL-based models is significantly affected by the distribution shift represented by different camera parameters, bone lengths, backgrounds, and occlusions. We tackle this problem through online adaptation, gradually correcting the model bias during testing. There are two main challenges: First, the lack of 3D annotations increases the training difficulty… 
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation
A pseudoground-truth annotator based on CLIFF is proposed, which provides high-quality 3D annotations for in-the-wild 2D datasets and offers crucial full supervision for regression-based methods.


Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
A new training algorithm named Bilevel Online Adaptation (BOA) is proposed, which divides the optimization process of overall multi-objective into two steps of weight probe and weight update in a training iteration, and leads to state-of-the-art results on two human mesh reconstruction benchmarks1.
Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation
A skeleton-disentangled based framework is proposed, which divides this task into multi-level spatial and temporal granularity in a decoupling manner, and an effective and pluggable "disentangling the skeleton from the details" (DSD) module is proposed.
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
A Pyramidal Mesh Alignment Feedback (PyMAF) loop to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status in the authors' deep regressor is proposed.
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction
This paper addresses the problem of 3D human pose and shape estimation from a single image by proposing a graph-based mesh regression, which outperform the comparable baselines relying on model parameter regression, and achieves state-of-the-art results among model-based pose estimation approaches.
Towards Robust RGB-D Human Mesh Recovery
This work proposes a new method that uses RGB-D data to estimate a parametric human mesh model, and designs a new dynamic data fusion module that facilitates learning with a combination of RGB-only andRGB-D datasets, and a new constraint generator module that provides SMPL supervisory signals when explicit SMPL annotations are not available.
End-to-End Human Pose and Mesh Reconstruction with Transformers
A new method to reconstruct 3D human pose and mesh vertices from a single image using a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, which generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets.
DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare
A novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image and a large-scale synthetic dataset utilizing web-crawled Mocap sequences, 3D scans and animations is constructed.
Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
The first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single unconstrained image is described, showing superior pose accuracy with respect to the state of the art.
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation
This paper proposes a novel hybrid inverse kinematics solution (HybrIK) that surpasses the state-of-the-art methods by a large margin on various 3D human pose and shape benchmarks and preserves both the accuracy of 3D pose and the realistic body structure of the parametric human model.
Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation
This paper augments existing 2D datasets with high-quality 3D pose fits by augmenting them with Exemplar Fine-Tuning (EFT), and shows that EFT produces 3D annotations that result in better downstream performance and are qualitatively preferable in an extensive human-based assessment.