• DOI:10.48550/arXiv.2203.15798
• Corpus ID: 247779353

DRaCoN - Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars

@article{Raj2022DRaCoND,
  title={DRaCoN - Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars},
  author={Amit Raj and Umar Iqbal and Koki Nagano and S. Khamis and Pavlo Molchanov and James Hays and Jan Kautz},
  journal={ArXiv},
  year={2022},
  volume={abs/2203.15798}
}

• Amit Raj, Umar Iqbal, +4 authors J. Kautz
• Published 29 March 2022
• Computer Science
• ArXiv

Acquisition and creation of digital human avatars is an important problem with applications to virtual telep-resence, gaming, and human modeling. Most contempo-rary approaches for avatar generation can be viewed ei-ther as 3D-based methods, which use multi-view data to learn a 3D representation with appearance (such as a mesh, implicit surface, or volume), or 2D-based methods which learn photo-realistic renderings of avatars but lack accu-rate 3D representations. In this work, we present, DRaCoN, a framework for learning full-body volumetric avatars which exploits the advantages of both the 2D and 3D neural rendering techniques. It consists of a Differentiable Rasterization module, DiffRas, that synthesizes a low-resolution version of the target image along with additional latent features guided by a parametric body model. The output of DiffRas is then used as conditioning to our conditional neural 3D representation module (c-NeRF) which generates the final high-res image along with body geometry using volumetric rendering. While DiffRas helps in obtaining photorealistic image quality, c-NeRF, which employs signed distance fields (SDF) for 3D representations, helps to obtain fine 3D geometric details. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods both in terms of error metrics and visual quality.