PrincetonComputer SciencePIXL GroupPublications → [Wei et al. 2022] Local Access
Self-supervised Neural Articulated Shape and Appearance Models

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022

Fangyin Wei, Rohan Chabra, Lingni Ma,
Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz,
Chris Sweeney, Richard Newcombe, Mira Slavcheva
Our self-supervised method learns the shape and appearance of articulated object classes. After training from multi-view synthetic images of different states of object instances, our model can reconstruct and animate objects from static real-world images.
Abstract

Learning geometry, motion, and appearance priors of object classes is important for the solution of a large variety of computer vision problems. While the majority of approaches has focused on static objects, dynamic objects, especially with controllable articulation, are less explored. We propose a novel approach for learning a representation of the geometry, appearance, and motion of a class of articulated objects given only a set of color images as input. In a self-supervised manner, our novel representation learns shape, appearance, and articulation codes that enable independent control of these semantic dimensions. Our model is trained end-to-end without requiring any articulation annotations. Experiments show that our approach performs well for different joint types, such as revolute and prismatic joints, as well as different combinations of these joints. Compared to state of the art that uses direct 3D supervision and does not output appearance, we recover more faithful geometry and appearance from 2D observations only. In addition, our representation enables a large variety of applications, such as few-shot reconstruction, the generation of novel articulations, and novel view-synthesis.
Paper
Links
Citation

Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz, Chris Sweeney, Richard Newcombe, and Mira Slavcheva.
"Self-supervised Neural Articulated Shape and Appearance Models."
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.

BibTeX

@inproceedings{Wei:2022:SNA,
   author = "Fangyin Wei and Rohan Chabra and Lingni Ma and Christoph Lassner and
      Michael Zollh{\"o}fer and Szymon Rusinkiewicz and Chris Sweeney
      and Richard Newcombe and Mira Slavcheva",
   title = "Self-supervised Neural Articulated Shape and Appearance Models",
   booktitle = "IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",
   year = "2022",
   month = jun
}