We introduce a novel RGB-D patch descriptor designed for detecting coplanar surfaces in SLAM reconstruction. The core of our method is a deep convolutional neural network that takes in RGB, depth, and normal information of a planar patch in an image and outputs a descriptor that can be used to find coplanar patches from other images. We train the network on 10 million triplets of coplanar and non-coplanar patches, and evaluate on a new coplanarity benchmark created from commodity RGB-D scans. Experiments show that our learned descriptor outperforms alternatives extended for this new task by a significant margin. In addition, we demonstrate the benefits of coplanarity matching in a robust RGBD reconstruction formulation. We find that coplanarity constraints detected with our method are sufficient to get reconstruction results comparable to state-of-the-art frameworks on most scenes, but outperform other methods on established benchmarks when combined with traditional keypoint matching.

@inproceedings{Shi:2018:PPC,
   author = "Yifei Shi and Kai Xu and Matthias Nie{\ss}ner and Szymon Rusinkiewicz
      and Thomas Funkhouser",
   title = "{PlaneMatch}: Patch Coplanarity Prediction for Robust {RGB-D}
      Reconstruction",
   booktitle = "European Conference on Computer Vision (ECCV) oral presentation",
   year = "2018",
   month = sep
}