Clutter Detection and Removal in 3D Scenes with View-Consistent Inpainting

International Conference on Computer Vision (ICCV), October 2023

Fangyin Wei, Thomas Funkhouser, Szymon Rusinkiewicz

Clutter detection and removal in 3D Scenes with view-consistent inpainting. Beginning from an input scene (left), our first task is to predict clutter regions. Traditional semantic segmentation datasets and methods [9] (top center) were not designed for this problem, and contain missing or incorrect labels (especially for small objects) as well as semantic categories that only sometimes correspond to clutter. In contrast, our method (bottom center) correctly identifies clutter segments, which we then remove and inpaint with coherent texture and geometry (right).

Removing clutter from scenes is essential in many applications, ranging from privacy-concerned content filtering to data augmentation. In this work, we present an automatic system that removes clutter from 3D scenes and inpaints with coherent geometry and texture. We propose techniques for its two key components: 3D segmentation based on shared properties and 3D inpainting, both of which are important problems. The definition of 3D scene clutter (frequently-moving objects) is not well captured by commonly-studied object categories in computer vision. To tackle the lack of well-defined clutter annotations, we group noisy fine-grained labels, leverage virtual rendering, and impose an instance-level area-sensitive loss. Once clutter is removed, we inpaint geometry and texture in the resulting holes by merging inpainted RGB-D images. This requires novel voting and pruning strategies that guarantee multi-view consistency across individually inpainted images for mesh reconstruction. Experiments on ScanNet and Matterport3D dataset show that our method outperforms baselines for clutter segmentation and 3D inpainting, both visually and quantitatively.

Paper: PDF file
Supplementary material PDF file

Fangyin Wei, Thomas Funkhouser, and Szymon Rusinkiewicz.
"Clutter Detection and Removal in 3D Scenes with View-Consistent Inpainting."
International Conference on Computer Vision (ICCV), October 2023.

@inproceedings{Wei:2023:CDA,
   author = "Fangyin Wei and Thomas Funkhouser and Szymon Rusinkiewicz",
   title = "Clutter Detection and Removal in {3D} Scenes with View-Consistent
      Inpainting",
   booktitle = "International Conference on Computer Vision (ICCV)",
   year = "2023",
   month = oct
}