Interactive Scene Reconstruction

Project Page | Paper | Arxiv

This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments. The proposed pipeline reconstructs an interactive indoor scene from RGBD streams, where objects are replaced by (articulated) CAD models. Represented as a contact graph, the reconstructed scene naturally encodes actionable information in terms of environmental kinematics, and can be imported into various simulators to support robot interactions.

The pipeline consists of 3 modules:

• A robust panoptic mapping module that accurately reconstruct the semantics and geometry of objects and layouts, which is a modified version of Voxblox++ but with improved robustness. The 2D image segmentation is obtained using Detectron2
• An object-based reasoning module that constructs a contact graph from the dense panoptic map and replaces objects with aligned CAD models
• An interface that converts a contact graph into a kinematic tree in the URDF format, which can be imported into ROS-based simulators

Todo

• Upload code for panoptic mapping
• Upload code for CAD replacement
• Upload code for URDF conversion and scene visualization
• Upload dataset and use cases
• Update instructions

Installation and Usages

• Installation on Ubuntu
• Robust Panoptic Mapping
• Contact Graph and CAD Replacement
• Interactive Scene Generation and Gazebo Example

Citing

• Muzhi Han*, Zeyu Zhang*, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, and Hangxin Liu. Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments, IEEE International Conference on Robotics and Automation (ICRA), 2021

@inproceedings{han2021reconstructing,
  title={Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments},
  author={Han, Muzhi and Zhang, Zeyu and Jiao, Ziyuan and Xie, Xu and Zhu, Yixin and Zhu, Song-Chun and Liu, Hangxin},
  booktitle={ICRA},
  year={2021}
}