Abstract: We propose scaling up 3D scene reconstruction by training with synthesized data. At the core of our work is MegaSynth, a 3D dataset comprising 700K scenes (which takes only 3 days to generate) - 70 times larger than the prior real dataset DL3DV - dramatically scaling the training data. To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition. Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability. Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization. We explore training LRMs with both MegaSynth and available real data, enabling wide-coverage scene reconstruction within 0.3 second.
sudo apt install libxi6 libsm6 libxext6
pip install datasets opencv-python Pillow rich bpy==3.6.0 numpy scipy matplotlib
# install blender, then
export PATH=path/to/blender/:$PATH # needs blender binary in addition to bpy to run
. ./render.sh
# adjust your target number of scenes
# all see scene parameters in each file
- Modify the texture sampling logic. As we synthesize scenes with Adobe internal material maps, we cannot release with the internal materials. Now, we sample a single material for the whole scene as a quick demo, following Zeroverse. We will modify the sampling logic to enable different materials for each shape primmitive. See the sampling logic at L207 and L1252 of
create_shapes.py. - Release our internal rendering data.
If you find this code useful, please consider citing:
@article{jiang2024megasynth,
title={MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data},
author={Jiang, Hanwen and Xu, Zexiang and Xie, Desai and Chen, Ziwen and Jin, Haian and Luan, Fujun and Shu, Zhixin and Zhang, Kai and Bi, Sai and Sun, Xin and Gu, Jiuxiang and Huang, Qixing and Pavlakos, Georgios and Tan, Hao},
booktitle={arXiv preprint arXiv:2412.14166},
year={2024},
}
@article{xie2024lrm,
title={LRM-Zero: Training Large Reconstruction Models with Synthesized Data},
author={Xie, Desai and Bi, Sai and Shu, Zhixin and Zhang, Kai and Xu, Zexiang and Zhou, Yi and Pirk, S{\"o}ren and Kaufman, Arie and Sun, Xin and Tan, Hao},
journal={arXiv preprint arXiv:2406.09371},
year={2024}
}
@article{xu2018deep,
title={Deep image-based relighting from optimal sparse samples},
author={Xu, Zexiang and Sunkavalli, Kalyan and Hadap, Sunil and Ramamoorthi, Ravi},
journal={ACM Transactions on Graphics (TOG)},
volume={37},
number={4},
pages={126},
year={2018},
publisher={ACM}
}