DiffGS: Functional Gaussian Splatting Diffusion
(NeurIPS 2024)

Junsheng Zhou* · Weiqi Zhang* · Yu-Shen Liu

(* Equal Contribution)

Paper | Project Page

We release the code of the paper DiffGS: Functional Gaussian Splatting Diffusion in this repository.

Abstract

In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.

Method

Overview of DiffGS. (a) We disentangle the fitted 3DGS into three Gaussian Splatting Functions to model the Gaussian probability, colors and transforms. We then train a Gaussian VAE with a conditional latent diffusion model for generating these functions. (b) During generation, we first extract Gaussian geometry from the generated GauPF, followed by the GauCF and GauTF to obtain the Gaussian attributes.

Generation Results

Unconditional Generations

Visual Comparisons

Visualization Results

Text/Image-to-3D Generation

Gaussian Completion

Point-to-Gaussian Generation

Install

We recommend creating an anaconda environment using our provided environment.yml:

conda env create -f environment.yml
conda activate diffgs

Notice：Since the code uses the original repository of Gaussian Splatting, please follow the environment setup instructions provided in the official repository to install the required dependencies.

Pretrained model

We first provide the pretrained models: Gaussian VAE and Gaussian LDM of the chair unconditional model. Please download the pretrained models from Google Drive.

Inference

To inference pretrained model of ShapeNet Chair, save the downloaded model checkpoint to config/stage1 and config/stage2. Additionally, you also need to adjust the checkpoint path in config/genetate/specs.json, then run the following command:

python test.py -e config/generate/

Data preparation

1. Please refer to the Stanford ShapeNet Renderer repository to render the ShapeNet dataset. Next, perform point sampling on the mesh and modify the shapene_folder path in sample_points.py. The sampled points will be used as the initial positions for the Gaussians.

cd proecess_data
python sample_points.py

2. Run the Gaussian fitting script provided by us.

python train_gaussian.py -s <path to COLMAP or NeRF Synthetic dataset>

3. Run the conversion script convert.py provided by us to transform the Gaussians into data suitable for training, and perform sampling of the Gaussian probability field.

python convert_data.py

Training

1. Train Gaussian modulations

python train.py -e config/stage1/ -b 32 -w 8    # -b for batch size, -w for workers, -r to resume training

2. Train the diffusion model using the modulations extracted from the first stage

# extract the modulations / latent vectors, which will be saved in a "modulations" folder in the config directory
# the folder needs to correspond to "Data_path" in the diffusion config files

python test.py -e config/stage1/ -r {num epoch}

python train.py -e config/stage2 -b 32 -w 8 

Citation

If you find our code or paper useful, please consider citing

@inproceedings{diffgs,
    title={DiffGS: Functional Gaussian Splatting Diffusion},
    author={Zhou, Junsheng and Zhang, Weiqi and Liu, Yu-Shen},
    booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
    year={2024}
}