Neural Geometry Fields for Meshes

Training

Requirements: NVIDIA GPU (with CUDA support), Python, Assimp

The necessary python packages are listed in requirements.txt so installing them with pip install -r requirements.txt is simplest. If there are errors in install the library in extensions it can likely be resolved by installing PyTorch and/or Wheel beforehand.

Then run python source/train.py on any target mesh:

usage: train.py [-h] [--mesh MESH] [--lod LOD] [--features FEATURES] [--display DISPLAY] [--batch BATCH] [--fixed-seed]

options:
  -h, --help           show this help message and exit
  --mesh MESH          Target mesh
  --lod LOD            Number of patches to partition
  --features FEATURES  Feature vector size
  --display DISPLAY    Display the result after training
  --batch BATCH        Batch size for training
  --fixed-seed         Fixed random seed (for debugging)

The results of the training will be placed into a local results directory as follows:

results
├── binaries           (Binaries for trained neural geometry fields)
├── loss               (Loss plots)
├── meta               (Generic metadata)
├── quadrangulated     (Partitioned surfaces)
├── stl                (Final surfaces exported as STLs)
└── torched            (Pytorch binary data)

The memory usage is relatively modest (under 8 GB for the default 1K patches and 10 feature channels), but it can be adjusted with the batch size option.

Some tips to consider if errors appear:

• The STL format for meshes is most reliable; if the program complains from the meshio library, try again with a STL rather than an OBJ or etc.
• PyMeshLab is used for simplification and quadrangulation, but the execution is not always reliable. For this reason we time out the quadrangulation process after a minute. If this happens, the target mesh is likely too large.

Rasterization

Requirements: Vulkan, GLFW, CMake, Mesh shaders¹

Source code for the real-time rasterizer is provided in the rasterizer directory. Make sure that the submodules have also been cloned. We rely on CMake to compile the program:

cmake -B build .
cmake --build build -j

To run the rasterizer, execute the resulting build/testbed binary by providing a path to the neural geometry field binary file (e.g. within results/binaries). For example:

./build/testbed results/binaries/nefertiti-lod1000-f20.bin

A few binaries have been provided in the resources/samples directory to explore the rasterizer on pretrained NGFs.

Here are some performance statistics for the rasterizer:

GPU	Patch Count	Framerate (Frametime)
RTX 3060 Mobile	1K	250 FPS (4 ms)
RTX 3060 Mobile	2.5K	100 FPS (10 ms)
RTX 4090	1K	1200 FPS (0.8 ms)
RTX 4090	2.5K	600 FPS (1.6 ms)

Citation

@inproceedings{vs2024ngfs,
  title = {Neural Geometry Fields for Meshes},
  author = {Sivaram, Venkataram and Ramamoorthi, Ravi and Li, Tzu-Mao},
  numpages = {11},
  year = {2024},
  series = {SIGGRAPH '24}
}

Footnotes

1. Check vulkaninfo from the command-line or search for your GPU model here. ↩