Hongyu Zhou¹, Longzhong Lin¹, Jiabao Wang¹, Yichong Lu¹, Dongfeng Bai², Bingbing Liu², Yue Wang¹, Andreas Geiger^3,4, Yiyi Liao^1,†
1 Zhejiang University ² Huawei ³ University of Tübingen ⁴ Tübingen AI Center
^† Corresponding Authors

This is the official project repository of the paper HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving

---

TODO list

• Release sample data and results
• Release unicycle model part
• Release GUI
• Release more scenarios

Installation

Requirements are not in principle hard requirements, but there might be some differences (not tested):

• Linux
• Python 3.11
• CUDA 11.8
• colmap 3.10-dev
• Pytorch 2.5.1 / Pytorch 2.2.0

Create conda environment needed to run HUGSIM with:

conda create --name hugsim python=3.11

Please install these packages following official instructions

• Pytorch
• simple_waymo_open_dataset_reader
• tinycudann
• unidepth
• flow_vis_torch
• pytorch3d
• kitti360Scripts
• nuscenes-devkit (have to install from the source code for python 3.11)
• HUGSIM_splat

Please install packages required by InverseForm

Install remaining dependencies by:

pip install -r requirements.txt

To run the closed-loop simulation, the hug_sim gymnasium environment should be installed:

cd ./sim
pip install -e .

Data Preparation

Please refer to Data Preparation Document

We will provide sample sequence data and reconstructed results. The download link will be provided later.

Reconstruction

seq=${seq_name}
input_path=${datadir}/${seq}
output_path=${modeldir}/${seq}
mkdir -p ${output_path}
CUDA_VISIBLE_DEVICES=4 \
python -u train_ground.py --data_cfg ./configs/${dataset_name: [kitti360, waymo, nusc, pandaset]}.yaml \
        --source_path ${input_path} --model_path ${output_path}
CUDA_VISIBLE_DEVICES=4 \
python -u train.py --data_cfg ./configs/${dataset_name}.yaml \
        --source_path ${input_path} --model_path ${output_path}

Simulation

Before simulation, UniAD_SIM, VAD_SIM and NAVSIM client should be installed.

CUDA_VISIBLE_DEVICES=${sim_cuda} \
python closed_loop.py --scenario_path ${scenario_cfg_path} \
            --base_path ./configs/sim/${dataset_name}_base.yaml \
            --camera_path ./configs/sim/${dataset_name}_camera.yaml \
            --kinematic_path ./configs/sim/kinematic.yaml \
            --ad ${method_name: [uniad, vad, ltf]} \
            --ad_cuda ${ad_cuda}

Citation

If you find our paper and codes useful, please kindly cite us via:

@article{zhou2024hugsim,
  title={HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving},
  author={Zhou, Hongyu and Lin, Longzhong and Wang, Jiabao and Lu, Yichong and Bai, Dongfeng and Liu, Bingbing and Wang, Yue and Geiger, Andreas and Liao, Yiyi},
  journal={arXiv preprint arXiv:2412.01718},
  year={2024}
}