Coherent Reconstruction of Multiple Humans from a Single Image

Code repository for the paper:
Coherent Reconstruction of Multiple Humans from a Single Image
Wen Jiang*, Nikos Kolotouros*, Georgios Pavlakos, Xiaowei Zhou, Kostas Daniilidis
CVPR 2020 [paper] [project page]

Installation instructions

This codebase was adopted from early version of mmdetection and mmcv. Users of this repo are highly recommended to read the readme of mmcv and mmdetection before using this code.

To install mmcv and mmdetection:

conda env create -f environment.yml
cd neural_renderer/
python3 setup.py install
cd ../mmcv
python3 setup.py install
cd ../mmdetection
./compile.sh
python setup.py develop
cd ../sdf
python3 setup.py install

Fetch data

Download our model data and place them under mmdetection/data. This includes the model checkpoint and joint regressors. You also need to download the mean SMPL parameters from here Besides these files, you also need to download the SMPL model. You will need the neutral model for training evaluation and running the demo code. Please go to the websites for the corresponding projects and register to get access to the downloads section. In case you need to convert the models to be compatible with python3, please follow the instructions here.

After finishing with the installation and downloading the necessary data, you can continue with running the demo/evaluation/training code.

Run demo code

We provide code to evaluate our pretrained model on a folder of images by running:

cd mmdetection
python3 tools/demo.py --config=configs/smpl/tune.py --image_folder=demo_images/ --output_folder=results/ --ckpt data/checkpoint.pt

Prepare datasets

Please refer to DATASETS.md for the preparation of the dataset files. We will update the repo with instructions soon.

Run evaluation code

Besides the demo code, we also provide code to evaluate our models on the datasets we employ for our empirical evaluation. Before continuing, please make sure that you follow the preparation of test sets.

You could use either our pretrained checkpoint or the model trained by yourself to evaluate on Panoptic, MuPoTS-3D and h3.6m.

Example usage:

cd mmdetection
python3 tools/full_eval.py configs/smpl/tune.py full_h36m --ckpt ./work_dirs/tune/latest.pth

Running the above command will compute the MPJPE and Reconstruction Error on the Human3.6M dataset (Protocol I). The full_h36m option can be replaced with other dataset or sequences based on the type of evaluation you want to perform: haggling: haggling sequence of Panoptic mafia: mafia sequence of Panoptic ultimatum: ultimatum sequence of Panoptic haggling: haggling sequence of Panoptic mupots: MuPoTS-3D dataset posetrack: PoseTrack dataset PS:

• For Panoptic, the command will compute the MPJPE for each sequence.
• For MuPoTS-3D, the command will save the results to the work_dirs/tune/mupots.mat which can be taken as input for official mupots-3d test script/

Run training code

Please make sure you have prepared all datasets before running our training script. The training of our model would take three phases, pretrain -> baseline -> fine tuning. We prepared three configuration files under mmdetection/configs/smpl/ To train our model from scratch:

cd mmdetection
# Phase 1: pretraining
python3 tools/train.py configs/smpl/pretrain.py --create_dummy
while true:
do
    python3 tools/train.py configs/smpl/pretrain.py
done
# We could move to next phase after training for 240k iterations

# Phase 2: baseline
python3 tools/train.py configs/smpl/baseline.py --load_pretrain ./work_dirs/pretrain/latest.pth
while true:
do
    python3 tools/train.py configs/smpl/baseline.py 
done
# We could move to next phase after training for 180k iterations

# Phase 3: Fine-tuning
python3 tools/train.py configs/smpl/tune.py --load_pretrain ./work_dirs/baseline/latest.pth
while true:
do
    python3 tools/train.py configs/smpl/tune.py 
done
# It could be done after 100k iterations of training

All the checkpoints, evaluation results and logs would be saved to ./mmdetection/work_dirs/ + pretrain|baseline|tune respectively. Our training program will save the checkpoints and restart every 50 mins. You could change the time_limit in those configurations files for your convenience.

Citing

If you find this code useful for your research or the use data generated by our method, please consider citing the following paper:

@Inproceedings{jiang2020mpshape,
  Title          = {Coherent Reconstruction of Multiple Humans from a Single Image},
  Author         = {Jiang, Wen and Kolotouros, Nikos and Pavlakos, Georgios and Zhou, Xiaowei and Daniilidis, Kostas},
  Booktitle      = {CVPR},
  Year           = {2020}
}

Acknowledgements

This code uses these code (mmcv, mmdetection) as backbone. We gratefully appreciate the impact it had on our work. If you use our code, please consider citing the original paper as well.