WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild

Rolandos Alexandros Potamias¹   Jinglei Zhang²   Jiankang Deng¹   Stefanos Zafeiriou¹

¹Imperial College London, UK
²Shanghai Jiao Tong University, China

This is the official implementation of WiLoR, an state-of-the-art hand localization and reconstruction model:

Installation

[Update] Quick Installation

Thanks to @warmshao WiLoR can now be installed using a single pip command:

pip install git+https://github.com/warmshao/WiLoR-mini

Please head to WiLoR-mini for additional details.

Note: the above code is a simplified version of WiLoR and can be used for demo only. If you wish to use WiLoR for other tasks it is suggested to follow the original installation instructued bellow:

Original Installation

git clone --recursive https://github.com/rolpotamias/WiLoR.git
cd WiLoR

The code has been tested with PyTorch 2.0.0 and CUDA 11.7. It is suggested to use an anaconda environment to install the the required dependencies:

conda create --name wilor python=3.10
conda activate wilor

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu117
# Install requirements
pip install -r requirements.txt

Download the pretrained models using:

wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt -P ./pretrained_models/
wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/wilor_final.ckpt -P ./pretrained_models/

It is also required to download MANO model from MANO website. Create an account by clicking Sign Up and download the models (mano_v*_*.zip). Unzip and place the right hand model MANO_RIGHT.pkl under the mano_data/ folder. Note that MANO model falls under the MANO license.

Demo

python demo.py --img_folder demo_img --out_folder demo_out --save_mesh 

Start a local gradio demo

You can start a local demo for inference by running:

python gradio_demo.py

WHIM Dataset

The dataset will be released soon.

Acknowledgements

Parts of the code are taken or adapted from the following repos:

• HaMeR
• Ultralytics

License

WiLoR models fall under the CC-BY-NC--ND License. This repository depends also on Ultralytics library and MANO Model, which are fall under their own licenses. By using this repository, you must also comply with the terms of these external licenses.

Citing

If you find WiLoR useful for your research, please consider citing our paper:

@misc{potamias2024wilor,
    title={WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild},
    author={Rolandos Alexandros Potamias and Jinglei Zhang and Jiankang Deng and Stefanos Zafeiriou},
    year={2024},
    eprint={2409.12259},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}