FacialFlowNet

Official release of FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model

ACMMM2024

Jianzhi Lu, Ruian He, Shili Zhou, Weimin Tan, Bo Yan

Fudan, Shanghai, China

DecFlow

Environments

You will have to choose cudatoolkit version to match your compute environment. The code is tested on PyTorch 1.10.0 and cuda 11.8, but other versions might also work.

conda create --name decflow python==3.8
conda activate decflow
pip install torch==1.10.0+cu113 torchvision==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
pip install -r requirements.txt

Demos

python get_flow.py

Train

# stage1 train both flow decoder heads
python train.py --name decflow-facialflownet-part1 --stage facialflow --validation facialflow --output checkpoints --restore_ckpt checkpoints/gma-sintel.pth  --num_steps 10000 --lr 0.000125 --image_size 480 480 --wdecay 0.00001 --gamma 0.85 --gpus 0 --batch_size 6 --val_freq 1000 --print_freq 100 --mixed_precision
# stage2 frozen facialflow decoder and train headflow decoder
python train.py --name decflow --stage facialflow --validation facialflow  --output checkpoints --restore_ckpt checkpoints/decflow-facialflownet-part1.pth  --num_steps 10000 --lr 0.000125 --image_size 480 480 --wdecay 0.00001 --gamma 0.85 --gpus 0 --batch_size 6 --val_freq 1000 --print_freq 100 --mixed_precision --frozen

Evaluate

python evaluate.py --model ./checkpoints/decflow-facialflownet.pth --dataset facialflow

FacialFlowNet Dataset

You can download the FacialFlowNet dataset from here with the extraction code c2z2, and extract the compressed file to the following path:

├── FacialFlowNet
    ├── image
        ├── facial
            ├── train 
            ├── test 
            ├── val
        ├── head
            ├── ...
    ├── flow
        ├── facial
            ├── train 
            ├── test 
            ├── val
        ├── head
            ├── ...
    ├── mask
        ├── ...

Acknowledgement

Parts of code are adapted from the following repositories. We thank the authors for their great contribution to the community:

• RAFT
• GMA
• DAD-3DHeads

Citation

If you use the FacialFlowNet Dataset and/or DecFlow - implicitly or explicitly - for your research projects, please cite the following paper:

@inproceedings{lu2024facialflownet,
  title={FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model},
  author={Lu, Jianzhi and He, Ruian and Zhou, Shili and Tan, Weimin and Yan, Bo},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={2194--2203},
  year={2024}
}