FRESCO - Official PyTorch Implementation

FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
Shuai Yang, Yifan Zhou, Ziwei Liu and Chen Change Loy
in CVPR 2024
Project Page | Paper | Supplementary Video | Input Data and Video Results

Abstract: The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.

Features:

• Temporal consistency: use robust intra-and inter-frame constraint with better consistency and coverage than optical flow alone.
  • Compared with our previous work Rerender-A-Video, FRESCO is more robust to large and quick motion.
• Zero-shot: no training or fine-tuning required.
• Flexibility: compatible with off-the-shelf models (e.g., ControlNet, LoRA) for customized translation.

teasers.mp4

Updates

• [03/2023] Code is released.
• [03/2024] This website is created.

TODO

• Add webUI.
• Upload paper to arXiv, release related material
• Update readme

Installation

1. Clone the repository.

git clone https://github.com/williamyang1991/FRESCO.git
cd FRESCO

2. You can simply set up the environment with pip based on requirements.txt

   • We have tested on torch 2.0.0/2.1.0 and diffusers 0.19.3
   • If you use new versions of diffusers, you need to modify my_forward()

3. Run the installation script. The required models will be downloaded in ./model, ./src/ControlNet/annotator and ./src/ebsynth/deps/ebsynth/bin.

   • Requires access to huggingface.co

python install.py

4. You can run the demo with run_fresco.py

python run_fresco.py ./config/config_music.yaml

5. For issues with Ebsynth, please refer to issues

(1) Inference

WebUI (recommended)

TBA

Command Line

We provide a flexible script run_fresco.py to run our method.

Set the options via a config file. For example,

python run_fresco.py ./config/config_music.yaml

We provide some examples of the config in config directory. Most options in the config is the same as those in WebUI. Please check the explanations in the WebUI section.

(2) Results

Key frame translation

a red car turns in the winter	an African American boxer wearing black boxing gloves punches towards the camera, cartoon style	a cartoon spiderman in black suit, black shoes and white gloves is dancing	a beautiful woman holding her glasses in CG style

Full video translation

more_result_1.mp4

Citation

If you find this work useful for your research, please consider citing our paper:

@inproceedings{yang2024fresco,
 title = {FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation},
 author = {Yang, Shuai and Zhou, Yifan and Liu, Ziwei and and Loy, Chen Change},
 booktitle = {CVPR},
 year = {2024},
}

Acknowledgments

The code is mainly developed based on Rerender-A-Video, ControlNet, Stable Diffusion, GMFlow and Ebsynth.