FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
Shuai Yang, Yifan Zhou, Ziwei Liu and Chen Change Loy
in CVPR 2024
Project Page | Paper | Supplementary Video | Input Data and Video Results
Abstract: The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.
Features:
- Temporal consistency: use robust intra-and inter-frame constraint with better consistency and coverage than optical flow alone.
- Zero-shot: no training or fine-tuning required.
- Flexibility: compatible with off-the-shelf models (e.g., ControlNet, LoRA) for customized translation.
- [03/2023] Code is released.
- [03/2024] This website is created.
- Add webUI.
- Upload paper to arXiv, release related material
- Clone the repository.
git clone [email protected]:williamyang1991/FRESCO.git --recursive
cd FRESCO- You can simply set up the environment with pip based on the
requirements.txt
- We have tested on torch 2.0.0/2.1.0 and diffusers 0.19.3
- If you use new versions of diffusers, you need to modify my_forward()
- Run the installation script. The required models will be downloaded in
./model,./src/ControlNet/annotatorand./src/ebsynth/deps/ebsynth/bin.
- Requires access to huggingface.co
python install.py- You can run the demo with
run_fresco.py
python run_fresco.py ./config/config_music.yaml