Project Page: Magic-Me

Magic-Me: Identity-Specific Video Customized Diffusion

Ze Ma*, Daquan Zhou* †, Chun-Hsiao Yeh, Xue-She Wang, Xiuyu Li, Huanrui Yang, Zhen Dong †, Kurt Keutzer, Jiashi Feng (*Joint First Author, † Corresponding Author)

In this repository, we propose a series of controllable Video generation and editing methods. The first release includes Customized Diffusion (VCD). It includes three novel components that are essential for high-quality ID preservation: 1) an ID module trained with the cropped identity by prompt-to-segmentation to disentangle the ID information and the background noise for more accurate ID token learning; 2) a text-to-video (T2V) VCD module with 3D Gaussian Noise Prior for better inter-frame consistency and 3) video-to-video (V2V) Face VCD and Tiled VCD modules to deblur the face and upscale the video for higher resolution.

Video Demonstration

final_demo.mp4

Video Customization Diffusion Model Pipeline           

ID Specific Video Generation

Altman	Altman	Robert	Taylor
superman.mp4	walking.mp4	desert.mp4	light.mp4

ID Specific Video Editing

Original Video	Altman	Bengio	Zuck
happy_new_year_ori.mp4	altman_hapy_new_year.mp4	bengio_hapy_new_year.mp4	zuck_hapy_new_year.mp4
dance_ori.mp4	dance_altman.mp4	dance_bengio.mp4	dance_zuck.mp4

Next

More works will be released soon. Stay tuned.

1. Magic-Me

   • Support SD-XL
   • Integrate pose/depth/stretch control
   • Integrate multi-prompt / Prompt Travel
   • Release arxiv and codes
   • Release demo

2. Magic ID-ditting

   • Release arxiv and codes
   • Release demo

3. Magic-Me Instant

   • Release arxiv and codes
   • Release demo

4. Magic-Me Crowd

   • Release arxiv and codes
   • Release demo

Install

First, make sure that Anaconda is installed (refer to official Install Tutorial).

git clone https://github.com/Zhen-Dong/Magic-Me.git
cd Magic-Me
conda env create -f environment.yaml 

Download checkpoints

1. Stable Diffusion 1.5

   git lfs install
   git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/
   

2. Download motion module and realistic vision checkpoints

   source download_checkpoint.sh
   

Train and evaluate extended ID tokens

• Quick training with evaluation.

  conda activate magic
  source train.sh
  

• Customized training of a specific identity
  1. Select 5-10 images of a specific identity into a new diretory under dataset.
  2. Make a new .yaml out of configs/ceo.yaml.
  3. Run the training codes in conda env magic.
     python train.py --config configs/your_config.yaml
  4. The ID embeddings are available in the directory of outputs/magic-me-ceo-xxxxx/checkpoints.

Inference

• Option 1 (T2V VCD)
  • The generated videos from the prompts in the configs/ceo.yaml are saved during training in the directory of outputs/magic-me-ceo-xxxxx/samples.
• Option 2 (T2V + Face + Tiled VCD)

  • This requires you are subscribed to Google Colab. Users must comply with Google Colab's terms of service and respect copyright laws, ensuring that the resources provided are used responsibly and ethically for academic, educational, or research purposes only.

  • Use this Colab Notebook to run ComfyUI.

  • In the ComfyUI, one click on the "Queue Prompt" to generate the video.

  • Feel free to change the prompt inside ComfyUI (embedding:firstname man for male, embedding:firstname woman for female)

    

  • The available embeddings are cloned into the directory magic_factory/Magic-ComfyUI/models/embeddings.

    

    Feel free to put your newly trained embeddings, for example boy1.pt, in the same directory, and mention the mebdding as embedding:boy1 man in the ComfyUI prompt.

Interesting Prompts:

Use embedding:firstname man for the male character and embedding:firstname woman for the female.

1. a photo of embedding:firstname man in superman costume in the outer space, stars in the background.
2. a photo of embedding:firstname woman, desert, a striking tableau forms around a solitary figure who stoically stands against the whipping winds and stinging sands. Clad in a worn, tattered, dark brown cloak, with hood pulled low, only hinting at the determined visage beneath - this individual embodies fortitude and tenacity.
3. a photo of embedding:firstname woman, clad in exquisite hybrid armor, studded with iridescent gemstones, Cherry blossoms sway gently in the breeze, lithesome figure against a softly blurred backdrop. style of Gerald Parel, ward-winning, professional, highly detailed.
4. a photo of embedding:firstname man walking on the street, B&W, city, captured on a 35mm camera, black and white, classic film. inside Film Still, film grain, Movie Still, Film Still, Cinematic, Cinematic Shot, Cinematic Lighting
5. a photo of embedding:firstname man, amazing quality, knight, holding a sword, dazzling ,transparent ,polishing, waving sword, gold armor, glowing armor, glowing eyes, full armor, shine armor, dazzling armor, masterpiece, best quality, hyper detailed, ultra detailed, UHD, perfect anatomy, (in castle:1.2).

Disclaimer

This project is released for academic use. We disclaim responsibility for user-generated content. Users are solely liable for their actions. The project contributors are not legally affiliated with, nor accountable for, users' behaviors. Use the generative model responsibly, adhering to ethical and legal standards.

Acknowledgements

Codebase built upon Tune-a-Video and AnimateDiff