Data Engine of MINIMA (CVPR 2025) for Multi-modal Image Generation

MINIMA Data Engine is an open-source project that integrates cutting-edge methods from the community to enable multi-modal transformations.
Currently, the engine supports the following modalities:

Online Demo

The online demo is under development. Stay tuned!

Environment Setup

The engine requires the following dependencies:

pip install -r engine_extra_requirements.txt

Checkpoints Preparation

We recommend placing all checkpoints in the data_engine/weights directory.
The download_weights.sh script can be used to download all the required weights and place them in the correct directory following the instructions below:

cd data_engine
bash download_weights.sh

You can also download the weights manually and place them in the data_engine/weights directory.

Weights Structure

The directory structure should be like this:

weights/
├── stylebooth/
│ ├── step-210000/
│ └── stylebooth-tb-5000-0.bin
├── clip-vit-large-patch14/
│ ├── tokenizer.json
│ └── ...
├── depth_anything_v2/
│ └── depth_anything_v2_vitl.pth
├── dsine/
│ └── dsine.pt
├── paint_transformer/
│ └── model.pth
└── anime_to_sketch/
  └── improved.bin

Infrared Generation

The infrared generation code is based on scepter

Please download the weights from styleBooth weights, clip-vit-large-patch14. And our style tuner is available for download at step-210000

NOTE: Generation a 1024x1024 image requires a GPU with about 12GB of memory.

Depth Generation

The depth generation code is based on Depth-Anything-V2

Please download the weights from Depth-Anything-V2-Large

Event Generation

The event generation module is a simple simulation implemented with basic code.

NOTE: Since this is a simulated process, no checkpoint is required.

Normal Generation

The normal generation code is based on DSINE

Please download weights

Sketch Generation

The sketch generation code is based on Anime2Sketch

Please download the weights

Paint Generation

The paint generation code is based on PaintTransformer

Please download weights

MINIMA Data Engine

To run the engine, you can use the following command:

cd data_engine
python modality_engine.py --modality <modality> --input_path <input_path> --output_dir <output_dir>
# --modality: Choose from [infrared, depth, event, normal, sketch, paint]
# --input_path: Supports both a single image or a directory that contains images

# Example
python modality_engine.py --modality infrared --input_path ./figs/origin_image.jpg --output_dir './result'

Acknowledgements

We sincerely appreciate the contributions from the open-source community.

Special thanks to:

• Scepter
• Depth-Anything-V2
• DSINE
• Anime2Sketch
• PaintTransformer

Your support and feedback help improve MINIMA Data Engine. We welcome contributions and collaborations from the community!