Benchmarks

Run Different Baselines

In this part, you need to use the virtual environment of the original diffusers.

conda activate diffusers

Note that to run scripts of different image-edit pipelines, you should first generate the dataset (at least dataset/hetu_origin). And you should move the scripts to your workspace, so that it could read the dataset directly.

For instance, you could run the following commands to run an SDEdit pipeline.

cd dataset && mkdir SDEdit && cd ..
python SDEdit.py

Evaluate

You should first move the scripts from eval directory to your workspace. Then, to obtain the image CLIP embeddings, you should change the root in image_clip.py to evaluate on the images you have just generated (dataset/hetu_edit by default). Note that it's also necessary to obtain the text CLIP embeddings (only need once) before measuring the metrics.

python image_clip.py
python text_clip.py

Then, you could simply run the scripts after adjusting the root to your .pth files generated by image_clip.py and text_clip.py.

python image_image_similarity.py
python direction_similarity.py
python clip_accuracy.py