Unified Language-driven Zero-shot Domain Adaptation (CVPR 2024)

Senqiao Yang, Zhuotao Tian, Li Jiang, Jiaya Jia

[arXiv] [Page] [BibTeX] [Dataset]

News

• We have updated the Sand-Fire dataset, which can be downloaded here.
• We have uploaded the script_visualize.py to visualize the collected data and human annotations.

Video

You can also see the performance comparison video on our project-page.

Environment setup

conda env create -f environment.yml
conda activate ulda

Prepare Dataset

• CITYSCAPES: Follow the instructions in Cityscapes to download the images and semantic segmentation ground-truths. Please follow the dataset directory structure:

  <CITYSCAPES_DIR>/             % Cityscapes dataset root
  ├── leftImg8bit/              % input image (leftImg8bit_trainvaltest.zip)
  └── gtFine/                   % semantic segmentation labels (gtFine_trainvaltest.zip)

• ACDC: Download ACDC images and ground truths from ACDC. Please follow the dataset directory structure:

  <ACDC_DIR>/                   % ACDC dataset root
  ├── rbg_anon/                 % input image (rgb_anon_trainvaltest.zip)
  └── gt/                       % semantic segmentation labels (gt_trainval.zip)

• GTA5: Download GTA5 images and ground truths from GTA5. Please follow the dataset directory structure:

  <GTA5_DIR>/                   % GTA5 dataset root
  ├── images/                   % input image 
  └── labels/                   % semantic segmentation labels

Prepare checkpoint

Our checkpoints are downloaded from PODA

You can directly download it from here.

Run

Stage 1

Train the PIN to simulate the feature

bash bash/stage1.sh

Stage 2

Rectify and train the unified decoder (with evaluation)

bash bash/stage2.sh

Result

Due to the method's insensitivity to parameters, we randomly selected two sets of example parameters. Additionally, to maintain efficiency, we sampled only 100 images to train the PIN, which still achieved good performance. We encourage users to select parameters that best fit their specific tasks when utilizing our work.

The simulated PIN and adapted model can be downloaded from here.

Domain	Fog	Night	Rain	Snow	Average
mIoU	53.31	24.94	43.59	44.79	41.66

Test

python predict.py --ckpt <your path>

Citation

Please cite our work if you find it useful.

@inproceedings{yang2024unified,
  title={Unified Language-driven Zero-shot Domain Adaptation},
  author={Yang, Senqiao and Tian, Zhuotao and Jiang, Li and Jia, Jiaya},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={23407--23415},
  year={2024}
}