Dynamic Extension Nets for Few-shot Semantic Segmentation

Created by Lizhao Liu, Junyi Cao and Minqian Liu from South China University of Technology.

This repository contains the official PyTorch-implementation of our ACM MM 2020 paper Dynamic Extension Nets for Few-shot Semantic Segmentation. In particular, we release the code for training and testing the DENet and our re-implemented methods for few-shot semantic segmentation under {1, 2}-way and {1, 5}-shot settings.

Pretrained Models

For the convenience of relevant researchers to reproduce our results, we have release the pretrained models, tensorboard files and logs during training. They are all uploaded to the google drive: https://drive.google.com/uc?export=download&id=13SzrlZ6iTJL-pGhEDDrL-SjJuAGVhMid

The config files are in reproduce_configs. The run script is in run.sh.

Note that, it may take a while to fully load the tensorboard files after running the tensorboard command.

The reproduced results are somehow better than the ones in the original paper.

COCO-20i
	1-way 1-shot					1-way 5-shot
	fold0	fold1	fold2	fold3	mean	fold0	fold1	fold2	fold3	mean
Paper	42.90	45.78	42.16	40.22	42.77	45.40	44.86	41.57	40.26	43.02
Reproduced	43.14	45.84	42.07	40.80	42.96	46.46	44.95	40.28	40.99	43.17

---

Usage

Environment

• PyTorch 1.3.1
• torchvision 0.4.2
• tensorboardX for recording training information
• tqdm for displaying the training progress
• pyyaml for reading yml files
• A full list of dependencies are in requirements.txt, you can use

   pip install -r requirements.txt

  to download all the dependencies.

Dataset Preparation

1. To save your trouble, you can directly download our preprocessed datasets via google drive:

   • PASCAL VOC 5i (6.4GB): https://drive.google.com/uc?export=download&id=1uWNPSd_tEG5-WCRcG1sk9-5R0yI-O0XL
   • COCO 20i (23GB): The coco dataset is too large and I highly recommend you to download and preprocess it as the following instructions.

2. Enter the file dataset/voc_sbd.py and modify the ROOT to the path where you want to save the dataset.

   # specify root path for the data path
   ROOT = '/path/to/data'

3. Run the function run_VOCSBDSegmentation5i_from_scratch() to download and process 4 subfolds of PASCAL-5i dataset.

4. Enter the file dataset/coco.py and modify the ROOT to the path where you want to save the dataset.

   # specify root path for the data path
   ROOT = '/path/to/data'

5. Run the function run_COCOStuff20i_from_scratch() to download and process 4 subfolds of COCO-20i dataset.

Configurations

Please enter the config directory, we already provide you with several configuration templates for training, testing and performing inference using our DENet and some other re-implemented models. You need to create three empty files in config directory: train_config.yml, test_config.yml and inference_config.yml. Then, you can copy the content in one of the template files to fill the corresponding yml file. For example, you can copy the content in train_config_denet.template to train_config.yml for training with DENet. You still need to specify the path to the dataset in yml files.

Note: The provided templates use PASCAL-5i as the default dataset, you can change the corresponding arguments to use COCO-20i dataset.

Training

After the train_config.yml file is appropriately configured, training is quite simple with just one line of script:

python -u train.py

All the records of training information will be automatically saved to runs/${MODEL_NAME}/${ID}.

Testing

After the test_config.yml file is appropriately configured, you can run the following script to test the trained model:

python -u test.py

Likewise, all the records of testing information will be automatically saved to runs/${MODEL_NAME}/${ID}.

Inference

After the inference_config.yml file is appropriately configured, you can do inference with the trained model using the script:

python -u inference.py

The output images will be saved to results/${MODEL_NAME}/${ID}.

---

Quantitative Results

• All results below are in mIoU(%)

PASCAL-5i
1 way 1 shot	1 way 5 shot	2 way 1 shot	2 way 5 shot
60.08	60.46	52.17	53.62

COCO-20i
1 way 1 shot	1 way 5 shot	2 way 1 shot	2 way 5 shot
42.77	43.02	38.52	40.87

---

Qualitative Results

• 1 way

• 2 way

• failure cases

---

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{liu2020dynamic, 
title={Dynamic Extension Nets for Few-shot Semantic Segmentation},
author={Liu, Lizhao and Cao, Junyi and Liu, Minqian and Guo, Yong and Chen, Qi and Tan, Mingkui}, 
booktitle={Proceedings of the 28th ACM International Conference on Multimedia},  
year={2020}
}

---

Acknowledgments

This work was partially supported by Science and Technology Program of Guangzhou, China under Grants 202007030007, Key-Area Research and Development Program of Guangdong Province 2019B010155001, National Natural Science Foundation of China (NSFC) 61836003 (key project), Guangdong 2017ZT07X183, Fundamental Research Funds for the Central Universities D2191240.