Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

Panoptic SegFormer is accepted by CVPR'22 and we update our latest paper on arXiv

Install

Windows installation:

With conda or miniconda installed:

1. conda env create -f environment.yaml
2. conda activate panformer
3. git clone https://github.com/open-mmlab/mmcv.git
4. cd mmcv
5. git checkout v1.3.4
6. $env:MMCV_WITH_OPS=1
7. python setup.py build_ext
8. python setup.py develop

Linux Install

Prerequisites

• Linux
• Python 3.6+
• PyTorch 1.5+
• torchvision
• CUDA 9.2+ (If you build PyTorch from source, CUDA 9.0 is also compatible)
• GCC 5+
• mmcv-full==1.3.4
• mmdet==2.12.0 # higher version may not work
• timm==0.4.5
• einops==0.3.0
• Pillow==8.0.1
• opencv-python==4.5.2

note: PyTorch1.8 has a bug in its adamw.py and it is solved in PyTorch1.9(see), you can easily solve it by comparing the difference.

install Panoptic SegFormer

python setup.py install 

Results

results on COCO val

Backbone	Method	Lr Schd	PQ	Config	Download
R-50	Panoptic-SegFormer	1x	48.0	config	model
R-50	Panoptic-SegFormer	2x	49.6	config	model
R-101	Panoptic-SegFormer	2x	50.6	config	model
PVTv2-B5 (much lighter)	Panoptic-SegFormer	2x	55.6	config	model
Swin-L (window size 7)	Panoptic-SegFormer	2x	55.8	config	model

Datasets

When I began this project, mmdet dose not support panoptic segmentation officially. I convert the dataset from panoptic segmentation format to instance segmentation format for convenience.

1. prepare data (COCO)

cd Panoptic-SegFormer
mkdir datasets
cd datasets
ln -s path_to_coco coco
mkdir annotations/
cd annotations
wget http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip
unzip panoptic_annotations_trainval2017.zip

Then the directory structure should be the following:

Panoptic-SegFormer
├── datasets
│   ├── annotations/
│   │   ├── panoptic_train2017/
│   │   ├── panoptic_train2017.json
│   │   ├── panoptic_val2017/
│   │   └── panoptic_val2017.json
│   └── coco/ 
│
├── config
├── checkpoints
├── easymd
...

2. convert panoptic format to detection format

cd Panoptic-SegFormer
./tools/convert_panoptic_coco.sh coco

Then the directory structure should be the following:

Panoptic-SegFormer
├── datasets
│   ├── annotations/
│   │   ├── panoptic_train2017/
│   │   ├── panoptic_train2017_detection_format.json
│   │   ├── panoptic_train2017.json
│   │   ├── panoptic_val2017/
│   │   ├── panoptic_val2017_detection_format.json
│   │   └── panoptic_val2017.json
│   └── coco/ 
│
├── config
├── checkpoints
├── easymd
...

Run (panoptic segmentation)

train

single-machine with 8 gpus.

./tools/dist_train.sh ./configs/panformer/panformer_r50_24e_coco_panoptic.py 8

test

./tools/dist_test.sh ./configs/panformer/panformer_r50_24e_coco_panoptic.py path/to/model.pth 8

Citing

If you use Panoptic SegFormer in your research, please use the following BibTeX entry.

@misc{li2021panoptic,
      title={Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers}, 
      author={Zhiqi Li and Wenhai Wang and Enze Xie and Zhiding Yu and Anima Anandkumar and Jose M. Alvarez and Tong Lu and Ping Luo},
      year={2021},
      eprint={2109.03814},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

Mainly based on Defromable DETR from MMdet.

Thanks very much for other open source works: timm, Panoptic FCN, MaskFomer, QueryInst