Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification

Pingping Zhang* · Yuhao Wang · Yang Liu · Zhengzheng Tu · Huchuan Lu

CVPR 2024 Paper

Previous methods may be easily affected by irrelevant backgrounds and usually ignore the modality gaps. To address above issues, we propose a novel learning framework named EDITOR to sElect DIverse Tokens for multi-modal Object ReID. EDITOR prioritizes the selection of object-centric information, aiming to preserve the diverse features of different modalities while minimizing background interference. Our proposed EDITOR achieves competitive performance on three multi-modal object ReID benchmarks, i.e., RGBNT201, RGBNT100 and MSVR310.

News

Exciting news! Our paper has been accepted by the CVPR 2024! 🎉

Table of Contents

• Introduction
• Contributions
• Results
• Visualizations
• Reproduction
• Citation

Introduction

Multi-modal object ReID is crucial in scenarios where objects are captured through different image spectra, such as RGB, near-infrared, and thermal imaging. Previous multi-modal ReID methods typically adhere to the approach of extracting global features from all regions of images in different modalities and subsequently aggregating them. Nevertheless, these methods present two key limitations: (1) Within individual modalities, backgrounds introduce additional noise, especially in challenging visual environments. (2) Across different modalities, backgrounds introduce overhead in reducing modality gaps, which may amplify the difficulty in aggregating features. Hence, our method prioritizes the selection of object-centric information, aiming to preserve the diverse features of different modalities while minimizing background interference.

Contributions

• We introduce EDITOR, a novel learning framework for multi-modal object ReID. To our best knowledge, EDITOR represents the first attempt to enhance multi-modal object ReID through object-centric token selection.
• We propose a Spatial-Frequency Token Selection (SFTS) module and a Hierarchical Masked Aggregation (HMA) module. These modules effectively facilitate the selection and aggregation of multi-modal tokenized features.
• We propose two new loss functions with a Background Consistency Constraint (BCC) and an Object-Centric Feature Refinement (OCFR) to improve the feature discrimination with background suppressions.
• Extensive experiments are performed on three multi-modal object ReID benchmarks. The results fully validate the effectiveness of our proposed methods.

Results

Multi-modal Object ReID

Performance comparison with different modules [RGBNT201、RGBNT100]

Parameter Analysis of EDITOR [RGBNT100]

Visualizations

T-SNE

Similarity

Selection

Please check the paper for detailed information

Reproduction

Datasets

RGBNT201 link: https://drive.google.com/drive/folders/1EscBadX-wMAT56_It5lXY-S3-b5nK1wH
RGBNT100 link: https://pan.baidu.com/s/1xqqh7N4Lctm3RcUdskG0Ug code：rjin
MSVR310 link: https://pan.baidu.com/s/1soWMH10DyGG0krTKL7fsYg code: p6ch

Pretrained

ViT-B link: https://pan.baidu.com/s/1YE-24vSo5pv_wHOF-y4sfA

Configs

RGBNT201 file: EDITOR/configs/RGBNT201/EDITOR.yml
RGBNT100 file: EDITOR/configs/RGBNT100/EDITOR.yml
MSVR310 file: EDITOR/configs/MSVR310/EDITOR.yml

Bash

# python = 3.8
# cuda = 11.4
#!/bin/bash
source activate (your env)
cd ../(your path)
pip install -r requirements.txt
python train_net.py --config_file ../RGBNT201/EDITOR.yml

Star History

Citation

If you find EDITOR useful in your research, please consider citing:

@inproceedings{zhang2024magic,
    title     = {Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification},
    author    = {Pingping Zhang, Yuhao Wang, Yang Liu, Zhengzheng Tu, Huchuan Lu},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2024},
}