Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification (ECCV 2020)

Sanath Narayan^*, Akshita Gupta^*, Fahad Shahbaz Khan, Cees G. M. Snoek, Ling Shao

Paper: https://arxiv.org/abs/2003.07833

Video Presentation: Short summary , Overview

Finetuned features: https://drive.google.com/drive/folders/13-eyljOmGwVRUzfMZIf_19HmCj1yShf1?usp=sharing

Zero-shot learning strives to classify unseen categories for which no data is available during training. In the generalized variant, the test samples can further belong to seen or unseen categories. The stateof-the-art relies on Generative Adversarial Networks that synthesize unseen class features by leveraging class-specific semantic embeddings. During training, they generate semantically consistent features, but discard this constraint during feature synthesis and classification. We propose to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification. We first introduce a feedback loop, from a semantic embedding decoder, that iteratively refines the generated features during both the training and feature synthesis stages. The synthesized features together with their corresponding latent embeddings from the decoder are then transformed into discriminative features and utilized during classification to reduce ambiguities among categories. Experiments on (generalized) zero-shot object and action classification reveal the benefit of semantic consistency and iterative feedback, outperforming existing methods on six zero-shot learning benchmarks

Overall Architecture:

Overall Framework for TF-Vaegan

A feedback module, which utilizes the auxiliary decoder during both training and feature synthesis stages for improving semantic quality of synthesized feature.	A discriminative feature transformation that utilizes the auxiliary decoder during the classification stage for enhancing zero-shot classification.

Installation

The codebase is built on PyTorch 0.3.1 and tested on Ubuntu 16.04 environment (Python3.6, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

conda env create -f environment.yml
conda activate pytorch0.3.1

Directory Structure

zero-shot-images

• data/
• image-scripts/
  • run_awa_tfvaegan.py
  • run_cub_tfvaegan.py
  • run_flo_tfvaegan.py
  • run_sun_tfvaegan.py
• classifier.py
• config.py
• model.py
• train_tfvaegan_inductive.py
• util.py

zero-shot-actions

• action-scripts/
  • run_hmdb51.py
  • run_ucf101.py
• data/
• classifier.py
• classifier_entropy.py
• config.py
• model.py
• train_tfvaegan.py
• util.py

Training

Zero-Shot Image Classification

cd zero-shot-images
CUB : python image-scripts/run_cub_tfvaegan.py
AWA : python image_scripts/run_awa_tfvaegan.py
FLO : python image_scripts/run_flo_tfvaegan.py
SUN : python image_scripts/run_sun_tfvaegan.py

Zero-Shot Action Classification

cd zero-shot-actions
HMDB51 : python action_scripts/run_hmdb51_tfvaegan.py
UCF101 : python action_scripts/run_ucf101_tfvaegan.py

Results

Citation:

If you find this useful, please cite our work as follows:

@article{narayan2020latent,
  title={Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification},
  author={Narayan, Sanath and Gupta, Akshita and Khan, Fahad Shahbaz and Snoek, Cees GM and Shao, Ling},
  journal={arXiv preprint arXiv:2003.07833},
  year={2020}
}