ssl-for-slr

Collection of self-supervised models for speaker and language recognition tasks.

Models

• sslforslr.models.cpc.CPC
  "Representation Learning with Contrastive Predictive Coding" (arxiv)
  Aaron van den Oord, Yazhe Li, Oriol Vinyals

• sslforslr.models.lim.LIM
  "Learning Speaker Representations with Mutual Information" (arxiv)
  Mirco Ravanelli, Yoshua Bengio

• sslforslr.models.simclr.SimCLR
  "Contrastive Self-Supervised Learning for Text-Independent Speaker Verification" (sci-hub)
  Haoran Zhang, Yuexian Zou, Helin Wang

• sslforslr.models.moco.MoCo
  "Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning" (arxiv)
  Wei Xia, Chunlei Zhang, Chao Weng, Meng Yu, Dong Yu

Datasets

VoxCeleb1 and VoxCeleb2 are used for our experiments and we rely on MUSAN and Room Impulse Response and Noise Database for data augmentation.

To download, extract and prepare all datasets run python prepare_data.py data/. The data/ directory will have the structure detailed below.

data
├── musan_split/
├── simulated_rirs/
├── voxceleb1/
├── voxceleb2/
├── trials
├── voxceleb1_train_list
└── voxceleb2_train_list

Trials and train lists files are also automatically created with the following formats.

• trials

  1 id10270/x6uYqmx31kE/00001.wav id10270/8jEAjG6SegY/00008.wav
  ...
  0 id10309/0cYFdtyWVds/00005.wav id10296/Y-qKARMSO7k/00001.wav
  

• voxceleb1_train_list and voxceleb2_train_list

  id00012 voxceleb2/id00012/21Uxsk56VDQ/00001.wav
  ...
  id09272 voxceleb2/id09272/u7VNkYraCw0/00027.wav
  

Please refer to prepare_data.py script if you want further details about data preparation.

Usage

Start self-supervised training with python train.py configs/cpc-base.yml.

Then, you can evaluate model on speaker verification (EER, minDCF) with python evaluate.py configs/cpc-base.yml.

To-Do

• Pytorch implementation
• Change repo/project name -> ssl-for-sv?
• Make sure other models work (MoCo/XVectorEncoder, CPC/CPCEncoder, LIM/SincEncoder, Wav2Spk)
• Get model name with config filename
• CPC/LIM: @tf.function warning when doing tensor[1, :]
• Fix warning when loading: some weights are not used
• Allow restore optimizer