AutoTransition: Learning to Recommend Video Transition Effects

This is an official implementation of AutoTransition: Learning to Recommend Video Transition Effects.

Transition Dataset

We release the videos with annotated transitions extracted from the video editing template on online video editing platforms. Due to the privacy policy, we only release the link to the videos. The dataset can be downloaded from here: Download Link

Use the following command to download the source video in the dataset:

python3 tools/download_videos.py annotation.json ./template_download

The videos will be downloaded to ./template_download.

Usage

Prepare Data

To speed up the training, we convert videos to JPEG image and extract audio features before training. Run the following commands to finish these steps:

python3 preprocess/convert_video_folder.py ./path/to/template_root
python3 preprocess/extract_audio_features.py ./path/to/template_root path/to/annotation.json --model_path /path/to/audio/model.pth --cuda

The pretrained Harmonic CNN model could be downloaded from this link.

Train & Test

To train transition embeddings:

python3 tools/run_net.py --cfg configs/base/train_transition_embedding.yaml \
  DATASET.TRANSITION_CLASSIFICATION.JSON_ANNOTATION /path/to/annotation.json \
  DATASET.TRANSITION_CLASSIFICATION.TEMPLATE_ROOT /path/to/template_root

The transition embeddings can be found in ./log directory after training.

To train transition recommendation:

python3 tools/run_net.py --cfg configs/base/train_transition_recommendation.yaml \
  MODEL.TRANSITION_TRANSFORMER.EMBEDDING.PRETRAINED_EMBEDDING /path/to/pretrained/transition/embedding.pth \
  DATASET.TRANSITION_DATASET.JSON_ANNOTATION /path/to/annotation.json \
  DATASET.TRANSITION_DATASET.TEMPLATE_ROOT /path/to/template_root

tensorboard --logdir=./log