This is a Tensorflow implementation for the CVPR 2019 paper 'Fully Automatic Video Colorization with Self-Regularization and Diversity'.

More results are shown on our project website https://leichenyang.weebly.com/project-color.html

News

We propose a novel and general framework Deep-Video-Prior that can address the temporal inconsistency problem given an input video and a processed video. We can obtain high-quality video using a single-image colorization method and our novel framework.

Quick inference( without refinement network)

conda env create -f environment.yml
conda activate automatic-video-colorization
bash pretrained_models/download_models.sh
python test.py 

the results are saved in test_result/

Dependency

Environment

This code is based on tensorflow. It has been tested on Ubuntu 18.04 LTS.

Anaconda is recommended: Ubuntu 18.04 | Ubuntu 16.04

After installing Anaconda, you can setup the environment simply by

conda env create -f environment.yml
conda activate automatic-video-colorization

Pretrained-models and VGG-Models

bash download_pretrained.sh

Usage

Image colorization

We provide the ckpt to colorize a single image, the temporal consistency is not as good as video colorization but the colorization performance is better.

You can colorization a single image, e.g.:

python test.py --img_path PATH/TO/IMAGE
e.g.,
python test.py --img_path demo_imgs/ILSVRC2012_val_00040251.JPEG

or colorize the images in a folder by:

python test.py --img_path PATH/TO/FOLDER
e.g., 
python test.py --img_path demo_imgs/

The results are saved in test_result/

Video colorization without optical flow

Step1. the video should be split to frames first:

python video_utils.py --video2frames  --video_dir demo.mkv --out_frames_dir demo_framesdir

Step2. Colorizing blanc-and-white frames (you can also use the image colorization pretrained model to colorize the frames):

python test_div_video.py --use_gpu 1 --video_path demo_framesdir 

Results are saved in video_path_colorized, e.g., demo_framesdir_colorized

Step3. Convert colorized frames to video, note that you need to check the fps of original video

python video_utils.py --frames2video --colorized_video_dir demo_colorized.mp4 --colorized_frames_dir demo_framesdir_colorized --fps 24

Step4. Add the sound of original video (again, you need to make sure the fps of colorized vide is consistent with the original video)

python video_utils.py --add_sound --colorized_video_dir demo_colorized.mp4 --video_dir demo.mkv

Training

python main_whole.py --model YOUR_MODEL_NAME --data_dir data

Prepare the dataset

For the video dataset, please download the DAVIS dataset and generate the optical flow by PWC-Net by yourself. If you want to use FlowNet2 or other methods, please make sure the file format is the same.

At last, please arrange your data in the following format:

+data
-----+JPEGImages
----------------+480p
---------------------+VideoFrames1
---------------------+VideoFrames2
-----+FLOWImages
----------------+Forward
-----------------------+VideoFrames1
-----------------------+VideoFrames2
----------------+Backward
-----------------------+VideoFrames1
-----------------------+VideoFrames2
-----+FLOWImages_GRAY
--------------------+Forward
----------------------------+VideoFrames1
----------------------------+VideoFrames2
--------------------+Backward
----------------------------+VideoFrames1
----------------------------+VideoFrames2

For the image dataset, please download the ImageNet dataset.

Citation

If you use our code or paper, please cite:

@InProceedings{Lei_2019_CVPR,
author = {Lei, Chenyang and Chen, Qifeng},
title = {Fully Automatic Video Colorization With Self-Regularization and Diversity},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}

Contact

If you have any question, please feel free to contact me (Chenyang LEI, [email protected])