3D_Photo_Inpainting.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3D-Photo-Inpainting.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/fzantalis/colab_collection/blob/master/3D_Photo_Inpainting.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IY-s3ZpHBeAU",
        "colab_type": "text"
      },
      "source": [
        "**[CVPR 2020] 3D Photography using Context-aware Layered Depth Inpainting**\n",
        "\n",
        "[project website](https://shihmengli.github.io/3D-Photo-Inpainting/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LPQ0EWxgvMIc",
        "colab_type": "text"
      },
      "source": [
        "- **Prepare environment**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5o-EIMeaghU0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip3 install torch==1.4.0+cu100 torchvision==0.5.0+cu100 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "!pip3 install opencv-python==4.2.0.32\n",
        "!pip3 install vispy==0.6.4\n",
        "!pip3 install moviepy==1.0.2\n",
        "!pip3 install transforms3d==0.3.1\n",
        "!pip3 install networkx==2.3\n",
        "!sudo apt install sed"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-g7AeLMvY0R",
        "colab_type": "text"
      },
      "source": [
        "- **Download script and pretrained model**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cOFIBkWrBlNM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/\n",
        "!git clone https://github.com/vt-vl-lab/3d-photo-inpainting.git\n",
        "%cd 3d-photo-inpainting\n",
        "!sh download.sh"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_Fg_aMeDnmd",
        "colab_type": "text"
      },
      "source": [
        "- **Switch off off-screen rendering**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S9TmdOBRDXcV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!sed -i 's/offscreen_rendering: True/offscreen_rendering: False/g' argument.yml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cCN8L7GiDpZA",
        "colab_type": "text"
      },
      "source": [
        "- **Please upload `.jpg` files to `/content/3d-photo-inpainting-master/image/`**\n",
        "  - You can run this step multiple times to upload multiple `.jpg` files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n-Kl6fo1oows",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd image\n",
        "from google.colab import files\n",
        "uploaded = files.upload()\n",
        "for fn in uploaded.keys():\n",
        "  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
        "      name=fn, length=len(uploaded[fn])))\n",
        "%cd .."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wyLFopwVDzUg",
        "colab_type": "text"
      },
      "source": [
        "- **Execute the 3D Photo Inpainting**\n",
        "  - Note: The 3D photo generation process usually takes about 2-3 minutes or more depending on the available computing resources."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i5-MWEjfBjYx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python main.py --config argument.yml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPvkMT0msIJB",
        "colab_type": "text"
      },
      "source": [
        "- **The results are stored in the following directories**\n",
        "  - Corresponding depth map estimated by [MiDaS](https://github.com/intel-isl/MiDaS.git) \n",
        "      - E.g. ```/content/3d-photo-inpainting/depth/moon.npy```\n",
        "  - Inpainted 3D mesh\n",
        "      - E.g. ```/content/3d-photo-inpainting/mesh/moon.ply```\n",
        "  - Rendered videos with zoom-in motion\n",
        "      - E.g. ```/content/3d-photo-inpainting/video/moon_zoom-in.mp4```\n",
        "  - Rendered videos with swing motion\n",
        "      - E.g. ```/content/3d-photo-inpainting/video/moon_swing.mp4```\n",
        "  - Rendered videos with circle motion\n",
        "      - E.g. ```/content/3d-photo-inpainting/video/moon_circle.mp4```\n",
        "  \n",
        "    ![](https://drive.google.com/uc?id=1xBiZAeGfhsoAsG08N5UM8-f993p0G5HF)"
      ]
    }
  ]
}