week04_finetuning/seminar_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using pre-trained model\n",
    "\n",
    "Today we're going to build and fine-tune CNN based on weights pre-trained on ImageNet: the largest image classification dataset as of now.\n",
    "More about imagenet: http://image-net.org/\n",
    "Setup: classify from a set of 1000 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# class labels\n",
    "LABELS_URL = 'https://raw.githubusercontent.com/anishathalye/imagenet-simple-labels/master/imagenet-simple-labels.json'\n",
    "labels = {i: c for i, c in enumerate(requests.get(LABELS_URL).json())}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(list(labels.items())[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TorchVision\n",
    "PyTorch has several companion libraries, one of them being [torchvision](https://github.com/pytorch/vision/tree/master/) - it contains a number of popular vision datasets, preprocessing tools and most importantly, [pre-trained models](https://github.com/pytorch/vision/tree/master/torchvision/models).\n",
    "\n",
    "For now, we're going to use torch Inception-v3 module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We're gonna use the inception-v3 network:\n",
    "![img](https://hackathonprojects.files.wordpress.com/2016/09/googlenet_diagram.png?w=650&h=192)\n",
    "\n",
    "Let's first look at the code here: [url](https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.models.inception import inception_v3\n",
    "\n",
    "model = inception_v3(pretrained=True,      # load existing weights\n",
    "                     transform_input=True, # preprocess input image the same way as in training\n",
    "                    )\n",
    "\n",
    "model.aux_logits = False # don't predict intermediate logits (yellow layers at the bottom)\n",
    "model.train(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "dummy_x = torch.randn(5, 3, 299, 299)\n",
    "model(dummy_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict class probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If using Colab\n",
    "!mkdir sample_images\n",
    "!wget https://raw.githubusercontent.com/yandexdataschool/Practical_DL/sem4_spring19/week04_finetuning/sample_images/albatross.jpg -O sample_images/albatross.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from skimage.transform import resize\n",
    "%matplotlib inline\n",
    "\n",
    "img = resize(plt.imread('sample_images/albatross.jpg'), (299, 299))\n",
    "plt.imshow(img)\n",
    "plt.show()\n",
    "\n",
    "\n",
    "def transform_input(img):\n",
    "    return torch.as_tensor(img.reshape([1, 299, 299, 3]).transpose([0, 3, 1, 2]), dtype=torch.float32)\n",
    "\n",
    "\n",
    "def predict(img):\n",
    "    img = transform_input(img)\n",
    "    \n",
    "    probs = torch.nn.functional.softmax(model(img), dim=-1)\n",
    "    \n",
    "    probs = probs.data.numpy()\n",
    "    \n",
    "    top_ix = probs.ravel().argsort()[-1:-10:-1]\n",
    "    print ('top-10 classes are: \\n [prob : class label]')\n",
    "    for l in top_ix:\n",
    "        print ('%.4f :\\t%s' % (probs.ravel()[l], labels[l].split(',')[0]))\n",
    "        \n",
    "\n",
    "predict(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Having fun with pre-trained nets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget http://cdn.com.do/wp-content/uploads/2017/02/Donal-Trum-Derogar.jpeg -O img.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = resize(plt.imread('img.jpg'), (299, 299))\n",
    "plt.imshow(img)\n",
    "plt.show()\n",
    "\n",
    "predict(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grand-quest: Dogs Vs Cats\n",
    "* original competition\n",
    "* https://www.kaggle.com/c/dogs-vs-cats\n",
    "* 25k JPEG images of various size, 2 classes (guess what)\n",
    "\n",
    "### Your main objective\n",
    "* In this seminar your goal is to fine-tune a pre-trained model to distinguish between the two rivaling animals\n",
    "* The first step is to just reuse some network layer as features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -nc https://www.dropbox.com/s/ae1lq6dsfanse76/dogs_vs_cats.train.zip?dl=1 -O data.zip\n",
    "# if the link does not work, you can download the dataset manually from https://disk.yandex.ru/d/nuyCNaDrE1Bq0w\n",
    "!unzip -n data.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to get features\n",
    "During good old days in Torch7 you could access any intermediate output from the sequential model. Nowadays it's a bit more difficult though it's not Tensorflow where you need to compile another model for that. Here we're going to redefine the last layer... yes, to do nothing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from copy import deepcopy  # in case you still need original model\n",
    "\n",
    "embedding = deepcopy(model)\n",
    "\n",
    "class Identity(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Identity, self).__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        <YOUR CODE>\n",
    "        \n",
    "    \n",
    "# redefine the last layer to be Identity\n",
    "<YOUR CODE>\n",
    "\n",
    "assert embedding(transform_input(img)).data.numpy().shape == (1, 2048), \"your output for single image should have shape (1, 2048)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# for starters\n",
    "* Train sklearn model, evaluate validation accuracy (should be >80%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extract features from images\n",
    "from tqdm import tqdm\n",
    "from skimage.io import imread\n",
    "import PIL.Image as Image\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "X = []\n",
    "Y = []\n",
    "\n",
    "batch_size = <YOUR VALUE>\n",
    "imgs = np.zeros([batch_size, 299, 299, 3])\n",
    "batch_index = 0\n",
    "\n",
    "for fname in tqdm(os.listdir('train')):\n",
    "    \n",
    "    y = fname.startswith(\"cat\")\n",
    "    Y.append(y)\n",
    "    \n",
    "    img = imread(os.path.join(\"train\", fname))\n",
    "    \n",
    "    img = np.array(Image.fromarray(img).resize((299, 299))) / 255.\n",
    "    imgs[batch_index] = img\n",
    "    \n",
    "    if batch_index == batch_size - 1:\n",
    "        input_tensor = torch.as_tensor(imgs.transpose([0,3,1,2]), dtype=torch.float32)\n",
    "    \n",
    "        # use your embedding model to produce feature vector\n",
    "        features = <YOUR CODE>\n",
    "        \n",
    "        X.append(features)\n",
    "        \n",
    "        batch_index = 0\n",
    "        continue\n",
    "        \n",
    "    batch_index += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.concatenate(X) #stack all [1xfeatures] matrices into one. \n",
    "assert X.ndim == 2\n",
    "#WARNING! the concatenate works for [1xN] matrices. If you have other format, stack them yourself.\n",
    "\n",
    "#crop if we ended prematurely\n",
    "Y = np.array(Y[:len(X)])\n",
    "\n",
    "print(X.shape, Y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<split data here or use cross-validation>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__load our dakka__\n",
    "![img](https://s-media-cache-ak0.pinimg.com/564x/80/a1/81/80a1817a928744a934a7d32e7c03b242.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier,ExtraTreesClassifier,GradientBoostingClassifier,AdaBoostClassifier\n",
    "from sklearn.linear_model import LogisticRegression, RidgeClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<YOUR CODE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Main quest\n",
    "\n",
    "* Get the score improved!\n",
    "* You have to reach __at least 95%__ on the test set. More = better.\n",
    "\n",
    "No methods are illegal: ensembling, data augmentation, NN hacks. \n",
    "Just don't let test data slip into training.\n",
    "\n",
    "\n",
    "### Split the raw image data\n",
    "  * please do train/validation/test instead of just train/test\n",
    "  * reasonable but not optimal split is 20k/2.5k/2.5k or 15k/5k/5k\n",
    "\n",
    "### Build a few layers on top of chosen \"neck\" layers.\n",
    "  * a good idea is to just stack more layers inside the same network\n",
    "  * alternative: stack on top of get_output\n",
    "\n",
    "### Train the newly added layers for some iterations\n",
    "  * you can selectively train some weights by sending the correct parameters in the optimizer\n",
    "      * `opt = torch.optim.Adam([head_only.parameters()])``\n",
    "  * it's cruicial to monitor the network performance at this and following steps\n",
    "\n",
    "### Fine-tune the network body\n",
    "  * probably a good idea to SAVE your new network weights now 'cuz it's easy to mess things up.\n",
    "  * Moreover, saving weights periodically is a no-nonsense idea\n",
    "  * even more cruicial to monitor validation performance\n",
    "  * main network body may need a separate, much lower learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: #deepdream\n",
    "\n",
    "https://twitter.com/search?q=%23deepdream&src=typd\n",
    "\n",
    "Code is heavily based on https://github.com/thesemicolonguy/deep-dream-pytorch\n",
    "\n",
    "Original blogpost where more ideas can be taken from: https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image, ImageFilter, ImageChops\n",
    "from torchvision import transforms\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modulelist = list(model.children())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess = transforms.Compose([\n",
    "#    transforms.Resize((299, 299)),  # do we really need this now?\n",
    "    transforms.ToTensor()#,\n",
    "    ])\n",
    "\n",
    "\n",
    "def dd_helper(image, layer, iterations, lr):\n",
    "    input_var = torch.as_tensor(preprocess(image).unsqueeze(0), requires_grad=True, dtype=torch.float32)\n",
    "    model.zero_grad()\n",
    "    for i in range(iterations):\n",
    "        out = input_var\n",
    "        for j in range(layer):\n",
    "            out = modulelist[j](out)\n",
    "        loss = out.norm()\n",
    "        loss.backward()\n",
    "        input_var.data = input_var.data + lr * input_var.grad.data\n",
    "    \n",
    "    input_im = input_var.data.squeeze()\n",
    "    input_im.transpose_(0, 1)\n",
    "    input_im.transpose_(1, 2)\n",
    "    input_im = np.clip(input_im, 0, 1)\n",
    "    im = Image.fromarray(np.uint8(input_im * 255))\n",
    "    return im"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deep_dream(image, layer, iterations, lr, octave_scale, num_octaves):\n",
    "    if num_octaves>0:\n",
    "        image1 = image.filter(ImageFilter.GaussianBlur(2))\n",
    "        if (image1.size[0] / octave_scale < 1 or image1.size[1] / octave_scale < 1):\n",
    "            size = image1.size\n",
    "        else:\n",
    "            size = (int(image1.size[0] / octave_scale), int(image1.size[1] / octave_scale))\n",
    "            \n",
    "        image1 = image1.resize(size, Image.ANTIALIAS)\n",
    "        image1 = deep_dream(image1, layer, iterations, lr, octave_scale, num_octaves-1)\n",
    "        size = (image.size[0], image.size[1])\n",
    "        image1 = image1.resize(size, Image.ANTIALIAS)\n",
    "        image = ImageChops.blend(image, image1, 0.6)\n",
    "    print(\"-------------- Recursive level: \", num_octaves, '--------------')\n",
    "    img_result = dd_helper(image, layer, iterations, lr)\n",
    "    img_result = img_result.resize(image.size)\n",
    "    plt.imshow(img_result)\n",
    "    return img_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = Image.fromarray(plt.imread('img.jpg'))\n",
    "img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = deep_dream(img, 5, 5, 0.3, 2, 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = deep_dream(img, 12, 5, 0.2, 2, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#can you implement one class probability optimization to make model dream about bananas\n",
    "<YOUR CODE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus 2: Adversarial Attack\n",
    "\n",
    "Original PyTorch tutorial is [here](https://pytorch.org/tutorials/beginner/fgsm_tutorial.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the Deep Dream step function\n",
    "# to make it doing adversarial example from original image\n",
    "\n",
    "\n",
    "def dd_helper_modified(image, layer, iterations, lr):\n",
    "    input_var = torch.as_tensor(preprocess(image).unsqueeze(0), requires_grad=True, dtype=torch.float32)\n",
    "    model.zero_grad()\n",
    "    for i in range(iterations):\n",
    "        out = input_var\n",
    "        for j in range(layer):  # maybe change this\n",
    "            out = modulelist[j](out)\n",
    "        loss = out.norm()\n",
    "        loss.backward()\n",
    "        input_var.data = input_var.data + lr * input_var.grad.data # and probably this\n",
    "    input_im = input_var.data.squeeze()\n",
    "    input_im.transpose_(0, 1)\n",
    "    input_im.transpose_(1, 2)\n",
    "    input_im = np.clip(input_im, 0, 1)\n",
    "    im = Image.fromarray(np.uint8(input_im * 255))\n",
    "    return im"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = Image.fromarray(plt.imread('img.jpg'))\n",
    "img_adv = dd_helper(img, ?, ?, ?)\n",
    "img_adv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict(resize(np.asarray(img_adv), (299, 299)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}