2-cars-classification-advancedmodel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O7xCwK47U1fA"
   },
   "source": [
    "# Importing libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we need to import some external libraries and our own helper files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SZ-QxVjNU1fC"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing libraries\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "print(\"Importing libraries\")\n",
    "# system libraries\n",
    "import os\n",
    "import logging\n",
    "\n",
    "# third party libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "# Own helper files\n",
    "from ml_webinar import preprocessing, evaluate, plots\n",
    "\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xxwgjKykU1fG"
   },
   "source": [
    "# File input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OpKyhp9HU1fH"
   },
   "source": [
    "Next, we indicate where our data can be found, several variables for the algorithm, and the correct labels for our data. Since we use this script for several demo's with different datasets, we indicate what the variable DATA_FOLDER should be for each demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L1-FAhHzNk_f"
   },
   "outputs": [],
   "source": [
    "#Quiz 2: 'data/data_rotation_orig1'\n",
    "#Quiz 3: 'data/data_rotation_orig2'\n",
    "#Quiz 4: 'data/data_rotation_orig3'\n",
    "DATA_FOLDER = 'data/data_rotation_orig3'\n",
    "IMG_PIX = 256\n",
    "\n",
    "if DATA_FOLDER == \"data/data_colors\":\n",
    "    CLASS_DICT = {\n",
    "        \"c_black\": {\"index\": 0, \"name\": \"Black\"},\n",
    "        \"c_blue\": {\"index\": 1, \"name\": \"Blue\"},\n",
    "        \"c_red\": {\"index\": 2, \"name\": \"Red\"},\n",
    "        \"c_white\": {\"index\": 3, \"name\": \"White\"}\n",
    "    }\n",
    "else:\n",
    "    CLASS_DICT = {\n",
    "        \"F\": {\"index\": 0, 'name': \"Front\"},\n",
    "        \"B\": {\"index\": 1, 'name': \"Back\"}, \n",
    "        \"R\": {\"index\": 2, 'name': \"Right\"},\n",
    "        \"L\": {\"index\": 3, 'name': \"Left\"},\n",
    "        \"FR\": {\"index\": 4, 'name': \"Front Right\"},\n",
    "        \"FL\": {\"index\": 5, 'name': \"Front Left\"},\n",
    "    }\n",
    "    \n",
    "    \n",
    "EPOCHS = {\n",
    "    'data/data_rotation_orig1': 5,\n",
    "    'data/data_rotation_orig2': 5,\n",
    "    'data/data_rotation_orig3': 20,\n",
    "    'data/data_colors': 10\n",
    "}.get(DATA_FOLDER)\n",
    "\n",
    "if DATA_FOLDER in ['data/data_rotation_orig3', 'data/data_colors']:\n",
    "    WIDTH_SHIFT, HEIGHT_SHIFT = 0.1, 0.1\n",
    "else:\n",
    "    WIDTH_SHIFT, HEIGHT_SHIFT = 0, 0\n",
    "    \n",
    "SEED = 1234\n",
    "np.random.seed(seed=SEED)\n",
    "tf.random.set_seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6_66B7syU1fV"
   },
   "source": [
    "# Creating a training set and test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pictures in the dataset need to be reformatted before they can be used: we rescale the pixel values, indicate we want the RGB color model, and assign the images to either the training set or test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_generator(phase, width_shift=0, height_shift=0):\n",
    "    image_gen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255,\n",
    "                                                            width_shift_range=height_shift,\n",
    "                                                            height_shift_range=width_shift)\n",
    "    return image_gen.flow_from_directory(directory=os.path.join(DATA_FOLDER, phase),\n",
    "                                                     batch_size=32,\n",
    "                                                     shuffle=True,\n",
    "                                                     target_size=(IMG_PIX, IMG_PIX),\n",
    "                                                     classes=list(CLASS_DICT.keys()),\n",
    "                                                     class_mode='sparse',\n",
    "                                                     color_mode=\"rgb\",\n",
    "                                                     seed=SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Executing this function for the test set and training set shows us how many images end up in each set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1473 images belonging to 6 classes.\n",
      "Found 134 images belonging to 6 classes.\n"
     ]
    }
   ],
   "source": [
    "train_data_gen = make_generator('train', width_shift=WIDTH_SHIFT, height_shift=HEIGHT_SHIFT)\n",
    "test_data_gen = make_generator('test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vZ4XPbt6U1fY"
   },
   "source": [
    "# Building the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the data preparation is done, we can start building the model. In this example, we use a more advanced Neural Network with seven middle layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2MWJuo9XU1fY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compiling the model\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "cv_model = keras.Sequential([\n",
    "    keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_PIX, IMG_PIX, 3), name='block1_conv1'),\n",
    "    keras.layers.MaxPooling2D((2, 2), name='block1_maxpool'),\n",
    "    keras.layers.Conv2D(64, (3, 3), activation='relu', name='block2_conv1'),\n",
    "    keras.layers.MaxPooling2D((2, 2), name='block2_maxpool'),\n",
    "    keras.layers.Conv2D(32, (3, 3), activation='relu', name='block3_pool'),\n",
    "    keras.layers.Flatten(name='flatten_6'),\n",
    "    keras.layers.Dense(64, activation='relu', name='fc1'),\n",
    "    keras.layers.Dense(len(CLASS_DICT.keys()), name='fc2'),\n",
    "])\n",
    "\n",
    "print(\"Compiling the model\")\n",
    "cv_model.compile(optimizer='adam', \n",
    "                 loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), \n",
    "                 metrics=['accuracy'])\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can ask TensorFlow to give us some more information about the model we just created. In the webinar, however, we already show this in the slides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cv_model.summary()\n",
    "#keras.utils.plot_model(cv_model, to_file='reports/model_2.png', show_shapes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h9J0u9nCU1fa"
   },
   "source": [
    "# Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are finally at the point where we can feed the model data to teach it to classify our images correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hKs6Z8B8U1fa",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "47/47 [==============================] - 105s 2s/step - loss: 2.0219 - accuracy: 0.2396\n",
      "Epoch 2/20\n",
      "47/47 [==============================] - 96s 2s/step - loss: 1.6764 - accuracy: 0.3014\n",
      "Epoch 3/20\n",
      "47/47 [==============================] - 94s 2s/step - loss: 1.6228 - accuracy: 0.3157\n",
      "Epoch 4/20\n",
      "47/47 [==============================] - 97s 2s/step - loss: 1.5144 - accuracy: 0.3659\n",
      "Epoch 5/20\n",
      "47/47 [==============================] - 93s 2s/step - loss: 1.3379 - accuracy: 0.4474\n",
      "Epoch 6/20\n",
      "39/47 [=======================>......] - ETA: 16s - loss: 1.2632 - accuracy: 0.4610"
     ]
    }
   ],
   "source": [
    "cv_model.fit(train_data_gen, epochs=EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we let the model train on the training data several times, we can add the final output layer which allows us to make easier predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "probability_model = tf.keras.Sequential([cv_model, tf.keras.layers.Softmax(name='softmax_9')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like before, we can ask TensorFlow to give us some more information about the (now trained) model. This can be useful for debugging, but is irrelevant for the webinar demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#keras.utils.plot_model(probability_model, to_file='reports/model_2prob.png', show_shapes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a trained model, we can see how well the model performs when we make it pretend the training set is the test set. This allows us to discover problems such as overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_images, train_labels = evaluate.get_images('train', DATA_FOLDER, CLASS_DICT, IMG_PIX)\n",
    "insample_classification = probability_model.predict(train_images)\n",
    "df_confusion_train = evaluate.make_confusion_matrix(train_labels, insample_classification, CLASS_DICT)\n",
    "df_confusion_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classification performance a model gets on the data that is used to train the model is called the train accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_accuracy = evaluate.get_accuracy(df_confusion_train)\n",
    "print(' train accuracy %3.2f' % train_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k0c5c64LU1fd"
   },
   "source": [
    "# Making predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With all the training out of the way, we can feed the Neural Network test data. The algorithm has not seen these images before, but it can use the correct labels to determine the overall accuracy of its predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_model.evaluate(test_data_gen)\n",
    "test_images, test_labels = evaluate.get_images('test', DATA_FOLDER, CLASS_DICT, IMG_PIX)\n",
    "predictions = probability_model.predict(test_images)\n",
    "\n",
    "df_confusion_test = evaluate.make_confusion_matrix(test_labels, predictions, CLASS_DICT)\n",
    "df_confusion_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fVuFEAylU1fd"
   },
   "source": [
    "The classification performance a model gets on the test set data is called the test accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_accuracy = evaluate.get_accuracy(df_confusion_test)\n",
    "print(' test accuracy %3.2f' % test_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8Lupdd0SU1fi"
   },
   "source": [
    "# Graphing the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UH6QVyABU1fl"
   },
   "outputs": [],
   "source": [
    "from random import randrange\n",
    "idx_img = randrange(len(predictions))\n",
    "plots.plot_image_and_predictions(test_images[idx_img], test_labels[idx_img], predictions[idx_img], CLASS_DICT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ml6Zmu_Y7E_U"
   },
   "source": [
    "In the plot below, Blue means a correct prediction, and Red means an incorrect prediction. The label below each image shows what the model predicted, and what the actual label is. Can you spot some improvements we can make to the model or dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CJwl0cpp7Itz",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plots.plot_multiple_images_and_predictions(predictions, test_labels, test_images, CLASS_DICT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acknowledgements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Copyright (c) 2017 François Chollet\n",
    "#\n",
    "# Permission is hereby granted, free of charge, to any person obtaining a\n",
    "# copy of this software and associated documentation files (the \"Software\"),\n",
    "# to deal in the Software without restriction, including without limitation\n",
    "# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
    "# and/or sell copies of the Software, and to permit persons to whom the\n",
    "# Software is furnished to do so, subject to the following conditions:\n",
    "#\n",
    "# The above copyright notice and this permission notice shall be included in\n",
    "# all copies or substantial portions of the Software.\n",
    "#\n",
    "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
    "# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
    "# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
    "# DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "2020 Summer School V3 Joost.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}