Copy_of_C3_W1_Lab_1_Keras_Tuner.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Copy of C3_W1_Lab_1_Keras_Tuner.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/angelainfanta1312/ML-in-Production/blob/main/Copy_of_C3_W1_Lab_1_Keras_Tuner.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFdPvlXBOdUN"
      },
      "source": [
        "# Ungraded Lab: Intro to Keras Tuner"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "Developing machine learning models is usually an iterative process. You start with an initial design then reconfigure until you get a model that can be trained efficiently in terms of time and compute resources. As you may already know, these settings that you adjust are called _hyperparameters_. These are the variables that govern the training process and the topology of an ML model. These remain constant over the training process and directly impact the performance of your ML program. \n",
        "\n",
        "The process of finding the optimal set of hyperparameters is called *hyperparameter tuning* or *hypertuning*, and it is an essential part of a machine learning pipeline. Without it, you might end up with a model that has unnecessary parameters and take too long to train.\n",
        "\n",
        "Hyperparameters are of two types:\n",
        "1. *Model hyperparameters* which influence model selection such as the number and width of hidden layers\n",
        "\n",
        "2. *Algorithm hyperparameters* which influence the speed and quality of the learning algorithm such as the learning rate for Stochastic Gradient Descent (SGD) and the number of nearest neighbors for a k Nearest Neighbors (KNN) classifier.\n",
        "\n",
        "For more complex models, the number of hyperparameters can increase dramatically and tuning them manually can be quite challenging.\n",
        "\n",
        "In this lab, you will practice hyperparameter tuning with [Keras Tuner](https://keras-team.github.io/keras-tuner/), a package from the Keras team that automates this process. For comparison, you will first train a baseline model with pre-selected hyperparameters, then redo the process with tuned hyperparameters. Some of the examples and discussions here are taken from the [official tutorial provided by Tensorflow](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/keras/keras_tuner.ipynb#scrollTo=sKwLOzKpFGAj) but we've expounded on a few key parts for clarity.\n",
        "\n",
        "Let's begin!\n",
        "\n",
        "**Note: The notebooks in this course are shared with read-only access. To be able to save your work, kindly select File > Save a Copy in Drive from the Colab menu and run the notebook from there. You will need a Gmail account to save a copy.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ReV_UXOgCZvx"
      },
      "source": [
        "## Download and prepare the dataset\n",
        "\n",
        "Let us first load the [Fashion MNIST dataset](https://github.com/zalandoresearch/fashion-mnist) into your workspace. You will use this to train a machine learning model that classifies images of clothing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ysAmHLZoDld7"
      },
      "source": [
        "# Import keras\n",
        "from tensorflow import keras"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OHlHs9Wj_PUM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e77a837-1cee-4abc-e48a-7288032e2f5d"
      },
      "source": [
        "# Download the dataset and split into train and test sets\n",
        "(img_train, label_train), (img_test, label_test) = keras.datasets.fashion_mnist.load_data()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
            "32768/29515 [=================================] - 0s 0us/step\n",
            "40960/29515 [=========================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
            "26427392/26421880 [==============================] - 1s 0us/step\n",
            "26435584/26421880 [==============================] - 1s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
            "16384/5148 [===============================================================================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
            "4423680/4422102 [==============================] - 0s 0us/step\n",
            "4431872/4422102 [==============================] - 0s 0us/step\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nHkQOzHLoKNA"
      },
      "source": [
        "For preprocessing, you will normalize the pixel values to make the training converge faster."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bLVhXs3xrUD0"
      },
      "source": [
        "# Normalize pixel values between 0 and 1\n",
        "img_train = img_train.astype('float32') / 255.0\n",
        "img_test = img_test.astype('float32') / 255.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hM19_JWD6eF"
      },
      "source": [
        "## Baseline Performance\n",
        "\n",
        "As mentioned, you will first have a baseline performance using arbitrarily handpicked parameters so you can compare the results later. In the interest of time and resource limits provided by Colab, you will just build a shallow dense neural network (DNN) as shown below. This is to demonstrate the concepts without involving huge datasets and long tuning and training times. As you'll see later, even small models can take some time to tune. You can extend the concepts here when you get to build more complex models in your own projects. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sqbYwwukkA6z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e484112d-c65d-471d-c232-beca19c57738"
      },
      "source": [
        "# Build the baseline model using the Sequential API\n",
        "b_model = keras.Sequential()\n",
        "b_model.add(keras.layers.Flatten(input_shape=(28, 28)))\n",
        "b_model.add(keras.layers.Dense(units=512, activation='relu', name='dense_1')) # You will tune this layer later\n",
        "b_model.add(keras.layers.Dropout(0.2))\n",
        "b_model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "\n",
        "# Print model summary\n",
        "b_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " flatten (Flatten)           (None, 784)               0         \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 512)               401920    \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 512)               0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 10)                5130      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 407,050\n",
            "Trainable params: 407,050\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WAlb_KxTK50d"
      },
      "source": [
        "As shown, we hardcoded all the hyperparameters when declaring the layers. These include the number of hidden units, activation, and dropout. You will see how you can automatically tune some of these a bit later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RM354GIBKdf0"
      },
      "source": [
        "Let's then setup the loss, metrics, and the optimizer. The learning rate is also a hyperparameter you can tune automatically but for now, let's set it at `0.001`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lp58Ety3pLj2"
      },
      "source": [
        "# Setup the training parameters\n",
        "b_model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001),\n",
        "            loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "            metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FxeAlZlLpHI"
      },
      "source": [
        "With all settings set, you can start training the model. We've set the number of epochs to 10 but feel free to increase it if you have more time to go through the notebook. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K1JjZ-FdLXZ3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "40395feb-70f3-4bb6-88b4-7eee85b94b85"
      },
      "source": [
        "# Number of training epochs.\n",
        "NUM_EPOCHS = 10\n",
        "\n",
        "# Train the model\n",
        "b_model.fit(img_train, label_train, epochs=NUM_EPOCHS, validation_split=0.2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "1500/1500 [==============================] - 7s 3ms/step - loss: 0.5178 - accuracy: 0.8143 - val_loss: 0.3950 - val_accuracy: 0.8516\n",
            "Epoch 2/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3918 - accuracy: 0.8552 - val_loss: 0.3546 - val_accuracy: 0.8736\n",
            "Epoch 3/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3530 - accuracy: 0.8687 - val_loss: 0.3562 - val_accuracy: 0.8712\n",
            "Epoch 4/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3314 - accuracy: 0.8774 - val_loss: 0.3435 - val_accuracy: 0.8734\n",
            "Epoch 5/10\n",
            "1500/1500 [==============================] - 5s 3ms/step - loss: 0.3175 - accuracy: 0.8817 - val_loss: 0.3653 - val_accuracy: 0.8712\n",
            "Epoch 6/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3032 - accuracy: 0.8864 - val_loss: 0.3237 - val_accuracy: 0.8805\n",
            "Epoch 7/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2928 - accuracy: 0.8904 - val_loss: 0.3287 - val_accuracy: 0.8803\n",
            "Epoch 8/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2805 - accuracy: 0.8960 - val_loss: 0.3220 - val_accuracy: 0.8877\n",
            "Epoch 9/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2730 - accuracy: 0.8980 - val_loss: 0.3253 - val_accuracy: 0.8860\n",
            "Epoch 10/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2652 - accuracy: 0.8986 - val_loss: 0.3183 - val_accuracy: 0.8893\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f406027e690>"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S6LALxGwMtkV"
      },
      "source": [
        "Finally, you want to see how this baseline model performs against the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kBnZ2tFbpxgC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "25a1f77a-437e-4107-ae14-d00ef5b2f72c"
      },
      "source": [
        "# Evaluate model on the test set\n",
        "b_eval_dict = b_model.evaluate(img_test, label_test, return_dict=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "313/313 [==============================] - 1s 3ms/step - loss: 0.3432 - accuracy: 0.8809\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9YCfzg0IM9b6"
      },
      "source": [
        "Let's define a helper function for displaying the results so it's easier to compare later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vt2dWs0NxnUn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6008b4dc-18ec-468c-deac-b75759393078"
      },
      "source": [
        "# Define helper function\n",
        "def print_results(model, model_name, eval_dict):\n",
        "  '''\n",
        "  Prints the values of the hyparameters to tune, and the results of model evaluation\n",
        "\n",
        "  Args:\n",
        "    model (Model) - Keras model to evaluate\n",
        "    model_name (string) - arbitrary string to be used in identifying the model\n",
        "    eval_dict (dict) -  results of model.evaluate\n",
        "  '''\n",
        "  print(f'\\n{model_name}:')\n",
        "\n",
        "  print(f'number of units in 1st Dense layer: {model.get_layer(\"dense_1\").units}')\n",
        "  print(f'learning rate for the optimizer: {model.optimizer.lr.numpy()}')\n",
        "\n",
        "  for key,value in eval_dict.items():\n",
        "    print(f'{key}: {value}')\n",
        "\n",
        "# Print results for baseline model\n",
        "print_results(b_model, 'BASELINE MODEL', b_eval_dict)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "BASELINE MODEL:\n",
            "number of units in 1st Dense layer: 512\n",
            "learning rate for the optimizer: 0.0010000000474974513\n",
            "loss: 0.34320804476737976\n",
            "accuracy: 0.8809000253677368\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AH-RLK3Wxt_X"
      },
      "source": [
        "That's it for getting the results for a single set of hyperparameters. As you can see, this process can be tedious if you want to try different sets of parameters. For example, will your model improve if you use `learning_rate=0.00001` and `units=128`? What if `0.001` paired with `256`? The process will be even more difficult if you decide to also tune the dropout and try out other activation functions as well. Keras Tuner solves this problem by having an API to automatically search for the optimal set. You will just need to set it up once then wait for the results. You will see how this is done in the next sections."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7oyczDXqtWjI"
      },
      "source": [
        "## Keras Tuner\n",
        "\n",
        "To perform hypertuning with Keras Tuner, you will need to:\n",
        "\n",
        "* Define the model\n",
        "* Select which hyperparameters to tune\n",
        "* Define its search space\n",
        "* Define the search strategy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "### Install and import packages\n",
        "\n",
        "You will start by installing and importing the required packages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hpMLpbt9jcO6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "626b8b4f-6b73-47a3-9b9a-d1f14fb4c113"
      },
      "source": [
        "# Install Keras Tuner\n",
        "!pip install -q -U keras-tuner"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l\r\u001b[K     |██▌                             | 10 kB 27.1 MB/s eta 0:00:01\r\u001b[K     |█████                           | 20 kB 30.1 MB/s eta 0:00:01\r\u001b[K     |███████▍                        | 30 kB 33.0 MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 40 kB 35.6 MB/s eta 0:00:01\r\u001b[K     |████████████▎                   | 51 kB 37.5 MB/s eta 0:00:01\r\u001b[K     |██████████████▊                 | 61 kB 41.5 MB/s eta 0:00:01\r\u001b[K     |█████████████████▏              | 71 kB 33.0 MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 81 kB 33.8 MB/s eta 0:00:01\r\u001b[K     |██████████████████████          | 92 kB 35.9 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▌       | 102 kB 38.2 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 112 kB 38.2 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▍  | 122 kB 38.2 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▉| 133 kB 38.2 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 133 kB 38.2 MB/s \n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_leAIdFKAxAD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c75b4aa-0c0b-4227-f738-022f5a83fb75"
      },
      "source": [
        "# Import required packages\n",
        "import tensorflow as tf\n",
        "import kerastuner as kt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:3: DeprecationWarning: `import kerastuner` is deprecated, please use `import keras_tuner`.\n",
            "  This is separate from the ipykernel package so we can avoid doing imports until\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5YEL2H2Ax3e"
      },
      "source": [
        "### Define the model\n",
        "\n",
        "The model you set up for hypertuning is called a *hypermodel*. When you build this model, you define the hyperparameter search space in addition to the model architecture. \n",
        "\n",
        "You can define a hypermodel through two approaches:\n",
        "\n",
        "* By using a model builder function\n",
        "* By [subclassing the `HyperModel` class](https://keras-team.github.io/keras-tuner/#you-can-use-a-hypermodel-subclass-instead-of-a-model-building-function) of the Keras Tuner API\n",
        "\n",
        "\n",
        "In this lab, you will take the first approach: you will use a model builder function to define the image classification model. This function returns a compiled model and uses hyperparameters you define inline to hypertune the model. \n",
        "\n",
        "The function below basically builds the same model you used earlier. The difference is there are two hyperparameters that are setup for tuning:\n",
        "\n",
        "* the number of hidden units of the first Dense layer\n",
        "* the learning rate of the Adam optimizer\n",
        "\n",
        "You will see that this is done with a HyperParameters object which configures the hyperparameter you'd like to tune. For this exercise, you will: \n",
        "\n",
        "* use its `Int()` method to define the search space for the Dense units. This allows you to set a minimum and maximum value, as well as the step size when incrementing between these values. \n",
        "\n",
        "* use its `Choice()` method for the learning rate. This allows you to define discrete values to include in the search space when hypertuning.\n",
        "\n",
        "You can view all available methods and its sample usage in the [official documentation](https://keras-team.github.io/keras-tuner/documentation/hyperparameters/#hyperparameters)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZQKodC-jtsva"
      },
      "source": [
        "def model_builder(hp):\n",
        "  '''\n",
        "  Builds the model and sets up the hyperparameters to tune.\n",
        "\n",
        "  Args:\n",
        "    hp - Keras tuner object\n",
        "\n",
        "  Returns:\n",
        "    model with hyperparameters to tune\n",
        "  '''\n",
        "\n",
        "  # Initialize the Sequential API and start stacking the layers\n",
        "  model = keras.Sequential()\n",
        "  model.add(keras.layers.Flatten(input_shape=(28, 28)))\n",
        "\n",
        "  # Tune the number of units in the first Dense layer\n",
        "  # Choose an optimal value between 32-512\n",
        "  hp_units = hp.Int('units', min_value=32, max_value=512, step=32)\n",
        "  model.add(keras.layers.Dense(units=hp_units, activation='relu', name='dense_1'))\n",
        "\n",
        "  # Add next layers\n",
        "  model.add(keras.layers.Dropout(0.2))\n",
        "  model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "\n",
        "  # Tune the learning rate for the optimizer\n",
        "  # Choose an optimal value from 0.01, 0.001, or 0.0001\n",
        "  hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])\n",
        "\n",
        "  model.compile(optimizer=keras.optimizers.Adam(learning_rate=hp_learning_rate),\n",
        "                loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "                metrics=['accuracy'])\n",
        "\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0J1VYw4q3x0b"
      },
      "source": [
        "## Instantiate the Tuner and perform hypertuning\n",
        "\n",
        "Now that you have the model builder, you can then define how the tuner can find the optimal set of hyperparameters, also called the search strategy. Keras Tuner has [four tuners](https://keras-team.github.io/keras-tuner/documentation/tuners/) available with built-in strategies - `RandomSearch`, `Hyperband`, `BayesianOptimization`, and `Sklearn`. \n",
        "\n",
        "In this tutorial, you will use the Hyperband tuner. Hyperband is an algorithm specifically developed for hyperparameter optimization. It uses adaptive resource allocation and early-stopping to quickly converge on a high-performing model. This is done using a sports championship style bracket wherein the algorithm trains a large number of models for a few epochs and carries forward only the top-performing half of models to the next round. You can read about the intuition behind the algorithm in section 3 of [this paper](https://arxiv.org/pdf/1603.06560.pdf).\n",
        "\n",
        "Hyperband determines the number of models to train in a bracket by computing 1 + log<sub>`factor`</sub>(`max_epochs`) and rounding it up to the nearest integer. You will see these parameters (i.e. `factor` and `max_epochs` passed into the initializer below). In addition, you will also need to define the following to instantiate the Hyperband tuner:\n",
        "\n",
        "* the hypermodel (built by your model builder function)\n",
        "* the `objective` to optimize (e.g. validation accuracy)\n",
        "* a `directory` to save logs and checkpoints for every trial (model configuration) run during the hyperparameter search. If you re-run the hyperparameter search, the Keras Tuner uses the existing state from these logs to resume the search. To disable this behavior, pass an additional `overwrite=True` argument while instantiating the tuner.\n",
        "* the `project_name` to differentiate with other runs. This will be used as a subdirectory name under the `directory`.\n",
        "\n",
        "You can refer to the [documentation](https://keras.io/api/keras_tuner/tuners/hyperband/) for other arguments you can pass in."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oichQFly6Y46"
      },
      "source": [
        "# Instantiate the tuner\n",
        "tuner = kt.Hyperband(model_builder,\n",
        "                     objective='val_accuracy',\n",
        "                     max_epochs=10,\n",
        "                     factor=3,\n",
        "                     directory='kt_dir',\n",
        "                     project_name='angelas_kt_hyperband')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ij3hGcp4e8QG"
      },
      "source": [
        "Let's see a summary of the hyperparameters that you will tune:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JmkJOPp5WkiG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "438a76aa-b971-4a8a-b516-48490ef9408c"
      },
      "source": [
        "# Display hypertuning settings\n",
        "tuner.search_space_summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Search space summary\n",
            "Default search space size: 2\n",
            "units (Int)\n",
            "{'default': None, 'conditions': [], 'min_value': 32, 'max_value': 512, 'step': 32, 'sampling': None}\n",
            "learning_rate (Choice)\n",
            "{'default': 0.01, 'conditions': [], 'values': [0.01, 0.001, 0.0001], 'ordered': True}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cwhBdXx0Ekj8"
      },
      "source": [
        "You can pass in a callback to stop training early when a metric is not improving. Below, we define an [EarlyStopping](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) callback to monitor the validation loss and stop training if it's not improving after 5 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WT9IkS9NEjLc"
      },
      "source": [
        "stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UKghEo15Tduy"
      },
      "source": [
        "You will now run the hyperparameter search. The arguments for the search method are the same as those used for `tf.keras.model.fit` in addition to the callback above. This will take around 10 minutes to run."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dSBQcTHF9cKt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fd2d8fb6-05f4-4ad9-8d01-0f7d6c40c820"
      },
      "source": [
        "# Perform hypertuning\n",
        "tuner.search(img_train, label_train, epochs=NUM_EPOCHS, validation_split=0.2, callbacks=[stop_early])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Trial 30 Complete [00h 00m 43s]\n",
            "val_accuracy: 0.8840000033378601\n",
            "\n",
            "Best val_accuracy So Far: 0.8861666917800903\n",
            "Total elapsed time: 00h 10m 44s\n",
            "INFO:tensorflow:Oracle triggered exit\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ewN6WBDYWvRw"
      },
      "source": [
        "You can get the top performing model with the [get_best_hyperparameters()](https://keras-team.github.io/keras-tuner/documentation/tuners/#get_best_hyperparameters-method)  method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iG0zIuP5WuTI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc525d5f-4f43-4d90-ffa4-abc6cab10cb3"
      },
      "source": [
        "# Get the optimal hyperparameters from the results\n",
        "best_hps=tuner.get_best_hyperparameters()[0]\n",
        "\n",
        "print(f\"\"\"\n",
        "The hyperparameter search is complete. The optimal number of units in the first densely-connected\n",
        "layer is {best_hps.get('units')} and the optimal learning rate for the optimizer\n",
        "is {best_hps.get('learning_rate')}.\n",
        "\"\"\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "The hyperparameter search is complete. The optimal number of units in the first densely-connected\n",
            "layer is 288 and the optimal learning rate for the optimizer\n",
            "is 0.001.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lak_ylf88xBv"
      },
      "source": [
        "## Build and train the model\n",
        "\n",
        "Now that you have the best set of hyperparameters, you can rebuild the hypermodel with these values and retrain it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "McO82AXOuxXh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1e595e26-42aa-4c3c-ac1d-759623b082aa"
      },
      "source": [
        "# Build the model with the optimal hyperparameters\n",
        "h_model = tuner.hypermodel.build(best_hps)\n",
        "h_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " flatten_2 (Flatten)         (None, 784)               0         \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 288)               226080    \n",
            "                                                                 \n",
            " dropout_2 (Dropout)         (None, 288)               0         \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 10)                2890      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 228,970\n",
            "Trainable params: 228,970\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l64WP7Rau1lm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d7163ca6-6d9e-4884-b014-310cbd48b52d"
      },
      "source": [
        "# Train the hypertuned model\n",
        "h_model.fit(img_train, label_train, epochs=NUM_EPOCHS, validation_split=0.2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.5261 - accuracy: 0.8113 - val_loss: 0.4109 - val_accuracy: 0.8560\n",
            "Epoch 2/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3965 - accuracy: 0.8546 - val_loss: 0.3910 - val_accuracy: 0.8533\n",
            "Epoch 3/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3626 - accuracy: 0.8666 - val_loss: 0.3539 - val_accuracy: 0.8748\n",
            "Epoch 4/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3402 - accuracy: 0.8752 - val_loss: 0.3398 - val_accuracy: 0.8769\n",
            "Epoch 5/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3234 - accuracy: 0.8807 - val_loss: 0.3588 - val_accuracy: 0.8711\n",
            "Epoch 6/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.3091 - accuracy: 0.8855 - val_loss: 0.3248 - val_accuracy: 0.8852\n",
            "Epoch 7/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2967 - accuracy: 0.8889 - val_loss: 0.3303 - val_accuracy: 0.8818\n",
            "Epoch 8/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2896 - accuracy: 0.8915 - val_loss: 0.3214 - val_accuracy: 0.8845\n",
            "Epoch 9/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2783 - accuracy: 0.8962 - val_loss: 0.3262 - val_accuracy: 0.8846\n",
            "Epoch 10/10\n",
            "1500/1500 [==============================] - 4s 3ms/step - loss: 0.2664 - accuracy: 0.9011 - val_loss: 0.3285 - val_accuracy: 0.8853\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f3fe2a58a10>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MqU5ZVAaag2v"
      },
      "source": [
        "You will then get its performance against the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9E0BTp9Ealjb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "372a6aa0-5230-448a-d9d3-843a72267e8c"
      },
      "source": [
        "# Evaluate the hypertuned model against the test set\n",
        "h_eval_dict = h_model.evaluate(img_test, label_test, return_dict=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "313/313 [==============================] - 1s 3ms/step - loss: 0.3590 - accuracy: 0.8778\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQRpPHZsz-eC"
      },
      "source": [
        "We can compare the results we got with the baseline model we used at the start of the notebook. Results may vary but you will usually get a model that has less units in the dense layer, while having comparable loss and accuracy. This indicates that you reduced the model size and saved compute resources while still having more or less the same accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BjVYPOw6MH5d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "528b1633-824d-4ac2-d410-768ddbdf983b"
      },
      "source": [
        "# Print results of the baseline and hypertuned model\n",
        "print_results(b_model, 'BASELINE MODEL', b_eval_dict)\n",
        "print_results(h_model, 'HYPERTUNED MODEL', h_eval_dict)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "BASELINE MODEL:\n",
            "number of units in 1st Dense layer: 512\n",
            "learning rate for the optimizer: 0.0010000000474974513\n",
            "loss: 0.34320804476737976\n",
            "accuracy: 0.8809000253677368\n",
            "\n",
            "HYPERTUNED MODEL:\n",
            "number of units in 1st Dense layer: 288\n",
            "learning rate for the optimizer: 0.0010000000474974513\n",
            "loss: 0.3590010106563568\n",
            "accuracy: 0.8777999877929688\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKn4g_HzP2KS"
      },
      "source": [
        "## Bonus Challenges (optional)\n",
        "\n",
        "If you want to keep practicing with Keras Tuner in this notebook, you can do a factory reset (`Runtime > Factory reset runtime`) and take on any of the following:\n",
        "\n",
        "- hypertune the dropout layer with `hp.Float()` or `hp.Choice()`\n",
        "- hypertune the activation function of the 1st dense layer with `hp.Choice()`\n",
        "- determine the optimal number of Dense layers you can add to improve the model. You can use the code [here](https://keras.io/guides/keras_tuner/getting_started/#the-search-space-may-contain-conditional-hyperparameters) as reference.\n",
        "- explore pre-defined `HyperModel` classes - [HyperXception and HyperResNet](https://keras-team.github.io/keras-tuner/documentation/hypermodels/#hyperresnet-class) for computer vision applications."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sKwLOzKpFGAj"
      },
      "source": [
        "## Wrap Up\n",
        "\n",
        "In this tutorial, you used Keras Tuner to conveniently tune hyperparameters. You defined which ones to tune, the search space, and search strategy to arrive at the optimal set of hyperparameters. These concepts will again be discussed in the next sections but in the context of AutoML, a package that automates the entire machine learning pipeline. On to the next!\n"
      ]
    }
  ]
}