LSTM_1_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LSTM 1.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Wxtc9pILfBe",
        "colab_type": "text"
      },
      "source": [
        "## Water Extraction Forecasting - **Daily Frequency**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rS4XwfAZfSM_",
        "colab_type": "text"
      },
      "source": [
        "Walkthrough video explaining step by step how to reproduce the notebook (if desired):\n",
        "https://www.loom.com/share/5b87cb54e4ee4362a88fafb309b635a5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3GdNvJ-MzHC",
        "colab_type": "text"
      },
      "source": [
        "The data includes the date-time from 2013 to 2018 in a daily frequency. \n",
        "\n",
        "The complete feature list is as follows:\n",
        "\n",
        " \n",
        "* Date: Day - Month - Year format\n",
        "* Q: Volumetric flowrate used during that day from the natural sources to provide the city with the demanded water [DESIRED FORECAST FEATURE] \n",
        "* DHIAv: Direct Horizontal Irradiance - Average\n",
        "* DHIMax: Direct Horizontal Irradiance - Max \n",
        "* DNIAv: Direct Normal Irradiance - Average\n",
        "* DNIMax: Direct Normal Irradiance - Max\n",
        "* GHIAv: Global Horizontal Irradiance - Average\n",
        "* GHIMax: Global Horizontal Irradiance - Max\n",
        "* DPMin: Dew Point - Min\n",
        "* DPAv: Dew Point - Average\n",
        "* DPMax: Dew Point - Max\n",
        "* WSMin: Wind Speed - Min\n",
        "* WSAv: Wind Speed - Average\n",
        "* WSMax: Wind Speed - Max\n",
        "* RainMin: Rain - Min\n",
        "* RainAv: Rain - Average\n",
        "* RainMax: Rain - Max\n",
        "* RHMin: Relative Humidity - Min\n",
        "* RHAv: Relative Humidity - Average\n",
        "* RHMax: Relative Humidity - Max\n",
        "* Tmin: Temperature - Min\n",
        "* TAv: Temperature - Average\n",
        "* TMax: Temperature - Max\n",
        "* PMin: Pressure - Min\n",
        "* PAv: Pressure - Average\n",
        "* PMax: Pressure - Max\n",
        "* Weekday: Weekday (1: yes) \n",
        "* Weekend: Weekend (1: yes)\n",
        "* Festive: Festive (free labor day) (1: yes)\n",
        "* year: Year, goes from 2013 to 2018\n",
        "* day_of_month: Month, goes from 1 to 31\n",
        "* day_of_week: Goes from 1 to 7\n",
        "* month: Month, goes from 1 to 12\n",
        "\n",
        "Total data points: 2191\n",
        "Total features: 32\n",
        "\n",
        "Based on:\n",
        "* https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6brkRao6bsEQ",
        "colab_type": "code",
        "outputId": "cb42016e-37f7-4b74-aebb-1157900bddfe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "#Basic libraries\n",
        "from numpy import concatenate\n",
        "from matplotlib import pyplot\n",
        "from datetime import datetime\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from math import sqrt\n",
        "\n",
        "#Pandas\n",
        "import pandas as pd\n",
        "from pandas import read_csv\n",
        "from pandas import DataFrame\n",
        "from pandas import concat\n",
        "\n",
        "#Sklearn\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.metrics import mean_squared_error\n",
        "\n",
        "#Tensorflow\n",
        "import tensorflow as tf\n",
        "\n",
        "#Seaborn for graphs\n",
        "import seaborn as sns \n",
        "%matplotlib inline\n",
        "%config InlineBackend.figure_format='retina'\n",
        "sns.set(style='whitegrid', palette='muted', font_scale=1.5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6aL02mBlOhMb",
        "colab_type": "text"
      },
      "source": [
        "##Connection of our Google Drive to the notebook\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VyXdU7gybyHE",
        "colab_type": "code",
        "outputId": "218fd762-90f7-482c-fa1e-0a942c68ddde",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CDHxSVKWOvHQ",
        "colab_type": "text"
      },
      "source": [
        "####This root path is specific for OUR GOOGLE DRIVE folder. If you want to reproduce this notebook, change root_path to where datasets are stored. We will provide these datasets in our github. You can download them, and upload them to a folder in your google drive for testing. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WhDeB7hIb2pp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "root_path = '/content/gdrive/Shared drives/AguaYDrenaje/Datasets'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r8d1G6EMb9g2",
        "colab_type": "code",
        "outputId": "a89ed700-f0ae-4a89-8661-e09eab2ed4cc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 407
        }
      },
      "source": [
        "dataset = read_csv(root_path+\"/postQData2.csv\", parse_dates=['date'], index_col='date')\n",
        "dataset"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-4-3105645d2cb0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdataset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mroot_path\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m\"/postQData2.csv\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparse_dates\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'date'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex_col\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'date'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    674\u001b[0m         )\n\u001b[1;32m    675\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 676\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    678\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    446\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    447\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 448\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp_or_buf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    449\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    450\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    878\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 880\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    881\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    882\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1112\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"c\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1113\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"c\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1114\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1115\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1116\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"python\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   1889\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"usecols\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0musecols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1890\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1891\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1892\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1893\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] File /content/gdrive/Shared drives/AguaYDrenaje/Datasets/postQData2.csv does not exist: '/content/gdrive/Shared drives/AguaYDrenaje/Datasets/postQData2.csv'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ViOrDZ9tPQxp",
        "colab_type": "text"
      },
      "source": [
        "##LSTM Data Preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hkZQWHjOPbOh",
        "colab_type": "text"
      },
      "source": [
        "This involves framing the dataset as a supervised learning problem and normalizing the input variables.\n",
        "\n",
        "### Important: We will frame the supervised learning problem as predicting the Q (volumetric flowrate) of the current day (t) given today's features AND yesterday's Q. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-8_p55uldIph",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# convert series to supervised learning\n",
        "def series_to_supervised(data, n_in=1, n_out=1, dropnan=True):\n",
        "    n_vars = 1 if type(data) is list else data.shape[1]\n",
        "    df = DataFrame(data)\n",
        "    cols, names = list(), list()\n",
        "    # input sequence (t-n, ... t-1)\n",
        "    for i in range(n_in, 0, -1):\n",
        "        cols.append(df.shift(i))\n",
        "        names += [('var%d(t-%d)' % (j+1, i)) for j in range(n_vars)]\n",
        "    # forecast sequence (t, t+1, ... t+n)\n",
        "    for i in range(0, n_out):\n",
        "        cols.append(df.shift(-i))\n",
        "        if i == 0:\n",
        "            names += [('var%d(t)' % (j+1)) for j in range(n_vars)]\n",
        "        else:\n",
        "            names += [('var%d(t+%d)' % (j+1, i)) for j in range(n_vars)]\n",
        "    # put it all together\n",
        "    agg = concat(cols, axis=1)\n",
        "    agg.columns = names\n",
        "    # drop rows with NaN values\n",
        "    if dropnan:\n",
        "        agg.dropna(inplace=True)\n",
        "    return agg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zzt_b4BedP6r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Change dataframe into array for faster processing\n",
        "values = dataset.values\n",
        "\n",
        "# ensure all data is float\n",
        "values = values.astype('float32')\n",
        "\n",
        "# normalize features\n",
        "scaler = MinMaxScaler(feature_range=(0, 1))\n",
        "scaled = scaler.fit(values)\n",
        "scaled = scaler.transform(values)\n",
        "\n",
        "# frame as supervised learning\n",
        "reframed = series_to_supervised(scaled, 1, 1)\n",
        "\n",
        "# drop columns we don't want to predict\n",
        "reframed.drop(reframed.columns[[33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]], axis=1, inplace=True)\n",
        "\n",
        "print(reframed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rnrqL7OxP-vz",
        "colab_type": "text"
      },
      "source": [
        "##Define and Fit the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jLMuFVM_QB3o",
        "colab_type": "text"
      },
      "source": [
        "In this section, we will fit a LSTM on the multivariate input data.\n",
        "\n",
        "First, we must split the prepared dataset into train and test sets. For this demonstration, we will fit the model on the first 5 years of data, then evaluate it on the remaining 1 year of data.\n",
        "\n",
        "The example below splits the dataset into train and test sets, then splits the train and test sets into input and output variables. Finally, the inputs (X) are reshaped into the 3D format expected by LSTMs, namely [samples, timesteps, features]."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHB48AomQW3Y",
        "colab_type": "text"
      },
      "source": [
        "### Split into train and test sets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1049gTSEd7YT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# split into train and test sets\n",
        "values = reframed.values\n",
        "n_train_days = 365*5\n",
        "train = values[:n_train_days, :]\n",
        "test = values[n_train_days:, :]\n",
        "\n",
        "# split into input and outputs\n",
        "X_train, y_train = train[:, :-1], train[:, -1]\n",
        "X_test, y_test = test[:, :-1], test[:, -1]\n",
        "\n",
        "# reshape input to be 3D [samples, timesteps, features]\n",
        "X_train = X_train.reshape((X_train.shape[0], 1, X_train.shape[1]))\n",
        "X_test = X_test.reshape((X_test.shape[0], 1, X_test.shape[1]))\n",
        "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFY_w5H9Qc7Q",
        "colab_type": "text"
      },
      "source": [
        "We will define the LSTM with **70 neurons** in the **first hidden layer** and **1 neuron** in the **output** layer for predicting Q. The input shape will be 1 time step with 32 features.\n",
        "\n",
        "We will use the **Mean Absolute Error (MAE)** loss function and the efficient **Adam version of stochastic gradient descent**.\n",
        "\n",
        "The model will be fit for **50 training epochs** with a **batch size of True**. Remember that the internal state of the LSTM in Keras is reset at the end of each batch, so an internal state that is a function of a number of days may be helpful (try testing this).\n",
        "\n",
        "Finally, we keep track of both the training and test loss during training by setting the validation_data argument in the fit() function. At the end of the run both the training and test loss are plotted."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d8r0KgKud_f4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# design network\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.LSTM(70, input_shape=(X_train.shape[1], X_train.shape[2])))\n",
        "model.add(tf.keras.layers.Dense(1))\n",
        "model.compile(loss='mae', optimizer='adam')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "11b4qQpMeBBg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# fit network\n",
        "history = model.fit(X_train, y_train, epochs=50, batch_size=True, validation_data=(X_test, y_test), verbose=False, shuffle=False)\n",
        "\n",
        "# plot history\n",
        "pyplot.plot(history.history['loss'], label='train')\n",
        "pyplot.plot(history.history['val_loss'], label='test')\n",
        "plt.xlabel(\"epochs\")\n",
        "plt.ylabel(\"loss\")\n",
        "pyplot.legend()\n",
        "pyplot.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7A2fG9gQs84",
        "colab_type": "text"
      },
      "source": [
        "Running the example first creates a plot showing the train and test loss during training.\n",
        "\n",
        "Interestingly, we can see that test loss is above training loss. This suggests that the model is NOT overfitting the data. \n",
        "\n",
        "The Train and test loss are printed at the end of each training epoch. At the end of the run, the final RMSE of the model on the test dataset is printed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2B2euVGNRGRA",
        "colab_type": "text"
      },
      "source": [
        "## Prediction & Reverse Scaling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pjUC2u7JeCrr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# make a prediction\n",
        "y_pred = model.predict(X_test)\n",
        "X_test = X_test.reshape((X_test.shape[0], X_test.shape[2]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bgXSek8leIO2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# invert scaling for forecast\n",
        "y_pred_inv = concatenate((y_pred, X_test[:, 1:]), axis=1)\n",
        "y_pred_inv = scaler.inverse_transform(y_pred_inv)\n",
        "y_pred_inv = y_pred_inv[:,0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n4VPA_iQeJa_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# invert scaling for actual\n",
        "y_test = y_test.reshape((len(y_test), 1))\n",
        "y_inv = concatenate((y_test, X_test[:, 1:]), axis=1)\n",
        "y_inv = scaler.inverse_transform(y_inv)\n",
        "y_inv = y_inv[:,0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wl34KAl3RCsE",
        "colab_type": "text"
      },
      "source": [
        "## Evaluate the model\n",
        "\n",
        "After the model is fit, we can forecast for the entire test dataset.\n",
        "\n",
        "We combine the forecast with the test dataset and invert the scaling. We also invert scaling on the test dataset with the expected pollution numbers.\n",
        "\n",
        "With forecasts and actual values in their original scale, we can then calculate an error score for the model. In this case, we calculate the Root Mean Squared Error (RMSE) that gives error in the same units as the variable itself."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BdJ-4rlFeLDu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# calculate RMSE\n",
        "rmse = sqrt(mean_squared_error(y_inv, y_pred_inv))\n",
        "print('Test RMSE: %.3f' % rmse)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7X06cS87RSyN",
        "colab_type": "text"
      },
      "source": [
        "## Q predicted & real comparison"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hIjSpP7MeQ7U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Let's get together predicted and real into a DataFrame for easier comparison\n",
        "y_comparison = pd.DataFrame({\"Q real\": y_inv,\n",
        "                            \"Q forecast\": y_pred_inv})\n",
        "y_comparison.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAuQ8WSFRf6q",
        "colab_type": "text"
      },
      "source": [
        "## Plotting results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FcPQ-n8jeSU7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.figure(figsize=(14, 8))\n",
        "plt.title(\"Q real vs Q forecast\")\n",
        "plt.plot(y_comparison[\"Q real\"], label = \"Q real\")\n",
        "plt.plot(y_comparison[\"Q forecast\"], label = \"Q forecast\")\n",
        "plt.xlabel(\"Time\")\n",
        "plt.ylabel(\"Q\")\n",
        "plt.legend()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XkCHu8k9eTt2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}