rnntf_sample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eIrvnAbGZ1wP"
   },
   "source": [
    "##### Copyright 2019 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "_A4IPZ-WZ9H7"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zfiHTzhkmNwd"
   },
   "source": [
    "# Recurrent Neural Networks (RNN) with Keras\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/keras/rnn\">\n",
    "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
    "    View on TensorFlow.org</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/keras/rnn.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
    "    Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/keras/rnn.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
    "    View source on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/keras/rnn.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />\n",
    "    Download notebook</a>\n",
    "  </td>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jfOdaQLhXLDR"
   },
   "source": [
    "Recurrent neural networks (RNN) are a class of neural networks that is powerful for modeling sequence data such as time series or natural language.\n",
    "\n",
    "Schematically, a RNN layer uses a `for` loop to iterate over the timesteps of a sequence, while maintaining an internal state that encodes information about the timesteps it has seen so far.\n",
    "\n",
    "The Keras RNN API is designed with a focus on:\n",
    "\n",
    "- **Ease of use**: the built-in `tf.keras.layers.RNN`, `tf.keras.layers.LSTM`, `tf.keras.layers.GRU` layers enable you to quickly build recurrent models without having to make difficult configuration choices.\n",
    "  \n",
    "- **Ease of customization**: You can also define your own RNN cell layer (the inner part of the `for` loop) with custom behavior, and use it with the generic `tf.keras.layers.RNN` layer (the `for` loop itself). This allows you to quickly prototype different research ideas in a flexible way with minimal code.\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QGJH5EKYoSHZ"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wJEBe8hTlB6W"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import collections\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DznzjxWCilt4"
   },
   "source": [
    "## Build a simple model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H5tPG7KJirBj"
   },
   "source": [
    "There are three built-in RNN layers in Keras:\n",
    "\n",
    "1. `tf.keras.layers.SimpleRNN`, a fully-connected RNN where the output from previous timestep is to be fed to next timestep.\n",
    "\n",
    "2. `tf.keras.layers.GRU`, first proposed in [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078).\n",
    "\n",
    "3. `tf.keras.layers.LSTM`, first proposed in [Long Short-Term Memory](https://www.bioinf.jku.at/publications/older/2604.pdf).\n",
    "\n",
    "In early 2015, Keras had the first reusable open-source Python implementations of LSTM and GRU.\n",
    "\n",
    "Here is a simple example of a `Sequential` model that processes sequences of integers, embeds each integer into a 64-dimensional vector, then processes the sequence of vectors using a `LSTM` layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QHdAFEATnFpn"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 64)          64000     \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 128)               98816     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 164,106\n",
      "Trainable params: 164,106\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "# Add an Embedding layer expecting input vocab of size 1000, and\n",
    "# output embedding dimension of size 64.\n",
    "model.add(layers.Embedding(input_dim=1000, output_dim=64))\n",
    "\n",
    "# Add a LSTM layer with 128 internal units.\n",
    "model.add(layers.LSTM(128))\n",
    "\n",
    "# Add a Dense layer with 10 units and softmax activation.\n",
    "model.add(layers.Dense(10, activation='softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sVT4R7O3qDXM"
   },
   "source": [
    "## Outputs and states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IOQnPR9eqLwk"
   },
   "source": [
    "By default, the output of a RNN layer contain a single vector per sample. This vector is the RNN cell output corresponding to the last timestep, containing information about the entire input sequence. The shape of this output is `(batch_size, units)` where `units` corresponds to the `units` argument passed to the layer's constructor. \n",
    "\n",
    "A RNN layer can also return the entire sequence of outputs for each sample (one vector per timestep per sample), if you set `return_sequences=True`. The shape of this output is `(batch_size, timesteps, units)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wNlkR8oXpNEx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, None, 64)          64000     \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (None, None, 256)         247296    \n",
      "_________________________________________________________________\n",
      "simple_rnn (SimpleRNN)       (None, 128)               49280     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 361,866\n",
      "Trainable params: 361,866\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "model.add(layers.Embedding(input_dim=1000, output_dim=64))\n",
    "\n",
    "# The output of GRU will be a 3D tensor of shape (batch_size, timesteps, 256)\n",
    "model.add(layers.GRU(256, return_sequences=True))\n",
    "\n",
    "# The output of SimpleRNN will be a 2D tensor of shape (batch_size, 128)\n",
    "model.add(layers.SimpleRNN(128))\n",
    "\n",
    "model.add(layers.Dense(10, activation='softmax'))\n",
    "\n",
    "model.summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1HagyjYos5rD"
   },
   "source": [
    "In addition, a RNN layer can return its final internal state(s). The returned states can be used to resume the RNN execution later, or [to initialize another RNN](https://arxiv.org/abs/1409.3215). This setting is commonly used in the encoder-decoder sequence-to-sequence model, where the encoder final state is used as the initial state of the decoder.\n",
    "\n",
    "To configure a RNN layer to return its internal state, set the `return_state` parameter to `True` when creating the layer. Note that `LSTM` has 2 state  tensors, but `GRU` only has one.\n",
    "\n",
    "To configure the initial state of the layer, just call the layer with additional keyword argument `initial_state`.\n",
    "Note that the shape of the state needs to match the unit size of the layer, like in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2_HsGBrDvaea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, None, 64)     64000       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, None, 64)     128000      input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "encoder (LSTM)                  [(None, 64), (None,  33024       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "decoder (LSTM)                  (None, 64)           33024       embedding_3[0][0]                \n",
      "                                                                 encoder[0][1]                    \n",
      "                                                                 encoder[0][2]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 10)           650         decoder[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 258,698\n",
      "Trainable params: 258,698\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "encoder_vocab = 1000\n",
    "decoder_vocab = 2000\n",
    "\n",
    "encoder_input = layers.Input(shape=(None, ))\n",
    "encoder_embedded = layers.Embedding(input_dim=encoder_vocab, output_dim=64)(encoder_input)\n",
    "\n",
    "# Return states in addition to output\n",
    "output, state_h, state_c = layers.LSTM(\n",
    "    64, return_state=True, name='encoder')(encoder_embedded)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "decoder_input = layers.Input(shape=(None, ))\n",
    "decoder_embedded = layers.Embedding(input_dim=decoder_vocab, output_dim=64)(decoder_input)\n",
    "\n",
    "# Pass the 2 states to a new LSTM layer, as initial state\n",
    "decoder_output = layers.LSTM(\n",
    "    64, name='decoder')(decoder_embedded, initial_state=encoder_state)\n",
    "output = layers.Dense(10, activation='softmax')(decoder_output)\n",
    "\n",
    "model = tf.keras.Model([encoder_input, decoder_input], output)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kJDJSXjZ2VaY"
   },
   "source": [
    "## RNN layers and RNN cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hQRxLRSS2gDf"
   },
   "source": [
    "In addition to the built-in RNN layers, the RNN API also provides cell-level APIs. Unlike RNN layers, which processes whole batches of input sequences, the RNN cell only processes a single timestep.\n",
    "\n",
    "The cell is the inside of the `for` loop of a RNN layer. Wrapping a cell inside a `tf.keras.layers.RNN` layer gives you a layer capable of processing batches of sequences, e.g. `RNN(LSTMCell(10))`.\n",
    "\n",
    "Mathematically, `RNN(LSTMCell(10))` produces the same result as `LSTM(10)`. In fact, the implementation of this layer in TF v1.x was just creating the corresponding RNN cell and wrapping it in a RNN layer.  However using the built-in `GRU` and `LSTM` layers enables the use of CuDNN and you may see better performance.\n",
    "\n",
    "There are three built-in RNN cells, each of them corresponding to the matching RNN layer.\n",
    "\n",
    "- `tf.keras.layers.SimpleRNNCell` corresponds to the `SimpleRNN` layer.\n",
    "\n",
    "- `tf.keras.layers.GRUCell` corresponds to the `GRU` layer.\n",
    "\n",
    "- `tf.keras.layers.LSTMCell` corresponds to the `LSTM` layer.\n",
    "\n",
    "The cell abstraction, together with the generic `tf.keras.layers.RNN` class, make it very easy to implement custom RNN architectures for your research.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "veiCKSUU-ina"
   },
   "source": [
    "## Cross-batch statefulness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EvAaiMJbWR2A"
   },
   "source": [
    "When processing very long sequences (possibly infinite), you may want to use the pattern of **cross-batch statefulness**.\n",
    "\n",
    "Normally, the internal state of a RNN layer is reset every time it sees a new batch (i.e. every sample seen by the layer is assume to be independent from the past). The layer will only maintain a state while processing a given sample.\n",
    "\n",
    "If you have very long sequences though, it is useful to break them into shorter sequences, and to feed these shorter sequences sequentially into a RNN layer without resetting the layer's state. That way, the layer can retain information about the entirety of the sequence, even though it's only seeing one sub-sequence at a time.\n",
    "\n",
    "You can do this by setting `stateful=True` in the constructor.\n",
    "\n",
    "If you have a sequence `s = [t0, t1, ... t1546, t1547]`, you would split it into e.g.\n",
    "\n",
    "```\n",
    "s1 = [t0, t1, ... t100]\n",
    "s2 = [t101, ... t201]\n",
    "...\n",
    "s16 = [t1501, ... t1547]\n",
    "```\n",
    "\n",
    "Then you would process it via:\n",
    "\n",
    "```python\n",
    "lstm_layer = layers.LSTM(64, stateful=True)\n",
    "for s in sub_sequences:\n",
    "  output = lstm_layer(s)\n",
    "```\n",
    "\n",
    "When you want to clear the state, you  can use `layer.reset_states()`.\n",
    "\n",
    "\n",
    "> Note: In this setup, sample `i` in a given batch is assumed to be the continuation of sample `i` in the previous batch. This means that all batches should contain the same number of samples (batch size). E.g. if a batch contains `[sequence_A_from_t0_to_t100,  sequence_B_from_t0_to_t100]`, the next batch should contain `[sequence_A_from_t101_to_t200,  sequence_B_from_t101_to_t200]`.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Here is a complete example:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "E6TsLXJ0X3Xd"
   },
   "outputs": [],
   "source": [
    "paragraph1 = np.random.random((20, 10, 50)).astype(np.float32)\n",
    "paragraph2 = np.random.random((20, 10, 50)).astype(np.float32)\n",
    "paragraph3 = np.random.random((20, 10, 50)).astype(np.float32)\n",
    "\n",
    "lstm_layer = layers.LSTM(64, stateful=True)\n",
    "output = lstm_layer(paragraph1)\n",
    "output = lstm_layer(paragraph2)\n",
    "output = lstm_layer(paragraph3)\n",
    "\n",
    "# reset_states() will reset the cached state to the original initial_state.\n",
    "# If no initial_state was provided, zero-states will be used by default.\n",
    "lstm_layer.reset_states()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7AtPur5BDzb4"
   },
   "source": [
    "##Bidirectional RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OsdEIXXREL_N"
   },
   "source": [
    "For sequences other than time series (e.g. text), it is often the case that a RNN model can perform better if it not only processes sequence from start to end, but also backwards. For example, to predict the next word in a sentence, it is often useful to have the context around the word, not only just the words that come before it.\n",
    "\n",
    "Keras provides an easy API for you to build such bidirectional RNNs: the `tf.keras.layers.Bidirectional` wrapper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MNhYIAXqYl3B"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "bidirectional (Bidirectional (None, 5, 128)            38400     \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 64)                41216     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 80,266\n",
      "Trainable params: 80,266\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "\n",
    "model.add(layers.Bidirectional(layers.LSTM(64, return_sequences=True), \n",
    "                               input_shape=(5, 10)))\n",
    "model.add(layers.Bidirectional(layers.LSTM(32)))\n",
    "model.add(layers.Dense(10, activation='softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ThwlodTjZCU0"
   },
   "source": [
    "Under the hood, `Bidirectional` will copy the RNN layer passed in, and flip the `go_backwards` field of the newly copied layer, so that it will process the inputs in reverse order.\n",
    "\n",
    "The output of the `Bidirectional` RNN will be, by default, the sum of the forward layer output and the backward layer output. If you need a different merging behavior, e.g. concatenation, change the `merge_mode` parameter in the `Bidirectional` wrapper constructor. For more details about `Bidirectional`, please check [the API docs](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Bidirectional)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ANGN956w6FRs"
   },
   "source": [
    "## Performance optimization and CuDNN kernels in TensorFlow 2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "76xAs7epaX21"
   },
   "source": [
    "In TensorFlow 2.0, the built-in LSTM and GRU layers have been updated to leverage CuDNN kernels by default when a GPU is available. With this change, the prior `keras.layers.CuDNNLSTM/CuDNNGRU` layers have been deprecated, and you can build your model without worrying about the hardware it will run on.\n",
    "\n",
    "Since the CuDNN kernel is built with certain assumptions, this means the layer **will not be able to use the CuDNN kernel if you change the defaults of the built-in LSTM or GRU layers**. E.g.:\n",
    "\n",
    "- Changing the `activation` function from `tanh` to something else.\n",
    "- Changing the `recurrent_activation` function from `sigmoid` to something else.\n",
    "- Using `recurrent_dropout` > 0.\n",
    "- Setting `unroll` to True, which forces LSTM/GRU to decompose the inner `tf.while_loop` into an unrolled `for` loop.\n",
    "- Setting `use_bias` to False.\n",
    "- Using masking when the input data is not strictly right padded (if the mask corresponds to strictly right padded data, CuDNN can still be used. This is the most common case).\n",
    "\n",
    "For the detailed list of constraints, please see the documentation for the [LSTM](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/LSTM) and [GRU](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/GRU) layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ybd73JmvqLp4"
   },
   "source": [
    "### Using CuDNN kernels when available\n",
    "\n",
    "Let's build a simple LSTM model to demonstrate the performance difference.\n",
    "\n",
    "We'll use as input sequences the sequence of rows of MNIST digits (treating each row of pixels as a timestep), and we'll predict the digit's label.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m9kM9hwRsxMx"
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "# Each MNIST image batch is a tensor of shape (batch_size, 28, 28).\n",
    "# Each input sequence will be of size (28, 28) (height is treated like time).\n",
    "input_dim = 28\n",
    "\n",
    "units = 64\n",
    "output_size = 10  # labels are from 0 to 9\n",
    "\n",
    "# Build the RNN model\n",
    "def build_model(allow_cudnn_kernel=True):\n",
    "  # CuDNN is only available at the layer level, and not at the cell level.\n",
    "  # This means `LSTM(units)` will use the CuDNN kernel,\n",
    "  # while RNN(LSTMCell(units)) will run on non-CuDNN kernel.\n",
    "  if allow_cudnn_kernel:\n",
    "    # The LSTM layer with default options uses CuDNN.\n",
    "    lstm_layer = tf.keras.layers.LSTM(units, input_shape=(None, input_dim))\n",
    "  else:\n",
    "    # Wrapping a LSTMCell in a RNN layer will not use CuDNN.\n",
    "    lstm_layer = tf.keras.layers.RNN(\n",
    "        tf.keras.layers.LSTMCell(units),\n",
    "        input_shape=(None, input_dim))\n",
    "  model = tf.keras.models.Sequential([\n",
    "      lstm_layer,\n",
    "      tf.keras.layers.BatchNormalization(),\n",
    "      tf.keras.layers.Dense(output_size, activation='softmax')]\n",
    "  )\n",
    "  return model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uuztNezFh0BL"
   },
   "source": [
    "### Load MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m_kZTLDobchi"
   },
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "sample, sample_label = x_train[0], y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UXF8elCuib8k"
   },
   "source": [
    "### Create a model instance and compile it\n",
    "We choose `sparse_categorical_crossentropy` as the loss function for the model. The output of the model has shape of `[batch_size, 10]`. The target for the model is a integer vector, each of the integer is in the range of 0 to 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "klgv6dfK0KNb"
   },
   "outputs": [],
   "source": [
    "model = build_model(allow_cudnn_kernel=True)\n",
    "\n",
    "model.compile(loss='sparse_categorical_crossentropy', \n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qzeeo65r25CU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 8s 126us/sample - loss: 0.9247 - accuracy: 0.7083 - val_loss: 0.5664 - val_accuracy: 0.8170\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 5s 85us/sample - loss: 0.3930 - accuracy: 0.8821 - val_loss: 0.3145 - val_accuracy: 0.9002\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 5s 85us/sample - loss: 0.2515 - accuracy: 0.9241 - val_loss: 0.2888 - val_accuracy: 0.9070\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 5s 86us/sample - loss: 0.1961 - accuracy: 0.9407 - val_loss: 0.1784 - val_accuracy: 0.9428\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 6s 93us/sample - loss: 0.1649 - accuracy: 0.9510 - val_loss: 0.1326 - val_accuracy: 0.9573\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f6a980abb38>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "          validation_data=(x_test, y_test),\n",
    "          batch_size=batch_size,\n",
    "          epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kvCWAssZjsdW"
   },
   "source": [
    "### Build a new model without CuDNN kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "H2JfHDOhOFtx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "60000/60000 [==============================] - 21s 349us/sample - loss: 0.1457 - accuracy: 0.9567 - val_loss: 0.1881 - val_accuracy: 0.9387\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f6a5d4f0b00>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_model = build_model(allow_cudnn_kernel=False)\n",
    "slow_model.set_weights(model.get_weights())\n",
    "slow_model.compile(loss='sparse_categorical_crossentropy', \n",
    "                   optimizer='sgd', \n",
    "                   metrics=['accuracy'])\n",
    "slow_model.fit(x_train, y_train, \n",
    "               validation_data=(x_test, y_test), \n",
    "               batch_size=batch_size,\n",
    "               epochs=1)  # We only train for one epoch because it's slower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zx8QLf81dTVr"
   },
   "source": [
    "As you can see, the model built with CuDNN is much faster to train compared to the model that use the regular TensorFlow kernel.\n",
    "\n",
    "The same CuDNN-enabled model can also be use to run inference in a CPU-only environment. The `tf.device` annotation below is just forcing the device placement. The model will run on CPU by default if no GPU is available.\n",
    "\n",
    "You simply don't have to worry about the hardware you're running on anymore. Isn't that pretty cool?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "z_z1eRh1fMBL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted result is: [5], target result is: 5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAN9klEQVR4nO3df4xV9ZnH8c+zWP6QojBrOhKKSyEGg8ZON4gbl6w1hvojGhw1TSexoZE4/YNJaLIhNewf1WwwZBU2SzTNTKMWNl1qEzUgaQouoOzGhDgiKo5LdQ2mTEaowZEf/mCHefaPezBTnfu9w7nn3nOZ5/1Kbu6957nnnicnfDi/7pmvubsATH5/VXYDAJqDsANBEHYgCMIOBEHYgSAuaubCzIxT/0CDubuNN72uLbuZ3Wpmh8zsPTN7sJ7vAtBYlvc6u5lNkfRHSUslHZH0qqQudx9IzMOWHWiwRmzZF0t6z93fd/czkn4raVkd3weggeoJ+2xJfxrz/kg27S+YWbeZ9ZtZfx3LAlCnhp+gc/c+SX0Su/FAmerZsg9KmjPm/bezaQBaUD1hf1XSlWb2HTObKulHkrYV0xaAouXejXf3ETPrkbRD0hRJT7n724V1BqBQuS+95VoYx+xAwzXkRzUALhyEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBJF7yGZcGKZMmZKsX3rppQ1dfk9PT9XaxRdfnJx3wYIFyfrKlSuT9ccee6xqraurKznv559/nqyvW7cuWX/44YeT9TLUFXYzOyzppKSzkkbcfVERTQEoXhFb9pvc/aMCvgdAA3HMDgRRb9hd0k4ze83Musf7gJl1m1m/mfXXuSwAdah3N36Juw+a2bckvWhm/+Pue8d+wN37JPVJkpl5ncsDkFNdW3Z3H8yej0l6XtLiIpoCULzcYTezaWY2/dxrST+QdLCoxgAUq57d+HZJz5vZue/5D3f/QyFdTTJXXHFFsj516tRk/YYbbkjWlyxZUrU2Y8aM5Lz33HNPsl6mI0eOJOsbN25M1js7O6vWTp48mZz3jTfeSNZffvnlZL0V5Q67u78v6bsF9gKggbj0BgRB2IEgCDsQBGEHgiDsQBDm3rwftU3WX9B1dHQk67t3707WG32baasaHR1N1u+///5k/dSpU7mXPTQ0lKx//PHHyfqhQ4dyL7vR3N3Gm86WHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeC4Dp7Adra2pL1ffv2Jevz5s0rsp1C1ep9eHg4Wb/pppuq1s6cOZOcN+rvD+rFdXYgOMIOBEHYgSAIOxAEYQeCIOxAEIQdCIIhmwtw/PjxZH316tXJ+h133JGsv/7668l6rT+pnHLgwIFkfenSpcn66dOnk/Wrr766am3VqlXJeVEstuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EAT3s7eASy65JFmvNbxwb29v1dqKFSuS8953333J+pYtW5J1tJ7c97Ob2VNmdszMDo6Z1mZmL5rZu9nzzCKbBVC8iezG/1rSrV+Z9qCkXe5+paRd2XsALaxm2N19r6Sv/h50maRN2etNku4quC8ABcv72/h2dz83WNaHktqrfdDMuiV151wOgILUfSOMu3vqxJu790nqkzhBB5Qp76W3o2Y2S5Ky52PFtQSgEfKGfZuk5dnr5ZK2FtMOgEapuRtvZlskfV/SZWZ2RNIvJK2T9DszWyHpA0k/bGSTk92JEyfqmv+TTz7JPe8DDzyQrD/zzDPJeq0x1tE6aobd3buqlG4uuBcADcTPZYEgCDsQBGEHgiDsQBCEHQiCW1wngWnTplWtvfDCC8l5b7zxxmT9tttuS9Z37tyZrKP5GLIZCI6wA0EQdiAIwg4EQdiBIAg7EARhB4LgOvskN3/+/GR9//79yfrw8HCyvmfPnmS9v7+/au2JJ55IztvMf5uTCdfZgeAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIrrMH19nZmaw//fTTyfr06dNzL3vNmjXJ+ubNm5P1oaGhZD0qrrMDwRF2IAjCDgRB2IEgCDsQBGEHgiDsQBBcZ0fSNddck6xv2LAhWb/55vyD/fb29ibra9euTdYHBwdzL/tClvs6u5k9ZWbHzOzgmGkPmdmgmR3IHrcX2SyA4k1kN/7Xkm4dZ/q/untH9vh9sW0BKFrNsLv7XknHm9ALgAaq5wRdj5m9me3mz6z2ITPrNrN+M6v+x8gANFzesP9S0nxJHZKGJK2v9kF373P3Re6+KOeyABQgV9jd/ai7n3X3UUm/krS42LYAFC1X2M1s1pi3nZIOVvssgNZQ8zq7mW2R9H1Jl0k6KukX2fsOSS7psKSfunvNm4u5zj75zJgxI1m/8847q9Zq3StvNu7l4i/t3r07WV+6dGmyPllVu85+0QRm7Bpn8pN1dwSgqfi5LBAEYQeCIOxAEIQdCIKwA0FwiytK88UXXyTrF12Uvlg0MjKSrN9yyy1Vay+99FJy3gsZf0oaCI6wA0EQdiAIwg4EQdiBIAg7EARhB4KoedcbYrv22muT9XvvvTdZv+6666rWal1Hr2VgYCBZ37t3b13fP9mwZQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBILjOPsktWLAgWe/p6UnW77777mT98ssvP++eJurs2bPJ+tBQ+q+Xj46OFtnOBY8tOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwXX2C0Cta9ldXeMNtFtR6zr63Llz87RUiP7+/mR97dq1yfq2bduKbGfSq7llN7M5ZrbHzAbM7G0zW5VNbzOzF83s3ex5ZuPbBZDXRHbjRyT9o7svlPR3klaa2UJJD0ra5e5XStqVvQfQomqG3d2H3H1/9vqkpHckzZa0TNKm7GObJN3VqCYB1O+8jtnNbK6k70naJ6nd3c/9OPlDSe1V5umW1J2/RQBFmPDZeDP7pqRnJf3M3U+MrXlldMhxB2109z53X+Tui+rqFEBdJhR2M/uGKkH/jbs/l00+amazsvosScca0yKAItTcjTczk/SkpHfcfcOY0jZJyyWty563NqTDSaC9fdwjnC8tXLgwWX/88ceT9auuuuq8eyrKvn37kvVHH320am3r1vQ/GW5RLdZEjtn/XtKPJb1lZgeyaWtUCfnvzGyFpA8k/bAxLQIoQs2wu/t/Sxp3cHdJNxfbDoBG4eeyQBCEHQiCsANBEHYgCMIOBMEtrhPU1tZWtdbb25uct6OjI1mfN29erp6K8MorryTr69evT9Z37NiRrH/22Wfn3RMagy07EARhB4Ig7EAQhB0IgrADQRB2IAjCDgQR5jr79ddfn6yvXr06WV+8eHHV2uzZs3P1VJRPP/20am3jxo3JeR955JFk/fTp07l6Quthyw4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQYS5zt7Z2VlXvR4DAwPJ+vbt25P1kZGRZD11z/nw8HByXsTBlh0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgjB3T3/AbI6kzZLaJbmkPnf/NzN7SNIDkv6cfXSNu/++xnelFwagbu4+7qjLEwn7LEmz3H2/mU2X9Jqku1QZj/2Uuz820SYIO9B41cI+kfHZhyQNZa9Pmtk7ksr90ywAztt5HbOb2VxJ35O0L5vUY2ZvmtlTZjazyjzdZtZvZv11dQqgLjV347/8oNk3Jb0saa27P2dm7ZI+UuU4/p9V2dW/v8Z3sBsPNFjuY3ZJMrNvSNouaYe7bxinPlfSdne/psb3EHagwaqFveZuvJmZpCclvTM26NmJu3M6JR2st0kAjTORs/FLJP2XpLckjWaT10jqktShym78YUk/zU7mpb6LLTvQYHXtxheFsAONl3s3HsDkQNiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQii2UM2fyTpgzHvL8umtaJW7a1V+5LoLa8ie/ubaoWm3s/+tYWb9bv7otIaSGjV3lq1L4ne8mpWb+zGA0EQdiCIssPeV/LyU1q1t1btS6K3vJrSW6nH7ACap+wtO4AmIexAEKWE3cxuNbNDZvaemT1YRg/VmNlhM3vLzA6UPT5dNobeMTM7OGZam5m9aGbvZs/jjrFXUm8Pmdlgtu4OmNntJfU2x8z2mNmAmb1tZquy6aWuu0RfTVlvTT9mN7Mpkv4oaamkI5JeldTl7gNNbaQKMzssaZG7l/4DDDP7B0mnJG0+N7SWmf2LpOPuvi77j3Kmu/+8RXp7SOc5jHeDeqs2zPhPVOK6K3L48zzK2LIvlvSeu7/v7mck/VbSshL6aHnuvlfS8a9MXiZpU/Z6kyr/WJquSm8twd2H3H1/9vqkpHPDjJe67hJ9NUUZYZ8t6U9j3h9Ra4337pJ2mtlrZtZddjPjaB8zzNaHktrLbGYcNYfxbqavDDPeMusuz/Dn9eIE3dctcfe/lXSbpJXZ7mpL8soxWCtdO/2lpPmqjAE4JGl9mc1kw4w/K+ln7n5ibK3MdTdOX01Zb2WEfVDSnDHvv51NawnuPpg9H5P0vCqHHa3k6LkRdLPnYyX38yV3P+ruZ919VNKvVOK6y4YZf1bSb9z9uWxy6etuvL6atd7KCPurkq40s++Y2VRJP5K0rYQ+vsbMpmUnTmRm0yT9QK03FPU2Scuz18slbS2xl7/QKsN4VxtmXCWvu9KHP3f3pj8k3a7KGfn/lfRPZfRQpa95kt7IHm+X3ZukLars1v2fKuc2Vkj6a0m7JL0r6T8ltbVQb/+uytDeb6oSrFkl9bZElV30NyUdyB63l73uEn01Zb3xc1kgCE7QAUEQdiAIwg4EQdiBIAg7EARhB4Ig7EAQ/w8ie3GmjcGk5QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with tf.device('CPU:0'):\n",
    "  cpu_model = build_model(allow_cudnn_kernel=True)\n",
    "  cpu_model.set_weights(model.get_weights())\n",
    "  result = tf.argmax(cpu_model.predict_on_batch(tf.expand_dims(sample, 0)), axis=1)\n",
    "  print('Predicted result is: %s, target result is: %s' % (result.numpy(), sample_label))\n",
    "  plt.imshow(sample, cmap=plt.get_cmap('gray'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2mCetBoTiqcB"
   },
   "source": [
    "## RNNs with list/dict inputs, or nested inputs\n",
    "\n",
    "Nested structures allow implementers to include more information within a single timestep. For example, a video frame could have audio and video input at the same time. The data shape in this case could be:\n",
    "\n",
    "`[batch, timestep, {\"video\": [height, width, channel], \"audio\": [frequency]}]`\n",
    "\n",
    "In another example, handwriting data could have both coordinates x and y for the current position of the pen, as well as pressure information. So the data representation could be:\n",
    "\n",
    "`[batch, timestep, {\"location\": [x, y], \"pressure\": [force]}]`\n",
    "\n",
    "The following code provides an example of how to build a custom RNN cell that accepts such structured inputs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A1IkIxWykSZQ"
   },
   "source": [
    "### Define a custom cell that support nested input/output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6yOT8nSqzp4A"
   },
   "outputs": [],
   "source": [
    "NestedInput = collections.namedtuple('NestedInput', ['feature1', 'feature2'])\n",
    "NestedState = collections.namedtuple('NestedState', ['state1', 'state2'])\n",
    "\n",
    "class NestedCell(tf.keras.layers.Layer):\n",
    "\n",
    "  def __init__(self, unit_1, unit_2, unit_3, **kwargs):\n",
    "    self.unit_1 = unit_1\n",
    "    self.unit_2 = unit_2\n",
    "    self.unit_3 = unit_3\n",
    "    self.state_size = NestedState(state1=unit_1, \n",
    "                                  state2=tf.TensorShape([unit_2, unit_3]))\n",
    "    self.output_size = (unit_1, tf.TensorShape([unit_2, unit_3]))\n",
    "    super(NestedCell, self).__init__(**kwargs)\n",
    "\n",
    "  def build(self, input_shapes):\n",
    "    # expect input_shape to contain 2 items, [(batch, i1), (batch, i2, i3)]\n",
    "    input_1 = input_shapes.feature1[1]\n",
    "    input_2, input_3 = input_shapes.feature2[1:]\n",
    "\n",
    "    self.kernel_1 = self.add_weight(\n",
    "        shape=(input_1, self.unit_1), initializer='uniform', name='kernel_1')\n",
    "    self.kernel_2_3 = self.add_weight(\n",
    "        shape=(input_2, input_3, self.unit_2, self.unit_3),\n",
    "        initializer='uniform',\n",
    "        name='kernel_2_3')\n",
    "\n",
    "  def call(self, inputs, states):\n",
    "    # inputs should be in [(batch, input_1), (batch, input_2, input_3)]\n",
    "    # state should be in shape [(batch, unit_1), (batch, unit_2, unit_3)]\n",
    "    input_1, input_2 = tf.nest.flatten(inputs)\n",
    "    s1, s2 = states\n",
    "\n",
    "    output_1 = tf.matmul(input_1, self.kernel_1)\n",
    "    output_2_3 = tf.einsum('bij,ijkl->bkl', input_2, self.kernel_2_3)\n",
    "    state_1 = s1 + output_1\n",
    "    state_2_3 = s2 + output_2_3\n",
    "\n",
    "    output = [output_1, output_2_3]\n",
    "    new_states = NestedState(state1=state_1, state2=state_2_3)\n",
    "\n",
    "    return output, new_states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BJHOrrybk6Zy"
   },
   "source": [
    "### Build a RNN model with nested input/output\n",
    "\n",
    "Let's build a Keras model that uses a `tf.keras.layers.RNN` layer and the custom cell we just defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "itrDe0Y2qPjP"
   },
   "outputs": [],
   "source": [
    "unit_1 = 10\n",
    "unit_2 = 20\n",
    "unit_3 = 30\n",
    "\n",
    "input_1 = 32\n",
    "input_2 = 64\n",
    "input_3 = 32\n",
    "batch_size = 64\n",
    "num_batch = 100\n",
    "timestep = 50\n",
    "\n",
    "cell = NestedCell(unit_1, unit_2, unit_3)\n",
    "rnn = tf.keras.layers.RNN(cell)\n",
    "\n",
    "inp_1 = tf.keras.Input((None, input_1))\n",
    "inp_2 = tf.keras.Input((None, input_2, input_3))\n",
    "\n",
    "outputs = rnn(NestedInput(feature1=inp_1, feature2=inp_2))\n",
    "\n",
    "model = tf.keras.models.Model([inp_1, inp_2], outputs)\n",
    "\n",
    "model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2MaihTM2mDcp"
   },
   "source": [
    "### Train the model with randomly generated data\n",
    "\n",
    "Since there isn't a good candidate dataset for this model, we use random Numpy data for demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lN-imRqElz2S"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 6400 samples\n",
      "6400/6400 [==============================] - 5s 804us/sample - loss: 0.3786 - rnn_1_loss: 0.1166 - rnn_1_1_loss: 0.2620 - rnn_1_accuracy: 0.0997 - rnn_1_1_accuracy: 0.0333\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f667c8a4b00>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_1_data = np.random.random((batch_size * num_batch, timestep, input_1))\n",
    "input_2_data = np.random.random((batch_size * num_batch, timestep, input_2, input_3))\n",
    "target_1_data = np.random.random((batch_size * num_batch, unit_1))\n",
    "target_2_data = np.random.random((batch_size * num_batch, unit_2, unit_3))\n",
    "input_data = [input_1_data, input_2_data]\n",
    "target_data = [target_1_data, target_2_data]\n",
    "\n",
    "model.fit(input_data, target_data, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oDdrwgBWnjYp"
   },
   "source": [
    "With the Keras `tf.keras.layers.RNN` layer, You are only expected to define the math logic for individual step within the sequence, and the `tf.keras.layers.RNN` layer will handle the sequence iteration for you. It's an incredibly powerful way to quickly prototype new kinds of RNNs (e.g. a LSTM variant).\n",
    "\n",
    "For more details, please visit the [API docs](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/RNN)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "rnn.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}