day_6_old.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Day 6: Deep Learning 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('/Users/davidbuchacaprats/Documents/lxmls2015/lxmls-toolkit_fork')\n",
    "import lxmls\n",
    "import scipy\n",
    "#path_inside_lxmls_toolkit_student = \"/Users/davidbuchacaprats/Dropbox/lxmls2015/lxmls-toolkit-student\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get in contact with the multi-layer perceptron (MLP) class in Numpy and see that for a single layer this is simply a log-linear model. Revisit the sentiment classification exercise of day one. Reformulate train and test data in a way suitable for the exercises of today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000\n",
      "1600\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import lxmls.readers.sentiment_reader as srs\n",
    "scr = srs.SentimentCorpus(\"books\")\n",
    "\n",
    "train_x = scr.train_X.T\n",
    "train_y = scr.train_y[:, 0]\n",
    "test_x = scr.test_X.T\n",
    "test_y = scr.test_y[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       ..., \n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((13989, 1600), array([ 0.,  0.,  0., ...,  0.,  0.,  0.]), 5.0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x.shape, train_x[0], train_x[0].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the MLP and SGD code and create a single layer model by specifying the number of inputs, outputs and the type of layer. Note that the number of inputs equals the number of features and the number of outputs the number of classes (2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Define MLP (log linear)\n",
    "import lxmls.deep_learning.mlp as dl \n",
    "import lxmls.deep_learning.sgd as sgd # Model parameters\n",
    "geometry = [train_x.shape[0], 2] \n",
    "actvfunc = ['softmax']\n",
    "\n",
    "# Instantiate model\n",
    "mlp      = dl.NumpyMLP(geometry, actvfunc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'actvfunc': ['softmax'],\n",
       " 'n_layers': 1,\n",
       " 'params': [array([[-0.05110566,  0.02022964, -0.01031658, ..., -0.04864428,\n",
       "           0.02159766, -0.03177295],\n",
       "         [ 0.01863954, -0.02708282,  0.07601295, ...,  0.02542927,\n",
       "          -0.02739964,  0.03537327]]), array([[ 0.],\n",
       "         [ 0.]])]}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "mlp.__dict__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Put a breakpoint inside of the lxmls/deep learning/mlp.py function and debug step by step. Identify the forward pass in Eq: 6.12 and the computation of the gradients (to be completed in the next exercise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Untrained Log-linear Accuracy train: 0.518750 test: 0.542500\n"
     ]
    }
   ],
   "source": [
    "# Play with the untrained MLP forward\n",
    "hat_train_y = mlp.forward(train_x)\n",
    "hat_test_y = mlp.forward(test_x)\n",
    "\n",
    "# Compute accuracy\n",
    "acc_train = sgd.class_acc(hat_train_y, train_y)[0]\n",
    "acc_test = sgd.class_acc(hat_test_y, test_y)[0]\n",
    "print \"Untrained Log-linear Accuracy train: %f test: %f\"%(acc_train,acc_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to ```lxmls/deep_learning/mlp.py``` in the class  NumpyMLP and check grads(). Complete the code of the NumpyMLP class with the Backpropagation recursion that we just saw. Once you are done. Try different network geometries by increasing the number of layers and layer sizes e.g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "geometry = [train_x.shape[0], 20, 2]\n",
    "actvfunc = ['sigmoid', 'softmax']\n",
    "\n",
    "# Instantiate model\n",
    "mlp      = dl.NumpyMLP(geometry, actvfunc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can test the different models with the same sentiment analysis problem as in Exercise 6.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch 321/321 (100%)   Epoch  1/ 2 in 5.45 seg\n",
      "Batch 321/321 (100%)   Epoch  2/ 2 in 5.40 seg\n",
      "\n",
      "MLP ([13989, 20, 2]) Amazon Sentiment Accuracy train: 0.954375 test: 0.747500\n"
     ]
    }
   ],
   "source": [
    "n_iter = 2\n",
    "bsize  = 5\n",
    "lrate  = 0.01\n",
    "\n",
    "\n",
    "# Train\n",
    "sgd.SGD_train(mlp, n_iter, bsize=bsize, lrate=lrate, train_set=(train_x, train_y))\n",
    "acc_train = sgd.class_acc(mlp.forward(train_x), train_y)[0]\n",
    "acc_test = sgd.class_acc(mlp.forward(test_x), test_y)[0]\n",
    "print \"MLP (%s) Amazon Sentiment Accuracy train: %f test: %f\" % (geometry, acc_train, acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that if the batch size is increased the code can take advantage from the paralelization\n",
    "of the vectorized functions build in numpy. Try bsize=256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch 13/13 (100%)   Epoch  1/ 6 in 3.07 seg\n",
      "Batch 13/13 (100%)   Epoch  2/ 6 in 2.84 seg\n",
      "Batch 13/13 (100%)   Epoch  3/ 6 in 2.79 seg\n",
      "Batch 13/13 (100%)   Epoch  4/ 6 in 2.80 seg\n",
      "Batch 13/13 (100%)   Epoch  5/ 6 in 3.03 seg\n",
      "Batch 13/13 (100%)   Epoch  6/ 6 in 3.19 seg\n",
      "\n",
      "MLP ([13989, 20, 2]) Amazon Sentiment Accuracy train: 1.000000 test: 0.825000\n"
     ]
    }
   ],
   "source": [
    "n_iter = 6\n",
    "bsize  = 128\n",
    "lrate  = 0.01\n",
    "\n",
    "\n",
    "# Train\n",
    "sgd.SGD_train(mlp, n_iter, bsize=bsize, lrate=lrate, train_set=(train_x, train_y))\n",
    "acc_train = sgd.class_acc(mlp.forward(train_x), train_y)[0]\n",
    "acc_test = sgd.class_acc(mlp.forward(test_x), test_y)[0]\n",
    "print \"MLP (%s) Amazon Sentiment Accuracy train: %f test: %f\" % (geometry, acc_train, acc_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Get in contact with Theano. Learn the difference between a symbolic representation and a function. Start by implementing the first layer of our previous MLP in Numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       ..., \n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = test_x "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Numpy code\n",
    "x = test_x \n",
    "W1, b1 = mlp.params[:2]       # Weights and bias of the first layer\n",
    "z1 = np.dot(W1,x) + b1        # Linear transformation\n",
    "tilde_z1 = 1/(1+np.exp(-z1))  # Non-linear transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will implement this in Theano. We start by creating the variables over which we will produce the operations. For example the symbolic input is defined as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Theano code.\n",
    "# NOTE: We use undescore to denote symbolic equivalents to Numpy variables. # This is no Python convention!.\n",
    "import theano\n",
    "import theano.tensor as T\n",
    "_x = T.matrix('x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this variable does not have any particular value, nor a space reserved in memory for it. It contains just a symbolic definition of what the variable can contain. The particular values will be given when we use it to compile a function.\n",
    "\n",
    "We could actually use the same definition format to define the weights and give their particular values as inputs to the compiled function. However, since we will be using a more complicated format in later exercises, we will use it here as well. The shared class allows to define variables that are shared across functions. They are also given a concrete value so that we do not need to give it for each function call. This format is therefore ideal for the weights of our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_W1 = theano.shared(value=W1, name='W1', borrow=True)\n",
    "_b1 = theano.shared(value=b1, name='b1', borrow=True, broadcastable=(False, True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets describe the operations we want to do with the variables. Again only symbolically. This is done by replacing our usual operations by Theano symbolic ones when necessary e. g. the internal product dot() or the sigmoid. Some operations like e.g. + are automatically recognized by Theano (operator overloading)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_z1            = T.dot(_W1, _x) + _b1\n",
    "_tilde_z1      = T.nnet.sigmoid(_z1)\n",
    "# Keep in mind that naming variables is useful when debugging\n",
    "_z1.name       = 'z1'\n",
    "_tilde_z1.name = 'tilde_z1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When debugging the code it is often useful to print the graph of computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigmoid [@A] 'tilde_z1'   \n",
      " |Elemwise{add,no_inplace} [@B] 'z1'   \n",
      "   |dot [@C] ''   \n",
      "   | |W1 [@D]\n",
      "   | |x [@E]\n",
      "   |b1 [@F]\n"
     ]
    }
   ],
   "source": [
    "theano.printing.debugprint(_tilde_z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to keep in mind that, until this point, we do not have a function we can use to produce any practical input. In order to obtain this we have to compile this function by calling\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "layer1 = theano.function([_x], _tilde_z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the use of [ ] for the input variables, even if we just specify one variable. We can now do a test to compare the Numpy and Theano implementations and see that they give the same outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Numpy and Theano Perceptrons are equivalent\n"
     ]
    }
   ],
   "source": [
    "# Check Numpy and Theano match\n",
    "if np.allclose(tilde_z1, layer1(x.astype(theano.config.floatX))):\n",
    "    print \"\\nNumpy and Theano Perceptrons are equivalent\"\n",
    "else:\n",
    "    raise ValueError, \"Numpy and Theano Perceptrons are different\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complete the method forward() inside of the ```lxmls/deep_learning/mlp.py```  class TheanoMLP. Note that this is called only once at the initialization of the class. To debug your implementation put a breakpoint at the ￼ ￼ init ￼ ￼ function call. Hint: Note that this is very similar to ```NumpyMLP.forward()```. You just need to keep track of the symbolic variable representing the output of the network after each layer is applied and compile the function at the end. After you are finished instantiate a Theano class and check that Numpy and Theano forward pass are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "mlp_a = dl.NumpyMLP(geometry, actvfunc)\n",
    "mlp_b = dl.TheanoMLP(geometry, actvfunc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first see an example that does not use any of the code in TheanoMLP but rather continues from what you wrote in exercise 6.3. In this exercise you completed a sigmoid layer with Theano. To get some values for the weights we used the first layer of the network you trained in 6.2. now we are going to use the second layer as well. This is thus assuming that your network in 6.2 has only two layers e.g. the recommended geometry (I, 20, 2). Make sure this is the case before starting this exercise.\n",
    "\n",
    "For the sake of clarity, lets write here the part of Ex. 6.2 that we had completed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get the values from our MLP from Ex 6.2\n",
    "W1, b1 = mlp.params[:2] # Weights and bias of fist layer\n",
    "# First layer symbolic variables\n",
    "_x = T.matrix('x')\n",
    "_W1 = theano.shared(value=W1, name='W1', borrow=True)\n",
    "_b1 = theano.shared(value=b1, name='b1', borrow=True, broadcastable=(False, True)) # First layer symbolic expressions\n",
    "_z1 = T.dot(_W1, _x) + _b1\n",
    "_tilde_z1 = T.nnet.sigmoid(_z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we just need to complete this with the second layer, using a softmax non-linearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "W2, b2 = mlp.params[2:] # Weights and bias of second (and last!) layer\n",
    "# Second layer symbolic variables\n",
    "_W2 = theano.shared(value=W2, name='W2', borrow=True)\n",
    "_b2 = theano.shared(value=b2, name='b2', borrow=True, broadcastable=(False, True)) # Second layer symbolic expressions\n",
    "_z2       = T.dot(_W2, _tilde_z1) + _b2\n",
    "_tilde_z2 = T.nnet.softmax(_z2.T).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "With this, we could compile a function to obtain the output of the network symb tilde z2 for a given input symb x. In this exercise we are however interested in obtaining the misclassification cost. This is given in Eq: 6.6. First we are going to need the symbolic variable for the correct output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_y = T.ivector('y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The minus posterior probability of the class given the input is the same as selecting the k(m)-th softmax output, were k(m) is the index of the correct class for xm. If we want to do this for a vector y containing M different examples, we can write this as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_F = -T.mean(T.log(_tilde_z2[_y, T.arange(_y.shape[0])]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now obtaining a function that computes the gradient could not be easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_nabla_F = T.grad(_F, _W1)\n",
    "nabla_F = theano.function([_x,_y], _nabla_F)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To finish this exercise have a look at the TheanoMLP class. As you may realise it just implements what is shown above for the generic case of N layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s first have an understanding of handling train/test data inside the Theano computation graph. One important aspect to take into account is that both type and shape of the data have to match their corresponding graph variables. This is the main source of errors when you are starting with Theano."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Cast data into the types and shapes used in the Theano graph\n",
    "train_x = train_x.astype(theano.config.floatX)\n",
    "train_y = train_y.astype('int32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the Theano type theano.config.floatX. This will automatically switch between float32 (GPU) and float64 (CPU). To use data in a Theano computation graph, we use the theano.shared variable. This will also push data into the GPU,\n",
    "if used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_train_x = theano.shared(train_x, \"train_x\", borrow=True)\n",
    "_train_y = theano.shared(train_y, \"train_y\", borrow=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this is done, we can create and compile functions using these variables. One function that will be useful in the future will be one returning a batch of instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_i = T.lscalar()\n",
    "get_tr_batch_y  = theano.function([_i], _train_y[_i * bsize:(_i+1)*bsize])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " The mini-batch function in the previous exercise is the key to fast batch update. This is combined with the updates argument of theano.function. The input to this argument, is a list of tuples with each parameter and update rule. This can be compactly defined using list comprehensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mlp_c   = dl.TheanoMLP(geometry, actvfunc)\n",
    "_x      = T.matrix('x')\n",
    "_y      = T.ivector('y')\n",
    "_F      = mlp_c._cost(_x, _y)\n",
    "updates = [(par, par - lrate*T.grad(_F, par)) for par in mlp_c.params]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be now combined with the givens argument of theano.function. This maps input and target to other variables. In this case a mini-batch of inputs and targets given an index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_j      = T.lscalar()\n",
    "givens  = { _x : _train_x[:, _j*bsize:(_j+1)*bsize],\n",
    "            _y : _train_y[_j*bsize:(_j+1)*bsize] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With updates and givens, we can now define the batch update function. This will return the cost of each batch and update the MLP parameters at the same time using updates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "batch_up = theano.function([_j], _F, updates=updates, givens=givens)\n",
    "n_batch  = train_x.shape[1]/bsize  + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have defined this, we can compare speed and accuracy of the Numpy and simple gradient versions using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch 321/321 (100%)   Epoch  1/ 2 in 8.36 seg\n",
      "Batch 321/321 (100%)   Epoch  2/ 2 in 8.28 seg\n",
      "\n",
      "\n",
      "Numpy version took 16.64 sec\n",
      "Amazon Sentiment Accuracy train: 0.954375 test: 0.747500\n",
      "\n",
      "Batch 321/321 (100%)   Epoch  1/ 2 in 4.48 seg\n",
      "Batch 321/321 (100%)   Epoch  2/ 2 in 4.52 seg\n",
      "\n",
      "\n",
      "Compiled gradient version took 9.00 sec\n",
      "Amazon Sentiment Accuracy train: 0.954375 test: 0.747500\n",
      "\n",
      "Batch 321/321 (100%)   Epoch  1/ 2 in 1.16 seg\n",
      "Batch 321/321 (100%)   Epoch  2/ 2 in 1.24 seg\n",
      "\n",
      "\n",
      "Theano compiled batch update version took 2.40\n",
      "Amazon Sentiment Accuracy train: 0.954375 test: 0.747500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "# Model\n",
    "geometry = [train_x.shape[0], 20, 2] \n",
    "actvfunc = ['sigmoid', 'softmax']\n",
    "\n",
    "# Numpy MLP\n",
    "mlp_a = dl.NumpyMLP(geometry, actvfunc)\n",
    "init_t = time.clock()\n",
    "sgd.SGD_train(mlp_a, n_iter, bsize=bsize, lrate=lrate, train_set=(train_x, train_y)) \n",
    "print \"\\nNumpy version took %2.2f sec\" % (time.clock() - init_t)\n",
    "\n",
    "acc_train = sgd.class_acc(mlp_a.forward(train_x), train_y)[0]\n",
    "acc_test = sgd.class_acc(mlp_a.forward(test_x), test_y)[0]\n",
    "print \"Amazon Sentiment Accuracy train: %f test: %f\\n\" % (acc_train, acc_test)\n",
    "# Theano grads\n",
    "mlp_b = dl.TheanoMLP(geometry, actvfunc)\n",
    "init_t = time.clock()\n",
    "sgd.SGD_train(mlp_b, n_iter, bsize=bsize, lrate=lrate, train_set=(train_x, train_y)) \n",
    "print \"\\nCompiled gradient version took %2.2f sec\" % (time.clock() - init_t) \n",
    "\n",
    "acc_train = sgd.class_acc(mlp_b.forward(train_x), train_y)[0]\n",
    "acc_test = sgd.class_acc(mlp_b.forward(test_x), test_y)[0]\n",
    "print \"Amazon Sentiment Accuracy train: %f test: %f\\n\" % (acc_train, acc_test)\n",
    "# Theano batch update\n",
    "init_t = time.clock()\n",
    "sgd.SGD_train(mlp_c, n_iter, batch_up=batch_up, n_batch=n_batch)\n",
    "print \"\\nTheano compiled batch update version took %2.2f\" % (time.clock() - init_t) \n",
    "acc_train = sgd.class_acc(mlp_c.forward(train_x), train_y)[0]\n",
    "acc_test = sgd.class_acc(mlp_c.forward(test_x), test_y)[0]\n",
    "print \"Amazon Sentiment Accuracy train: %f test: %f\\n\"%(acc_train,acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you may observe, just computing the gradients with Theano may not lead to a decrease, but rather an increase in computing speed. To maximally exploit the power of Theano, it is necessary to bundle both computations and data together using approaches like the compiled batch update."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "geometry ="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}