Skip to content

Commit

Permalink
Move code samples into markdown cells
Browse files Browse the repository at this point in the history
My understanding is that these code snippets are not meant to be
executed. Moving them into markdown cells makes it harder for the user
to mistakenly run the cells, since doing so would add an unwanted extra
Dense layer to the user's model.

The result of running the `model.add(..)` cell is that when the user
later pops the last layer for fine-tuning, they pop this extraneous
Dense layer rather than the original Vgg16 output layer. If the user
later saves the model weights and attempts to load them into a fresh
model in lesson3, they'll get an exception about trying to load a
weights file with 17 layers into a model with 16 layers.

Possibly related forum thread:

http://forums.fast.ai/t/lesson-3-discussion/186/35
  • Loading branch information
appleby committed May 27, 2017
1 parent 84b917d commit 622c891
Showing 1 changed file with 12 additions and 36 deletions.
48 changes: 12 additions & 36 deletions deeplearning1/nbs/lesson2.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -1287,42 +1287,18 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Do you remember how we defined our linear model? Here it is again for reference:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"lm = Sequential([ Dense(2, activation='softmax', input_shape=(1000,)) ])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And do you remember the definition of a fully connected layer in the original VGG?:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"model.add(Dense(4096, activation='relu'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Do you remember how we defined our linear model? Here it is again for reference:\n",
"\n",
"```python\n",
"lm = Sequential([ Dense(2, activation='softmax', input_shape=(1000,)) ])\n",
"```\n",
"\n",
"And do you remember the definition of a fully connected layer in the original VGG?:\n",
"\n",
"```python\n",
"model.add(Dense(4096, activation='relu'))\n",
"```\n",
"\n",
"You might we wondering, what's going on with that *activation* parameter? Adding an 'activation' parameter to a layer in Keras causes an additional function to be called after the layer is calculated. You'll recall that we had no such parameter in our most basic linear model at the start of this lesson - that's because a simple linear model has no *activation function*. But nearly all deep model layers have an activation function - specifically, a *non-linear* activation function, such as tanh, sigmoid (```1/(1+exp(x))```), or relu (```max(0,x)```, called the *rectified linear* function). Why?\n",
"\n",
"The reason for this is that if you stack purely linear layers on top of each other, then you just end up with a linear layer! For instance, if your first layer was ```2*x```, and your second was ```-2*x```, then the combination is: ```-2*(2*x) = -4*x```. If that's all we were able to do with deep learning, it wouldn't be very deep! But what if we added a relu activation after our first layer? Then the combination would be: ```-2 * max(0, 2*x)```. As you can see, that does not simplify to just a linear function like the previous example--and indeed we can stack as many of these on top of each other as we wish, to create arbitrarily complex functions.\n",
Expand Down

0 comments on commit 622c891

Please sign in to comment.