03-autograd_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autograd: automatic differentiation\n",
    "\n",
    "The ``autograd`` package provides automatic differentiation for all operations\n",
    "on Tensors. It is a define-by-run framework, which means that your backprop is\n",
    "defined by how your code is run, and that every single iteration can be\n",
    "different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a 2x2 tensor with gradient-accumulation capabilities\n",
    "x = torch.tensor([[1, 2], [3, 4]], requires_grad=True, dtype=torch.float32)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do an operation on the tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deduct 2 from all elements\n",
    "y = x - 2\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``y`` was created as a result of an operation, so it has a ``grad_fn``.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What's happening here?\n",
    "print(x.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's dig further...\n",
    "y.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.grad_fn.next_functions[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.grad_fn.next_functions[0][0].variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do more operations on y\n",
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's visualise the computational graph! (thks @szagoruyko)\n",
    "from torchviz import make_dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_dot(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradients\n",
    "\n",
    "Let's backprop now `out.backward()` is equivalent to doing `out.backward(torch.tensor([1.0]))`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Backprop\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print gradients $\\frac{\\text{d}o}{\\text{d}x}$.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute it by hand BEFORE executing this\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do many crazy things with autograd!\n",
    "> With Great *Flexibility* Comes Great Responsibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dynamic graphs!\n",
    "x = torch.randn(3, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "i = 0\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "    i += 1\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we don't run backward on a scalar we need to specify the grad_output\n",
    "gradients = torch.FloatTensor([0.1, 1.0, 0.0001])\n",
    "y.backward(gradients)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# BEFORE executing this, can you tell what would you expect it to print?\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This variable decides the tensor's range below\n",
    "n = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Both x and w that allows gradient accumulation\n",
    "x = torch.arange(1., n + 1, requires_grad=True)\n",
    "w = torch.ones(n, requires_grad=True)\n",
    "z = w @ x\n",
    "z.backward()\n",
    "print(x.grad, w.grad, sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only w that allows gradient accumulation\n",
    "x = torch.arange(1., n + 1)\n",
    "w = torch.ones(n, requires_grad=True)\n",
    "z = w @ x\n",
    "z.backward()\n",
    "print(x.grad, w.grad, sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Regardless of what you do in this loop, all torch tensors will not have gradient accumulation\n",
    "with torch.no_grad():\n",
    "    x = torch.arange(1., n + 1)\n",
    "    w = torch.ones(n, requires_grad=True)\n",
    "    z = w @ x\n",
    "    z.backward()\n",
    "    print(x.grad, w.grad, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More stuff\n",
    "\n",
    "Documentation of the automatic differentiation package is at\n",
    "http://pytorch.org/docs/autograd."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}