07.2 梯度下降和随机梯度下降.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "在本节中，我们将介绍梯度下降（gradient descent）的工作原理。虽然梯度下降在深度学习中很少被直接使用，但理解梯度的意义以及沿着梯度反方向更新自变量可能降低目标函数值的原因是学习后续优化算法的基础。随后，我们将引出随机梯度下降（stochastic gradient descent）。\n",
    "\n",
    "## 一维梯度下降\n",
    "\n",
    "我们先以简单的一维梯度下降为例，解释梯度下降算法可能降低目标函数值的原因。假设连续可导的函数$f: \\mathbb{R} \\rightarrow \\mathbb{R}$的输入和输出都是标量。给定绝对值足够小的数$\\epsilon$，根据泰勒展开公式（参见附录中[“数学基础”](11.02%20数学基础.ipynb)一节），我们得到以下的近似：\n",
    "\n",
    "$$f(x + \\epsilon) \\approx f(x) + \\epsilon f'(x) .$$\n",
    "\n",
    "这里$f'(x)$是函数$f$在$x$处的梯度。一维函数的梯度是一个标量，也称导数。\n",
    "\n",
    "接下来，找到一个常数$\\eta > 0$，使得$\\left|\\eta f'(x)\\right|$足够小，那么可以将$\\epsilon$替换为$-\\eta f'(x)$并得到\n",
    "\n",
    "$$f(x - \\eta f'(x)) \\approx f(x) -  \\eta f'(x)^2.$$\n",
    "\n",
    "如果导数$f'(x) \\neq 0$，那么$\\eta f'(x)^2>0$，所以\n",
    "\n",
    "$$f(x - \\eta f'(x)) \\lesssim f(x).$$\n",
    "\n",
    "这意味着，如果通过\n",
    "\n",
    "$$x \\leftarrow x - \\eta f'(x)$$\n",
    "\n",
    "来迭代$x$，函数$f(x)$的值可能会降低。因此在梯度下降中，我们先选取一个初始值$x$和常数$\\eta > 0$，然后不断通过上式来迭代$x$，直到达到停止条件，例如$f'(x)^2$的值已足够小或迭代次数已达到某个值。\n",
    "\n",
    "下面我们以目标函数$f(x)=x^2$为例来看一看梯度下降是如何工作的。虽然我们知道最小化$f(x)$的解为$x=0$，这里依然使用这个简单函数来观察$x$是如何被迭代的。首先，导入本节实验所需的包或模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:33:31.437131Z",
     "start_time": "2019-11-29T06:33:30.460898Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import d2lzh as d2l\n",
    "import math\n",
    "from mxnet import nd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来使用$x=10$作为初始值，并设$\\eta=0.2$。使用梯度下降对$x$迭代10次，可见最终$x$的值较接近最优解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:33:39.964377Z",
     "start_time": "2019-11-29T06:33:39.953002Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 0.06046617599999997\n"
     ]
    }
   ],
   "source": [
    "def gd(eta):\n",
    "    x = 10\n",
    "    results = [x]\n",
    "    for i in range(10):\n",
    "        x -= eta * 2 * x  # f(x) = x * x的导数为f'(x) = 2 * x\n",
    "        results.append(x)\n",
    "    print('epoch 10, x:', x)\n",
    "    return results\n",
    "\n",
    "res = gd(0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面将绘制出自变量$x$的迭代轨迹。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:33:43.571263Z",
     "start_time": "2019-11-29T06:33:43.378407Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"182.259375pt\" version=\"1.1\" viewBox=\"0 0 248.771875 182.259375\" width=\"248.771875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M -0 182.259375 \n",
       "L 248.771875 182.259375 \n",
       "L 248.771875 0 \n",
       "L -0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.271875 143.1 \n",
       "L 241.571875 143.1 \n",
       "L 241.571875 7.2 \n",
       "L 46.271875 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"ma8b82b7bbd\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.149148\" xlink:href=\"#ma8b82b7bbd\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 62.0625 27.6875 \n",
       "L 12.3125 27.6875 \n",
       "L 12.3125 35.15625 \n",
       "L 62.0625 35.15625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-8722\"/>\n",
       "       <path d=\"M 52.59375 0 \n",
       "L 9.234375 0 \n",
       "L 9.234375 8.25 \n",
       "L 26.171875 8.25 \n",
       "L 26.171875 65.671875 \n",
       "L 8.796875 60.640625 \n",
       "L 8.796875 69.4375 \n",
       "L 35.6875 77.25 \n",
       "L 35.6875 8.25 \n",
       "L 52.59375 8.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-49\"/>\n",
       "       <path d=\"M 4.4375 36.328125 \n",
       "Q 4.4375 56.34375 11.078125 66.625 \n",
       "Q 17.71875 76.90625 30.328125 76.90625 \n",
       "Q 54.34375 76.90625 54.34375 38.140625 \n",
       "Q 54.34375 19 47.578125 8.859375 \n",
       "Q 40.828125 -1.265625 28.609375 -1.265625 \n",
       "Q 17.09375 -1.265625 10.765625 8.34375 \n",
       "Q 4.4375 17.96875 4.4375 36.328125 \n",
       "z\n",
       "M 14.203125 36.765625 \n",
       "Q 14.203125 6.640625 29.5 6.640625 \n",
       "Q 44.53125 6.640625 44.53125 37.25 \n",
       "Q 44.53125 68.953125 29.78125 68.953125 \n",
       "Q 14.203125 68.953125 14.203125 36.765625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(45.576491 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"132.8125\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"99.535511\" xlink:href=\"#ma8b82b7bbd\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 8.640625 11.625 \n",
       "Q 16.265625 6.640625 24.515625 6.640625 \n",
       "Q 32.328125 6.640625 37.109375 11 \n",
       "Q 41.890625 15.375 41.890625 22.609375 \n",
       "Q 41.890625 29.890625 37.03125 33.890625 \n",
       "Q 32.171875 37.890625 22.953125 37.890625 \n",
       "Q 18.5 37.890625 10.984375 37.203125 \n",
       "L 10.984375 75.640625 \n",
       "L 48.1875 75.640625 \n",
       "L 48.1875 67.1875 \n",
       "L 19.875 67.1875 \n",
       "L 19.875 45.796875 \n",
       "Q 24.125 46.046875 26.265625 46.046875 \n",
       "Q 38.234375 46.046875 44.921875 39.984375 \n",
       "Q 51.609375 33.9375 51.609375 23.296875 \n",
       "Q 51.609375 12.3125 44.421875 5.515625 \n",
       "Q 37.25 -1.265625 24.609375 -1.265625 \n",
       "Q 14.015625 -1.265625 8.640625 1.859375 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-53\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(92.894886 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"143.921875\" xlink:href=\"#ma8b82b7bbd\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(140.989844 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"188.308239\" xlink:href=\"#ma8b82b7bbd\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(185.376207 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"232.694602\" xlink:href=\"#ma8b82b7bbd\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(226.83054 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 49.421875 54 \n",
       "L 31.34375 26.8125 \n",
       "L 49.171875 0 \n",
       "L 38.03125 0 \n",
       "L 27.734375 17.1875 \n",
       "Q 26.90625 18.5625 25.390625 21.390625 \n",
       "L 25.203125 21.390625 \n",
       "Q 25.140625 21.09375 22.75 17.1875 \n",
       "L 12.25 0 \n",
       "L 1.265625 0 \n",
       "L 19.625 26.515625 \n",
       "L 2.09375 54 \n",
       "L 13.234375 54 \n",
       "L 23.34375 35.9375 \n",
       "Q 24.265625 34.28125 25.59375 31.640625 \n",
       "L 25.78125 31.640625 \n",
       "L 38.921875 54 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(141.3875 172.573438)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m262a12ff3c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m262a12ff3c\" y=\"136.922727\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(33.407813 140.919602)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m262a12ff3c\" y=\"112.213636\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 51.171875 0 \n",
       "L 5.171875 0 \n",
       "L 5.171875 8.296875 \n",
       "L 27.15625 30.21875 \n",
       "Q 36.234375 39.3125 39.3125 44.609375 \n",
       "Q 42.390625 49.90625 42.390625 55.5625 \n",
       "Q 42.390625 62.015625 38.765625 65.453125 \n",
       "Q 35.15625 68.890625 28.328125 68.890625 \n",
       "Q 18.21875 68.890625 9.03125 60.296875 \n",
       "L 9.03125 70.015625 \n",
       "Q 17.96875 76.90625 29.828125 76.90625 \n",
       "Q 40.046875 76.90625 45.953125 71.390625 \n",
       "Q 51.859375 65.875 51.859375 56.546875 \n",
       "Q 51.859375 49.515625 48.0625 42.75 \n",
       "Q 44.28125 35.984375 33.734375 25.53125 \n",
       "L 16.5 8.6875 \n",
       "L 16.5 8.5 \n",
       "L 51.171875 8.5 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 116.210511)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m262a12ff3c\" y=\"87.504545\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 55.078125 19.625 \n",
       "L 45.40625 19.625 \n",
       "L 45.40625 0 \n",
       "L 36.140625 0 \n",
       "L 36.140625 19.625 \n",
       "L 0.875 19.625 \n",
       "L 0.875 25.828125 \n",
       "L 34.328125 75.640625 \n",
       "L 45.40625 75.640625 \n",
       "L 45.40625 27.203125 \n",
       "L 55.078125 27.203125 \n",
       "z\n",
       "M 36.140625 27.203125 \n",
       "L 36.140625 58.890625 \n",
       "Q 36.140625 62.25 36.328125 66.65625 \n",
       "L 36.140625 66.65625 \n",
       "Q 35.453125 64.796875 33.25 60.84375 \n",
       "L 10.75 27.203125 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 91.50142)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-52\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m262a12ff3c\" y=\"62.795455\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <defs>\n",
       "       <path d=\"M 49.703125 65.921875 \n",
       "Q 43.796875 68.953125 37.359375 68.953125 \n",
       "Q 27.546875 68.953125 21.578125 60.40625 \n",
       "Q 15.625 51.859375 15.53125 37.0625 \n",
       "L 15.765625 37.0625 \n",
       "Q 21.140625 47.46875 32.90625 47.46875 \n",
       "Q 42.71875 47.46875 48.578125 41.0625 \n",
       "Q 54.4375 34.671875 54.4375 24.078125 \n",
       "Q 54.4375 12.984375 47.671875 5.859375 \n",
       "Q 40.921875 -1.265625 30.421875 -1.265625 \n",
       "Q 18.890625 -1.265625 12.328125 7.734375 \n",
       "Q 5.765625 16.75 5.765625 33.296875 \n",
       "Q 5.765625 53.328125 14.375 65.109375 \n",
       "Q 23 76.90625 37.0625 76.90625 \n",
       "Q 45.125 76.90625 49.703125 74.703125 \n",
       "z\n",
       "M 16.109375 24.75 \n",
       "Q 16.109375 17.28125 20.09375 11.953125 \n",
       "Q 24.078125 6.640625 30.671875 6.640625 \n",
       "Q 36.921875 6.640625 40.828125 11.234375 \n",
       "Q 44.734375 15.828125 44.734375 23.046875 \n",
       "Q 44.734375 30.859375 41.046875 35.203125 \n",
       "Q 37.359375 39.546875 30.5625 39.546875 \n",
       "Q 24.125 39.546875 20.109375 35.15625 \n",
       "Q 16.109375 30.765625 16.109375 24.75 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-54\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 66.79233)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-54\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m262a12ff3c\" y=\"38.086364\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <defs>\n",
       "       <path d=\"M 21.390625 39.796875 \n",
       "Q 7.859375 46.09375 7.859375 58.0625 \n",
       "Q 7.859375 66.40625 14.203125 71.65625 \n",
       "Q 20.5625 76.90625 30.421875 76.90625 \n",
       "Q 39.453125 76.90625 45.3125 71.96875 \n",
       "Q 51.171875 67.046875 51.171875 59.234375 \n",
       "Q 51.171875 46.625 37.0625 39.84375 \n",
       "L 37.0625 39.65625 \n",
       "Q 53.609375 33.6875 53.609375 19.34375 \n",
       "Q 53.609375 9.90625 46.796875 4.3125 \n",
       "Q 39.984375 -1.265625 27.78125 -1.265625 \n",
       "Q 17.78125 -1.265625 11.46875 4.25 \n",
       "Q 5.171875 9.765625 5.171875 18.5625 \n",
       "Q 5.171875 33.015625 21.390625 39.59375 \n",
       "z\n",
       "M 41.5 58.109375 \n",
       "Q 41.5 63.1875 38.328125 66.109375 \n",
       "Q 35.15625 69.046875 29.734375 69.046875 \n",
       "Q 24.65625 69.046875 21.09375 65.984375 \n",
       "Q 17.53125 62.9375 17.53125 58.203125 \n",
       "Q 17.53125 48.734375 29.390625 43.796875 \n",
       "Q 41.5 48.828125 41.5 58.109375 \n",
       "z\n",
       "M 28.8125 35.25 \n",
       "Q 14.796875 29.59375 14.796875 19.046875 \n",
       "Q 14.796875 13.578125 18.890625 10.0625 \n",
       "Q 23 6.546875 29.59375 6.546875 \n",
       "Q 35.984375 6.546875 39.984375 10.03125 \n",
       "Q 44 13.53125 44 18.75 \n",
       "Q 44 29.734375 28.8125 35.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-56\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 42.083239)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-56\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m262a12ff3c\" y=\"13.377273\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(21.679688 17.374148)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"117.285156\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 36.234375 71.78125 \n",
       "Q 33.546875 73.1875 30.421875 73.1875 \n",
       "Q 21.4375 73.1875 21.4375 62.0625 \n",
       "L 21.4375 54 \n",
       "L 34.078125 54 \n",
       "L 34.078125 46.09375 \n",
       "L 21.4375 46.09375 \n",
       "L 21.4375 0 \n",
       "L 11.96875 0 \n",
       "L 11.96875 46.09375 \n",
       "L 2.734375 46.09375 \n",
       "L 2.734375 54 \n",
       "L 11.96875 54 \n",
       "L 11.96875 62.5 \n",
       "Q 11.96875 71.1875 16.96875 76.140625 \n",
       "Q 21.96875 81.109375 29.6875 81.109375 \n",
       "Q 33.796875 81.109375 36.234375 80.171875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-102\"/>\n",
       "      <path d=\"M 23.921875 -17.1875 \n",
       "Q 7.46875 1.5625 7.46875 28.859375 \n",
       "Q 7.46875 56.15625 23.921875 75.640625 \n",
       "L 32.375 75.640625 \n",
       "Q 15.875 55.375 15.875 28.953125 \n",
       "Q 15.875 2.59375 32.28125 -17.1875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-40\"/>\n",
       "      <path d=\"M 1.21875 -17.1875 \n",
       "Q 17.671875 2.546875 17.671875 28.953125 \n",
       "Q 17.671875 55.421875 1.125 75.640625 \n",
       "L 9.625 75.640625 \n",
       "Q 26.125 56.296875 26.125 28.859375 \n",
       "Q 26.125 1.421875 9.625 -17.1875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-41\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(15.19375 82.758594)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-102\"/>\n",
       "      <use x=\"34.667969\" xlink:href=\"#MicrosoftYaHei-40\"/>\n",
       "      <use x=\"68.066406\" xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"118.75\" xlink:href=\"#MicrosoftYaHei-41\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#p7395a43caa)\" d=\"M 55.149148 13.377273 \n",
       "L 59.587784 25.422955 \n",
       "L 64.02642 36.850909 \n",
       "L 68.465057 47.661136 \n",
       "L 72.903693 57.853636 \n",
       "L 77.34233 67.428409 \n",
       "L 81.780966 76.385455 \n",
       "L 85.331875 83.106327 \n",
       "L 88.882784 89.431855 \n",
       "L 92.433693 95.362036 \n",
       "L 95.984602 100.896873 \n",
       "L 99.535511 106.036364 \n",
       "L 103.08642 110.780509 \n",
       "L 106.63733 115.129309 \n",
       "L 110.188239 119.082764 \n",
       "L 113.739148 122.640873 \n",
       "L 117.290057 125.803636 \n",
       "L 119.953239 127.916264 \n",
       "L 122.61642 129.806509 \n",
       "L 125.279602 131.474373 \n",
       "L 127.942784 132.919855 \n",
       "L 130.605966 134.142955 \n",
       "L 133.269148 135.143673 \n",
       "L 135.93233 135.922009 \n",
       "L 138.595511 136.477964 \n",
       "L 141.258693 136.811536 \n",
       "L 143.921875 136.922727 \n",
       "L 146.585057 136.811536 \n",
       "L 149.248239 136.477964 \n",
       "L 151.91142 135.922009 \n",
       "L 154.574602 135.143673 \n",
       "L 157.237784 134.142955 \n",
       "L 159.900966 132.919855 \n",
       "L 162.564148 131.474373 \n",
       "L 165.22733 129.806509 \n",
       "L 167.890511 127.916264 \n",
       "L 170.553693 125.803636 \n",
       "L 173.216875 123.468627 \n",
       "L 176.767784 120.009355 \n",
       "L 180.318693 116.154736 \n",
       "L 183.869602 111.904773 \n",
       "L 187.420511 107.259464 \n",
       "L 190.97142 102.218809 \n",
       "L 194.52233 96.782809 \n",
       "L 198.073239 90.951464 \n",
       "L 201.624148 84.724773 \n",
       "L 205.175057 78.102736 \n",
       "L 208.725966 71.085355 \n",
       "L 213.164602 61.757673 \n",
       "L 217.603239 51.812264 \n",
       "L 222.041875 41.249127 \n",
       "L 226.480511 30.068264 \n",
       "L 230.919148 18.269673 \n",
       "L 231.806875 15.835827 \n",
       "L 231.806875 15.835827 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_13\">\n",
       "    <path clip-path=\"url(#p7395a43caa)\" d=\"M 232.694602 13.377273 \n",
       "L 197.185511 92.446364 \n",
       "L 175.880057 120.911236 \n",
       "L 163.096784 131.158591 \n",
       "L 155.42682 134.847638 \n",
       "L 150.824842 136.175695 \n",
       "L 148.063655 136.653796 \n",
       "L 146.406943 136.825912 \n",
       "L 145.412916 136.887874 \n",
       "L 144.8165 136.91018 \n",
       "L 144.45865 136.91821 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m565d4626fc\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p7395a43caa)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"232.694602\" xlink:href=\"#m565d4626fc\" y=\"13.377273\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"197.185511\" xlink:href=\"#m565d4626fc\" y=\"92.446364\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"175.880057\" xlink:href=\"#m565d4626fc\" y=\"120.911236\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"163.096784\" xlink:href=\"#m565d4626fc\" y=\"131.158591\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"155.42682\" xlink:href=\"#m565d4626fc\" y=\"134.847638\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"150.824842\" xlink:href=\"#m565d4626fc\" y=\"136.175695\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"148.063655\" xlink:href=\"#m565d4626fc\" y=\"136.653796\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"146.406943\" xlink:href=\"#m565d4626fc\" y=\"136.825912\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"145.412916\" xlink:href=\"#m565d4626fc\" y=\"136.887874\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"144.8165\" xlink:href=\"#m565d4626fc\" y=\"136.91018\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"144.45865\" xlink:href=\"#m565d4626fc\" y=\"136.91821\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 46.271875 143.1 \n",
       "L 46.271875 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 241.571875 143.1 \n",
       "L 241.571875 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 46.271875 143.1 \n",
       "L 241.571875 143.1 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 46.271875 7.2 \n",
       "L 241.571875 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p7395a43caa\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"46.271875\" y=\"7.2\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_trace(res):\n",
    "    n = max(abs(min(res)), abs(max(res)), 10)\n",
    "    f_line = np.arange(-n, n, 0.1)\n",
    "    d2l.set_figsize()\n",
    "    d2l.plt.plot(f_line, [x * x for x in f_line])\n",
    "    d2l.plt.plot(res, [x * x for x in res], '-o')\n",
    "    d2l.plt.xlabel('x')\n",
    "    d2l.plt.ylabel('f(x)')\n",
    "\n",
    "show_trace(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学习率\n",
    "\n",
    "上述梯度下降算法中的正数$\\eta$通常叫作学习率。这是一个超参数，需要人工设定。如果使用过小的学习率，会导致$x$更新缓慢从而需要更多的迭代才能得到较好的解。\n",
    "\n",
    "下面展示使用学习率$\\eta=0.05$时自变量$x$的迭代轨迹。可见，同样迭代10次后，当学习率过小时，最终$x$的值依然与最优解存在较大偏差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:33:51.314826Z",
     "start_time": "2019-11-29T06:33:51.129533Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "6"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 3.4867844009999995\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"182.259375pt\" version=\"1.1\" viewBox=\"0 0 248.771875 182.259375\" width=\"248.771875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M -0 182.259375 \n",
       "L 248.771875 182.259375 \n",
       "L 248.771875 0 \n",
       "L -0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.271875 143.1 \n",
       "L 241.571875 143.1 \n",
       "L 241.571875 7.2 \n",
       "L 46.271875 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m29175522a0\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.149148\" xlink:href=\"#m29175522a0\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 62.0625 27.6875 \n",
       "L 12.3125 27.6875 \n",
       "L 12.3125 35.15625 \n",
       "L 62.0625 35.15625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-8722\"/>\n",
       "       <path d=\"M 52.59375 0 \n",
       "L 9.234375 0 \n",
       "L 9.234375 8.25 \n",
       "L 26.171875 8.25 \n",
       "L 26.171875 65.671875 \n",
       "L 8.796875 60.640625 \n",
       "L 8.796875 69.4375 \n",
       "L 35.6875 77.25 \n",
       "L 35.6875 8.25 \n",
       "L 52.59375 8.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-49\"/>\n",
       "       <path d=\"M 4.4375 36.328125 \n",
       "Q 4.4375 56.34375 11.078125 66.625 \n",
       "Q 17.71875 76.90625 30.328125 76.90625 \n",
       "Q 54.34375 76.90625 54.34375 38.140625 \n",
       "Q 54.34375 19 47.578125 8.859375 \n",
       "Q 40.828125 -1.265625 28.609375 -1.265625 \n",
       "Q 17.09375 -1.265625 10.765625 8.34375 \n",
       "Q 4.4375 17.96875 4.4375 36.328125 \n",
       "z\n",
       "M 14.203125 36.765625 \n",
       "Q 14.203125 6.640625 29.5 6.640625 \n",
       "Q 44.53125 6.640625 44.53125 37.25 \n",
       "Q 44.53125 68.953125 29.78125 68.953125 \n",
       "Q 14.203125 68.953125 14.203125 36.765625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(45.576491 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"132.8125\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"99.535511\" xlink:href=\"#m29175522a0\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 8.640625 11.625 \n",
       "Q 16.265625 6.640625 24.515625 6.640625 \n",
       "Q 32.328125 6.640625 37.109375 11 \n",
       "Q 41.890625 15.375 41.890625 22.609375 \n",
       "Q 41.890625 29.890625 37.03125 33.890625 \n",
       "Q 32.171875 37.890625 22.953125 37.890625 \n",
       "Q 18.5 37.890625 10.984375 37.203125 \n",
       "L 10.984375 75.640625 \n",
       "L 48.1875 75.640625 \n",
       "L 48.1875 67.1875 \n",
       "L 19.875 67.1875 \n",
       "L 19.875 45.796875 \n",
       "Q 24.125 46.046875 26.265625 46.046875 \n",
       "Q 38.234375 46.046875 44.921875 39.984375 \n",
       "Q 51.609375 33.9375 51.609375 23.296875 \n",
       "Q 51.609375 12.3125 44.421875 5.515625 \n",
       "Q 37.25 -1.265625 24.609375 -1.265625 \n",
       "Q 14.015625 -1.265625 8.640625 1.859375 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-53\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(92.894886 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"143.921875\" xlink:href=\"#m29175522a0\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(140.989844 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"188.308239\" xlink:href=\"#m29175522a0\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(185.376207 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"232.694602\" xlink:href=\"#m29175522a0\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(226.83054 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 49.421875 54 \n",
       "L 31.34375 26.8125 \n",
       "L 49.171875 0 \n",
       "L 38.03125 0 \n",
       "L 27.734375 17.1875 \n",
       "Q 26.90625 18.5625 25.390625 21.390625 \n",
       "L 25.203125 21.390625 \n",
       "Q 25.140625 21.09375 22.75 17.1875 \n",
       "L 12.25 0 \n",
       "L 1.265625 0 \n",
       "L 19.625 26.515625 \n",
       "L 2.09375 54 \n",
       "L 13.234375 54 \n",
       "L 23.34375 35.9375 \n",
       "Q 24.265625 34.28125 25.59375 31.640625 \n",
       "L 25.78125 31.640625 \n",
       "L 38.921875 54 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(141.3875 172.573438)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m523261b6e7\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m523261b6e7\" y=\"136.922727\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(33.407813 140.919602)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m523261b6e7\" y=\"112.213636\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 51.171875 0 \n",
       "L 5.171875 0 \n",
       "L 5.171875 8.296875 \n",
       "L 27.15625 30.21875 \n",
       "Q 36.234375 39.3125 39.3125 44.609375 \n",
       "Q 42.390625 49.90625 42.390625 55.5625 \n",
       "Q 42.390625 62.015625 38.765625 65.453125 \n",
       "Q 35.15625 68.890625 28.328125 68.890625 \n",
       "Q 18.21875 68.890625 9.03125 60.296875 \n",
       "L 9.03125 70.015625 \n",
       "Q 17.96875 76.90625 29.828125 76.90625 \n",
       "Q 40.046875 76.90625 45.953125 71.390625 \n",
       "Q 51.859375 65.875 51.859375 56.546875 \n",
       "Q 51.859375 49.515625 48.0625 42.75 \n",
       "Q 44.28125 35.984375 33.734375 25.53125 \n",
       "L 16.5 8.6875 \n",
       "L 16.5 8.5 \n",
       "L 51.171875 8.5 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 116.210511)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m523261b6e7\" y=\"87.504545\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 55.078125 19.625 \n",
       "L 45.40625 19.625 \n",
       "L 45.40625 0 \n",
       "L 36.140625 0 \n",
       "L 36.140625 19.625 \n",
       "L 0.875 19.625 \n",
       "L 0.875 25.828125 \n",
       "L 34.328125 75.640625 \n",
       "L 45.40625 75.640625 \n",
       "L 45.40625 27.203125 \n",
       "L 55.078125 27.203125 \n",
       "z\n",
       "M 36.140625 27.203125 \n",
       "L 36.140625 58.890625 \n",
       "Q 36.140625 62.25 36.328125 66.65625 \n",
       "L 36.140625 66.65625 \n",
       "Q 35.453125 64.796875 33.25 60.84375 \n",
       "L 10.75 27.203125 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 91.50142)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-52\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m523261b6e7\" y=\"62.795455\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <defs>\n",
       "       <path d=\"M 49.703125 65.921875 \n",
       "Q 43.796875 68.953125 37.359375 68.953125 \n",
       "Q 27.546875 68.953125 21.578125 60.40625 \n",
       "Q 15.625 51.859375 15.53125 37.0625 \n",
       "L 15.765625 37.0625 \n",
       "Q 21.140625 47.46875 32.90625 47.46875 \n",
       "Q 42.71875 47.46875 48.578125 41.0625 \n",
       "Q 54.4375 34.671875 54.4375 24.078125 \n",
       "Q 54.4375 12.984375 47.671875 5.859375 \n",
       "Q 40.921875 -1.265625 30.421875 -1.265625 \n",
       "Q 18.890625 -1.265625 12.328125 7.734375 \n",
       "Q 5.765625 16.75 5.765625 33.296875 \n",
       "Q 5.765625 53.328125 14.375 65.109375 \n",
       "Q 23 76.90625 37.0625 76.90625 \n",
       "Q 45.125 76.90625 49.703125 74.703125 \n",
       "z\n",
       "M 16.109375 24.75 \n",
       "Q 16.109375 17.28125 20.09375 11.953125 \n",
       "Q 24.078125 6.640625 30.671875 6.640625 \n",
       "Q 36.921875 6.640625 40.828125 11.234375 \n",
       "Q 44.734375 15.828125 44.734375 23.046875 \n",
       "Q 44.734375 30.859375 41.046875 35.203125 \n",
       "Q 37.359375 39.546875 30.5625 39.546875 \n",
       "Q 24.125 39.546875 20.109375 35.15625 \n",
       "Q 16.109375 30.765625 16.109375 24.75 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-54\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 66.79233)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-54\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m523261b6e7\" y=\"38.086364\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <defs>\n",
       "       <path d=\"M 21.390625 39.796875 \n",
       "Q 7.859375 46.09375 7.859375 58.0625 \n",
       "Q 7.859375 66.40625 14.203125 71.65625 \n",
       "Q 20.5625 76.90625 30.421875 76.90625 \n",
       "Q 39.453125 76.90625 45.3125 71.96875 \n",
       "Q 51.171875 67.046875 51.171875 59.234375 \n",
       "Q 51.171875 46.625 37.0625 39.84375 \n",
       "L 37.0625 39.65625 \n",
       "Q 53.609375 33.6875 53.609375 19.34375 \n",
       "Q 53.609375 9.90625 46.796875 4.3125 \n",
       "Q 39.984375 -1.265625 27.78125 -1.265625 \n",
       "Q 17.78125 -1.265625 11.46875 4.25 \n",
       "Q 5.171875 9.765625 5.171875 18.5625 \n",
       "Q 5.171875 33.015625 21.390625 39.59375 \n",
       "z\n",
       "M 41.5 58.109375 \n",
       "Q 41.5 63.1875 38.328125 66.109375 \n",
       "Q 35.15625 69.046875 29.734375 69.046875 \n",
       "Q 24.65625 69.046875 21.09375 65.984375 \n",
       "Q 17.53125 62.9375 17.53125 58.203125 \n",
       "Q 17.53125 48.734375 29.390625 43.796875 \n",
       "Q 41.5 48.828125 41.5 58.109375 \n",
       "z\n",
       "M 28.8125 35.25 \n",
       "Q 14.796875 29.59375 14.796875 19.046875 \n",
       "Q 14.796875 13.578125 18.890625 10.0625 \n",
       "Q 23 6.546875 29.59375 6.546875 \n",
       "Q 35.984375 6.546875 39.984375 10.03125 \n",
       "Q 44 13.53125 44 18.75 \n",
       "Q 44 29.734375 28.8125 35.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-56\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(27.54375 42.083239)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-56\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.271875\" xlink:href=\"#m523261b6e7\" y=\"13.377273\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(21.679688 17.374148)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"117.285156\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 36.234375 71.78125 \n",
       "Q 33.546875 73.1875 30.421875 73.1875 \n",
       "Q 21.4375 73.1875 21.4375 62.0625 \n",
       "L 21.4375 54 \n",
       "L 34.078125 54 \n",
       "L 34.078125 46.09375 \n",
       "L 21.4375 46.09375 \n",
       "L 21.4375 0 \n",
       "L 11.96875 0 \n",
       "L 11.96875 46.09375 \n",
       "L 2.734375 46.09375 \n",
       "L 2.734375 54 \n",
       "L 11.96875 54 \n",
       "L 11.96875 62.5 \n",
       "Q 11.96875 71.1875 16.96875 76.140625 \n",
       "Q 21.96875 81.109375 29.6875 81.109375 \n",
       "Q 33.796875 81.109375 36.234375 80.171875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-102\"/>\n",
       "      <path d=\"M 23.921875 -17.1875 \n",
       "Q 7.46875 1.5625 7.46875 28.859375 \n",
       "Q 7.46875 56.15625 23.921875 75.640625 \n",
       "L 32.375 75.640625 \n",
       "Q 15.875 55.375 15.875 28.953125 \n",
       "Q 15.875 2.59375 32.28125 -17.1875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-40\"/>\n",
       "      <path d=\"M 1.21875 -17.1875 \n",
       "Q 17.671875 2.546875 17.671875 28.953125 \n",
       "Q 17.671875 55.421875 1.125 75.640625 \n",
       "L 9.625 75.640625 \n",
       "Q 26.125 56.296875 26.125 28.859375 \n",
       "Q 26.125 1.421875 9.625 -17.1875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-41\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(15.19375 82.758594)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-102\"/>\n",
       "      <use x=\"34.667969\" xlink:href=\"#MicrosoftYaHei-40\"/>\n",
       "      <use x=\"68.066406\" xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"118.75\" xlink:href=\"#MicrosoftYaHei-41\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#p37510b29d4)\" d=\"M 55.149148 13.377273 \n",
       "L 59.587784 25.422955 \n",
       "L 64.02642 36.850909 \n",
       "L 68.465057 47.661136 \n",
       "L 72.903693 57.853636 \n",
       "L 77.34233 67.428409 \n",
       "L 81.780966 76.385455 \n",
       "L 85.331875 83.106327 \n",
       "L 88.882784 89.431855 \n",
       "L 92.433693 95.362036 \n",
       "L 95.984602 100.896873 \n",
       "L 99.535511 106.036364 \n",
       "L 103.08642 110.780509 \n",
       "L 106.63733 115.129309 \n",
       "L 110.188239 119.082764 \n",
       "L 113.739148 122.640873 \n",
       "L 117.290057 125.803636 \n",
       "L 119.953239 127.916264 \n",
       "L 122.61642 129.806509 \n",
       "L 125.279602 131.474373 \n",
       "L 127.942784 132.919855 \n",
       "L 130.605966 134.142955 \n",
       "L 133.269148 135.143673 \n",
       "L 135.93233 135.922009 \n",
       "L 138.595511 136.477964 \n",
       "L 141.258693 136.811536 \n",
       "L 143.921875 136.922727 \n",
       "L 146.585057 136.811536 \n",
       "L 149.248239 136.477964 \n",
       "L 151.91142 135.922009 \n",
       "L 154.574602 135.143673 \n",
       "L 157.237784 134.142955 \n",
       "L 159.900966 132.919855 \n",
       "L 162.564148 131.474373 \n",
       "L 165.22733 129.806509 \n",
       "L 167.890511 127.916264 \n",
       "L 170.553693 125.803636 \n",
       "L 173.216875 123.468627 \n",
       "L 176.767784 120.009355 \n",
       "L 180.318693 116.154736 \n",
       "L 183.869602 111.904773 \n",
       "L 187.420511 107.259464 \n",
       "L 190.97142 102.218809 \n",
       "L 194.52233 96.782809 \n",
       "L 198.073239 90.951464 \n",
       "L 201.624148 84.724773 \n",
       "L 205.175057 78.102736 \n",
       "L 208.725966 71.085355 \n",
       "L 213.164602 61.757673 \n",
       "L 217.603239 51.812264 \n",
       "L 222.041875 41.249127 \n",
       "L 226.480511 30.068264 \n",
       "L 230.919148 18.269673 \n",
       "L 231.806875 15.835827 \n",
       "L 231.806875 15.835827 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_13\">\n",
       "    <path clip-path=\"url(#p37510b29d4)\" d=\"M 232.694602 13.377273 \n",
       "L 223.81733 36.850909 \n",
       "L 215.827784 55.864555 \n",
       "L 208.637193 71.265607 \n",
       "L 202.165661 83.74046 \n",
       "L 196.341283 93.845091 \n",
       "L 191.099342 102.029842 \n",
       "L 186.381595 108.65949 \n",
       "L 182.135623 114.029505 \n",
       "L 178.314248 118.379217 \n",
       "L 174.875011 121.902484 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mc2baf64e69\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p37510b29d4)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"232.694602\" xlink:href=\"#mc2baf64e69\" y=\"13.377273\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"223.81733\" xlink:href=\"#mc2baf64e69\" y=\"36.850909\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"215.827784\" xlink:href=\"#mc2baf64e69\" y=\"55.864555\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"208.637193\" xlink:href=\"#mc2baf64e69\" y=\"71.265607\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"202.165661\" xlink:href=\"#mc2baf64e69\" y=\"83.74046\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"196.341283\" xlink:href=\"#mc2baf64e69\" y=\"93.845091\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"191.099342\" xlink:href=\"#mc2baf64e69\" y=\"102.029842\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"186.381595\" xlink:href=\"#mc2baf64e69\" y=\"108.65949\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"182.135623\" xlink:href=\"#mc2baf64e69\" y=\"114.029505\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"178.314248\" xlink:href=\"#mc2baf64e69\" y=\"118.379217\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"174.875011\" xlink:href=\"#mc2baf64e69\" y=\"121.902484\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 46.271875 143.1 \n",
       "L 46.271875 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 241.571875 143.1 \n",
       "L 241.571875 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 46.271875 143.1 \n",
       "L 241.571875 143.1 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 46.271875 7.2 \n",
       "L 241.571875 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p37510b29d4\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"46.271875\" y=\"7.2\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(0.05))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果使用过大的学习率，$\\left|\\eta f'(x)\\right|$可能会过大从而使前面提到的一阶泰勒展开公式不再成立：这时我们无法保证迭代$x$会降低$f(x)$的值。\n",
    "\n",
    "举个例子，当设学习率$\\eta=1.1$时，可以看到$x$不断越过（overshoot）最优解$x=0$并逐渐发散。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:34:00.728298Z",
     "start_time": "2019-11-29T06:34:00.560342Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 61.917364224000096\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"185.436172pt\" version=\"1.1\" viewBox=\"0 0 254.635938 185.436172\" width=\"254.635938pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 185.436172 \n",
       "L 254.635938 185.436172 \n",
       "L 254.635938 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 52.135938 146.276797 \n",
       "L 247.435938 146.276797 \n",
       "L 247.435938 10.376797 \n",
       "L 52.135938 10.376797 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mc27364c461\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"78.099482\" xlink:href=\"#mc27364c461\" y=\"146.276797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −50 -->\n",
       "      <defs>\n",
       "       <path d=\"M 62.0625 27.6875 \n",
       "L 12.3125 27.6875 \n",
       "L 12.3125 35.15625 \n",
       "L 62.0625 35.15625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-8722\"/>\n",
       "       <path d=\"M 8.640625 11.625 \n",
       "Q 16.265625 6.640625 24.515625 6.640625 \n",
       "Q 32.328125 6.640625 37.109375 11 \n",
       "Q 41.890625 15.375 41.890625 22.609375 \n",
       "Q 41.890625 29.890625 37.03125 33.890625 \n",
       "Q 32.171875 37.890625 22.953125 37.890625 \n",
       "Q 18.5 37.890625 10.984375 37.203125 \n",
       "L 10.984375 75.640625 \n",
       "L 48.1875 75.640625 \n",
       "L 48.1875 67.1875 \n",
       "L 19.875 67.1875 \n",
       "L 19.875 45.796875 \n",
       "Q 24.125 46.046875 26.265625 46.046875 \n",
       "Q 38.234375 46.046875 44.921875 39.984375 \n",
       "Q 51.609375 33.9375 51.609375 23.296875 \n",
       "Q 51.609375 12.3125 44.421875 5.515625 \n",
       "Q 37.25 -1.265625 24.609375 -1.265625 \n",
       "Q 14.015625 -1.265625 8.640625 1.859375 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-53\"/>\n",
       "       <path d=\"M 4.4375 36.328125 \n",
       "Q 4.4375 56.34375 11.078125 66.625 \n",
       "Q 17.71875 76.90625 30.328125 76.90625 \n",
       "Q 54.34375 76.90625 54.34375 38.140625 \n",
       "Q 54.34375 19 47.578125 8.859375 \n",
       "Q 40.828125 -1.265625 28.609375 -1.265625 \n",
       "Q 17.09375 -1.265625 10.765625 8.34375 \n",
       "Q 4.4375 17.96875 4.4375 36.328125 \n",
       "z\n",
       "M 14.203125 36.765625 \n",
       "Q 14.203125 6.640625 29.5 6.640625 \n",
       "Q 44.53125 6.640625 44.53125 37.25 \n",
       "Q 44.53125 68.953125 29.78125 68.953125 \n",
       "Q 14.203125 68.953125 14.203125 36.765625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(68.526826 161.270547)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "       <use x=\"132.8125\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"113.94271\" xlink:href=\"#mc27364c461\" y=\"146.276797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −25 -->\n",
       "      <defs>\n",
       "       <path d=\"M 51.171875 0 \n",
       "L 5.171875 0 \n",
       "L 5.171875 8.296875 \n",
       "L 27.15625 30.21875 \n",
       "Q 36.234375 39.3125 39.3125 44.609375 \n",
       "Q 42.390625 49.90625 42.390625 55.5625 \n",
       "Q 42.390625 62.015625 38.765625 65.453125 \n",
       "Q 35.15625 68.890625 28.328125 68.890625 \n",
       "Q 18.21875 68.890625 9.03125 60.296875 \n",
       "L 9.03125 70.015625 \n",
       "Q 17.96875 76.90625 29.828125 76.90625 \n",
       "Q 40.046875 76.90625 45.953125 71.390625 \n",
       "Q 51.859375 65.875 51.859375 56.546875 \n",
       "Q 51.859375 49.515625 48.0625 42.75 \n",
       "Q 44.28125 35.984375 33.734375 25.53125 \n",
       "L 16.5 8.6875 \n",
       "L 16.5 8.5 \n",
       "L 51.171875 8.5 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(104.370054 161.270547)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "       <use x=\"132.8125\" xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"149.785938\" xlink:href=\"#mc27364c461\" y=\"146.276797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(146.853906 161.270547)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"185.629165\" xlink:href=\"#mc27364c461\" y=\"146.276797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 25 -->\n",
       "      <g transform=\"translate(179.765103 161.270547)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"221.472393\" xlink:href=\"#mc27364c461\" y=\"146.276797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(215.60833 161.270547)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-53\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 49.421875 54 \n",
       "L 31.34375 26.8125 \n",
       "L 49.171875 0 \n",
       "L 38.03125 0 \n",
       "L 27.734375 17.1875 \n",
       "Q 26.90625 18.5625 25.390625 21.390625 \n",
       "L 25.203125 21.390625 \n",
       "Q 25.140625 21.09375 22.75 17.1875 \n",
       "L 12.25 0 \n",
       "L 1.265625 0 \n",
       "L 19.625 26.515625 \n",
       "L 2.09375 54 \n",
       "L 13.234375 54 \n",
       "L 23.34375 35.9375 \n",
       "Q 24.265625 34.28125 25.59375 31.640625 \n",
       "L 25.78125 31.640625 \n",
       "L 38.921875 54 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(147.251563 175.750235)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m37be0da8fd\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.135938\" xlink:href=\"#m37be0da8fd\" y=\"140.099534\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(39.271875 144.096409)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.135938\" xlink:href=\"#m37be0da8fd\" y=\"107.873869\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 1000 -->\n",
       "      <defs>\n",
       "       <path d=\"M 52.59375 0 \n",
       "L 9.234375 0 \n",
       "L 9.234375 8.25 \n",
       "L 26.171875 8.25 \n",
       "L 26.171875 65.671875 \n",
       "L 8.796875 60.640625 \n",
       "L 8.796875 69.4375 \n",
       "L 35.6875 77.25 \n",
       "L 35.6875 8.25 \n",
       "L 52.59375 8.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-49\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.679688 111.870744)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"117.285156\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"175.927734\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.135938\" xlink:href=\"#m37be0da8fd\" y=\"75.648205\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 2000 -->\n",
       "      <g transform=\"translate(21.679688 79.64508)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"117.285156\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"175.927734\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.135938\" xlink:href=\"#m37be0da8fd\" y=\"43.42254\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 3000 -->\n",
       "      <defs>\n",
       "       <path d=\"M 6.984375 12.640625 \n",
       "Q 14.75 6.640625 24.515625 6.640625 \n",
       "Q 32.328125 6.640625 36.890625 10.46875 \n",
       "Q 41.453125 14.3125 41.453125 20.75 \n",
       "Q 41.453125 35.109375 20.90625 35.109375 \n",
       "L 14.59375 35.109375 \n",
       "L 14.59375 43.015625 \n",
       "L 20.609375 43.015625 \n",
       "Q 38.8125 43.015625 38.8125 56.5 \n",
       "Q 38.8125 68.953125 24.90625 68.953125 \n",
       "Q 16.9375 68.953125 9.90625 63.578125 \n",
       "L 9.90625 72.609375 \n",
       "Q 17.328125 76.90625 27.25 76.90625 \n",
       "Q 36.921875 76.90625 42.78125 71.84375 \n",
       "Q 48.640625 66.796875 48.640625 58.734375 \n",
       "Q 48.640625 43.890625 33.5 39.65625 \n",
       "L 33.5 39.453125 \n",
       "Q 41.703125 38.578125 46.453125 33.671875 \n",
       "Q 51.21875 28.765625 51.21875 21.4375 \n",
       "Q 51.21875 11.234375 43.875 4.984375 \n",
       "Q 36.53125 -1.265625 24.3125 -1.265625 \n",
       "Q 13.578125 -1.265625 6.984375 2.734375 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-51\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.679688 47.419415)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-51\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"117.285156\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"175.927734\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.135938\" xlink:href=\"#m37be0da8fd\" y=\"11.196875\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 4000 -->\n",
       "      <defs>\n",
       "       <path d=\"M 55.078125 19.625 \n",
       "L 45.40625 19.625 \n",
       "L 45.40625 0 \n",
       "L 36.140625 0 \n",
       "L 36.140625 19.625 \n",
       "L 0.875 19.625 \n",
       "L 0.875 25.828125 \n",
       "L 34.328125 75.640625 \n",
       "L 45.40625 75.640625 \n",
       "L 45.40625 27.203125 \n",
       "L 55.078125 27.203125 \n",
       "z\n",
       "M 36.140625 27.203125 \n",
       "L 36.140625 58.890625 \n",
       "Q 36.140625 62.25 36.328125 66.65625 \n",
       "L 36.140625 66.65625 \n",
       "Q 35.453125 64.796875 33.25 60.84375 \n",
       "L 10.75 27.203125 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.679688 15.19375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-52\"/>\n",
       "       <use x=\"58.642578\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"117.285156\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "       <use x=\"175.927734\" xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 36.234375 71.78125 \n",
       "Q 33.546875 73.1875 30.421875 73.1875 \n",
       "Q 21.4375 73.1875 21.4375 62.0625 \n",
       "L 21.4375 54 \n",
       "L 34.078125 54 \n",
       "L 34.078125 46.09375 \n",
       "L 21.4375 46.09375 \n",
       "L 21.4375 0 \n",
       "L 11.96875 0 \n",
       "L 11.96875 46.09375 \n",
       "L 2.734375 46.09375 \n",
       "L 2.734375 54 \n",
       "L 11.96875 54 \n",
       "L 11.96875 62.5 \n",
       "Q 11.96875 71.1875 16.96875 76.140625 \n",
       "Q 21.96875 81.109375 29.6875 81.109375 \n",
       "Q 33.796875 81.109375 36.234375 80.171875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-102\"/>\n",
       "      <path d=\"M 23.921875 -17.1875 \n",
       "Q 7.46875 1.5625 7.46875 28.859375 \n",
       "Q 7.46875 56.15625 23.921875 75.640625 \n",
       "L 32.375 75.640625 \n",
       "Q 15.875 55.375 15.875 28.953125 \n",
       "Q 15.875 2.59375 32.28125 -17.1875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-40\"/>\n",
       "      <path d=\"M 1.21875 -17.1875 \n",
       "Q 17.671875 2.546875 17.671875 28.953125 \n",
       "Q 17.671875 55.421875 1.125 75.640625 \n",
       "L 9.625 75.640625 \n",
       "Q 26.125 56.296875 26.125 28.859375 \n",
       "Q 26.125 1.421875 9.625 -17.1875 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-41\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(15.19375 85.935391)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-102\"/>\n",
       "      <use x=\"34.667969\" xlink:href=\"#MicrosoftYaHei-40\"/>\n",
       "      <use x=\"68.066406\" xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"118.75\" xlink:href=\"#MicrosoftYaHei-41\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_11\">\n",
       "    <path clip-path=\"url(#p23ce1b8c10)\" d=\"M 61.01321 16.55407 \n",
       "L 65.601143 28.99418 \n",
       "L 70.045704 40.416169 \n",
       "L 74.346891 50.879977 \n",
       "L 78.648078 60.763723 \n",
       "L 82.805893 69.76663 \n",
       "L 86.820334 77.94477 \n",
       "L 90.834776 85.617612 \n",
       "L 94.705844 92.537873 \n",
       "L 98.43354 98.757758 \n",
       "L 102.161236 104.541952 \n",
       "L 105.745558 109.6928 \n",
       "L 109.186508 114.25864 \n",
       "L 112.627458 118.45324 \n",
       "L 115.925035 122.124706 \n",
       "L 119.079239 125.317509 \n",
       "L 122.233443 128.198368 \n",
       "L 125.244274 130.65728 \n",
       "L 128.255105 132.831963 \n",
       "L 131.122564 134.638838 \n",
       "L 133.990022 136.187908 \n",
       "L 136.85748 137.479173 \n",
       "L 139.581565 138.467083 \n",
       "L 142.305651 139.222323 \n",
       "L 145.029736 139.744894 \n",
       "L 147.753821 140.034795 \n",
       "L 150.477906 140.092027 \n",
       "L 153.201992 139.91659 \n",
       "L 155.926077 139.508484 \n",
       "L 158.650162 138.867708 \n",
       "L 161.374248 137.994264 \n",
       "L 164.098333 136.888149 \n",
       "L 166.822418 135.549366 \n",
       "L 169.689876 133.88876 \n",
       "L 172.557335 131.970348 \n",
       "L 175.568166 129.678554 \n",
       "L 178.578997 127.102528 \n",
       "L 181.733201 124.09898 \n",
       "L 184.887405 120.783488 \n",
       "L 188.184982 116.983756 \n",
       "L 191.482559 112.843076 \n",
       "L 194.923509 108.158862 \n",
       "L 198.364459 103.103407 \n",
       "L 201.948781 97.442544 \n",
       "L 205.676477 91.127935 \n",
       "L 209.404173 84.377634 \n",
       "L 213.275241 76.906558 \n",
       "L 217.289683 68.662499 \n",
       "L 221.304124 59.913142 \n",
       "L 225.461939 50.318618 \n",
       "L 229.763126 39.822854 \n",
       "L 234.207686 28.367844 \n",
       "L 238.508874 16.69262 \n",
       "L 238.508874 16.69262 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#p23ce1b8c10)\" d=\"M 164.123229 136.876968 \n",
       "L 132.581188 135.459038 \n",
       "L 170.431637 133.41722 \n",
       "L 125.011099 130.477002 \n",
       "L 179.515744 126.243088 \n",
       "L 114.110169 120.146252 \n",
       "L 192.596859 111.366808 \n",
       "L 98.412831 98.724408 \n",
       "L 211.433665 80.519353 \n",
       "L 75.808665 54.304073 \n",
       "L 238.558665 16.55407 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mce341722ae\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p23ce1b8c10)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"164.123229\" xlink:href=\"#mce341722ae\" y=\"136.876968\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"132.581188\" xlink:href=\"#mce341722ae\" y=\"135.459038\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"170.431637\" xlink:href=\"#mce341722ae\" y=\"133.41722\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"125.011099\" xlink:href=\"#mce341722ae\" y=\"130.477002\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"179.515744\" xlink:href=\"#mce341722ae\" y=\"126.243088\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"114.110169\" xlink:href=\"#mce341722ae\" y=\"120.146252\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"192.596859\" xlink:href=\"#mce341722ae\" y=\"111.366808\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"98.412831\" xlink:href=\"#mce341722ae\" y=\"98.724408\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.433665\" xlink:href=\"#mce341722ae\" y=\"80.519353\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"75.808665\" xlink:href=\"#mce341722ae\" y=\"54.304073\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"238.558665\" xlink:href=\"#mce341722ae\" y=\"16.55407\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 52.135938 146.276797 \n",
       "L 52.135938 10.376797 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 247.435938 146.276797 \n",
       "L 247.435938 10.376797 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 52.135938 146.276797 \n",
       "L 247.435938 146.276797 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 52.135938 10.376797 \n",
       "L 247.435938 10.376797 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p23ce1b8c10\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"52.135938\" y=\"10.376797\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多维梯度下降\n",
    "\n",
    "在了解了一维梯度下降之后，我们再考虑一种更广义的情况：目标函数的输入为向量，输出为标量。假设目标函数$f: \\mathbb{R}^d \\rightarrow \\mathbb{R}$的输入是一个$d$维向量$\\boldsymbol{x} = [x_1, x_2, \\ldots, x_d]^\\top$。目标函数$f(\\boldsymbol{x})$有关$\\boldsymbol{x}$的梯度是一个由$d$个偏导数组成的向量：\n",
    "\n",
    "$$\\nabla_{\\boldsymbol{x}} f(\\boldsymbol{x}) = \\bigg[\\frac{\\partial f(\\boldsymbol{x})}{\\partial x_1}, \\frac{\\partial f(\\boldsymbol{x})}{\\partial x_2}, \\ldots, \\frac{\\partial f(\\boldsymbol{x})}{\\partial x_d}\\bigg]^\\top.$$\n",
    "\n",
    "\n",
    "为表示简洁，我们用$\\nabla f(\\boldsymbol{x})$代替$\\nabla_{\\boldsymbol{x}} f(\\boldsymbol{x})$。梯度中每个偏导数元素$\\partial f(\\boldsymbol{x})/\\partial x_i$代表着$f$在$\\boldsymbol{x}$有关输入$x_i$的变化率。为了测量$f$沿着单位向量$\\boldsymbol{u}$（即$\\|\\boldsymbol{u}\\|=1$）方向上的变化率，在多元微积分中，我们定义$f$在$\\boldsymbol{x}$上沿着$\\boldsymbol{u}$方向的方向导数为\n",
    "\n",
    "$$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x}) = \\lim_{h \\rightarrow 0}  \\frac{f(\\boldsymbol{x} + h \\boldsymbol{u}) - f(\\boldsymbol{x})}{h}.$$\n",
    "\n",
    "依据方向导数性质 \\[1，14.6节定理三\\]，以上方向导数可以改写为\n",
    "\n",
    "$$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x}) = \\nabla f(\\boldsymbol{x}) \\cdot \\boldsymbol{u}.$$\n",
    "\n",
    "方向导数$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x})$给出了$f$在$\\boldsymbol{x}$上沿着所有可能方向的变化率。为了最小化$f$，我们希望找到$f$能被降低最快的方向。因此，我们可以通过单位向量$\\boldsymbol{u}$来最小化方向导数$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x})$。\n",
    "\n",
    "由于$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x}) = \\|\\nabla f(\\boldsymbol{x})\\| \\cdot \\|\\boldsymbol{u}\\|  \\cdot \\text{cos} (\\theta) = \\|\\nabla f(\\boldsymbol{x})\\|  \\cdot \\text{cos} (\\theta)$，\n",
    "其中$\\theta$为梯度$\\nabla f(\\boldsymbol{x})$和单位向量$\\boldsymbol{u}$之间的夹角，当$\\theta = \\pi$时，$\\text{cos}(\\theta)$取得最小值$-1$。因此，当$\\boldsymbol{u}$在梯度方向$\\nabla f(\\boldsymbol{x})$的相反方向时，方向导数$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x})$被最小化。因此，我们可能通过梯度下降算法来不断降低目标函数$f$的值：\n",
    "\n",
    "$$\\boldsymbol{x} \\leftarrow \\boldsymbol{x} - \\eta \\nabla f(\\boldsymbol{x}).$$\n",
    "\n",
    "同样，其中$\\eta$（取正数）称作学习率。\n",
    "\n",
    "下面我们构造一个输入为二维向量$\\boldsymbol{x} = [x_1, x_2]^\\top$和输出为标量的目标函数$f(\\boldsymbol{x})=x_1^2+2x_2^2$。那么，梯度$\\nabla f(\\boldsymbol{x}) = [2x_1, 4x_2]^\\top$。我们将观察梯度下降从初始位置$[-5,-2]$开始对自变量$\\boldsymbol{x}$的迭代轨迹。我们先定义两个辅助函数，第一个函数使用给定的自变量更新函数，从初始位置$[-5,-2]$开始迭代自变量$\\boldsymbol{x}$共20次，第二个函数对自变量$\\boldsymbol{x}$的迭代轨迹进行可视化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:34:09.919092Z",
     "start_time": "2019-11-29T06:34:09.905055Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "10"
    }
   },
   "outputs": [],
   "source": [
    "def train_2d(trainer):  # 本函数将保存在d2lzh包中方便以后使用\n",
    "    x1, x2, s1, s2 = -5, -2, 0, 0  # s1和s2是自变量状态，本章后续几节会使用\n",
    "    results = [(x1, x2)]\n",
    "    for i in range(20):\n",
    "        x1, x2, s1, s2 = trainer(x1, x2, s1, s2)\n",
    "        results.append((x1, x2))\n",
    "    print('epoch %d, x1 %f, x2 %f' % (i + 1, x1, x2))\n",
    "    return results\n",
    "\n",
    "def show_trace_2d(f, results):  # 本函数将保存在d2lzh包中方便以后使用\n",
    "    d2l.plt.plot(*zip(*results), '-o', color='#ff7f0e')\n",
    "    x1, x2 = np.meshgrid(np.arange(-5.5, 1.0, 0.1), np.arange(-3.0, 1.0, 0.1))\n",
    "    d2l.plt.contour(x1, x2, f(x1, x2), colors='#1f77b4')\n",
    "    d2l.plt.xlabel('x1')\n",
    "    d2l.plt.ylabel('x2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，观察学习率为$0.1$时自变量的迭代轨迹。使用梯度下降对自变量$\\boldsymbol{x}$迭代20次后，可见最终$\\boldsymbol{x}$的值较接近最优解$[0,0]$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:34:15.031792Z",
     "start_time": "2019-11-29T06:34:14.830991Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "15"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.057646, x2 -0.000073\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"182.259375pt\" version=\"1.1\" viewBox=\"0 0 244.460938 182.259375\" width=\"244.460938pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 182.259375 \n",
       "L 244.460938 182.259375 \n",
       "L 244.460938 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 41.960938 143.1 \n",
       "L 237.260938 143.1 \n",
       "L 237.260938 7.2 \n",
       "L 41.960938 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mfc526a53ea\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"87.734375\" xlink:href=\"#mfc526a53ea\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 62.0625 27.6875 \n",
       "L 12.3125 27.6875 \n",
       "L 12.3125 35.15625 \n",
       "L 62.0625 35.15625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-8722\"/>\n",
       "       <path d=\"M 55.078125 19.625 \n",
       "L 45.40625 19.625 \n",
       "L 45.40625 0 \n",
       "L 36.140625 0 \n",
       "L 36.140625 19.625 \n",
       "L 0.875 19.625 \n",
       "L 0.875 25.828125 \n",
       "L 34.328125 75.640625 \n",
       "L 45.40625 75.640625 \n",
       "L 45.40625 27.203125 \n",
       "L 55.078125 27.203125 \n",
       "z\n",
       "M 36.140625 27.203125 \n",
       "L 36.140625 58.890625 \n",
       "Q 36.140625 62.25 36.328125 66.65625 \n",
       "L 36.140625 66.65625 \n",
       "Q 35.453125 64.796875 33.25 60.84375 \n",
       "L 10.75 27.203125 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(81.09375 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"148.765625\" xlink:href=\"#mfc526a53ea\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 51.171875 0 \n",
       "L 5.171875 0 \n",
       "L 5.171875 8.296875 \n",
       "L 27.15625 30.21875 \n",
       "Q 36.234375 39.3125 39.3125 44.609375 \n",
       "Q 42.390625 49.90625 42.390625 55.5625 \n",
       "Q 42.390625 62.015625 38.765625 65.453125 \n",
       "Q 35.15625 68.890625 28.328125 68.890625 \n",
       "Q 18.21875 68.890625 9.03125 60.296875 \n",
       "L 9.03125 70.015625 \n",
       "Q 17.96875 76.90625 29.828125 76.90625 \n",
       "Q 40.046875 76.90625 45.953125 71.390625 \n",
       "Q 51.859375 65.875 51.859375 56.546875 \n",
       "Q 51.859375 49.515625 48.0625 42.75 \n",
       "Q 44.28125 35.984375 33.734375 25.53125 \n",
       "L 16.5 8.6875 \n",
       "L 16.5 8.5 \n",
       "L 51.171875 8.5 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(142.125 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"209.796875\" xlink:href=\"#mfc526a53ea\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 4.4375 36.328125 \n",
       "Q 4.4375 56.34375 11.078125 66.625 \n",
       "Q 17.71875 76.90625 30.328125 76.90625 \n",
       "Q 54.34375 76.90625 54.34375 38.140625 \n",
       "Q 54.34375 19 47.578125 8.859375 \n",
       "Q 40.828125 -1.265625 28.609375 -1.265625 \n",
       "Q 17.09375 -1.265625 10.765625 8.34375 \n",
       "Q 4.4375 17.96875 4.4375 36.328125 \n",
       "z\n",
       "M 14.203125 36.765625 \n",
       "Q 14.203125 6.640625 29.5 6.640625 \n",
       "Q 44.53125 6.640625 44.53125 37.25 \n",
       "Q 44.53125 68.953125 29.78125 68.953125 \n",
       "Q 14.203125 68.953125 14.203125 36.765625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(206.864844 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 49.421875 54 \n",
       "L 31.34375 26.8125 \n",
       "L 49.171875 0 \n",
       "L 38.03125 0 \n",
       "L 27.734375 17.1875 \n",
       "Q 26.90625 18.5625 25.390625 21.390625 \n",
       "L 25.203125 21.390625 \n",
       "Q 25.140625 21.09375 22.75 17.1875 \n",
       "L 12.25 0 \n",
       "L 1.265625 0 \n",
       "L 19.625 26.515625 \n",
       "L 2.09375 54 \n",
       "L 13.234375 54 \n",
       "L 23.34375 35.9375 \n",
       "Q 24.265625 34.28125 25.59375 31.640625 \n",
       "L 25.78125 31.640625 \n",
       "L 38.921875 54 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-120\"/>\n",
       "      <path d=\"M 52.59375 0 \n",
       "L 9.234375 0 \n",
       "L 9.234375 8.25 \n",
       "L 26.171875 8.25 \n",
       "L 26.171875 65.671875 \n",
       "L 8.796875 60.640625 \n",
       "L 8.796875 69.4375 \n",
       "L 35.6875 77.25 \n",
       "L 35.6875 8.25 \n",
       "L 52.59375 8.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-49\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(134.144531 172.573438)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"50.683594\" xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m7515ad5ed4\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7515ad5ed4\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −3 -->\n",
       "      <defs>\n",
       "       <path d=\"M 6.984375 12.640625 \n",
       "Q 14.75 6.640625 24.515625 6.640625 \n",
       "Q 32.328125 6.640625 36.890625 10.46875 \n",
       "Q 41.453125 14.3125 41.453125 20.75 \n",
       "Q 41.453125 35.109375 20.90625 35.109375 \n",
       "L 14.59375 35.109375 \n",
       "L 14.59375 43.015625 \n",
       "L 20.609375 43.015625 \n",
       "Q 38.8125 43.015625 38.8125 56.5 \n",
       "Q 38.8125 68.953125 24.90625 68.953125 \n",
       "Q 16.9375 68.953125 9.90625 63.578125 \n",
       "L 9.90625 72.609375 \n",
       "Q 17.328125 76.90625 27.25 76.90625 \n",
       "Q 36.921875 76.90625 42.78125 71.84375 \n",
       "Q 48.640625 66.796875 48.640625 58.734375 \n",
       "Q 48.640625 43.890625 33.5 39.65625 \n",
       "L 33.5 39.453125 \n",
       "Q 41.703125 38.578125 46.453125 33.671875 \n",
       "Q 51.21875 28.765625 51.21875 21.4375 \n",
       "Q 51.21875 11.234375 43.875 4.984375 \n",
       "Q 36.53125 -1.265625 24.3125 -1.265625 \n",
       "Q 13.578125 -1.265625 6.984375 2.734375 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-51\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.679688 147.096875)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-51\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7515ad5ed4\" y=\"108.253846\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(21.679688 112.250721)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7515ad5ed4\" y=\"73.407692\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(21.679688 77.404567)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7515ad5ed4\" y=\"38.561538\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.096875 42.558413)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(15.19375 80.616406)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"50.683594\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\"/>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 237.260938 104.627963 \n",
       "L 236.722426 104.769231 \n",
       "L 234.209375 105.394675 \n",
       "L 231.157813 106.064793 \n",
       "L 228.10625 106.645562 \n",
       "L 225.054688 107.136982 \n",
       "L 222.003125 107.539053 \n",
       "L 218.951563 107.851775 \n",
       "L 215.9 108.075148 \n",
       "L 212.848438 108.209172 \n",
       "L 209.796875 108.253846 \n",
       "L 209.796875 108.253846 \n",
       "L 209.796875 108.253846 \n",
       "L 206.745313 108.209172 \n",
       "L 203.69375 108.075148 \n",
       "L 200.642188 107.851775 \n",
       "L 197.590625 107.539053 \n",
       "L 194.539063 107.136982 \n",
       "L 191.4875 106.645562 \n",
       "L 188.435938 106.064793 \n",
       "L 185.384375 105.394675 \n",
       "L 182.871324 104.769231 \n",
       "L 182.332813 104.627963 \n",
       "L 179.28125 103.733264 \n",
       "L 176.229688 102.744387 \n",
       "L 173.178125 101.661331 \n",
       "L 172.201625 101.284615 \n",
       "L 170.126563 100.438352 \n",
       "L 167.075 99.094286 \n",
       "L 164.339116 97.8 \n",
       "L 164.023438 97.641608 \n",
       "L 160.971875 96.004895 \n",
       "L 158.012784 94.315385 \n",
       "L 157.920313 94.259181 \n",
       "L 154.86875 92.29206 \n",
       "L 152.724409 90.830769 \n",
       "L 151.817188 90.169894 \n",
       "L 148.765625 87.82679 \n",
       "L 148.170198 87.346154 \n",
       "L 145.714063 85.216667 \n",
       "L 144.223765 83.861538 \n",
       "L 142.6625 82.328308 \n",
       "L 140.76375 80.376923 \n",
       "L 139.610938 79.08913 \n",
       "L 137.728059 76.892308 \n",
       "L 136.559375 75.398901 \n",
       "L 135.064732 73.407692 \n",
       "L 133.507813 71.115182 \n",
       "L 132.729963 69.923077 \n",
       "L 130.695588 66.438462 \n",
       "L 130.45625 65.973846 \n",
       "L 128.959257 62.953846 \n",
       "L 127.462264 59.469231 \n",
       "L 127.404688 59.310839 \n",
       "L 126.239545 55.984615 \n",
       "L 125.240852 52.5 \n",
       "L 124.464091 49.015385 \n",
       "L 124.353125 48.318462 \n",
       "L 123.924836 45.530769 \n",
       "L 123.603618 42.046154 \n",
       "L 123.496546 38.561538 \n",
       "L 123.603618 35.076923 \n",
       "L 123.924836 31.592308 \n",
       "L 124.353125 28.804615 \n",
       "L 124.464091 28.107692 \n",
       "L 125.240852 24.623077 \n",
       "L 126.239545 21.138462 \n",
       "L 127.404688 17.812238 \n",
       "L 127.462264 17.653846 \n",
       "L 128.959257 14.169231 \n",
       "L 130.45625 11.149231 \n",
       "L 130.695588 10.684615 \n",
       "L 132.729963 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 237.260938 134.578531 \n",
       "L 234.209375 135.117063 \n",
       "L 231.157813 135.592238 \n",
       "L 228.10625 136.004056 \n",
       "L 226.996591 136.130769 \n",
       "L 225.054688 136.344737 \n",
       "L 222.003125 136.619838 \n",
       "L 218.951563 136.833806 \n",
       "L 215.9 136.98664 \n",
       "L 212.848438 137.07834 \n",
       "L 209.796875 137.108907 \n",
       "L 206.745313 137.07834 \n",
       "L 203.69375 136.98664 \n",
       "L 200.642188 136.833806 \n",
       "L 197.590625 136.619838 \n",
       "L 194.539063 136.344737 \n",
       "L 192.597159 136.130769 \n",
       "L 191.4875 136.004056 \n",
       "L 188.435938 135.592238 \n",
       "L 185.384375 135.117063 \n",
       "L 182.332813 134.578531 \n",
       "L 179.28125 133.976643 \n",
       "L 176.229688 133.311399 \n",
       "L 173.443478 132.646154 \n",
       "L 173.178125 132.580406 \n",
       "L 170.126563 131.758563 \n",
       "L 167.075 130.870972 \n",
       "L 164.023438 129.917634 \n",
       "L 161.759375 129.161538 \n",
       "L 160.971875 128.888235 \n",
       "L 157.920313 127.76086 \n",
       "L 154.86875 126.565158 \n",
       "L 152.724409 125.676923 \n",
       "L 151.817188 125.285793 \n",
       "L 148.765625 123.899058 \n",
       "L 145.714063 122.441209 \n",
       "L 145.217297 122.192308 \n",
       "L 142.6625 120.857774 \n",
       "L 139.610938 119.189607 \n",
       "L 138.766888 118.707692 \n",
       "L 136.559375 117.391282 \n",
       "L 133.507813 115.494103 \n",
       "L 133.088971 115.223077 \n",
       "L 130.45625 113.44025 \n",
       "L 128.038031 111.738462 \n",
       "L 127.404688 111.271013 \n",
       "L 124.353125 108.933771 \n",
       "L 123.496546 108.253846 \n",
       "L 121.301563 106.422189 \n",
       "L 119.387871 104.769231 \n",
       "L 118.25 103.733264 \n",
       "L 115.648668 101.284615 \n",
       "L 115.198438 100.836593 \n",
       "L 112.24375 97.8 \n",
       "L 112.146875 97.694406 \n",
       "L 109.14226 94.315385 \n",
       "L 109.095313 94.259181 \n",
       "L 106.317024 90.830769 \n",
       "L 106.04375 90.470292 \n",
       "L 103.744022 87.346154 \n",
       "L 102.992188 86.249145 \n",
       "L 101.401937 83.861538 \n",
       "L 99.940625 81.492 \n",
       "L 99.271789 80.376923 \n",
       "L 97.348887 76.892308 \n",
       "L 96.889063 75.97967 \n",
       "L 95.62775 73.407692 \n",
       "L 94.081625 69.923077 \n",
       "L 93.8375 69.308145 \n",
       "L 92.727841 66.438462 \n",
       "L 91.53892 62.953846 \n",
       "L 90.785938 60.407396 \n",
       "L 90.515546 59.469231 \n",
       "L 89.665744 55.984615 \n",
       "L 88.970451 52.5 \n",
       "L 88.429668 49.015385 \n",
       "L 88.043394 45.530769 \n",
       "L 87.81163 42.046154 \n",
       "L 87.734375 38.561538 \n",
       "L 87.81163 35.076923 \n",
       "L 88.043394 31.592308 \n",
       "L 88.429668 28.107692 \n",
       "L 88.970451 24.623077 \n",
       "L 89.665744 21.138462 \n",
       "L 90.515546 17.653846 \n",
       "L 90.785938 16.71568 \n",
       "L 91.53892 14.169231 \n",
       "L 92.727841 10.684615 \n",
       "L 93.8375 7.814932 \n",
       "L 94.081625 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 135.064732 143.1 \n",
       "L 133.507813 142.361734 \n",
       "L 130.45625 140.855671 \n",
       "L 128.038031 139.615385 \n",
       "L 127.404688 139.27915 \n",
       "L 124.353125 137.597976 \n",
       "L 121.783388 136.130769 \n",
       "L 121.301563 135.845664 \n",
       "L 118.25 133.976643 \n",
       "L 116.148924 132.646154 \n",
       "L 115.198438 132.021553 \n",
       "L 112.146875 129.950508 \n",
       "L 111.020144 129.161538 \n",
       "L 109.095313 127.76086 \n",
       "L 106.317024 125.676923 \n",
       "L 106.04375 125.463579 \n",
       "L 102.992188 123.010126 \n",
       "L 102.003653 122.192308 \n",
       "L 99.940625 120.41293 \n",
       "L 98.017723 118.707692 \n",
       "L 96.889063 117.662308 \n",
       "L 94.32575 115.223077 \n",
       "L 93.8375 114.736852 \n",
       "L 90.90483 111.738462 \n",
       "L 90.785938 111.610976 \n",
       "L 87.734375 108.253846 \n",
       "L 87.734375 108.253846 \n",
       "L 84.795833 104.769231 \n",
       "L 84.682813 104.627963 \n",
       "L 82.07244 101.284615 \n",
       "L 81.63125 100.687253 \n",
       "L 79.549007 97.8 \n",
       "L 78.579688 96.374476 \n",
       "L 77.211746 94.315385 \n",
       "L 75.528125 91.617618 \n",
       "L 75.048104 90.830769 \n",
       "L 73.059445 87.346154 \n",
       "L 72.476563 86.249145 \n",
       "L 71.235817 83.861538 \n",
       "L 69.559135 80.376923 \n",
       "L 69.425 80.073913 \n",
       "L 68.046875 76.892308 \n",
       "L 66.66875 73.407692 \n",
       "L 66.373438 72.582389 \n",
       "L 65.441908 69.923077 \n",
       "L 64.34977 66.438462 \n",
       "L 63.386118 62.953846 \n",
       "L 63.321875 62.685799 \n",
       "L 62.566849 59.469231 \n",
       "L 61.874742 55.984615 \n",
       "L 61.308473 52.5 \n",
       "L 60.868041 49.015385 \n",
       "L 60.553447 45.530769 \n",
       "L 60.364691 42.046154 \n",
       "L 60.301772 38.561538 \n",
       "L 60.364691 35.076923 \n",
       "L 60.553447 31.592308 \n",
       "L 60.868041 28.107692 \n",
       "L 61.308473 24.623077 \n",
       "L 61.874742 21.138462 \n",
       "L 62.566849 17.653846 \n",
       "L 63.321875 14.437278 \n",
       "L 63.386118 14.169231 \n",
       "L 64.34977 10.684615 \n",
       "L 65.441908 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 95.62775 143.1 \n",
       "L 93.8375 141.800652 \n",
       "L 90.90483 139.615385 \n",
       "L 90.785938 139.523684 \n",
       "L 87.734375 137.108907 \n",
       "L 86.528819 136.130769 \n",
       "L 84.682813 134.578531 \n",
       "L 82.440098 132.646154 \n",
       "L 81.63125 131.922932 \n",
       "L 78.615588 129.161538 \n",
       "L 78.579688 129.127376 \n",
       "L 75.528125 126.155204 \n",
       "L 75.048104 125.676923 \n",
       "L 72.476563 123.010126 \n",
       "L 71.705288 122.192308 \n",
       "L 69.425 119.671522 \n",
       "L 68.571875 118.707692 \n",
       "L 66.373438 116.11359 \n",
       "L 65.634638 115.223077 \n",
       "L 63.321875 112.305725 \n",
       "L 62.881443 111.738462 \n",
       "L 60.301772 108.253846 \n",
       "L 60.270312 108.209172 \n",
       "L 57.896875 104.769231 \n",
       "L 57.21875 103.733264 \n",
       "L 55.647649 101.284615 \n",
       "L 54.167188 98.845385 \n",
       "L 53.545024 97.8 \n",
       "L 51.589654 94.315385 \n",
       "L 51.115625 93.416129 \n",
       "L 49.77875 90.830769 \n",
       "L 48.093125 87.346154 \n",
       "L 48.064063 87.281624 \n",
       "L 46.552541 83.861538 \n",
       "L 45.126577 80.376923 \n",
       "L 45.0125 80.073913 \n",
       "L 43.836669 76.892308 \n",
       "L 42.660837 73.407692 \n",
       "L 41.960938 71.115182 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 66.66875 143.1 \n",
       "L 66.373438 142.834224 \n",
       "L 63.321875 140.028814 \n",
       "L 62.881443 139.615385 \n",
       "L 60.270313 137.07834 \n",
       "L 59.314773 136.130769 \n",
       "L 57.21875 133.976643 \n",
       "L 55.949783 132.646154 \n",
       "L 54.167188 130.706604 \n",
       "L 52.774727 129.161538 \n",
       "L 51.115625 127.248416 \n",
       "L 49.77875 125.676923 \n",
       "L 48.064063 123.579042 \n",
       "L 46.951811 122.192308 \n",
       "L 45.0125 119.671522 \n",
       "L 44.284604 118.707692 \n",
       "L 41.960938 115.494103 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_7\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 42.660837 143.1 \n",
       "L 41.960938 142.361734 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_8\"/>\n",
       "   <g id=\"line2d_8\">\n",
       "    <path clip-path=\"url(#pbc9aec93e5)\" d=\"M 57.21875 108.253846 \n",
       "L 87.734375 80.376923 \n",
       "L 112.146875 63.650769 \n",
       "L 131.676875 53.615077 \n",
       "L 147.300875 47.593662 \n",
       "L 159.800075 43.980812 \n",
       "L 169.799435 41.813103 \n",
       "L 177.798923 40.512477 \n",
       "L 184.198513 39.732102 \n",
       "L 189.318186 39.263876 \n",
       "L 193.413924 38.982941 \n",
       "L 196.690514 38.81438 \n",
       "L 199.311786 38.713243 \n",
       "L 201.408804 38.652561 \n",
       "L 203.086418 38.616152 \n",
       "L 204.428509 38.594307 \n",
       "L 205.502183 38.581199 \n",
       "L 206.361121 38.573335 \n",
       "L 207.048272 38.568616 \n",
       "L 207.597992 38.565785 \n",
       "L 208.037769 38.564087 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mb3ff0e5b83\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pbc9aec93e5)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"57.21875\" xlink:href=\"#mb3ff0e5b83\" y=\"108.253846\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"87.734375\" xlink:href=\"#mb3ff0e5b83\" y=\"80.376923\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"112.146875\" xlink:href=\"#mb3ff0e5b83\" y=\"63.650769\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"131.676875\" xlink:href=\"#mb3ff0e5b83\" y=\"53.615077\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"147.300875\" xlink:href=\"#mb3ff0e5b83\" y=\"47.593662\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"159.800075\" xlink:href=\"#mb3ff0e5b83\" y=\"43.980812\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"169.799435\" xlink:href=\"#mb3ff0e5b83\" y=\"41.813103\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"177.798923\" xlink:href=\"#mb3ff0e5b83\" y=\"40.512477\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"184.198513\" xlink:href=\"#mb3ff0e5b83\" y=\"39.732102\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"189.318186\" xlink:href=\"#mb3ff0e5b83\" y=\"39.263876\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"193.413924\" xlink:href=\"#mb3ff0e5b83\" y=\"38.982941\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"196.690514\" xlink:href=\"#mb3ff0e5b83\" y=\"38.81438\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"199.311786\" xlink:href=\"#mb3ff0e5b83\" y=\"38.713243\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"201.408804\" xlink:href=\"#mb3ff0e5b83\" y=\"38.652561\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"203.086418\" xlink:href=\"#mb3ff0e5b83\" y=\"38.616152\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"204.428509\" xlink:href=\"#mb3ff0e5b83\" y=\"38.594307\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"205.502183\" xlink:href=\"#mb3ff0e5b83\" y=\"38.581199\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"206.361121\" xlink:href=\"#mb3ff0e5b83\" y=\"38.573335\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"207.048272\" xlink:href=\"#mb3ff0e5b83\" y=\"38.568616\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"207.597992\" xlink:href=\"#mb3ff0e5b83\" y=\"38.565785\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"208.037769\" xlink:href=\"#mb3ff0e5b83\" y=\"38.564087\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 41.960938 143.1 \n",
       "L 41.960938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.260938 143.1 \n",
       "L 237.260938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 41.960938 143.1 \n",
       "L 237.260938 143.1 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 41.960938 7.2 \n",
       "L 237.260938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pbc9aec93e5\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"41.960938\" y=\"7.2\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "eta = 0.1\n",
    "\n",
    "def f_2d(x1, x2):  # 目标函数\n",
    "    return x1 ** 2 + 2 * x2 ** 2\n",
    "\n",
    "def gd_2d(x1, x2, s1, s2):\n",
    "    return (x1 - eta * 2 * x1, x2 - eta * 4 * x2, 0, 0)\n",
    "\n",
    "show_trace_2d(f_2d, train_2d(gd_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 随机梯度下降\n",
    "\n",
    "在深度学习里，目标函数通常是训练数据集中有关各个样本的损失函数的平均。设$f_i(\\boldsymbol{x})$是有关索引为$i$的训练数据样本的损失函数，$n$是训练数据样本数，$\\boldsymbol{x}$是模型的参数向量，那么目标函数定义为\n",
    "\n",
    "$$f(\\boldsymbol{x}) = \\frac{1}{n} \\sum_{i = 1}^n f_i(\\boldsymbol{x}).$$\n",
    "\n",
    "目标函数在$\\boldsymbol{x}$处的梯度计算为\n",
    "\n",
    "$$\\nabla f(\\boldsymbol{x}) = \\frac{1}{n} \\sum_{i = 1}^n \\nabla f_i(\\boldsymbol{x}).$$\n",
    "\n",
    "如果使用梯度下降，每次自变量迭代的计算开销为$\\mathcal{O}(n)$，它随着$n$线性增长。因此，当训练数据样本数很大时，梯度下降每次迭代的计算开销很高。\n",
    "\n",
    "随机梯度下降（stochastic gradient descent，SGD）减少了每次迭代的计算开销。在随机梯度下降的每次迭代中，我们随机均匀采样的一个样本索引$i\\in\\{1,\\ldots,n\\}$，并计算梯度$\\nabla f_i(\\boldsymbol{x})$来迭代$\\boldsymbol{x}$：\n",
    "\n",
    "$$\\boldsymbol{x} \\leftarrow \\boldsymbol{x} - \\eta \\nabla f_i(\\boldsymbol{x}).$$\n",
    "\n",
    "这里$\\eta$同样是学习率。可以看到每次迭代的计算开销从梯度下降的$\\mathcal{O}(n)$降到了常数$\\mathcal{O}(1)$。值得强调的是，随机梯度$\\nabla f_i(\\boldsymbol{x})$是对梯度$\\nabla f(\\boldsymbol{x})$的无偏估计：\n",
    "\n",
    "$$E_i \\nabla f_i(\\boldsymbol{x}) = \\frac{1}{n} \\sum_{i = 1}^n \\nabla f_i(\\boldsymbol{x}) = \\nabla f(\\boldsymbol{x}).$$\n",
    "\n",
    "这意味着，平均来说，随机梯度是对梯度的一个良好的估计。\n",
    "\n",
    "下面我们通过在梯度中添加均值为0的随机噪声来模拟随机梯度下降，以此来比较它与梯度下降的区别。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-29T06:34:26.430493Z",
     "start_time": "2019-11-29T06:34:26.185618Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "17"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.185784, x2 -0.155319\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"182.259375pt\" version=\"1.1\" viewBox=\"0 0 244.460938 182.259375\" width=\"244.460938pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 182.259375 \n",
       "L 244.460938 182.259375 \n",
       "L 244.460938 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 41.960938 143.1 \n",
       "L 237.260938 143.1 \n",
       "L 237.260938 7.2 \n",
       "L 41.960938 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"md3117885ff\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"87.734375\" xlink:href=\"#md3117885ff\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 62.0625 27.6875 \n",
       "L 12.3125 27.6875 \n",
       "L 12.3125 35.15625 \n",
       "L 62.0625 35.15625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-8722\"/>\n",
       "       <path d=\"M 55.078125 19.625 \n",
       "L 45.40625 19.625 \n",
       "L 45.40625 0 \n",
       "L 36.140625 0 \n",
       "L 36.140625 19.625 \n",
       "L 0.875 19.625 \n",
       "L 0.875 25.828125 \n",
       "L 34.328125 75.640625 \n",
       "L 45.40625 75.640625 \n",
       "L 45.40625 27.203125 \n",
       "L 55.078125 27.203125 \n",
       "z\n",
       "M 36.140625 27.203125 \n",
       "L 36.140625 58.890625 \n",
       "Q 36.140625 62.25 36.328125 66.65625 \n",
       "L 36.140625 66.65625 \n",
       "Q 35.453125 64.796875 33.25 60.84375 \n",
       "L 10.75 27.203125 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(81.09375 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"148.765625\" xlink:href=\"#md3117885ff\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 51.171875 0 \n",
       "L 5.171875 0 \n",
       "L 5.171875 8.296875 \n",
       "L 27.15625 30.21875 \n",
       "Q 36.234375 39.3125 39.3125 44.609375 \n",
       "Q 42.390625 49.90625 42.390625 55.5625 \n",
       "Q 42.390625 62.015625 38.765625 65.453125 \n",
       "Q 35.15625 68.890625 28.328125 68.890625 \n",
       "Q 18.21875 68.890625 9.03125 60.296875 \n",
       "L 9.03125 70.015625 \n",
       "Q 17.96875 76.90625 29.828125 76.90625 \n",
       "Q 40.046875 76.90625 45.953125 71.390625 \n",
       "Q 51.859375 65.875 51.859375 56.546875 \n",
       "Q 51.859375 49.515625 48.0625 42.75 \n",
       "Q 44.28125 35.984375 33.734375 25.53125 \n",
       "L 16.5 8.6875 \n",
       "L 16.5 8.5 \n",
       "L 51.171875 8.5 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(142.125 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"209.796875\" xlink:href=\"#md3117885ff\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 4.4375 36.328125 \n",
       "Q 4.4375 56.34375 11.078125 66.625 \n",
       "Q 17.71875 76.90625 30.328125 76.90625 \n",
       "Q 54.34375 76.90625 54.34375 38.140625 \n",
       "Q 54.34375 19 47.578125 8.859375 \n",
       "Q 40.828125 -1.265625 28.609375 -1.265625 \n",
       "Q 17.09375 -1.265625 10.765625 8.34375 \n",
       "Q 4.4375 17.96875 4.4375 36.328125 \n",
       "z\n",
       "M 14.203125 36.765625 \n",
       "Q 14.203125 6.640625 29.5 6.640625 \n",
       "Q 44.53125 6.640625 44.53125 37.25 \n",
       "Q 44.53125 68.953125 29.78125 68.953125 \n",
       "Q 14.203125 68.953125 14.203125 36.765625 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(206.864844 158.09375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 49.421875 54 \n",
       "L 31.34375 26.8125 \n",
       "L 49.171875 0 \n",
       "L 38.03125 0 \n",
       "L 27.734375 17.1875 \n",
       "Q 26.90625 18.5625 25.390625 21.390625 \n",
       "L 25.203125 21.390625 \n",
       "Q 25.140625 21.09375 22.75 17.1875 \n",
       "L 12.25 0 \n",
       "L 1.265625 0 \n",
       "L 19.625 26.515625 \n",
       "L 2.09375 54 \n",
       "L 13.234375 54 \n",
       "L 23.34375 35.9375 \n",
       "Q 24.265625 34.28125 25.59375 31.640625 \n",
       "L 25.78125 31.640625 \n",
       "L 38.921875 54 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-120\"/>\n",
       "      <path d=\"M 52.59375 0 \n",
       "L 9.234375 0 \n",
       "L 9.234375 8.25 \n",
       "L 26.171875 8.25 \n",
       "L 26.171875 65.671875 \n",
       "L 8.796875 60.640625 \n",
       "L 8.796875 69.4375 \n",
       "L 35.6875 77.25 \n",
       "L 35.6875 8.25 \n",
       "L 52.59375 8.25 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-49\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(134.144531 172.573438)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"50.683594\" xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m7ff2048d89\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7ff2048d89\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −3 -->\n",
       "      <defs>\n",
       "       <path d=\"M 6.984375 12.640625 \n",
       "Q 14.75 6.640625 24.515625 6.640625 \n",
       "Q 32.328125 6.640625 36.890625 10.46875 \n",
       "Q 41.453125 14.3125 41.453125 20.75 \n",
       "Q 41.453125 35.109375 20.90625 35.109375 \n",
       "L 14.59375 35.109375 \n",
       "L 14.59375 43.015625 \n",
       "L 20.609375 43.015625 \n",
       "Q 38.8125 43.015625 38.8125 56.5 \n",
       "Q 38.8125 68.953125 24.90625 68.953125 \n",
       "Q 16.9375 68.953125 9.90625 63.578125 \n",
       "L 9.90625 72.609375 \n",
       "Q 17.328125 76.90625 27.25 76.90625 \n",
       "Q 36.921875 76.90625 42.78125 71.84375 \n",
       "Q 48.640625 66.796875 48.640625 58.734375 \n",
       "Q 48.640625 43.890625 33.5 39.65625 \n",
       "L 33.5 39.453125 \n",
       "Q 41.703125 38.578125 46.453125 33.671875 \n",
       "Q 51.21875 28.765625 51.21875 21.4375 \n",
       "Q 51.21875 11.234375 43.875 4.984375 \n",
       "Q 36.53125 -1.265625 24.3125 -1.265625 \n",
       "Q 13.578125 -1.265625 6.984375 2.734375 \n",
       "z\n",
       "\" id=\"MicrosoftYaHei-51\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.679688 147.096875)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-51\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7ff2048d89\" y=\"108.253846\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(21.679688 112.250721)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7ff2048d89\" y=\"73.407692\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(21.679688 77.404567)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-8722\"/>\n",
       "       <use x=\"74.169922\" xlink:href=\"#MicrosoftYaHei-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"41.960938\" xlink:href=\"#m7ff2048d89\" y=\"38.561538\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.096875 42.558413)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#MicrosoftYaHei-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(15.19375 80.616406)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#MicrosoftYaHei-120\"/>\n",
       "      <use x=\"50.683594\" xlink:href=\"#MicrosoftYaHei-50\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\"/>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 237.260938 104.627963 \n",
       "L 236.722426 104.769231 \n",
       "L 234.209375 105.394675 \n",
       "L 231.157813 106.064793 \n",
       "L 228.10625 106.645562 \n",
       "L 225.054688 107.136982 \n",
       "L 222.003125 107.539053 \n",
       "L 218.951563 107.851775 \n",
       "L 215.9 108.075148 \n",
       "L 212.848438 108.209172 \n",
       "L 209.796875 108.253846 \n",
       "L 209.796875 108.253846 \n",
       "L 209.796875 108.253846 \n",
       "L 206.745313 108.209172 \n",
       "L 203.69375 108.075148 \n",
       "L 200.642188 107.851775 \n",
       "L 197.590625 107.539053 \n",
       "L 194.539063 107.136982 \n",
       "L 191.4875 106.645562 \n",
       "L 188.435938 106.064793 \n",
       "L 185.384375 105.394675 \n",
       "L 182.871324 104.769231 \n",
       "L 182.332813 104.627963 \n",
       "L 179.28125 103.733264 \n",
       "L 176.229688 102.744387 \n",
       "L 173.178125 101.661331 \n",
       "L 172.201625 101.284615 \n",
       "L 170.126563 100.438352 \n",
       "L 167.075 99.094286 \n",
       "L 164.339116 97.8 \n",
       "L 164.023438 97.641608 \n",
       "L 160.971875 96.004895 \n",
       "L 158.012784 94.315385 \n",
       "L 157.920313 94.259181 \n",
       "L 154.86875 92.29206 \n",
       "L 152.724409 90.830769 \n",
       "L 151.817188 90.169894 \n",
       "L 148.765625 87.82679 \n",
       "L 148.170198 87.346154 \n",
       "L 145.714063 85.216667 \n",
       "L 144.223765 83.861538 \n",
       "L 142.6625 82.328308 \n",
       "L 140.76375 80.376923 \n",
       "L 139.610938 79.08913 \n",
       "L 137.728059 76.892308 \n",
       "L 136.559375 75.398901 \n",
       "L 135.064732 73.407692 \n",
       "L 133.507813 71.115182 \n",
       "L 132.729963 69.923077 \n",
       "L 130.695588 66.438462 \n",
       "L 130.45625 65.973846 \n",
       "L 128.959257 62.953846 \n",
       "L 127.462264 59.469231 \n",
       "L 127.404688 59.310839 \n",
       "L 126.239545 55.984615 \n",
       "L 125.240852 52.5 \n",
       "L 124.464091 49.015385 \n",
       "L 124.353125 48.318462 \n",
       "L 123.924836 45.530769 \n",
       "L 123.603618 42.046154 \n",
       "L 123.496546 38.561538 \n",
       "L 123.603618 35.076923 \n",
       "L 123.924836 31.592308 \n",
       "L 124.353125 28.804615 \n",
       "L 124.464091 28.107692 \n",
       "L 125.240852 24.623077 \n",
       "L 126.239545 21.138462 \n",
       "L 127.404688 17.812238 \n",
       "L 127.462264 17.653846 \n",
       "L 128.959257 14.169231 \n",
       "L 130.45625 11.149231 \n",
       "L 130.695588 10.684615 \n",
       "L 132.729963 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 237.260938 134.578531 \n",
       "L 234.209375 135.117063 \n",
       "L 231.157813 135.592238 \n",
       "L 228.10625 136.004056 \n",
       "L 226.996591 136.130769 \n",
       "L 225.054688 136.344737 \n",
       "L 222.003125 136.619838 \n",
       "L 218.951563 136.833806 \n",
       "L 215.9 136.98664 \n",
       "L 212.848438 137.07834 \n",
       "L 209.796875 137.108907 \n",
       "L 206.745313 137.07834 \n",
       "L 203.69375 136.98664 \n",
       "L 200.642188 136.833806 \n",
       "L 197.590625 136.619838 \n",
       "L 194.539063 136.344737 \n",
       "L 192.597159 136.130769 \n",
       "L 191.4875 136.004056 \n",
       "L 188.435938 135.592238 \n",
       "L 185.384375 135.117063 \n",
       "L 182.332813 134.578531 \n",
       "L 179.28125 133.976643 \n",
       "L 176.229688 133.311399 \n",
       "L 173.443478 132.646154 \n",
       "L 173.178125 132.580406 \n",
       "L 170.126563 131.758563 \n",
       "L 167.075 130.870972 \n",
       "L 164.023438 129.917634 \n",
       "L 161.759375 129.161538 \n",
       "L 160.971875 128.888235 \n",
       "L 157.920313 127.76086 \n",
       "L 154.86875 126.565158 \n",
       "L 152.724409 125.676923 \n",
       "L 151.817188 125.285793 \n",
       "L 148.765625 123.899058 \n",
       "L 145.714063 122.441209 \n",
       "L 145.217297 122.192308 \n",
       "L 142.6625 120.857774 \n",
       "L 139.610938 119.189607 \n",
       "L 138.766888 118.707692 \n",
       "L 136.559375 117.391282 \n",
       "L 133.507813 115.494103 \n",
       "L 133.088971 115.223077 \n",
       "L 130.45625 113.44025 \n",
       "L 128.038031 111.738462 \n",
       "L 127.404688 111.271013 \n",
       "L 124.353125 108.933771 \n",
       "L 123.496546 108.253846 \n",
       "L 121.301563 106.422189 \n",
       "L 119.387871 104.769231 \n",
       "L 118.25 103.733264 \n",
       "L 115.648668 101.284615 \n",
       "L 115.198438 100.836593 \n",
       "L 112.24375 97.8 \n",
       "L 112.146875 97.694406 \n",
       "L 109.14226 94.315385 \n",
       "L 109.095313 94.259181 \n",
       "L 106.317024 90.830769 \n",
       "L 106.04375 90.470292 \n",
       "L 103.744022 87.346154 \n",
       "L 102.992188 86.249145 \n",
       "L 101.401937 83.861538 \n",
       "L 99.940625 81.492 \n",
       "L 99.271789 80.376923 \n",
       "L 97.348887 76.892308 \n",
       "L 96.889063 75.97967 \n",
       "L 95.62775 73.407692 \n",
       "L 94.081625 69.923077 \n",
       "L 93.8375 69.308145 \n",
       "L 92.727841 66.438462 \n",
       "L 91.53892 62.953846 \n",
       "L 90.785938 60.407396 \n",
       "L 90.515546 59.469231 \n",
       "L 89.665744 55.984615 \n",
       "L 88.970451 52.5 \n",
       "L 88.429668 49.015385 \n",
       "L 88.043394 45.530769 \n",
       "L 87.81163 42.046154 \n",
       "L 87.734375 38.561538 \n",
       "L 87.81163 35.076923 \n",
       "L 88.043394 31.592308 \n",
       "L 88.429668 28.107692 \n",
       "L 88.970451 24.623077 \n",
       "L 89.665744 21.138462 \n",
       "L 90.515546 17.653846 \n",
       "L 90.785938 16.71568 \n",
       "L 91.53892 14.169231 \n",
       "L 92.727841 10.684615 \n",
       "L 93.8375 7.814932 \n",
       "L 94.081625 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 135.064732 143.1 \n",
       "L 133.507813 142.361734 \n",
       "L 130.45625 140.855671 \n",
       "L 128.038031 139.615385 \n",
       "L 127.404688 139.27915 \n",
       "L 124.353125 137.597976 \n",
       "L 121.783388 136.130769 \n",
       "L 121.301563 135.845664 \n",
       "L 118.25 133.976643 \n",
       "L 116.148924 132.646154 \n",
       "L 115.198438 132.021553 \n",
       "L 112.146875 129.950508 \n",
       "L 111.020144 129.161538 \n",
       "L 109.095313 127.76086 \n",
       "L 106.317024 125.676923 \n",
       "L 106.04375 125.463579 \n",
       "L 102.992188 123.010126 \n",
       "L 102.003653 122.192308 \n",
       "L 99.940625 120.41293 \n",
       "L 98.017723 118.707692 \n",
       "L 96.889063 117.662308 \n",
       "L 94.32575 115.223077 \n",
       "L 93.8375 114.736852 \n",
       "L 90.90483 111.738462 \n",
       "L 90.785938 111.610976 \n",
       "L 87.734375 108.253846 \n",
       "L 87.734375 108.253846 \n",
       "L 84.795833 104.769231 \n",
       "L 84.682813 104.627963 \n",
       "L 82.07244 101.284615 \n",
       "L 81.63125 100.687253 \n",
       "L 79.549007 97.8 \n",
       "L 78.579688 96.374476 \n",
       "L 77.211746 94.315385 \n",
       "L 75.528125 91.617618 \n",
       "L 75.048104 90.830769 \n",
       "L 73.059445 87.346154 \n",
       "L 72.476563 86.249145 \n",
       "L 71.235817 83.861538 \n",
       "L 69.559135 80.376923 \n",
       "L 69.425 80.073913 \n",
       "L 68.046875 76.892308 \n",
       "L 66.66875 73.407692 \n",
       "L 66.373438 72.582389 \n",
       "L 65.441908 69.923077 \n",
       "L 64.34977 66.438462 \n",
       "L 63.386118 62.953846 \n",
       "L 63.321875 62.685799 \n",
       "L 62.566849 59.469231 \n",
       "L 61.874742 55.984615 \n",
       "L 61.308473 52.5 \n",
       "L 60.868041 49.015385 \n",
       "L 60.553447 45.530769 \n",
       "L 60.364691 42.046154 \n",
       "L 60.301772 38.561538 \n",
       "L 60.364691 35.076923 \n",
       "L 60.553447 31.592308 \n",
       "L 60.868041 28.107692 \n",
       "L 61.308473 24.623077 \n",
       "L 61.874742 21.138462 \n",
       "L 62.566849 17.653846 \n",
       "L 63.321875 14.437278 \n",
       "L 63.386118 14.169231 \n",
       "L 64.34977 10.684615 \n",
       "L 65.441908 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 95.62775 143.1 \n",
       "L 93.8375 141.800652 \n",
       "L 90.90483 139.615385 \n",
       "L 90.785938 139.523684 \n",
       "L 87.734375 137.108907 \n",
       "L 86.528819 136.130769 \n",
       "L 84.682813 134.578531 \n",
       "L 82.440098 132.646154 \n",
       "L 81.63125 131.922932 \n",
       "L 78.615588 129.161538 \n",
       "L 78.579688 129.127376 \n",
       "L 75.528125 126.155204 \n",
       "L 75.048104 125.676923 \n",
       "L 72.476563 123.010126 \n",
       "L 71.705288 122.192308 \n",
       "L 69.425 119.671522 \n",
       "L 68.571875 118.707692 \n",
       "L 66.373438 116.11359 \n",
       "L 65.634638 115.223077 \n",
       "L 63.321875 112.305725 \n",
       "L 62.881443 111.738462 \n",
       "L 60.301772 108.253846 \n",
       "L 60.270312 108.209172 \n",
       "L 57.896875 104.769231 \n",
       "L 57.21875 103.733264 \n",
       "L 55.647649 101.284615 \n",
       "L 54.167188 98.845385 \n",
       "L 53.545024 97.8 \n",
       "L 51.589654 94.315385 \n",
       "L 51.115625 93.416129 \n",
       "L 49.77875 90.830769 \n",
       "L 48.093125 87.346154 \n",
       "L 48.064063 87.281624 \n",
       "L 46.552541 83.861538 \n",
       "L 45.126577 80.376923 \n",
       "L 45.0125 80.073913 \n",
       "L 43.836669 76.892308 \n",
       "L 42.660837 73.407692 \n",
       "L 41.960938 71.115182 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 66.66875 143.1 \n",
       "L 66.373438 142.834224 \n",
       "L 63.321875 140.028814 \n",
       "L 62.881443 139.615385 \n",
       "L 60.270313 137.07834 \n",
       "L 59.314773 136.130769 \n",
       "L 57.21875 133.976643 \n",
       "L 55.949783 132.646154 \n",
       "L 54.167188 130.706604 \n",
       "L 52.774727 129.161538 \n",
       "L 51.115625 127.248416 \n",
       "L 49.77875 125.676923 \n",
       "L 48.064063 123.579042 \n",
       "L 46.951811 122.192308 \n",
       "L 45.0125 119.671522 \n",
       "L 44.284604 118.707692 \n",
       "L 41.960938 115.494103 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_7\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 42.660837 143.1 \n",
       "L 41.960938 142.361734 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_8\"/>\n",
       "   <g id=\"line2d_8\">\n",
       "    <path clip-path=\"url(#p229e17b4c9)\" d=\"M 57.21875 108.253846 \n",
       "L 89.462896 80.59606 \n",
       "L 118.343948 68.460942 \n",
       "L 136.138353 55.206655 \n",
       "L 154.972231 46.840684 \n",
       "L 162.332901 40.193927 \n",
       "L 167.88578 37.641348 \n",
       "L 174.696421 36.765353 \n",
       "L 176.981742 40.562021 \n",
       "L 183.571571 35.01201 \n",
       "L 186.771294 31.647658 \n",
       "L 189.392525 34.675585 \n",
       "L 195.137686 37.818535 \n",
       "L 195.193848 32.474049 \n",
       "L 199.552742 37.550392 \n",
       "L 198.808801 41.704479 \n",
       "L 203.247286 41.932446 \n",
       "L 203.056408 40.557277 \n",
       "L 201.958029 45.36386 \n",
       "L 204.809347 43.04063 \n",
       "L 204.127565 43.973802 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m66a4f070d0\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p229e17b4c9)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"57.21875\" xlink:href=\"#m66a4f070d0\" y=\"108.253846\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"89.462896\" xlink:href=\"#m66a4f070d0\" y=\"80.59606\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"118.343948\" xlink:href=\"#m66a4f070d0\" y=\"68.460942\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"136.138353\" xlink:href=\"#m66a4f070d0\" y=\"55.206655\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"154.972231\" xlink:href=\"#m66a4f070d0\" y=\"46.840684\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"162.332901\" xlink:href=\"#m66a4f070d0\" y=\"40.193927\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"167.88578\" xlink:href=\"#m66a4f070d0\" y=\"37.641348\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"174.696421\" xlink:href=\"#m66a4f070d0\" y=\"36.765353\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"176.981742\" xlink:href=\"#m66a4f070d0\" y=\"40.562021\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"183.571571\" xlink:href=\"#m66a4f070d0\" y=\"35.01201\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"186.771294\" xlink:href=\"#m66a4f070d0\" y=\"31.647658\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"189.392525\" xlink:href=\"#m66a4f070d0\" y=\"34.675585\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"195.137686\" xlink:href=\"#m66a4f070d0\" y=\"37.818535\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"195.193848\" xlink:href=\"#m66a4f070d0\" y=\"32.474049\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"199.552742\" xlink:href=\"#m66a4f070d0\" y=\"37.550392\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"198.808801\" xlink:href=\"#m66a4f070d0\" y=\"41.704479\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"203.247286\" xlink:href=\"#m66a4f070d0\" y=\"41.932446\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"203.056408\" xlink:href=\"#m66a4f070d0\" y=\"40.557277\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"201.958029\" xlink:href=\"#m66a4f070d0\" y=\"45.36386\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"204.809347\" xlink:href=\"#m66a4f070d0\" y=\"43.04063\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"204.127565\" xlink:href=\"#m66a4f070d0\" y=\"43.973802\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 41.960938 143.1 \n",
       "L 41.960938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.260938 143.1 \n",
       "L 237.260938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 41.960938 143.1 \n",
       "L 237.260938 143.1 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 41.960938 7.2 \n",
       "L 237.260938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p229e17b4c9\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"41.960938\" y=\"7.2\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sgd_2d(x1, x2, s1, s2):\n",
    "    return (x1 - eta * (2 * x1 + np.random.normal(0.1)),\n",
    "            x2 - eta * (4 * x2 + np.random.normal(0.1)), 0, 0)\n",
    "\n",
    "show_trace_2d(f_2d, train_2d(sgd_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，随机梯度下降中自变量的迭代轨迹相对于梯度下降中的来说更为曲折。这是由于实验所添加的噪声使模拟的随机梯度的准确度下降。在实际中，这些噪声通常指训练数据集中的无意义的干扰。\n",
    "\n",
    "\n",
    "## 小结\n",
    "\n",
    "* 使用适当的学习率，沿着梯度反方向更新自变量可能降低目标函数值。梯度下降重复这一更新过程直到得到满足要求的解。\n",
    "* 学习率过大或过小都有问题。一个合适的学习率通常是需要通过多次实验找到的。\n",
    "* 当训练数据集的样本较多时，梯度下降每次迭代的计算开销较大，因而随机梯度下降通常更受青睐。\n",
    "\n",
    "\n",
    "## 练习\n",
    "\n",
    "* 使用一个不同的目标函数，观察梯度下降和随机梯度下降中自变量的迭代轨迹。\n",
    "* 在二维梯度下降的实验中尝试使用不同的学习率，观察并分析实验现象。\n",
    "\n",
    "## 参考文献\n",
    "\n",
    "[1] Stewart, J. (2010). Calculus: early transcendentals. 7th ed. Cengage Learning.\n",
    "\n",
    "\n",
    "## 扫码直达[讨论区](https://discuss.gluon.ai/t/topic/1877)\n",
    "\n",
    "![](img/qr_gd-sgd.svg)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}