examples/01_rl_hello_world_cartpole.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Reinforcement Learning \"Hello World\"\n",
    "\n",
    "## Copyright\n",
    "\n",
    "*Copyright Geoscience DS&ML Special Interest Group, 2022.*\n",
    "*License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)*\n",
    "*Author(s): [Altay Sansal](https://github.com/tasansal)*\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "NOTE: This notebook requires wildcatter installed with the `hello_world` extra. To do this install wildcatter via:\n",
    "```shell\n",
    "pip install wildcatter[hello_world]\n",
    "```\n",
    "\n",
    "This notebook assumes familiarity with Reinforcement Learning concepts (agents, spaces, actions, ...) and knowledge of [OpenAI's Gym](https://www.gymlibrary.ml/) APIs and environments. If you are not familiar with RL or Gym at all, we refer the reader to the following resources:\n",
    "1. [OpenAI Gym Introduction](https://www.gymlibrary.ml/content/api/)\n",
    "2. [OpenAI Gym Introduction](https://www.gymlibrary.ml/content/api/)\n",
    "3. [OpenAI Spinning Up in Deep RL](https://spinningup.openai.com/en/latest/)\n",
    "    * [Part 1: Key Concepts in RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)\n",
    "    * [Part 2: Kinds of RL Algorithms](https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html)\n",
    "    * [Part 3: Intro to Policy Optimization](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html)\n",
    "    * [Key Papers](https://spinningup.openai.com/en/latest/spinningup/keypapers.html)\n",
    "    * [RL Algorithms](https://spinningup.openai.com/en/latest/user/algorithms.html)\n",
    "4. [David Silver's RL Course](https://www.davidsilver.uk/teaching/)\n",
    "5. [Berkeley's Deep RL Bootcamp](https://sites.google.com/view/deep-rl-bootcamp/lectures)\n",
    "6. [PyTorch DQN from Scratch](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html)\n",
    "7. [OpenAI Gym Tutorials](https://www.gymlibrary.ml/content/tutorials/)\n",
    "8. [More Resources](https://github.com/dennybritz/reinforcement-learning#resources)\n",
    "9. [If you like MATLAB...](https://www.mathworks.com/products/reinforcement-learning.html)\n",
    "\n",
    "First two resources are bare-minimum, but for the curious minded, we included more detailed resources!\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Here we will be using an environment from `Gym`. The environment we choose is the `CartPole-v1`, which is a part of the classic control system environments.\n",
    "\n",
    "The problem we are trying to solve is trying to keep a pole upright, which is attached to a frictionless cart with an un-actuated joint. The goal is to balance the pole by moving the cart to the left or to the right.\n",
    "\n",
    "Untrained environment looks like this:\n",
    "<img src=\"https://www.gymlibrary.ml/_images/cart_pole.gif\" width=\"350\"/>\n",
    "\n",
    "More information about the environment can be found [here](https://www.gymlibrary.ml/environments/classic_control/cart_pole/) and the source code for the environment can be found [here](https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py).\n",
    "\n",
    "The environment has the following action and observation spaces:\n",
    "\n",
    "| Property          | Details             |\n",
    "|-------------------|---------------------|\n",
    "| Action Space      | `Discrete(2)`       |\n",
    "| Observation Space | `Box(4, 'float32')` |\n",
    "\n",
    "Action Space:\n",
    "\n",
    "| Num | Action                 |\n",
    "|:---:|------------------------|\n",
    "|  0  | Push cart to the left  |\n",
    "|  1  | Push cart to the right |\n",
    "\n",
    "Observation Space:\n",
    "\n",
    "| Num | Action                |     Min      |     Max     |\n",
    "|:---:|-----------------------|:------------:|:-----------:|\n",
    "|  0  | Cart Position         |    `-4.8`    |    `4.8`    |\n",
    "|  1  | Cart Velocity         |    `-inf`    |    `inf`    |\n",
    "|  2  | Pole Angle            | `-0.418 rad` | `0.418 rad` |\n",
    "|  3  | Pole Angular Velocity |    `-inf`    |    `inf`    |\n",
    "\n",
    "The environment will terminate automatically (send a `done` signal) if the pole's angle is more than 12 degrees (0.20944 rad) or if the position of the cart is +/- 2.4, meaning cart reaches the end of the display.\n",
    "\n",
    "Let's take a more detailed look!\n",
    "\n",
    "We first import `gym` and make the environment. Then we will look at its pre-configured action space and observation space using built in attributes."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "\n",
    "env_id = \"CartPole-v1\"\n",
    "env = gym.make(env_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "data": {
      "text/plain": "Discrete(2)"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_space"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "data": {
      "text/plain": "Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_space"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here let's see what the observation state looks like if we reset the environment five times."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reset 1 [0.01656609 0.02576861 0.0039965  0.01861649]\n",
      "reset 2 [-0.03452398 -0.04903302 -0.0241876   0.03794585]\n",
      "reset 3 [-0.02339839  0.01897064  0.00293178 -0.03030473]\n",
      "reset 4 [ 0.0366796  -0.00294616  0.00407595 -0.00959057]\n",
      "reset 5 [-0.00534603  0.04765269 -0.00063244  0.04559017]\n"
     ]
    }
   ],
   "source": [
    "for idx in range(5):\n",
    "    print(f\"reset {idx + 1}\", env.reset())"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As expected, we can see the cart and pole parameters are all different once we reset the environment.\n",
    "\n",
    "Reminder: Values are `[cart_position, cart_velocity, pole_angle(rad), pole_angular_velocity]`.\n",
    "\n",
    "Now let's reset the environment and take up to 100 steps. Since there is no trained model, the actions we sample are going to be randomly selected from the action space. The expectation here is to fail after ~20 time steps.\n",
    "\n",
    "The boilerplate code is to visualize in Jupyter Notebook. Normally this would pop up a window using Pyglet and will visualize the current state of the system when the `.render()` method is called.\n",
    "\n",
    "Here, the returned variables are **observation** which is the current state, the **reward** is the current reward for the action we have taken (not cumulative), **done** states if the environment finished, or failed etc., and finally **info** is a diagnostic variable (if we included things in the `step` implementation). In the cartpole environment, the info is empty."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 720x720 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAHjCAYAAAA+KOkeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlt0lEQVR4nO3dfbDdZX33+89XSEAegxoBAwQPUBDsIUpu5QYtEARTQR4itVgfwDIGR6lavY9F7AzQqVMdDyLUI8ItELyrooOogI8UokJRNMGAKCKIoEEkIBiFTuXpOn9kJb1CAtk72Tt77/h6zezZa13rt9bv2uvXbt/88lvXrtZaAACAZZ4x1hMAAIDxRCADAEBHIAMAQEcgAwBARyADAEBHIAMAQGfUArmqZlfVrVV1e1WdPFr7AQCAkVSjsQ5yVW2U5GdJDkmyOMkPkryutfaTEd8ZAACMoNE6g/ySJLe31u5orT2S5OIkR47SvgAAYMSMViBPS/Kr7v7iwRgAAIxrG4/VjqtqbpK5SbL55pvvs8cee4zVVAAA+BO0cOHC+1trU588PlqBfHeSHbv7OwzGVmitnZfkvCSZOXNmW7BgwShNBQAAVlVVd61ufLQusfhBkt2q6vlVNTnJsUkuG6V9AQDAiBmVM8ittceq6qQk30iyUZILWms/Ho19AQDASBq1a5Bba19N8tXRen0AABgN/pIeAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQ2XpcnV9WdSf6Q5PEkj7XWZlbVs5J8LsnOSe5M8trW2oPrNk0AAFg/RuIM8kGttRmttZmD+ycnuaq1tluSqwb3AQBgQhiNSyyOTHLR4PZFSY4ahX0AAMCoWNdAbkm+WVULq2ruYGzb1to9g9u/SbLtOu4DAADWm3W6BjnJy1prd1fVc5NcWVU/7R9srbWqaqt74iCo5ybJTjvttI7TAACAkbFOZ5Bba3cPvi9J8sUkL0lyb1VtnySD70ue4rnntdZmttZmTp06dV2mAQAAI2atA7mqNq+qLZffTnJokpuTXJbkuMFmxyX58rpOEgAA1pd1ucRi2yRfrKrlr/OZ1trXq+oHST5fVSckuSvJa9d9mgAAsH6sdSC31u5Isvdqxn+b5OB1mRQAAIwVf0kPAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgKZce/OO+9MVeX4449fafz4449PVeXOO+8ctX2fdtppqap861vfGrV9AADji0BmRFXV037NmzdvrKc4rnzrW99KVeW0004btX2sj/+QWBfnn39+TjzxxLz0pS/NZpttlqrKP/7jPz7l9osWLcppp52W/fffP9tvv30mT56cadOm5XWve11uuOGG9ThzADZUG4/1BNgwnXrqqasdnzFjxrBfa9q0abnllluy9dZbr+OsGI/e8573ZOnSpdlmm23yvOc9Lz//+c+fdvu3vvWtuf7667PPPvtkzpw52WKLLbJo0aJcfPHFueSSS/K5z30uc+bMWU+zB2BDJJAZFSN5RnTSpEnZY489Ruz1GF8uvvjivOAFL8j06dMzb968vPnNb37a7V//+tfn3/7t37LrrruuNP7pT386b3jDGzJ37twcfvjhmTx58mhOG4ANmEssWO9+/etf55/+6Z+y//77Z7vttsvkyZPzvOc9L3/zN3+Tn/zkJ6ts/1TXID+d66+/Psccc8yK199xxx1z4okn5te//vVqt1+4cGFmz56dLbfcMltttVVe8YpX5Lvf/e6wf7af/exnOfnkkzNz5sxMnTo1m2yySaZPn565c+dm8eLFK217/PHH56CDDkqSnH766StdijKUa56vueaavPrVr84OO+yQTTbZJNttt1323XffnH766Su2qapcdNFFSZLnP//5K15/5513Xum1Hnjggbzvfe/LC17wgjzzmc/M1ltvnYMPPjjf/OY3V9nvvHnzVlwu85WvfCX77bdfNt9882yzzTY55phjcttttw3rPZs9e3amT58+5O3/7u/+bpU4TpaF82677Zbf/va3+dGPfjSsOQBAzxlk1rvvfOc7+eAHP5iDDjoor3nNa7LFFlvktttuyyWXXJLLLrss//Ef/5G99957rV//ggsuyNy5c7PJJpvkiCOOyI477pjbbrstn/zkJ3P55Zfne9/7XnbaaacV21933XV5xStekUceeSRz5szJrrvumkWLFuXAAw/MrFmzhrXvSy+9NJ/4xCdy0EEHZb/99svkyZPz4x//eMW+FyxYkGnTpiVJjjrqqCTJRRddlAMOOCAHHnjgitd5csA+2de//vUcdthh2WqrrXLEEUdk2rRpeeCBB3LLLbfk4x//+IpLXE499dR86Utfyo033ph3vvOdmTJlSpKs+J4kd911Vw488MDceeedefnLX57Zs2fn4YcfzhVXXJHZs2fn3HPPzVve8pbV/qxf+9rXcvTRR+fAAw/MokWL8oUvfCHz58/Pddddl913331Y791ImDRpUpJk4439agNgHbTWxvxrn332aWwYkrQk7dRTT13l68ILL2yttXbvvfe23//+96s8d9GiRW3zzTdvs2fPXmn8F7/4RUvSjjvuuJXGjzvuuJak/eIXv1gxduutt7ZJkya1XXbZpS1evHil7f/93/+9PeMZz2hHHXXUirEnnnii7b777i1J+9KXvrTS9h/96EdX/Dzz588f0s+/ePHi9l//9V+rjH/jG99oz3jGM9pb3/rWlcbnz5+/4v0ajjlz5rQkbdGiRas8dt999610f3XvU++AAw5oVdU++9nPrjT+4IMPtr333rttuumm7Te/+c2K8QsvvHDF+3L55Zev9Jzl79msWbOG9fM8+bXf//73D/u53/3ud1uSNm3atPbYY4+t1f4B+NOSZEFbTZs6zcKo6P+Zf7kDDjggxx9/fJ773Oeu9jl77713Zs2alW9+85t59NFHV5wNHI5zzjknjz76aM4666wVZ2qXO/jgg3PEEUfk8ssvzx/+8IdsueWWue6663LrrbfmL/7iL3LkkUeutP1JJ52Uf/3Xf13jh8Z6T97ncoceemj22muvfOMb3xj2z/R0nvnMZ64y9pznPGfIz7/xxhvz7W9/O8ccc0yOPfbYlR6bMmVKTj/99Bx11FH5whe+kLe97W0rPT5r1qwcfvjhK40tf8+uvvrq3HXXXcO6dGJdPPDAA3nTm96UJDnzzDOz0UYbrZf9ArBhEsiMimX/UfbUvvKVr+QTn/hEFixYkPvvvz+PPfbYSo/ff//92X777Ye93+XXDX/729/OD37wg1UeX7JkSR5//PH87Gc/yz777LNiWbADDjhglW032mijvOxlLxtWILfW8ulPfzrz5s3LjTfemAcffDCPP/74isdH6oNjr3/963PppZfmpS99af76r/86Bx10UPbff//ssMMOw3qd5e/X0qVLV/vByvvuuy9Jcsstt6zy2Jresx/+8IfrJZAffvjhHHnkkbntttvy3ve+N3/1V3816vsEYMMmkFnvzjrrrLzrXe/KNttsk0MOOSQ77bTTivVvl18v+8c//nGtXvu3v/1tkuTDH/7w02730EMPJVkWhkmy7bbbrna77bbbblj7f/e7352PfvSj2X777fPKV74y06ZNW3GWd968ebnrrruG9XpPZc6cObniiityxhln5IILLsi5556bJNlnn33yL//yLznkkEOG9DrL368rr7wyV1555VNut/z96q3pPVv+3o6mhx9+OIcddliuvfbavPvd786HPvShUd8nABs+gcx69dhjj+W0007LdtttlxtuuGGVs8Rrs3JEb/layUuXLs1WW2015O3vvffe1T7+m9/8Zsj7XrJkSc4+++y88IUvzHXXXZctt9xypcc/+9nPDvm1huKwww7LYYcdlocffjjXX399rrjiipxzzjk5/PDD88Mf/jB77rnnGl9j+c9/1lln5R3veMew9r+m92y0163+wx/+kMMOOyzXXHNN3vve94pjAEbMGpd5q6oLqmpJVd3cjT2rqq6sqtsG37cZjFdVnV1Vt1fVTVX14tGcPBPP/fffn9/97nfZb7/9Vonjhx56aJ3/Etq+++6bZNkSaEPx4hcv+z/Rb3/726s89vjjj+faa68d8r7vuOOOPPHEEzn00ENXiePFixfnjjvuWOU5y6+V7S/DGK7NN988s2bNykc+8pGccsopeeSRR/K1r31tSPsY7vvVW9N79qIXvWjYrzlUS5cuzaGHHpprrrkm73//+8UxACNqKOsgz0sy+0ljJye5qrW2W5KrBveT5C+T7Db4mpvknJGZJhuK5z73udlss82ycOHClf7Z/tFHH8073/nO3H///ev0+ieddFImTZqUv//7v8/PfvazVR5/5JFHVorB/fbbL7vvvnu+853v5Mtf/vJK237sYx8b1vXHy5dmu/baa1eK0YceeihvectbVrnOOkme/exnJ0l++ctfDnk/ybKl8lb3esvP6m622WZD2sfMmTPz8pe/PJdeemkuuOCC1e7rRz/6UZYsWbLK+NVXX50rrrhipbHl79lBBx00atcfP/jgg3nFK16R733vezn99NPzz//8z6OyHwD+dK3xEovW2neqaucnDR+Z5MDB7YuSfCvJPwzGPzVYNuN7VTWlqrZvrd0zYjNmQnvGM56Rd7zjHfngBz+YP//zP8+RRx6ZRx55JPPnz88DDzyQgw46KPPnz1/r199jjz1ywQUX5G//9m+z1157Zfbs2fmzP/uzPProo/nlL3+Za665JlOnTs1Pf/rTJMv+kMb555+fQw45JK95zWtWWgf5qquuyuzZs/P1r399SPvebrvtcuyxx+biiy/OjBkzcuihh2bp0qW58sors+mmm2bGjBlZtGjRSs/ZfffdM23atFx88cWZNGlSpk+fnqrKG9/4xqcNzHe84x25++67s//++2fnnXfO5MmTs3Dhwlx99dWZPn36SitSHHzwwfnwhz+ct7zlLXnNa16TLbfcMlOmTMlJJ52UJPnMZz6TWbNm5YQTTsjZZ5+dl770pZkyZUoWL16cm266KTfffHO++93vrrL6yKtf/eocffTROfroo1e8Z1/72tfyrGc9Kx//+MeH9J4lySc/+ckVZ51vv/32JMnll1++4g+r7LHHHjn55JNXbD9nzpwsWLAgu+yyS5544onVfrjwqKOOWqs/aw4ASYa2DnKSnZPc3N3/XXe7lt9PckWSl3WPXZVk5ppe3zrIG44M1sd9Oo8++mg744wz2gte8IK26aabtm233ba94Q1vaHfeeedq1+wdzjrIy910003tuOOOazvttFObPHly22abbdpee+3V5s6d26666qpVtl+wYEF75Stf2bbYYou2xRZbtIMPPrhdd9117dRTTx3WOsgPP/xwO+WUU9ouu+zSNtlkk7bDDju0t73tbe3+++9vBxxwwGrfm+9///tt1qxZbauttmpVNaT9fe5zn2vHHnts23XXXdvmm2/ettxyy7bXXnu1U045pS1ZsmSV7c8444y2xx57tMmTJ7ckbfr06Ss9/vvf/7594AMfaC9+8Yvb5ptv3jbddNO28847t1e96lXt3HPPbQ899NCKbZevVXzhhRe2yy+/vO27775ts802a1tvvXWbM2dOu/XWW4f0Xi23/Dg+1dcBBxyw0vbTp09/2u2Xzw0A1iRPsQ5ytTUsx5UkgzPIV7TWXji4/7vW2pTu8Qdba9tU1RVJPthau3YwflWSf2itLVjNa87NssswstNOO+0zUp/uB0bXvHnz8uY3vzkXXnjhsP78NwCMN1W1sLU288njQ7kGeXXurartBy+8fZLlFyjenWTHbrsdBmOraK2d11qb2VqbOXXq1LWcBgAAjKy1DeTLkhw3uH1cki93428arGaxb5KlzfXHAABMIGv8kF5VfTbLPpD3nKpanOTUJB9M8vmqOiHJXUleO9j8q0leleT2JP+Z5M2jMGcAABg1Q7oGebTNnDmzLViwymXKAAAwakb6GmQAANggCWQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOisMZCr6oKqWlJVN3djp1XV3VW1aPD1qu6x91XV7VV1a1W9crQmDgAAo2EoZ5DnJZm9mvEzW2szBl9fTZKq2jPJsUn2Gjzn41W10UhNFgAARtsaA7m19p0kDwzx9Y5McnFr7Y+ttV8kuT3JS9ZhfgAAsF6tyzXIJ1XVTYNLMLYZjE1L8qtum8WDMQAAmBDWNpDPSbJLkhlJ7klyxnBfoKrmVtWCqlpw3333reU0AABgZK1VILfW7m2tPd5aeyLJ/85/X0Zxd5Idu013GIyt7jXOa63NbK3NnDp16tpMAwAARtxaBXJVbd/dPTrJ8hUuLktybFVtUlXPT7Jbku+v2xQBAGD92XhNG1TVZ5McmOQ5VbU4yalJDqyqGUlakjuTnJgkrbUfV9Xnk/wkyWNJ3t5ae3xUZg4AAKOgWmtjPYfMnDmzLViwYKynAQDAn5CqWtham/nkcX9JDwAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADobj/UEAP7ULDzvxNWO7zP33PU8EwBWRyADjKKnimEAxi+XWAAAQEcgAwBARyADjBMuxwAYHwQywCjywTuAiUcgAwBARyADAEBHIAMAQEcgAwBARyADjCNWsgAYewIZAAA6AhlglFnqDWBiEcgAANARyAAA0BHIAADQEcgA44yVLADGlkAGWA98UA9g4hDIAADQEcgAANARyAAA0BHIAADQEcgAANARyADj0MLzTrTcG8AYEcgA64ml3gAmBoEMAACdNQZyVe1YVfOr6idV9eOqeudg/FlVdWVV3Tb4vs1gvKrq7Kq6vapuqqoXj/YPAQAAI2UoZ5AfS/Ke1tqeSfZN8vaq2jPJyUmuaq3tluSqwf0k+cskuw2+5iY5Z8RnDQAAo2SNgdxau6e1dsPg9h+S3JJkWpIjk1w02OyiJEcNbh+Z5FNtme8lmVJV24/0xAH+FPigHsD6N6xrkKtq5yQvSnJ9km1ba/cMHvpNkm0Ht6cl+VX3tMWDMYA/eT6oBzD+DTmQq2qLJF9I8q7W2u/7x1prLUkbzo6ram5VLaiqBffdd99wngoAAKNmSIFcVZOyLI4/3Vq7dDB87/JLJwbflwzG706yY/f0HQZjK2mtnddam9lamzl16tS1nT8AAIyooaxiUUnOT3JLa+0j3UOXJTlucPu4JF/uxt80WM1i3yRLu0sxAABgXNt4CNvsn+SNSX5UVYsGY6ck+WCSz1fVCUnuSvLawWNfTfKqJLcn+c8kbx7JCQMAwGhaYyC31q5NUk/x8MGr2b4lefs6zgsAAMaEv6QHMM5Z6g1g/RLIAOvZPnPPtdwbwDgmkAEAoCOQAQCgI5ABAKAjkAEAoCOQASYAK1kArD8CGWCMWMkCYHwSyAAA0BHIAADQEcgAANARyAAA0BHIABOElSwA1g+BDAAAHYEMMIYs9QYw/ghkAADoCGQAAOgIZAAA6AhkgAnEShYAo08gA4wxH9QDGF8EMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0BDLABGOpN4DRJZABxgFLvQGMHwIZAAA6AhkAADoCGQAAOgIZYALyQT2A0SOQAQCgI5ABxgkrWQCMDwIZAAA6AhkAADoCGQAAOgIZAAA6AhlggrLUG8DoEMgA44iVLADGnkAGAICOQAYAgI5ABgCAjkAGAICOQAaYwBaed6LVLABGmEAGGGesZAEwtgQyAAB0BDIAAHTWGMhVtWNVza+qn1TVj6vqnYPx06rq7qpaNPh6Vfec91XV7VV1a1W9cjR/AAAAGEkbD2Gbx5K8p7V2Q1VtmWRhVV05eOzM1tr/229cVXsmOTbJXkmel+Tfq+rPWmuPj+TEAQBgNKzxDHJr7Z7W2g2D239IckuSaU/zlCOTXNxa+2Nr7RdJbk/ykpGYLAAAjLZhXYNcVTsneVGS6wdDJ1XVTVV1QVVtMxibluRX3dMW5+mDGoB1ZKk3gJEz5ECuqi2SfCHJu1prv09yTpJdksxIck+SM4az46qaW1ULqmrBfffdN5ynAmzw9pl7ruXeAMbIkAK5qiZlWRx/urV2aZK01u5trT3eWnsiyf/Of19GcXeSHbun7zAYW0lr7bzW2szW2sypU6euy88AAAAjZiirWFSS85Pc0lr7SDe+fbfZ0UluHty+LMmxVbVJVT0/yW5Jvj9yUwYAgNEzlFUs9k/yxiQ/qqpFg7FTkryuqmYkaUnuTHJikrTWflxVn0/ykyxbAePtVrAAAGCiWGMgt9auTVKreeirT/OcDyT5wDrMC4BhWnjeia5bBhgB/pIewDgmeAHWP4EMAAAdgQwAAB2BDAAAHYEMAAAdgQwAAB2BDLABWXjeiWM9BYAJTyADjHOWegNYvwQyAAB0BDIAAHQEMgAAdAQywAbGB/UA1o1ABgCAjkAGmACsZAGw/ghkAADoCGQAAOgIZAAA6AhkAADoCGSADZCl3gDWnkAGmCCsZAGwfghkAADoCGQAAOgIZAAA6AhkAADoCGSADZSVLADWjkAGmECsZAEw+gQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMsAGzFJvAMMnkAEmGEu9AYwugQwAAB2BDDDKqmrEv8Z6/2s7F4CJQCADAEBHIANs4BacOzcLzp071tMAmDA2HusJADB8M088b6XovfzXqw/gVz/vvPU1JYANhjPIABPY5b+e+5RxvPxxAIZHIAMAQMclFgAbuP8+i+xyC4ChcAYZYIJy+QTA6BDIAADQEcgAfyIs9QYwNAIZYII6/fSZYz0FgA2SD+kBbOCshQwwPM4gA0xgr37eeU8bwOIYYPjWeAa5qjZN8p0kmwy2v6S1dmpVPT/JxUmenWRhkje21h6pqk2SfCrJPkl+m+SvW2t3jtL8AciqITzzxGX3Tx+LyQBMcEM5g/zHJLNaa3snmZFkdlXtm+RDSc5sre2a5MEkJwy2PyHJg4PxMwfbAQDAhLDGM8ittZbkocHdSYOvlmRWkr8ZjF+U5LQk5yQ5cnA7SS5J8rGqqsHrADCClp8pBmDkDOka5KraqKoWJVmS5MokP0/yu9baY4NNFieZNrg9LcmvkmTw+NIsuwwDAADGvSGtYtFaezzJjKqakuSLSfZY1x1X1dwkc7v76/qSAIwBv7+BDc2wlnlrrf2uquYn+Z9JplTVxoOzxDskuXuw2d1JdkyyuKo2TrJ1ln1Y78mvdV6S85Jk5syZbcGCBWv/UwCMYxt6QLqCDpionur38xovsaiqqYMzx6mqZyY5JMktSeYnOWaw2XFJvjy4fdngfgaPX+36YwAAJoqhnEHePslFVbVRlgX151trV1TVT5JcXFX/nOSHSc4fbH9+kv9TVbcneSDJsaMwbwAAGBVDWcXipiQvWs34HUlesprx/0ryVyMyOwAAWM/8JT0AAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADoCGQAAOgIZAAA6AhkAADobDzWEwDY0LXWxnoKAAyDM8gAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANARyAAA0BHIAADQEcgAANBZYyBX1aZV9f2qurGqflxVpw/G51XVL6pq0eBrxmC8qursqrq9qm6qqheP8s8AAAAjZuMhbPPHJLNaaw9V1aQk11bV1waP/T+ttUuetP1fJtlt8PXSJOcMvgMAwLi3xjPIbZmHBncnDb7a0zzlyCSfGjzve0mmVNX26z5VAAAYfUO6BrmqNqqqRUmWJLmytXb94KEPDC6jOLOqNhmMTUvyq+7piwdjAAAw7g0pkFtrj7fWZiTZIclLquqFSd6XZI8k/yPJs5L8w3B2XFVzq2pBVS247777hjdrAAAYJcNaxaK19rsk85PMbq3dM7iM4o9JLkzyksFmdyfZsXvaDoOxJ7/Wea21ma21mVOnTl2ryQMAwEgbyioWU6tqyuD2M5MckuSny68rrqpKclSSmwdPuSzJmwarWeybZGlr7Z5RmDsAAIy4oaxisX2Si6pqoywL6s+31q6oqquramqSSrIoyVsH2381yauS3J7kP5O8ecRnDQAAo2SNgdxauynJi1YzPusptm9J3r7uUwMAgPXPX9IDAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCAjkAGAICOQAYAgI5ABgCATrXWxnoOqar7kjyc5P6xngtr7Tlx/CYyx29ic/wmNsdvYnP8JrbprbWpTx4cF4GcJFW1oLU2c6znwdpx/CY2x29ic/wmNsdvYnP8NkwusQAAgI5ABgCAzngK5PPGegKsE8dvYnP8JjbHb2Jz/CY2x28DNG6uQQYAgPFgPJ1BBgCAMTfmgVxVs6vq1qq6vapOHuv5sHpVdUFVLamqm7uxZ1XVlVV12+D7NoPxqqqzB8f0pqp68djNnKrasarmV9VPqurHVfXOwbjjNwFU1aZV9f2qunFw/E4fjD+/qq4fHKfPVdXkwfgmg/u3Dx7feUx/AJIkVbVRVf2wqq4Y3Hf8JoiqurOqflRVi6pqwWDM788N3JgGclVtlOT/S/KXSfZM8rqq2nMs58RTmpdk9pPGTk5yVWtttyRXDe4ny47nboOvuUnOWU9zZPUeS/Ke1tqeSfZN8vbB/585fhPDH5PMaq3tnWRGktlVtW+SDyU5s7W2a5IHk5ww2P6EJA8Oxs8cbMfYe2eSW7r7jt/EclBrbUa3nJvfnxu4sT6D/JIkt7fW7mitPZLk4iRHjvGcWI3W2neSPPCk4SOTXDS4fVGSo7rxT7VlvpdkSlVtv14myipaa/e01m4Y3P5Dlv2P9LQ4fhPC4Dg8NLg7afDVksxKcslg/MnHb/lxvSTJwVVV62e2rE5V7ZDksCSfHNyvOH4Tnd+fG7ixDuRpSX7V3V88GGNi2La1ds/g9m+SbDu47biOU4N/rn1Rkuvj+E0Yg3+eX5RkSZIrk/w8ye9aa48NNumP0YrjN3h8aZJnr9cJ82QfTfLeJE8M7j87jt9E0pJ8s6oWVtXcwZjfnxu4jcd6AmwYWmutqiyJMo5V1RZJvpDkXa213/cnpRy/8a219niSGVU1JckXk+wxtjNiqKrq8CRLWmsLq+rAMZ4Oa+dlrbW7q+q5Sa6sqp/2D/r9uWEa6zPIdyfZsbu/w2CMieHe5f90NPi+ZDDuuI4zVTUpy+L40621SwfDjt8E01r7XZL5Sf5nlv3T7fKTHP0xWnH8Bo9vneS363emdPZPckRV3ZlllxHOSnJWHL8Jo7V29+D7kiz7D9SXxO/PDd5YB/IPkuw2+DTv5CTHJrlsjOfE0F2W5LjB7eOSfLkbf9Pg07z7Jlna/VMU69ng+sXzk9zSWvtI95DjNwFU1dTBmeNU1TOTHJJl15HPT3LMYLMnH7/lx/WYJFc3C96Pmdba+1prO7TWds6y/427urX2+jh+E0JVbV5VWy6/neTQJDfH788N3pj/oZCqelWWXZ+1UZILWmsfGNMJsVpV9dkkByZ5TpJ7k5ya5EtJPp9kpyR3JXlta+2BQZB9LMtWvfjPJG9urS0Yg2mTpKpeluSaJD/Kf18DeUqWXYfs+I1zVfV/Z9mHgDbKspMan2+t/VNV/V9ZdkbyWUl+mOQNrbU/VtWmSf5Pll1r/kCSY1trd4zN7OkNLrH4X621wx2/iWFwnL44uLtxks+01j5QVc+O358btDEPZAAAGE/G+hILAAAYVwQyAAB0BDIAAHQEMgAAdAQyAAB0BDIAAHQEMgAAdAQyAAB0/n9q0JK8LueeaQAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython import display\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "obs = env.reset()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "img = ax.imshow(env.render(mode=\"rgb_array\"), interpolation=\"none\")\n",
    "fig.tight_layout()\n",
    "\n",
    "done = False\n",
    "for idx in range(100):\n",
    "    img.set_data(env.render(mode=\"rgb_array\"))\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "\n",
    "    action = env.action_space.sample()\n",
    "    obs, reward, done, info = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        plt.text(100, 100, f\"Failed at step {idx}\", fontsize=20)\n",
    "        break\n",
    "\n",
    "if not done:\n",
    "    plt.text(100, 100, f\"Stable until step {idx + 1}\", fontsize=20)\n",
    "\n",
    "env.close()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Training An Agent\n",
    "\n",
    "We chose to use the Proximal Policy Optimization (PPO) algorithm with a Multi-Layer Perceptron (MLP, also known as Feedforward Fully-Connected Neural Network) policy to make our agent learn how to balance a stick! Some other popular algorithms are Deep Q-Network (DQN), Advantage Actor Critic (A2C), and more. A very extensive list of RL algorithms can be found [here](https://docs.ray.io/en/latest/rllib/rllib-algorithms.html).\n",
    "\n",
    "However, keep in mind, this is a very simple RL problem. Typically, RL algorithms require a lot of hyperparameter tuning.\n",
    "\n",
    "In this example, due to its simplicity, we are using the [Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/) library. This does allow many configuration options. However, for production scale RL and high performance training, the industry standard is [Ray](https://docs.ray.io/en/latest/index.html)'s [RLlib](https://docs.ray.io/en/latest/rllib/index.html).\n",
    "\n",
    "For tunable parameters of PPO, we recommend checking out the Stable Baselines3 documentation (below). The PPO explained article is also helpful in understanding the parameters and their meanings.\n",
    "\n",
    "More information about PPO:\n",
    "* [PPO Paper](https://arxiv.org/abs/1707.06347)\n",
    "* [Understanding PPO Plots](https://medium.com/aureliantactics/understanding-ppo-plots-in-tensorboard-cbc3199b9ba2)\n",
    "* [OpenAI Spinning Up PPO](https://spinningup.openai.com/en/latest/algorithms/ppo.html)\n",
    "* [PPO Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html)\n",
    "* [PPO Explained - Article](https://jonathan-hui.medium.com/rl-proximal-policy-optimization-ppo-explained-77f014ec3f12)\n",
    "* [PPO Keras from Scratch](https://keras.io/examples/rl/ppo_cartpole/)\n",
    "\n",
    "We will first let it train for 10,000 time-steps. We will also evaluate it on 10 episodes to and get its average score."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n",
      "Wrapping the env with a `Monitor` wrapper\n",
      "Wrapping the env in a DummyVecEnv.\n",
      "------------------------------------------\n",
      "| rollout/                |              |\n",
      "|    ep_len_mean          | 63.2         |\n",
      "|    ep_rew_mean          | 63.2         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 1657         |\n",
      "|    iterations           | 5            |\n",
      "|    time_elapsed         | 6            |\n",
      "|    total_timesteps      | 10240        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0070142997 |\n",
      "|    clip_fraction        | 0.0708       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -0.601       |\n",
      "|    explained_variance   | 0.254        |\n",
      "|    learning_rate        | 0.0003       |\n",
      "|    loss                 | 22.6         |\n",
      "|    n_updates            | 40           |\n",
      "|    policy_gradient_loss | -0.0167      |\n",
      "|    value_loss           | 70.3         |\n",
      "------------------------------------------\n",
      "10 evaluation runs mean reward = 436.3\n"
     ]
    }
   ],
   "source": [
    "from stable_baselines3 import PPO\n",
    "from stable_baselines3.common.evaluation import evaluate_policy\n",
    "\n",
    "model = PPO(\"MlpPolicy\", env, verbose=1)\n",
    "model.learn(total_timesteps=10_000, log_interval=5)\n",
    "\n",
    "mean_reward, std_reward = evaluate_policy(model, model.get_env(), n_eval_episodes=10)\n",
    "print(f\"10 evaluation runs mean reward = {mean_reward}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now let's visualize the trained agent.\n",
    "\n",
    "Note that now the action is being predicted by the model, instead of sampling randomly from the action space."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 720x720 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAHjCAYAAAA+KOkeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAApLklEQVR4nO3debSdVX0//vcmEwSBMIQhkBBkEKVKgPwQESFAZRAlUChiHRi0wVURrG2dqgW0Vq1Fqwu/SrTIoCCCQpjEIoSxioShIAEhQCBBCIRJQQQS9u+PnKQ75Ca5Se7l5sbXa62z7vPsZ/qcs5Obd56zzz6l1hoAAGCe1fq6AAAAWJkIyAAA0BCQAQCgISADAEBDQAYAgIaADAAAjV4LyKWU/Uopvy2lTCulfKq3rgMAAD2p9MY8yKWUAUnuSfL2JDOT3JTkPbXWqT1+MQAA6EG9dQd55yTTaq3311pfTPKjJON76VoAANBjeisgb5pkRrM+s9MGAAArtYF9deFSyoQkE5JkzTXX3Gnbbbftq1IAAPgzdPPNN8+utQ5/ZXtvBeSHk4xs1jfrtC1Qa52YZGKSjB07tk6ZMqWXSgEAgEWVUh7sqr23hljclGTrUsoWpZTBSQ5PclEvXQsAAHpMr9xBrrXOKaUcm+TnSQYkOa3WemdvXAsAAHpSr41BrrVeluSy3jo/AAD0Bt+kBwAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBj4IocXEqZnuQPSeYmmVNrHVtKWS/JuUlGJ5me5LBa61MrViYAALw6euIO8p611jG11rGd9U8lubLWunWSKzvrAADQL/TGEIvxSc7oLJ+R5KBeuAYAAPSKFQ3INcl/l1JuLqVM6LRtVGt9pLP8aJKNVvAaAADwqlmhMchJdqu1PlxK2TDJFaWUu9uNtdZaSqldHdgJ1BOSZNSoUStYBgAA9IwVuoNca3248/OxJBck2TnJrFLKJknS+fnYYo6dWGsdW2sdO3z48BUpAwAAesxyB+RSypqllLXmLyfZJ8lvklyU5IjObkckmbSiRQIAwKtlRYZYbJTkglLK/POcXWu9vJRyU5Ifl1I+mOTBJIeteJkAAPDqWO6AXGu9P8n2XbQ/kWTvFSkKAAD6im/SAwCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGRWKdOnT08pJUceeWS3jzn99NNTSsnpp5/ea3WtDMaNG5dSykJtV199dUopOfHEE/umKABYCQnI9Ki5c+fmu9/9bvbYY4+st956GTRoUDbccMO86U1vyoc+9KFcdNFFC+3/5xJOXw1HHnlkSimZPn16X5eSJCmlZNy4cX1dRpdmzpyZL37xi/nrv/7rbLXVVllttdVSSsm0adOWeNzzzz+fE044Ia973euy+uqrZ8MNN8xhhx2Wu+66a7HHPPnkk/nYxz6W0aNHZ8iQIRkxYkSOPvrozJw5s6efFgA9ZGBfF8CqY+7cuXnnO9+Zyy+/PMOGDcsBBxyQzTbbLC+++GLuvPPOnH322bn77rtz4IEH9nWpf5bOPPPM/PGPf+zrMlYKU6ZMyWc/+9mUUrLFFltknXXWydNPP73EY1544YW8/e1vzw033JCxY8fm+OOPz4wZM3Leeefl0ksvzVVXXZU3v/nNCx3zxBNPZNddd80999yTvfbaK4cffnjuvvvufP/738+ll16aX/7yl3nta1/bi88UgOUhINNjzjnnnFx++eXZfvvtc80112SdddZZaPsf//jH3HjjjX1UHaNGjerrElYaY8eOzbXXXpvtt98+a6+9dsaNG5drrrlmicd87Wtfyw033JBDDz005557blZbbd4bcO9+97tz0EEH5eijj84dd9yxoD1JPvOZz+See+7Jxz/+8Zx88skL2r/5zW/m+OOPz9/93d/l8ssv750nCcByM8SCHvM///M/Sea91f/KcJwkQ4cOzZ577rlgfdy4cTnqqKOSJEcddVRKKQse84cJ/O53v8vnP//5vPWtb83GG2+cwYMHZ8SIEfmbv/mbTJ06dYn13H333TnooIOy3nrrZc0118xuu+2W//7v/16m5zRz5swce+yxee1rX5shQ4Zk/fXXz4EHHpibbrqp2+dY2jjf0aNHZ/To0Qu1tUNPJk+enHHjxmWttdbK2muvnQMOOGCRt/RLKTnjjDOSJFtsscWC17E9b1djkJfViy++mG9+85vZcccds+6662bo0KEZPXp0xo8fn1/84hcL1Z4k11xzzUL9+srX4MYbb8yhhx66oG9HjhyZY445Jr/73e8Wufb8+l944YV89rOfzRZbbJEhQ4Zkyy23zEknnZQXX3yx289js802y9ve9rasvfba3dq/1prvfOc7SZJ///d/XygEjx8/Pm9729syderUhUL2s88+m7POOitrrrnmIs/72GOPzeabb56f//znuf/++7tdNwCvDneQ6THrr79+kuSee+7p1v5HHnlkhg0blkmTJmX8+PEZM2bMgm3Dhg1Lklx77bX58pe/nD333DOHHHJIXvOa1+Tee+/N+eefn4suuig33HBDtt9++0XO/cADD+Qtb3lL3vjGN+aYY47JI488knPPPTf7779/zj777Lz73e9ean233HJL9tlnnzz55JPZd99981d/9VeZPXt2Lrzwwuy222654IIL8o53vKNbz3V5XXLJJZk0aVL233//fPjDH87UqVNz2WWX5aabbsrUqVOzwQYbJElOOOGEXHjhhfnf//3fHH/88Qtev/k/e8qRRx6Zc845J3/xF3+RD3zgA1ljjTXyu9/9Ltdff30uv/zy/OVf/mXGjBmTE044ISeddFI233zzhT4w2Y5JPu200zJhwoQMGTIkBx54YEaOHJl777033/ve93LxxRfnV7/6VZd3vQ877LDcdNNNOfTQQzNo0KBMmjQpJ554YqZMmZKLLrpohf8T0JX77rsvDz30ULbZZptsscUWi2zff//9c9111+Wqq65a8J/AX/3qV3n++eezzz77ZK211lpo/9VWWy377rtvJk6cmMmTJxtmAbCyqbX2+WOnnXaq9H+33HJLHTRoUC2l1Pe97331Jz/5SZ0+ffoSj/n+979fk9Tvf//7XW6fNWtW/f3vf79I+2233VbXXHPNut9++y3U/sADD9QkNUn9x3/8x4W23XTTTXXgwIF12LBh9ZlnnlliDS+99FLdcsst65AhQ+rVV1+90HkefvjhOmLEiLrxxhvXP/3pT0t8frXWOnny5JqknnDCCV1u33zzzevmm2++UNv8mgYMGFB/8YtfLLTtU5/6VE1Sv/KVryzUfsQRR9Qk9YEHHujyOnvssUed91e++7W1nn766VpKqTvttFOdM2fOIttnz5690HqSuscee3R5rt/+9rd10KBBdcstt6wzZ85caNsvfvGLutpqq9WDDjqoy/q33nrr+uSTTy5of/755+suu+xSk9Qzzzxzqc+jK/PPfe+993a5/ZJLLqlJ6jvf+c4ut5933nk1ST3ssMMWtJ1yyik1ST322GO7POarX/1qTVI/8YlPLFfNAKy4JFNqF9nUEAt6zA477JAf/OAH2WijjfKDH/wghxxySEaPHp31118/Bx98cC6++OJlPueGG264yN23JNl+++2z1157ZfLkyXnppZcW2b7OOuvkX/7lXxZqGzt2bN773vfm6aefzgUXXLDE61566aW577778tGPfjR77LHHQttGjBiRT3ziE3n00Udz5ZVXLvNzWhaHH3549t5774XaJkyYkCT59a9/3avXfqVSSmqtGTJkyEJDDOab/w5Cd3z729/OSy+9lG984xvZdNNNF9q2995758ADD8zFF1+cP/zhD4sc+7nPfS7rrrvugvXVV189X/rSl5LMuyvdG5555pkk6XLoUNveftBveY4BYOVgiAU96rDDDsvBBx+cyZMn5/rrr8+tt96a66+/PhdeeGEuvPDCfOADH1hojGp3XHrppfnOd76TKVOmZPbs2ZkzZ85C22fPnp1NNtlkobYdd9yxy2A9bty4nHHGGbn11ltzxBFHLPaav/zlL5MkDz74YJdjh++9994kyV133dWrwyzGjh27SNvIkSOTJE899VSvXbcra6+9dt71rnfl4osvzpgxY3LIIYfkbW97W9785jdn6NChy3Su+a/vNddc0+V47sceeyxz587NPffck5122mmhba/8D0uS7LbbbhkwYEBuvfXWZaoDALoiINPjBg0alH322Sf77LNPknnTv/3kJz/J0UcfnTPPPDMHH3xwDjrooG6d6xvf+EY+9rGPZd11183b3/72jBo1KkOHDk0pZcGY2xdeeGGR4zbaaKMuz7fxxhsn+b+7e4vzxBNPJEnOO++8Je737LPPdudpLLeuxhAPHDjvr+3cuXN79dpdOffcc/OVr3wlZ599dk444YQk8+7gHnroofmP//iPxb7urzT/9f3qV7+6xP26en27usbAgQOzwQYb5LHHHuvW9ZfV/Lu9i/tzM7+97a/lOQaAlYOATK8bMGBADjvssNxxxx3513/911x11VXdCshz5szJiSeemI033ji33HLLIneJ59+F7MqsWbO6bH/00UeTLP5t7/nmb580adIKz9s8fzjCK+98z/f000/3m5C0xhpr5MQTT8yJJ56YGTNm5Nprr83pp5+eH/zgB5k+fXquu+66bp2nDY/dnUlivlmzZi3y4b05c+Zk9uzZy3yu7nrd616XZPEfQJ3/jsI222yzQscAsHJY6hjkUspppZTHSim/adrWK6VcUUq5t/Nz3U57KaV8s5QyrZRyeyllx94snv5l/pCHeWPi5xkwYECSru+Gzp49O08//XR23XXXRcLxs88+m1tuuWWx17rlllu6HL969dVXJ5k3XnpJdtlllyTpduBbkvnjZWfMmLHItmnTpi31bnZ3Lem17A0jR47Me9/73vz85z/PVlttleuvv37BneFk3n8MFlfLiry+Xc1XfP3112fu3LlL7dflteWWW2bUqFG555578sADDyyy/Wc/+1mSZK+99lrQtssuu2SNNdbIDTfcsMifxZdffnnBlIPt1IcArBy68yG905Ps94q2TyW5sta6dZIrO+tJsn+SrTuPCUm+3TNl0h+cc845ueKKK/Lyyy8vsu3RRx/Nd7/73STJ7rvvvqB9/ge7HnrooUWO2XDDDTN06NDcfPPNC73V/tJLL+X444/P7NmzF1vLM888k89//vMLtU2ZMiU//OEPs8466+Tggw9e4nMZP358ttxyy3zrW9/KZZdd1uU+v/zlL7v1zXTbbrtt1l577UyaNGmhIQDPP/98jjvuuKUe311Lei17wuOPP5477rhjkfbnnnsuzz77bAYOHJjBgwcvVE9X/ylI5s0DPGjQoPz93/99l3dYX3zxxcWG5y984QsLjb/+05/+lE9/+tNJsmBe7Z5WSsmHP/zhJMknPvGJhf6MT5o0Kdddd13e8IY3LDQ++jWveU3e//7357nnnltkHPspp5yS6dOnZ9999zXFG8BKaKlDLGqt15ZSRr+ieXyScZ3lM5JcneSTnfYzO9Nm/KqUMqyUskmt9ZEeq5iV1o033phvfOMb2XjjjbPbbrstmC/2gQceyKWXXprnn38+48ePz6GHHrrgmLe85S0ZOnRo/vM//zNPPPHEgjHCH/3oR7POOuvkuOOOy5e//OW88Y1vzPjx4/Piiy9m8uTJefLJJ7Pnnntm8uTJXday++6753vf+15uvPHGvPWtb10wD/LLL7+cU089dalvxQ8aNCg//elPs+++++aAAw7IrrvumjFjxmTo0KGZMWNGbrrpptx///155JFHlvoBtUGDBuX444/PF77wheywww45+OCDM2fOnFxxxRUZMWJERowYsSwv82Ltvffe+epXv5q//du/zSGHHJK11lorw4YNy7HHHtsj53/44Yezww475I1vfGPe9KY3ZeTIkfn973+fSy65JI8++miOO+64hT4Yuffee+dHP/pR3vWud2XHHXfMoEGDsvvuu2f33XfPtttum9NOOy1HH310tttuu+y3337ZZptt8tJLL+Whhx7Kddddl+HDh+fuu+9epI7Xv/712W677RaaB/m+++7LAQcckPe///3dfj7t/Mzzr/PJT35ywXP40Ic+lN12223BPh//+MdzySWX5Pzzz8+b3/zm7L333nnooYdy3nnnZejQoTnttNMWmd3j3/7t33L11Vfna1/7Wm677bbsvPPOueuuuzJp0qRsuOGG+da3vtXtegF4FXU199srH0lGJ/lNs/50s1zmrye5JMluzbYrk4xd2vnNg7xqeOihh+opp5xSDzrooLrNNtvUtdZaqw4aNKhuvPHGdf/9969nnXVWnTt37iLH/exnP6u77LJLXXPNNRfMYTx/Lt+XXnqpnnzyyfX1r399XX311etGG21U3/e+99Xp06d3Oe/v/HmQjzjiiDp16tR64IEH1mHDhtU11lij7rrrrvXyyy9f5PpLmot51qxZ9ZOf/GTdbrvt6hprrFHXXHPNutVWW9VDDjmknnXWWfWll17q1mvz8ssv1y996Uv1ta99bR00aFAdOXJk/ad/+qf63HPPLXEe5MXND53FzDF88skn12233bYOHjy4JlnovCs6D/JTTz1VTzrppLrnnnvWESNG1MGDB9eNN9647rHHHvXss8+uL7/88kL7z5o1q77nPe+pG264YV1ttdW6vM7tt99ejzjiiDpq1Kg6ePDguu6669btttuuTpgwoV555ZUL7Tu//j/96U/1n//5n+vo0aPr4MGD6xZbbFFPPPHEbs1J3Zr/Z21xj65e++eee65+7nOfq1tttVUdPHhw3WCDDeqhhx5a77zzzsVe54knnqjHHXdcHTVq1IK/D0cddVSdMWPGMtULQM/LYuZBLrUZD7o4nTvIl9Ra/6Kz/nStdViz/ala67qllEuSfLnWen2n/cokn6y1TuninBMybxhGRo0atdODDz641DqAP1/jxo3LNddck+78zgKA7iil3FxrXWRO1eX9opBZpZRNOifeJMn8gZUPJxnZ7LdZp20RtdaJtdaxtdaxw4cPX84yAACgZy1vQL4oyfxvWTgiyaSm/QOd2Sx2SfJMNf4YAIB+ZKkf0iulnJN5H8jboJQyM8kJSb6c5MellA8meTDJYZ3dL0vyjiTTkvwxSe98pBwAAHpJt8Yg97axY8fWKVMWGaYMAAC9pqfHIAMAwCpJQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0FhqQC6lnFZKeayU8pum7cRSysOllNs6j3c02z5dSplWSvltKWXf3iocAAB6Q3fuIJ+eZL8u2r9eax3TeVyWJKWUNyQ5PMl2nWP+XyllQE8VCwAAvW2pAbnWem2SJ7t5vvFJflRrfaHW+kCSaUl2XoH6AADgVbUiY5CPLaXc3hmCsW6nbdMkM5p9ZnbaAACgX1jegPztJFsmGZPkkSQnL+sJSikTSilTSilTHn/88eUsAwAAetZyBeRa66xa69xa68tJvpv/G0bxcJKRza6bddq6OsfEWuvYWuvY4cOHL08ZAADQ45YrIJdSNmlWD04yf4aLi5IcXkoZUkrZIsnWSX69YiUCAMCrZ+DSdiilnJNkXJINSikzk5yQZFwpZUySmmR6kmOSpNZ6Zynlx0mmJpmT5CO11rm9UjkAAPSCUmvt6xoyduzYOmXKlL4uAwCAPyOllJtrrWNf2e6b9AAAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMrNJunnhMbp54TF+XAUA/MrCvCwBYHkIvAL3FHWQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIQL+004RTl2l/37wHQHcJyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoLDUgl1JGllIml1KmllLuLKUc32lfr5RyRSnl3s7PdTvtpZTyzVLKtFLK7aWUHXv7SQB/nnaacGp2mnBqt/e/eeIxvVgNAKuK7txBnpPkH2qtb0iyS5KPlFLekORTSa6stW6d5MrOepLsn2TrzmNCkm/3eNUAANBLlhqQa62P1Fpv6Sz/IcldSTZNMj7JGZ3dzkhyUGd5fJIz6zy/SjKslLJJTxcOAAC9YZnGIJdSRifZIcmNSTaqtT7S2fRoko06y5smmdEcNrPTBgAAK71uB+RSymuS/CTJx2qtv2+31VprkrosFy6lTCilTCmlTHn88ceX5VAAAOg13QrIpZRBmReOf1hr/Wmnedb8oROdn4912h9OMrI5fLNO20JqrRNrrWNrrWOHDx++vPUDAECP6s4sFiXJfyW5q9b6tWbTRUmO6CwfkWRS0/6BzmwWuyR5phmKAQAAK7WB3djnrUnen+SOUsptnbbPJPlykh+XUj6Y5MEkh3W2XZbkHUmmJfljkqN6smAAAOhNSw3Itdbrk5TFbN67i/1rko+sYF0AANAnfJMeAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjLQ7+004dRu73vzxGN6sRIAVgUCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjKwSthpwqnd3vfmicf0YiUA9HcCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjKwythpwqnd3vfmicf0YiUA9GcCMgAANARkAABoCMgAANAQkAEAoLHUgFxKGVlKmVxKmVpKubOUcnyn/cRSysOllNs6j3c0x3y6lDKtlPLbUsq+vfkEAACgJw3sxj5zkvxDrfWWUspaSW4upVzR2fb1Wut/tDuXUt6Q5PAk2yUZkeQXpZRtaq1ze7JwAADoDUu9g1xrfaTWektn+Q9J7kqy6RIOGZ/kR7XWF2qtDySZlmTnnigWAAB62zKNQS6ljE6yQ5IbO03HllJuL6WcVkpZt9O2aZIZzWEzs+RADQAAK41uB+RSymuS/CTJx2qtv0/y7SRbJhmT5JEkJy/LhUspE0opU0opUx5//PFlORQAAHpNtwJyKWVQ5oXjH9Zaf5oktdZZtda5tdaXk3w3/zeM4uEkI5vDN+u0LaTWOrHWOrbWOnb48OEr8hwAAKDHdGcWi5Lkv5LcVWv9WtO+SbPbwUl+01m+KMnhpZQhpZQtkmyd5Nc9VzIAAPSe7sxi8dYk709yRynltk7bZ5K8p5QyJklNMj3JMUlSa72zlPLjJFMzbwaMj5jBAgCA/mKpAbnWen2S0sWmy5ZwzBeTfHEF6gIAgD7hm/QAAKAhIAMAQENABlYpO004tdv73jzxmF6sBID+SkAGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAN/1m6eeExflwDASkZABlY5O004ta9LAKAfE5ABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGSgz5VSevzR19df3loA6HsCMvBnb8qpEzLl1Al9XQYAKwkBGVgljT1mYl+XAEA/NbCvCwB4NVz8u67vEL9rhCANwMLcQQZWaRf/bsJiw/H87QDQEpABAKAhIAOrLHeHAVgexiADf/b+L0gbjwyAO8gAALAQARkAABqGWAB/9uZP9XZSH9cBwMrBHWRglWWOYwCWh4AMrNLeNWLiEoOyEA3AKy11iEUpZfUk1yYZ0tn//FrrCaWULZL8KMn6SW5O8v5a64ullCFJzkyyU5Inkry71jq9l+oH6JY2CLdfQ21YBQCv1J07yC8k2avWun2SMUn2K6XskuQrSb5ea90qyVNJPtjZ/4NJnuq0f72zHwAA9AtLvYNca61Jnu2sDuo8apK9kvxNp/2MJCcm+XaS8Z3lJDk/ySmllNI5D8Crpr1TDADd1a0xyKWUAaWU25I8luSKJPclebrWOqezy8wkm3aWN00yI0k625/JvGEYAACw0uvWNG+11rlJxpRShiW5IMm2K3rhUsqEJBOa9RU9JcBKye83gP5lmeZBrrU+XUqZnOQtSYaVUgZ27hJvluThzm4PJxmZZGYpZWCSdTLvw3qvPNfEdL7XdezYsXXKlCnL/yyAfm1VD5BGmAGsnBb3789Sh1iUUoZ37hynlLJGkrcnuSvJ5CSHdnY7IsmkzvJFnfV0tl9l/DEAAP1Fd+4gb5LkjFLKgMwL1D+utV5SSpma5EellH9NcmuS/+rs/19JziqlTEvyZJLDe6FuAADoFd2ZxeL2JDt00X5/kp27aP9Tkr/ukeoAAOBV5pv0AACgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAACNgX1dAECtta9LAIAF3EEGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAAjaUG5FLK6qWUX5dS/reUcmcp5aRO++mllAdKKbd1HmM67aWU8s1SyrRSyu2llB17+TkAAECPGdiNfV5Islet9dlSyqAk15dSftbZ9k+11vNfsf/+SbbuPN6c5NudnwAAsNJb6h3kOs+zndVBnUddwiHjk5zZOe5XSYaVUjZZ8VIBAKD3dWsMcillQCnltiSPJbmi1npjZ9MXO8Movl5KGdJp2zTJjObwmZ02AABY6XUrINda59ZaxyTZLMnOpZS/SPLpJNsm+f+SrJfkk8ty4VLKhFLKlFLKlMcff3zZqgYAgF6yTLNY1FqfTjI5yX611kc6wyheSPL9JDt3dns4ycjmsM06ba8818Ra69ha69jhw4cvV/EAANDTujOLxfBSyrDO8hpJ3p7k7vnjikspJclBSX7TOeSiJB/ozGaxS5Jnaq2P9ELtAADQ47ozi8UmSc4opQzIvED941rrJaWUq0opw5OUJLcl+XBn/8uSvCPJtCR/THJUj1cNAAC9ZKkBudZ6e5IdumjfazH71yQfWfHSAADg1eeb9AAAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0BCQAQCgISADAEBDQAYAgIaADAAADQEZAAAaAjIAADQEZAAAaAjIAADQEJABAKAhIAMAQENABgCAhoAMAAANARkAABoCMgAANARkAABoCMgAANAQkAEAoCEgAwBAQ0AGAIBGqbX2dQ0ppTye5Lkks/u6FpbbBtF//Zn+69/0X/+m//o3/de/bV5rHf7KxpUiICdJKWVKrXVsX9fB8tF//Zv+69/0X/+m//o3/bdqMsQCAAAaAjIAADRWpoA8sa8LYIXov/5N//Vv+q9/03/9m/5bBa00Y5ABAGBlsDLdQQYAgD7X5wG5lLJfKeW3pZRppZRP9XU9dK2Uclop5bFSym+atvVKKVeUUu7t/Fy3015KKd/s9OntpZQd+65ySikjSymTSylTSyl3llKO77Trv36glLJ6KeXXpZT/7fTfSZ32LUopN3b66dxSyuBO+5DO+rTO9tF9+gRIkpRSBpRSbi2lXNJZ13/9RClleinljlLKbaWUKZ02vz9XcX0akEspA5J8K8n+Sd6Q5D2llDf0ZU0s1ulJ9ntF26eSXFlr3TrJlZ31ZF5/bt15TEjy7VepRro2J8k/1FrfkGSXJB/p/D3Tf/3DC0n2qrVun2RMkv1KKbsk+UqSr9dat0ryVJIPdvb/YJKnOu1f7+xH3zs+yV3Nuv7rX/astY5ppnPz+3MV19d3kHdOMq3Wen+t9cUkP0oyvo9rogu11muTPPmK5vFJzugsn5HkoKb9zDrPr5IMK6Vs8qoUyiJqrY/UWm/pLP8h8/6R3jT6r1/o9MOzndVBnUdNsleS8zvtr+y/+f16fpK9Synl1amWrpRSNktyQJLvddZL9F9/5/fnKq6vA/KmSWY06zM7bfQPG9VaH+ksP5pko86yfl1Jdd6u3SHJjdF//Ubn7fnbkjyW5Iok9yV5utY6p7NL20cL+q+z/Zkk67+qBfNK/5nkE0le7qyvH/3Xn9Qk/11KubmUMqHT5vfnKm5gXxfAqqHWWksppkRZiZVSXpPkJ0k+Vmv9fXtTSv+t3Gqtc5OMKaUMS3JBkm37tiK6q5TyziSP1VpvLqWM6+NyWD671VofLqVsmOSKUsrd7Ua/P1dNfX0H+eEkI5v1zTpt9A+z5r911Pn5WKddv65kSimDMi8c/7DW+tNOs/7rZ2qtTyeZnOQtmffW7fybHG0fLei/zvZ1kjzx6lZK461JDiylTM+8YYR7JflG9F+/UWt9uPPzscz7D+rO8ftzldfXAfmmJFt3Ps07OMnhSS7q45rovouSHNFZPiLJpKb9A51P8+6S5JnmrSheZZ3xi/+V5K5a69eaTfqvHyilDO/cOU4pZY0kb8+8ceSTkxza2e2V/Te/Xw9NclU14X2fqbV+uta6Wa11dOb9G3dVrfW90X/9QillzVLKWvOXk+yT5Dfx+3OV1+dfFFJKeUfmjc8akOS0WusX+7QgulRKOSfJuCQbJJmV5IQkFyb5cZJRSR5Mclit9clOIDsl82a9+GOSo2qtU/qgbJKUUnZLcl2SO/J/YyA/k3njkPXfSq6U8qbM+xDQgMy7qfHjWuvnSymvzbw7kusluTXJ+2qtL5RSVk9yVuaNNX8yyeG11vv7pnpanSEW/1hrfaf+6x86/XRBZ3VgkrNrrV8spawfvz9XaX0ekAEAYGXS10MsAABgpSIgAwBAQ0AGAICGgAwAAA0BGQAAGgIyAAA0BGQAAGgIyAAA0Pj/Ab0EEj/hcMLWAAAAAElFTkSuQmCC\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython import display\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "obs = env.reset()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "img = ax.imshow(env.render(mode=\"rgb_array\"), interpolation=\"none\")\n",
    "fig.tight_layout()\n",
    "\n",
    "done = False\n",
    "for idx in range(100):\n",
    "    img.set_data(env.render(mode=\"rgb_array\"))\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "\n",
    "    action, _states = model.predict(obs)\n",
    "    obs, reward, done, info = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        plt.text(100, 100, f\"Failed at step {idx}\", fontsize=20)\n",
    "        break\n",
    "\n",
    "if not done:\n",
    "    plt.text(100, 100, f\"Stable until step {idx + 1}\", fontsize=20)\n",
    "\n",
    "env.close()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's train for more time."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n",
      "Wrapping the env with a `Monitor` wrapper\n",
      "Wrapping the env in a DummyVecEnv.\n",
      "------------------------------------------\n",
      "| rollout/                |              |\n",
      "|    ep_len_mean          | 236          |\n",
      "|    ep_rew_mean          | 236          |\n",
      "| time/                   |              |\n",
      "|    fps                  | 1526         |\n",
      "|    iterations           | 15           |\n",
      "|    time_elapsed         | 20           |\n",
      "|    total_timesteps      | 30720        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0048547667 |\n",
      "|    clip_fraction        | 0.0299       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -0.523       |\n",
      "|    explained_variance   | 0.052        |\n",
      "|    learning_rate        | 0.0003       |\n",
      "|    loss                 | 0.0786       |\n",
      "|    n_updates            | 140          |\n",
      "|    policy_gradient_loss | -0.000506    |\n",
      "|    value_loss           | 1.98         |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| rollout/                |              |\n",
      "|    ep_len_mean          | 465          |\n",
      "|    ep_rew_mean          | 465          |\n",
      "| time/                   |              |\n",
      "|    fps                  | 1508         |\n",
      "|    iterations           | 30           |\n",
      "|    time_elapsed         | 40           |\n",
      "|    total_timesteps      | 61440        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0048740814 |\n",
      "|    clip_fraction        | 0.024        |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -0.48        |\n",
      "|    explained_variance   | 0.00551      |\n",
      "|    learning_rate        | 0.0003       |\n",
      "|    loss                 | 0.00102      |\n",
      "|    n_updates            | 290          |\n",
      "|    policy_gradient_loss | -0.00275     |\n",
      "|    value_loss           | 0.00221      |\n",
      "------------------------------------------\n",
      "10 evaluation runs mean reward = 500.0\n"
     ]
    }
   ],
   "source": [
    "model = PPO(\"MlpPolicy\", env, verbose=1)\n",
    "model.learn(total_timesteps=60_000, log_interval=15)\n",
    "\n",
    "mean_reward, std_reward = evaluate_policy(model, model.get_env(), n_eval_episodes=10)\n",
    "print(f\"10 evaluation runs mean reward = {mean_reward}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 720x720 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAHjCAYAAAA+KOkeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAApHUlEQVR4nO3deZzdVWH///chGwSBsIQlkBBkEaVKgPwQESFAZRElUChiXVi0wUdFsLZ1qxbQWrUWrT7wq0SLLAoiKIRNFCGsVSQsggSEAIEEIRAgKIhAwvn9kZv0hEySSTLDJOH5fDzmkXvP53PvPXc+YXjlM2c+U2qtAQAA5lqtrycAAAArEoEMAAANgQwAAA2BDAAADYEMAAANgQwAAI1eC+RSyn6llN+XUqaUUj7VW68DAAA9qfTGdZBLKf2S3JPk7UmmJ7kpyXtqrZN7/MUAAKAH9dYZ5J2TTKm13l9rfSHJj5KM7aXXAgCAHtNbgbxpkmnN/emdMQAAWKH176sXLqWMSzIuSdZcc82dtt12276aCgAAr0I333zzzFrr0JeP91YgP5xkeHN/s87YfLXW8UnGJ8no0aPrpEmTemkqAACwsFLKg12N99YSi5uSbF1K2aKUMjDJ4Uku6qXXAgCAHtMrZ5BrrbNLKccm+XmSfklOq7Xe2RuvBQAAPanX1iDXWi9LcllvPT8AAPQGv0kPAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABr9l+fBpZSpSf6UZE6S2bXW0aWU9ZKcm2RkkqlJDqu1PrV80wQAgFdGT5xB3rPWOqrWOrpz/1NJrqy1bp3kys59AABYKfTGEouxSc7o3D4jyUG98BoAANArljeQa5JflFJuLqWM64xtVGt9pHP70SQbLedrAADAK2a51iAn2a3W+nApZcMkV5RS7m431lprKaV29cBOUI9LkhEjRiznNAAAoGcs1xnkWuvDnT8fS3JBkp2TzCilbJIknT8fW8Rjx9daR9daRw8dOnR5pgEAAD1mmQO5lLJmKWWtebeT7JPkd0kuSnJEZ7cjkkxY3kkCAMArZXmWWGyU5IJSyrznObvWenkp5aYkPy6lfDDJg0kOW/5pAgDAK2OZA7nWen+S7bsYfyLJ3sszKQAA6Ct+kx4AADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDKrlKlTp6aUkiOPPLLbjzn99NNTSsnpp5/ea/NaEYwZMyallAXGrr766pRScuKJJ/bNpABgBSSQ6VFz5szJd7/73eyxxx5Zb731MmDAgGy44YZ505velA996EO56KKLFtj/1RKnr4QjjzwypZRMnTq1r6eSJCmlZMyYMX09jS5Nnz49X/ziF/O3f/u32WqrrbLaaqullJIpU6Ys9nHPPfdcTjjhhLzuda/L6quvng033DCHHXZY7rrrrkU+5sknn8zHPvaxjBw5MoMGDcqwYcNy9NFHZ/r06T39tgDoIf37egKsOubMmZN3vvOdufzyyzNkyJAccMAB2WyzzfLCCy/kzjvvzNlnn5277747Bx54YF9P9VXpzDPPzJ///Oe+nsYKYdKkSfnsZz+bUkq22GKLrLPOOpk1a9ZiH/P888/n7W9/e2644YaMHj06xx9/fKZNm5bzzjsvl156aa666qq8+c1vXuAxTzzxRHbdddfcc8892WuvvXL44Yfn7rvvzve///1ceuml+dWvfpXXvva1vfhOAVgWApkec8455+Tyyy/P9ttvn2uuuSbrrLPOAtv//Oc/58Ybb+yj2TFixIi+nsIKY/To0bn22muz/fbbZ+21186YMWNyzTXXLPYxX/va13LDDTfk0EMPzbnnnpvVVpv7Dbh3v/vdOeigg3L00UfnjjvumD+eJJ/5zGdyzz335OMf/3hOPvnk+ePf/OY3c/zxx+cf/uEfcvnll/fOmwRgmVliQY/53//93yRzv9X/8jhOksGDB2fPPfecf3/MmDE56qijkiRHHXVUSinzP+YtE/jDH/6Qz3/+83nrW9+ajTfeOAMHDsywYcPyd3/3d5k8efJi53P33XfnoIMOynrrrZc111wzu+22W37xi18s1XuaPn16jj322Lz2ta/NoEGDsv766+fAAw/MTTfd1O3nWNI635EjR2bkyJELjLVLTyZOnJgxY8ZkrbXWytprr50DDjhgoW/pl1JyxhlnJEm22GKL+Z/H9nm7WoO8tF544YV885vfzI477ph11103gwcPzsiRIzN27Nj88pe/XGDuSXLNNdcscFxf/jm48cYbc+ihh84/tsOHD88xxxyTP/zhDwu99rz5P//88/nsZz+bLbbYIoMGDcqWW26Zk046KS+88EK338dmm22Wt73tbVl77bW7tX+tNd/5zneSJP/5n/+5QASPHTs2b3vb2zJ58uQFIvuZZ57JWWedlTXXXHOh933sscdm8803z89//vPcf//93Z43AK8MZ5DpMeuvv36S5J577unW/kceeWSGDBmSCRMmZOzYsRk1atT8bUOGDEmSXHvttfnyl7+cPffcM4ccckhe85rX5N57783555+fiy66KDfccEO23377hZ77gQceyFve8pa88Y1vzDHHHJNHHnkk5557bvbff/+cffbZefe7373E+d1yyy3ZZ5998uSTT2bffffN3/zN32TmzJm58MILs9tuu+WCCy7IO97xjm6912V1ySWXZMKECdl///3z4Q9/OJMnT85ll12Wm266KZMnT84GG2yQJDnhhBNy4YUX5re//W2OP/74+Z+/eX/2lCOPPDLnnHNO/uqv/iof+MAHssYaa+QPf/hDrr/++lx++eX567/+64waNSonnHBCTjrppGy++eYL/MBkuyb5tNNOy7hx4zJo0KAceOCBGT58eO69995873vfy8UXX5xf//rXXZ71Puyww3LTTTfl0EMPzYABAzJhwoSceOKJmTRpUi666KLl/kdAV+6777489NBD2WabbbLFFlsstH3//ffPddddl6uuumr+PwJ//etf57nnnss+++yTtdZaa4H9V1tttey7774ZP358Jk6caJkFwIqm1trnHzvttFNl5XfLLbfUAQMG1FJKfd/73ld/8pOf1KlTpy72Md///vdrkvr973+/y+0zZsyof/zjHxcav+222+qaa65Z99tvvwXGH3jggZqkJqn//M//vMC2m266qfbv378OGTKkPv3004udw4svvli33HLLOmjQoHr11Vcv8DwPP/xwHTZsWN14443rX/7yl8W+v1prnThxYk1STzjhhC63b7755nXzzTdfYGzenPr161d/+ctfLrDtU5/6VE1Sv/KVrywwfsQRR9Qk9YEHHujydfbYY4869z/57s+tNWvWrFpKqTvttFOdPXv2Qttnzpy5wP0kdY899ujyuX7/+9/XAQMG1C233LJOnz59gW2//OUv62qrrVYPOuigLue/9dZb1yeffHL++HPPPVd32WWXmqSeeeaZS3wfXZn33Pfee2+X2y+55JKapL7zne/scvt5551Xk9TDDjts/tgpp5xSk9Rjjz22y8d89atfrUnqJz7xiWWaMwDLL8mk2kWbWmJBj9lhhx3ygx/8IBtttFF+8IMf5JBDDsnIkSOz/vrr5+CDD87FF1+81M+54YYbLnT2LUm233777LXXXpk4cWJefPHFhbavs846+bd/+7cFxkaPHp33vve9mTVrVi644ILFvu6ll16a++67Lx/96Eezxx57LLBt2LBh+cQnPpFHH300V1555VK/p6Vx+OGHZ++9915gbNy4cUmS3/zmN7362i9XSkmtNYMGDVpgicE8876D0B3f/va38+KLL+Yb3/hGNt100wW27b333jnwwANz8cUX509/+tNCj/3c5z6Xddddd/791VdfPV/60peSzD0r3RuefvrpJOly6VA73v6g37I8BoAVgyUW9KjDDjssBx98cCZOnJjrr78+t956a66//vpceOGFufDCC/OBD3xggTWq3XHppZfmO9/5TiZNmpSZM2dm9uzZC2yfOXNmNtlkkwXGdtxxxy7DesyYMTnjjDNy66235ogjjljka/7qV79Kkjz44INdrh2+9957kyR33XVXry6zGD169EJjw4cPT5I89dRTvfa6XVl77bXzrne9KxdffHFGjRqVQw45JG9729vy5je/OYMHD16q55r3+b3mmmu6XM/92GOPZc6cObnnnnuy0047LbDt5f9gSZLddtst/fr1y6233rpU8wCArghketyAAQOyzz77ZJ999kky9/JvP/nJT3L00UfnzDPPzMEHH5yDDjqoW8/1jW98Ix/72Mey7rrr5u1vf3tGjBiRwYMHp5Qyf83t888/v9DjNtpooy6fb+ONN07yf2f3FuWJJ55Ikpx33nmL3e+ZZ57pzttYZl2tIe7ff+5/tnPmzOnV1+7Kueeem6985Ss5++yzc8IJJySZewb30EMPzX/9138t8vP+cvM+v1/96lcXu19Xn9+uXqN///7ZYIMN8thjj3Xr9ZfWvLO9i/p7M2+8PV7L8hgAVgwCmV7Xr1+/HHbYYbnjjjvy7//+77nqqqu6FcizZ8/OiSeemI033ji33HLLQmeJ552F7MqMGTO6HH/00UeTLPrb3vPM2z5hwoTlvm7zvOUILz/zPc+sWbNWmkhaY401cuKJJ+bEE0/MtGnTcu211+b000/PD37wg0ydOjXXXXddt56njcfuXklinhkzZiz0w3uzZ8/OzJkzl/q5uut1r3tdkkX/AOq87yhss802y/UYAFYMS1yDXEo5rZTyWCnld83YeqWUK0op93b+XLczXkop3yylTCml3F5K2bE3J8/KZd6Sh7lr4ufq169fkq7Phs6cOTOzZs3KrrvuulAcP/PMM7nlllsW+Vq33HJLl+tXr7766iRz10svzi677JIk3Q6+xZm3XnbatGkLbZsyZcoSz2Z31+I+l71h+PDhee9735uf//zn2WqrrXL99dfPPzOczP2HwaLmsjyf366uV3z99ddnzpw5Szyuy2rLLbfMiBEjcs899+SBBx5YaPvPfvazJMlee+01f2yXXXbJGmuskRtuuGGhv4svvfTS/EsOtpc+BGDF0J0f0js9yX4vG/tUkitrrVsnubJzP0n2T7J152Nckm/3zDRZGZxzzjm54oor8tJLLy207dFHH813v/vdJMnuu+8+f3zeD3Y99NBDCz1mww03zODBg3PzzTcv8K32F198Mccff3xmzpy5yLk8/fTT+fznP7/A2KRJk/LDH/4w66yzTg4++ODFvpexY8dmyy23zLe+9a1cdtllXe7zq1/9qlu/mW7bbbfN2muvnQkTJiywBOC5557Lcccdt8THd9fiPpc94fHHH88dd9yx0Pizzz6bZ555Jv3798/AgQMXmE9X/yhI5l4HeMCAAfnHf/zHLs+wvvDCC4uM5y984QsLrL/+y1/+kk9/+tNJMv+62j2tlJIPf/jDSZJPfOITC/wdnzBhQq677rq84Q1vWGB99Gte85q8//3vz7PPPrvQOvZTTjklU6dOzb777usSbwAroCUusai1XltKGfmy4bFJxnRun5Hk6iSf7Iyf2blsxq9LKUNKKZvUWh/psRmzwrrxxhvzjW98IxtvvHF22223+deLfeCBB3LppZfmueeey9ixY3PooYfOf8xb3vKWDB48OP/93/+dJ554Yv4a4Y9+9KNZZ511ctxxx+XLX/5y3vjGN2bs2LF54YUXMnHixDz55JPZc889M3HixC7nsvvuu+d73/tebrzxxrz1rW+dfx3kl156KaeeeuoSvxU/YMCA/PSnP82+++6bAw44ILvuumtGjRqVwYMHZ9q0abnpppty//3355FHHlniD6gNGDAgxx9/fL7whS9khx12yMEHH5zZs2fniiuuyLBhwzJs2LCl+TQv0t57752vfvWr+fu///sccsghWWuttTJkyJAce+yxPfL8Dz/8cHbYYYe88Y1vzJve9KYMHz48f/zjH3PJJZfk0UcfzXHHHbfAD0buvffe+dGPfpR3vetd2XHHHTNgwIDsvvvu2X333bPtttvmtNNOy9FHH53tttsu++23X7bZZpu8+OKLeeihh3Lddddl6NChufvuuxeax+tf//pst912C1wH+b777ssBBxyQ97///d1+P+31mee9zic/+cn57+FDH/pQdtttt/n7fPzjH88ll1yS888/P29+85uz995756GHHsp5552XwYMH57TTTlvo6h7/8R//kauvvjpf+9rXctttt2XnnXfOXXfdlQkTJmTDDTfMt771rW7PF4BXUFfXfnv5R5KRSX7X3J/V3C7z7ie5JMluzbYrk4xe0vO7DvKq4aGHHqqnnHJKPeigg+o222xT11prrTpgwIC68cYb1/3337+eddZZdc6cOQs97mc/+1ndZZdd6pprrjn/GsbzruX74osv1pNPPrm+/vWvr6uvvnrdaKON6vve9746derULq/7O+86yEcccUSdPHlyPfDAA+uQIUPqGmusUXfdddd6+eWXL/T6i7sW84wZM+onP/nJut1229U11lijrrnmmnWrrbaqhxxySD3rrLPqiy++2K3PzUsvvVS/9KUv1de+9rV1wIABdfjw4fVf/uVf6rPPPrvY6yAv6vrQWcQ1hk8++eS67bbb1oEDB9YkCzzv8l4H+amnnqonnXRS3XPPPeuwYcPqwIED68Ybb1z32GOPevbZZ9eXXnppgf1nzJhR3/Oe99QNN9ywrrbaal2+zu23316POOKIOmLEiDpw4MC67rrr1u22266OGzeuXnnllQvsO2/+f/nLX+q//uu/1pEjR9aBAwfWLbbYop544onduiZ1a97ftUV9dPW5f/bZZ+vnPve5utVWW9WBAwfWDTbYoB566KH1zjvvXOTrPPHEE/W4446rI0aMmP/fw1FHHVWnTZu2VPMFoOdlEddBLrVZD7oonTPIl9Ra/6pzf1atdUiz/ala67qllEuSfLnWen1n/Mokn6y1TuriOcdl7jKMjBgxYqcHH3xwifMAXr3GjBmTa665Jt35mgUA3VFKubnWutA1VZf1F4XMKKVs0nniTZLMW1j5cJLhzX6bdcYWUmsdX2sdXWsdPXTo0GWcBgAA9KxlDeSLksz7LQtHJJnQjH+gczWLXZI8Xa0/BgBgJbLEH9IrpZyTuT+Qt0EpZXqSE5J8OcmPSykfTPJgksM6u1+W5B1JpiT5c5Le+ZFyAADoJd1ag9zbRo8eXSdNWmiZMgAA9JqeXoMMAACrJIEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDAAADYEMAACNJQZyKeW0UspjpZTfNWMnllIeLqXc1vl4R7Pt06WUKaWU35dS9u2tiQMAQG/ozhnk05Ps18X412utozoflyVJKeUNSQ5Psl3nMf+vlNKvpyYLAAC9bYmBXGu9NsmT3Xy+sUl+VGt9vtb6QJIpSXZejvkBAMArannWIB9bSrm9swRj3c7YpkmmNftM74wBAMBKYVkD+dtJtkwyKskjSU5e2icopYwrpUwqpUx6/PHHl3EaAADQs5YpkGutM2qtc2qtLyX5bv5vGcXDSYY3u27WGevqOcbXWkfXWkcPHTp0WaYBAAA9bpkCuZSySXP34CTzrnBxUZLDSymDSilbJNk6yW+Wb4oAAPDK6b+kHUop5yQZk2SDUsr0JCckGVNKGZWkJpma5JgkqbXeWUr5cZLJSWYn+UitdU6vzBwAAHpBqbX29RwyevToOmnSpL6eBgAAryKllJtrraNfPu436QEAQEMgAwBAQyADAEBDIAMAQEMgAwBAQyADAEBDIAMAQEMgAwBAQyADAEBDIAMAQEMgAwBAQyADAEBDIAMAQEMgAwBAQyADAEBDIAMAQEMgAwBAQyADAEBDIAMAQEMgA7xK3Tz+mL6eAsAKqX9fTwCAniV8AZaPM8gAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAq5idxp3a7X1dEg5gYQIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGWAVtNO4U7u9783jj+nFmQCsfAQyAAA0lhjIpZThpZSJpZTJpZQ7SynHd8bXK6VcUUq5t/Pnup3xUkr5ZillSinl9lLKjr39JgAAoKd05wzy7CT/VGt9Q5JdknyklPKGJJ9KcmWtdeskV3buJ8n+SbbufIxL8u0enzUAAPSSJQZyrfWRWustndt/SnJXkk2TjE1yRme3M5Ic1Lk9NsmZda5fJxlSStmkpycOAAC9YanWIJdSRibZIcmNSTaqtT7S2fRoko06tzdNMq152PTOGAAArPC6HcillNck+UmSj9Va/9huq7XWJHVpXriUMq6UMqmUMunxxx9fmocCAECv6VYgl1IGZG4c/7DW+tPO8Ix5Syc6fz7WGX84yfDm4Zt1xhZQax1fax1dax09dOjQZZ0/AAD0qO5cxaIk+Z8kd9Vav9ZsuijJEZ3bRySZ0Ix/oHM1i12SPN0sxQAAgBVa/27s89Yk709yRynlts7YZ5J8OcmPSykfTPJgksM62y5L8o4kU5L8OclRPTlhAADoTUsM5Frr9UnKIjbv3cX+NclHlnNeAADQJ/wmPQAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAHLz+GNy8/hj+noaACsEgQywitpp3Kl9PQWAlZJABgCAhkAGAICGQAYAgIZABgCAhkAGAICGQAYAgIZABgCAhkAGAICGQAYAgIZABgCAhkAGAICGQAYAgIZABgCAhkAGAICGQAYAgIZABgCAhkAGAICGQAZYhe007tSl2v/m8cf00kwAVh4CGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhlgFbfTuFOz07hTu73/zeOP6cXZAKz4BDIAADQEMgAANJYYyKWU4aWUiaWUyaWUO0spx3fGTyylPFxKua3z8Y7mMZ8upUwppfy+lLJvb74BAADoSf27sc/sJP9Ua72llLJWkptLKVd0tn291vpf7c6llDckOTzJdkmGJfllKWWbWuucnpw4AAD0hiWeQa61PlJrvaVz+09J7kqy6WIeMjbJj2qtz9daH0gyJcnOPTFZAADobUu1BrmUMjLJDklu7AwdW0q5vZRyWill3c7YpkmmNQ+bnsUHNQAArDC6HcillNck+UmSj9Va/5jk20m2TDIqySNJTl6aFy6ljCulTCqlTHr88ceX5qEAANBruhXIpZQBmRvHP6y1/jRJaq0zaq1zaq0vJflu/m8ZxcNJhjcP36wztoBa6/ha6+ha6+ihQ4cuz3sAAIAe052rWJQk/5Pkrlrr15rxTZrdDk7yu87ti5IcXkoZVErZIsnWSX7Tc1MGAIDe052rWLw1yfuT3FFKua0z9pkk7ymljEpSk0xNckyS1FrvLKX8OMnkzL0CxkdcwQIAgJXFEgO51np9ktLFpssW85gvJvnicswLAAD6hN+kBwAADYEMAAANgQwAAA2BDAAADYEMAAANgQwAAA2BDMBCbh5/TF9PAaDPCGSAV4mdxp3a11MAWCkIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGSAV5Gdxp3a7X1vHn9ML84EYMUlkAEAoCGQAQCgIZABAKAhkAEAoCGQAQCgIZABAKAhkAEAoCGQAQCgIZABVkCllF77WBHmAbAiE8gAANAQyAAA0Ojf1xMAoG9c/Idxi9z2rmHjX8GZAKxYnEEGeBVaXBx3ZzvAqkwgAwBAQyAD0CVnkYFXK4EMAAANgQzwKuPMMMDiCWSAV5mTThrd7X0nnSqmgVcfgQwAAA2BDECXXAsZeLUSyACvQkuKX3EMvJot8TfplVJWT3JtkkGd/c+vtZ5QStkiyY+SrJ/k5iTvr7W+UEoZlOTMJDsleSLJu2utU3tp/gAso64iePQxc8dOeqUnA7AC6c4Z5OeT7FVr3T7JqCT7lVJ2SfKVJF+vtW6V5KkkH+zs/8EkT3XGv97ZDwAAVgpLPINca61JnuncHdD5qEn2SvJ3nfEzkpyY5NtJxnZuJ8n5SU4ppZTO8wCwAph3phiAhXVrDXIppV8p5bYkjyW5Isl9SWbVWmd3dpmeZNPO7U2TTEuSzvanM3cZBgAArPCWeAY5SWqtc5KMKqUMSXJBkm2X94VLKeOSjGvuL+9TArCS8DUfWJF1K5DnqbXOKqVMTPKWJENKKf07Z4k3S/JwZ7eHkwxPMr2U0j/JOpn7w3ovf67xScYnyejRo+ukSZOW/V0ArGJW9YC06g5YESzqa+0Sl1iUUoZ2zhynlLJGkrcnuSvJxCSHdnY7IsmEzu2LOvfT2X6V9ccAAKwsunMGeZMkZ5RS+mVuUP+41npJKWVykh+VUv49ya1J/qez//8kOauUMiXJk0kO74V5AwBAr+jOVSxuT7JDF+P3J9m5i/G/JPnbHpkdAAC8wvwmPQAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGj07+sJALCwWmtfTwHgVcsZZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaAhkAABoCGQAAGgIZAAAaCwxkEspq5dSflNK+W0p5c5Sykmd8dNLKQ+UUm7rfIzqjJdSyjdLKVNKKbeXUnbs5fcAAAA9pn839nk+yV611mdKKQOSXF9K+Vln27/UWs9/2f77J9m68/HmJN/u/AkAACu8JZ5BrnM907k7oPNRF/OQsUnO7Dzu10mGlFI2Wf6pAgBA7+vWGuRSSr9Sym1JHktyRa31xs6mL3aWUXy9lDKoM7ZpkmnNw6d3xgAAYIXXrUCutc6ptY5KslmSnUspf5Xk00m2TfL/JVkvySeX5oVLKeNKKZNKKZMef/zxpZs1AAD0kqW6ikWtdVaSiUn2q7U+0llG8XyS7yfZubPbw0mGNw/brDP28ucaX2sdXWsdPXTo0GWaPAAA9LTuXMViaCllSOf2GknenuTueeuKSyklyUFJftd5yEVJPtC5msUuSZ6utT7SC3MHAIAe152rWGyS5IxSSr/MDeof11ovKaVcVUoZmqQkuS3Jhzv7X5bkHUmmJPlzkqN6fNYAANBLlhjItdbbk+zQxfhei9i/JvnI8k8NAABeeX6THgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADQEMgAANAQyAAA0BDIAADRKrbWv55BSyuNJnk0ys6/nwjLbII7fyszxW7k5fis3x2/l5vit3DavtQ59+eAKEchJUkqZVGsd3dfzYNk4fis3x2/l5vit3By/lZvjt2qyxAIAABoCGQAAGitSII/v6wmwXBy/lZvjt3Jz/FZujt/KzfFbBa0wa5ABAGBFsCKdQQYAgD7X54FcStmvlPL7UsqUUsqn+no+dK2Uclop5bFSyu+asfVKKVeUUu7t/LluZ7yUUr7ZOaa3l1J27LuZU0oZXkqZWEqZXEq5s5RyfGfc8VsJlFJWL6X8ppTy287xO6kzvkUp5cbOcTq3lDKwMz6oc39KZ/vIPn0DJElKKf1KKbeWUi7p3Hf8VhKllKmllDtKKbeVUiZ1xnz9XMX1aSCXUvol+VaS/ZO8Icl7Silv6Ms5sUinJ9nvZWOfSnJlrXXrJFd27idzj+fWnY9xSb79Cs2Rrs1O8k+11jck2SXJRzr/nTl+K4fnk+xVa90+yagk+5VSdknylSRfr7VuleSpJB/s7P/BJE91xr/e2Y++d3ySu5r7jt/KZc9a66jmcm6+fq7i+voM8s5JptRa76+1vpDkR0nG9vGc6EKt9dokT75seGySMzq3z0hyUDN+Zp3r10mGlFI2eUUmykJqrY/UWm/p3P5T5v5PetM4fiuFznF4pnN3QOejJtkryfmd8Zcfv3nH9fwke5dSyiszW7pSStksyQFJvte5X+L4rex8/VzF9XUgb5pkWnN/emeMlcNGtdZHOrcfTbJR57bjuoLqfLt2hyQ3xvFbaXS+PX9bkseSXJHkviSzaq2zO7u0x2j+8etsfzrJ+q/ohHm5/07yiSQvde6vH8dvZVKT/KKUcnMpZVxnzNfPVVz/vp4Aq4Zaay2luCTKCqyU8pokP0nysVrrH9uTUo7fiq3WOifJqFLKkCQXJNm2b2dEd5VS3pnksVrrzaWUMX08HZbNbrXWh0spGya5opRyd7vR189VU1+fQX44yfDm/madMVYOM+Z966jz52Odccd1BVNKGZC5cfzDWutPO8OO30qm1jorycQkb8ncb93OO8nRHqP5x6+zfZ0kT7yyM6Xx1iQHllKmZu4ywr2SfCOO30qj1vpw58/HMvcfqDvH189VXl8H8k1Jtu78NO/AJIcnuaiP50T3XZTkiM7tI5JMaMY/0Plp3l2SPN18K4pXWGf94v8kuavW+rVmk+O3EiilDO2cOU4pZY0kb8/cdeQTkxza2e3lx2/ecT00yVXVBe/7TK3107XWzWqtIzP3/3FX1VrfG8dvpVBKWbOUsta820n2SfK7+Pq5yuvzXxRSSnlH5q7P6pfktFrrF/t0QnSplHJOkjFJNkgyI8kJSS5M8uMkI5I8mOSwWuuTnSA7JXOvevHnJEfVWif1wbRJUkrZLcl1Se7I/62B/EzmrkN2/FZwpZQ3Ze4PAfXL3JMaP661fr6U8trMPSO5XpJbk7yv1vp8KWX1JGdl7lrzJ5McXmu9v29mT6uzxOKfa63vdPxWDp3jdEHnbv8kZ9dav1hKWT++fq7S+jyQAQBgRdLXSywAAGCFIpABAKAhkAEAoCGQAQCgIZABAKAhkAEAoCGQAQCgIZABAKDx/wNBaQtVr5kGrgAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython import display\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "obs = env.reset()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "img = ax.imshow(env.render(mode=\"rgb_array\"), interpolation=\"none\")\n",
    "fig.tight_layout()\n",
    "\n",
    "done = False\n",
    "for idx in range(100):\n",
    "    img.set_data(env.render(mode=\"rgb_array\"))\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "\n",
    "    action, _states = model.predict(obs)\n",
    "    obs, reward, done, info = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        plt.text(100, 100, f\"Failed at step {idx}\", fontsize=20)\n",
    "        break\n",
    "\n",
    "if not done:\n",
    "    plt.text(100, 100, f\"Stable until step {idx + 1}\", fontsize=20)\n",
    "\n",
    "env.close()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The new agent is much better at balancing the pole. If we increased the training steps to above 100,000 it is hard to see that the pole is even moving!\n",
    "\n",
    "# Conclusions\n",
    "* We learned about the CartPole environment.\n",
    "* We trained a simple RL environment using PPO algorithm.\n",
    "\n",
    "# Next Steps\n",
    "If you want to improve your RL skills from here, we recommend:\n",
    "1. Use different OpenAI Gym environments to practice.\n",
    "2. Try different RL algorithms (links above).\n",
    "3. Try hyperparameter tuning with Ray RLlib (links above).\n",
    "4. Implement your own environment for wildcatter!\n",
    "5. Check out Stable Baselines3 or RLlib TensorBoard integration [here](https://stable-baselines3.readthedocs.io/en/master/guide/tensorboard.html) and [here](https://docs.ray.io/en/latest/rllib/rllib-training.html)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}