Lectures/02-DataScraping.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "## all imports\n",
    "from IPython.display import HTML\n",
    "import numpy as np\n",
    "import urllib2\n",
    "import bs4 #this is beautiful soup\n",
    "import time\n",
    "import operator\n",
    "import socket\n",
    "import cPickle\n",
    "import re # regular expressions\n",
    "\n",
    "from pandas import Series\n",
    "import pandas as pd\n",
    "from pandas import DataFrame\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set_context(\"talk\")\n",
    "sns.set_style(\"white\")\n",
    "\n",
    "from secret import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "API registrations\n",
    "=================\n",
    "\n",
    "If you would like to run all the examples in this notebook, you need to register for the following APIs:\n",
    "\n",
    "* Rotten Tomatoes\n",
    "\n",
    "http://developer.rottentomatoes.com/member/register\n",
    "\n",
    "* Twitter\n",
    "\n",
    "https://apps.twitter.com/app/new\n",
    "\n",
    "* Twitter instructions\n",
    "\n",
    "https://twittercommunity.com/t/how-to-get-my-api-key/7033"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "CS109\n",
    "=====\n",
    "\n",
    "Verena Kaynig-Fittkau, Joe Blitzstein, Hanspeter Pfister\n",
    "\n",
    "* vkaynig@seas.harvard.edu\n",
    "* staff@cs109.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Announcements\n",
    "==============\n",
    "\n",
    "* Over 400 sign ups on github!\n",
    "* If you are still missing, fill out the survey, time is running out!\n",
    "* Make sure you are on Piazza!\n",
    "\n",
    "\n",
    "* More [git help](https://www.youtube.com/channel/UC0-KaiZFXBlGOFN71YsEV8g/videos)\n",
    "\n",
    "\n",
    "* HW0 is due on Thursday \n",
    "* HW1 is coming out on Thursday\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Todays lecture:\n",
    "===============\n",
    "\n",
    "* introduction to pandas\n",
    "    - read a table\n",
    "    - do some plots\n",
    "\n",
    "* all about data scraping\n",
    "* ***What is it? ***\n",
    "* How to do it:\n",
    "    - from a website\n",
    "    - with an API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Answer: Data scraping is about obtaining data from webpages. There is low level scraping where you parse the data out of the html code of the webpage. There also is scraping over APIs from websites who try to make your life a bit easier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "IPython Notebooks:\n",
    "===================\n",
    "\n",
    "![IPython](images/ipython.png \"IPython\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "IPython Notebooks:\n",
    "===================\n",
    "\n",
    "* These slides are an IPython notebook!\n",
    "* https://github.com/damianavila/live_reveal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello CS109\n",
      "I love IPython\n"
     ]
    }
   ],
   "source": [
    "print \"Hello CS109\"\n",
    "\n",
    "print \"I love IPython\"\n",
    "\n",
    "# Ipython notebook have tab completion!\n",
    "# and inbuild help\n",
    "  \n",
    "a = np.zeros(3)\n",
    "a.clip?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "General advice about programming\n",
    "==================================\n",
    "\n",
    "* You will find nearly everything on google\n",
    "* Try: length of a list in python\n",
    "* A programmer is someone who can turn stack overflow snippets into running code\n",
    "* Use tab completion\n",
    "* Make your variable names meaningful\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How to load a table\n",
    "===================\n",
    "\n",
    "* we use Pandas for this\n",
    "* Pandas can do a __lot__ more\n",
    "* more about it later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The MovieLens data\n",
    "===================\n",
    "\n",
    "http://grouplens.org/datasets/movielens/\n",
    "\n",
    "![Grouplens](images/grouplens.jpg \"Grouplens\")\n",
    "\n",
    "Example inspired by [Greg Reda](http://www.gregreda.com/2013/10/26/using-pandas-on-the-movielens-dataset/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Read the user data\n",
    "=================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>M</td>\n",
       "      <td>technician</td>\n",
       "      <td>85711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>F</td>\n",
       "      <td>other</td>\n",
       "      <td>94043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>M</td>\n",
       "      <td>writer</td>\n",
       "      <td>32067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>M</td>\n",
       "      <td>technician</td>\n",
       "      <td>43537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>33</td>\n",
       "      <td>F</td>\n",
       "      <td>other</td>\n",
       "      <td>15213</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  age sex  occupation zip_code\n",
       "0        1   24   M  technician    85711\n",
       "1        2   53   F       other    94043\n",
       "2        3   23   M      writer    32067\n",
       "3        4   24   M  technician    43537\n",
       "4        5   33   F       other    15213"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pass in column names for each CSV\n",
    "u_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']\n",
    "\n",
    "users = pd.read_csv(\n",
    "    'http://files.grouplens.org/datasets/movielens/ml-100k/u.user', \n",
    "    sep='|', names=u_cols)\n",
    "\n",
    "users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Read the ratings\n",
    "============"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>unix_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  unix_timestamp\n",
       "0      196       242       3       881250949\n",
       "1      186       302       3       891717742\n",
       "2       22       377       1       878887116\n",
       "3      244        51       2       880606923\n",
       "4      166       346       1       886397596"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']\n",
    "ratings = pd.read_csv(\n",
    "    'http://files.grouplens.org/datasets/movielens/ml-100k/u.data', \n",
    "    sep='\\t', names=r_cols)\n",
    "\n",
    "ratings.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now data about the movies\n",
    "========================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>release_date</th>\n",
       "      <th>video_release_date</th>\n",
       "      <th>imdb_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Toy%20Story%2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>GoldenEye (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?GoldenEye%20(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Four Rooms (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Four%20Rooms%...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Get Shorty (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Get%20Shorty%...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Copycat (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Copycat%20(1995)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id              title release_date  video_release_date  \\\n",
       "0         1   Toy Story (1995)  01-Jan-1995                 NaN   \n",
       "1         2   GoldenEye (1995)  01-Jan-1995                 NaN   \n",
       "2         3  Four Rooms (1995)  01-Jan-1995                 NaN   \n",
       "3         4  Get Shorty (1995)  01-Jan-1995                 NaN   \n",
       "4         5     Copycat (1995)  01-Jan-1995                 NaN   \n",
       "\n",
       "                                            imdb_url  \n",
       "0  http://us.imdb.com/M/title-exact?Toy%20Story%2...  \n",
       "1  http://us.imdb.com/M/title-exact?GoldenEye%20(...  \n",
       "2  http://us.imdb.com/M/title-exact?Four%20Rooms%...  \n",
       "3  http://us.imdb.com/M/title-exact?Get%20Shorty%...  \n",
       "4  http://us.imdb.com/M/title-exact?Copycat%20(1995)  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the movies file contains columns indicating the movie's genres\n",
    "# let's only load the first five columns of the file with usecols\n",
    "m_cols = ['movie_id', 'title', 'release_date', \n",
    "            'video_release_date', 'imdb_url']\n",
    "\n",
    "movies = pd.read_csv(\n",
    "    'http://files.grouplens.org/datasets/movielens/ml-100k/u.item', \n",
    "    sep='|', names=m_cols, usecols=range(5))\n",
    "\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Get information about data\n",
    "======================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movie_id                int64\n",
      "title                  object\n",
      "release_date           object\n",
      "video_release_date    float64\n",
      "imdb_url               object\n",
      "dtype: object\n",
      "\n",
      "          movie_id  video_release_date\n",
      "count  1682.000000                   0\n",
      "mean    841.500000                 NaN\n",
      "std     485.695893                 NaN\n",
      "min       1.000000                 NaN\n",
      "25%     421.250000                 NaN\n",
      "50%     841.500000                 NaN\n",
      "75%    1261.750000                 NaN\n",
      "max    1682.000000                 NaN\n"
     ]
    }
   ],
   "source": [
    "print movies.dtypes\n",
    "print\n",
    "print movies.describe()\n",
    "# *** Why only those two columns? ***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Selecting data\n",
    "==============\n",
    "\n",
    "* DataFrame => group of Series with shared index\n",
    "* single DataFrame column => Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   occupation sex\n",
      "0  technician   M\n",
      "1       other   F\n",
      "2      writer   M\n",
      "3  technician   M\n",
      "4       other   F\n"
     ]
    }
   ],
   "source": [
    "users.head()\n",
    "users['occupation'].head()\n",
    "## *** Where did the nice design go? ***\n",
    "columns_you_want = ['occupation', 'sex'] \n",
    "print users[columns_you_want].head()\n",
    "\n",
    "#print users.head()\n",
    "\n",
    "#print users.iloc[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Filtering data\n",
    "==============\n",
    "\n",
    "Select users older than 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>F</td>\n",
       "      <td>other</td>\n",
       "      <td>94043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>33</td>\n",
       "      <td>F</td>\n",
       "      <td>other</td>\n",
       "      <td>15213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>42</td>\n",
       "      <td>M</td>\n",
       "      <td>executive</td>\n",
       "      <td>98101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>57</td>\n",
       "      <td>M</td>\n",
       "      <td>administrator</td>\n",
       "      <td>91344</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>36</td>\n",
       "      <td>M</td>\n",
       "      <td>administrator</td>\n",
       "      <td>05201</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  age sex     occupation zip_code\n",
       "1        2   53   F          other    94043\n",
       "4        5   33   F          other    15213\n",
       "5        6   42   M      executive    98101\n",
       "6        7   57   M  administrator    91344\n",
       "7        8   36   M  administrator    05201"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oldUsers = users[users.age > 25]\n",
    "oldUsers.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Quiz:\n",
    "=====\n",
    "\n",
    "* show users aged 40 and male\n",
    "\n",
    "* show the mean age of female programmers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>40</td>\n",
       "      <td>M</td>\n",
       "      <td>librarian</td>\n",
       "      <td>02138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>83</td>\n",
       "      <td>40</td>\n",
       "      <td>M</td>\n",
       "      <td>other</td>\n",
       "      <td>44133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>116</td>\n",
       "      <td>40</td>\n",
       "      <td>M</td>\n",
       "      <td>healthcare</td>\n",
       "      <td>97232</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     user_id  age sex  occupation zip_code\n",
       "18        19   40   M   librarian    02138\n",
       "82        83   40   M       other    44133\n",
       "115      116   40   M  healthcare    97232"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# users aged 40 AND male\n",
    "users[(users.age == 40) & (users.sex == 'M')].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          user_id        age\n",
      "count    6.000000   6.000000\n",
      "mean   411.166667  32.166667\n",
      "std    149.987222   5.115336\n",
      "min    292.000000  26.000000\n",
      "25%    313.000000  28.250000\n",
      "50%    378.000000  32.000000\n",
      "75%    416.750000  36.500000\n",
      "max    698.000000  38.000000\n",
      "32.1666666667\n",
      "32.1666666667\n"
     ]
    }
   ],
   "source": [
    "## users who are female and programmers\n",
    "selected_users = users[(users.sex == 'F') & \n",
    "                       (users.occupation == 'programmer')]\n",
    "\n",
    "## show statistic summary\n",
    "print selected_users.describe()\n",
    "\n",
    "## alternatives:\n",
    "print selected_users.age.mean()\n",
    "print selected_users['age'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Split-apply-combine\n",
    "===================\n",
    "\n",
    "* splitting the data into groups based on some criteria\n",
    "* applying a function to each group independently\n",
    "* combining the results into a data structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Split-apply-combine\n",
    "===================\n",
    "\n",
    "<img src=http://i.imgur.com/yjNkiwL.png></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Find Diligent Users\n",
    "===================\n",
    "\n",
    "* split data per user ID\n",
    "* count ratings\n",
    "* combine result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   user_id  movie_id  rating  unix_timestamp\n",
      "0      196       242       3       881250949\n",
      "1      186       302       3       891717742\n",
      "2       22       377       1       878887116\n",
      "3      244        51       2       880606923\n",
      "4      166       346       1       886397596\n"
     ]
    }
   ],
   "source": [
    "print ratings.head()\n",
    "## split data\n",
    "#grouped_data = ratings.groupby('user_id')\n",
    "grouped_data = ratings['movie_id'].groupby(ratings['user_id'])\n",
    "\n",
    "## count and combine\n",
    "#ratings_per_user = grouped_data.count()\n",
    "\n",
    "#ratings_per_user.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Quiz\n",
    "====\n",
    "\n",
    "* get the average rating per movie\n",
    "* advanced: get the movie titles with the highest average rating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average ratings:\n",
      "movie_id\n",
      "1    3.878319\n",
      "2    3.206107\n",
      "3    3.033333\n",
      "4    3.550239\n",
      "5    3.302326\n",
      "Name: rating, dtype: float64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## split data\n",
    "grouped_data = ratings['rating'].groupby(ratings['movie_id'])\n",
    "## average and combine\n",
    "average_ratings = grouped_data.mean()\n",
    "print \"Average ratings:\"\n",
    "print average_ratings.head()\n",
    "print\n",
    "\n",
    "maximum_rating = average_ratings.max()\n",
    "good_movie_ids = average_ratings[average_ratings == maximum_rating].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Good movie ids:\n",
      "Int64Index([814, 1122, 1189, 1201, 1293, 1467, 1500, 1536, 1599, 1653], dtype='int64', name=u'movie_id')\n",
      "\n",
      "Best movie titles\n",
      "813                         Great Day in Harlem, A (1994)\n",
      "1121                       They Made Me a Criminal (1939)\n",
      "1188                                   Prefontaine (1997)\n",
      "1200           Marlene Dietrich: Shadow and Light (1996) \n",
      "1292                                      Star Kid (1997)\n",
      "1466                 Saint of Fort Washington, The (1993)\n",
      "1499                            Santa with Muscles (1996)\n",
      "1535                                 Aiqing wansui (1994)\n",
      "1598                        Someone Else's America (1995)\n",
      "1652    Entertaining Angels: The Dorothy Day Story (1996)\n",
      "Name: title, dtype: object\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print \"Good movie ids:\"\n",
    "print good_movie_ids\n",
    "print\n",
    "\n",
    "print \"Best movie titles\"\n",
    "print movies[movies.movie_id.isin(good_movie_ids)].title\n",
    "print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of ratings per movie\n",
      "movie_id\n",
      "814     1\n",
      "1122    1\n",
      "1189    3\n",
      "1201    1\n",
      "1293    3\n",
      "1467    2\n",
      "1500    2\n",
      "1536    1\n",
      "1599    1\n",
      "1653    1\n",
      "Name: rating, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "how_many_ratings = grouped_data.count()\n",
    "print \"Number of ratings per movie\"\n",
    "print how_many_ratings[average_ratings == maximum_rating]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Passing a Function\n",
    "==================\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movie_id\n",
       "1    3.878319\n",
       "2    3.206107\n",
       "3    3.033333\n",
       "4    3.550239\n",
       "5    3.302326\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_ratings = grouped_data.apply(lambda f: f.mean())\n",
    "average_ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Quiz\n",
    "====\n",
    "\n",
    "* get the average rating per user\n",
    "* advanced: list all occupations and if they are male or female dominant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id\n",
       "1    3.610294\n",
       "2    3.709677\n",
       "3    2.796296\n",
       "4    4.333333\n",
       "5    2.874286\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped_data = ratings['rating'].groupby(ratings['user_id'])\n",
    "average_ratings = grouped_data.mean()\n",
    "average_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "occupation\n",
      "administrator     True\n",
      "artist            True\n",
      "doctor            True\n",
      "educator          True\n",
      "engineer          True\n",
      "entertainment     True\n",
      "executive         True\n",
      "healthcare       False\n",
      "homemaker        False\n",
      "lawyer            True\n",
      "librarian        False\n",
      "marketing         True\n",
      "none              True\n",
      "other             True\n",
      "programmer        True\n",
      "retired           True\n",
      "salesman          True\n",
      "scientist         True\n",
      "student           True\n",
      "technician        True\n",
      "writer            True\n",
      "dtype: bool\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "grouped_data = users['sex'].groupby(users['occupation'])\n",
    "male_dominant_occupations = grouped_data.apply(lambda f: \n",
    "                                               sum(f == 'M') > sum(f == 'F'))\n",
    "print male_dominant_occupations\n",
    "print '\\n'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of male users: \n",
      "670\n",
      "number of female users: \n",
      "273\n"
     ]
    }
   ],
   "source": [
    "print 'number of male users: '\n",
    "print sum(users['sex'] == 'M')\n",
    "\n",
    "print 'number of female users: '\n",
    "print sum(users['sex'] == 'F')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pandas \"wrapup\"\n",
    "==========\n",
    "\n",
    "- create data frames\n",
    "- get sub-frames\n",
    "- filter data \n",
    "- use group-by\n",
    "- apply a user defined function\n",
    "\n",
    "\n",
    "![cute panda](images/cute_panda.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Python data scraping\n",
    "====================\n",
    "\n",
    "* Why scrape the web?\n",
    "    - vast source of information\n",
    "    - automate tasks\n",
    "    - keep up with sites\n",
    "    - fun!\n",
    "\n",
    "** Can you think of examples ? **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "  \n",
    "Answer: Some examples we had were stock market monitoring, sports data, or airline prices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Read and Tweet!\n",
    "=================\n",
    "\n",
    "![ReadTweet](http://developer.nytimes.com/files/readtweet.jpg)\n",
    "\n",
    "* by Justin Blinder\n",
    "* http://projects.justinblinder.com/We-Read-We-Tweet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "“We Read, We Tweet” geographically visualizes the dissemination of New York Times articles through Twitter. Each line connects the location of a tweet to the contextual location of the New York Times article it referenced. The lines are generated in a sequence based on the time in which a tweet occurs. The project explores digital news distribution in a temporal and spatial context through the social space of Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Twitter Sentiments\n",
    "=================\n",
    "\n",
    "![TwitterSentiments](http://www.csc.ncsu.edu/faculty/healey/tweet_viz/figs/tweet-viz-ex.png\n",
    " \"Twitter Sentiments\")\n",
    "\n",
    "* by Healey and Ramaswamy\n",
    "* http://www.csc.ncsu.edu/faculty/healey/tweet_viz/tweet_app/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Type a keyword into the input field, then click the Query button. Recent tweets that contain your keyword are pulled from Twitter and visualized in the Sentiment tab as circles. Hover your mouse over a tweet or click on it to see its text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Python data scraping\n",
    "====================\n",
    "\n",
    "* copyrights and permission:\n",
    "    - be careful and polite\n",
    "    - give credit\n",
    "    - care about media law\n",
    "    - don't be evil (no spam, overloading sites, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Robots.txt\n",
    "==========\n",
    "\n",
    "![Robots.txt](images/robots_txt.jpg \"Robots.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Robots.txt\n",
    "==========\n",
    "\n",
    "* specified by web site owner\n",
    "* gives instructions to web robots (aka your script)\n",
    "* is located at the top-level directory of the web server\n",
    "\n",
    "http://www.example.com/robots.txt\n",
    "\n",
    "If you want you can also have a look at\n",
    "\n",
    "http://google.com/robots.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Robots.txt\n",
    "==========\n",
    "\n",
    "*** What does this one do? ***"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\n",
    "User-agent: Google\n",
    "Disallow:\n",
    "\n",
    "User-agent: *\n",
    "Disallow: /"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Answer: This file allows google to search through everything on the server, while all others should stay completely away."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Things to consider:\n",
    "-------------------\n",
    "\n",
    "* can be just ignored\n",
    "* can be a security risk - *** Why? ***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Answer: You are basically telling everybody who cares to look into the file where you have stored sensitive information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scraping with Python:\n",
    "=====================\n",
    "\n",
    "* scraping is all about HTML tags\n",
    "* bad news: \n",
    "    - need to learn about tags\n",
    "    - websites can be ugly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "HTML\n",
    "=====\n",
    "\n",
    "* HyperText Markup Language\n",
    "\n",
    "* standard for creating webpages\n",
    "\n",
    "* HTML tags \n",
    "    - have angle brackets\n",
    "    - typically come in pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "This is an example for a minimal webpage defined in HTML tags. The root tag is `<html>` and then you have the `<head>` tag. This part of the page typically includes the title of the page and might also have other meta information like the author or keywords that are important for search engines. The `<body>` tag marks the actual content of the page. You can play around with the `<h2>` tag trying different header levels. They range from 1 to 6. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html>\n",
       "<html>\n",
       "  <head>\n",
       "    <title>This is a title</title>\n",
       "  </head>\n",
       "  <body>\n",
       "    <h2> Test </h2>\n",
       "    <p>Hello world!</p>\n",
       "  </body>\n",
       "</html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "htmlString = \"\"\"<!DOCTYPE html>\n",
    "<html>\n",
    "  <head>\n",
    "    <title>This is a title</title>\n",
    "  </head>\n",
    "  <body>\n",
    "    <h2> Test </h2>\n",
    "    <p>Hello world!</p>\n",
    "  </body>\n",
    "</html>\"\"\"\n",
    "\n",
    "htmlOutput = HTML(htmlString)\n",
    "htmlOutput"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Useful Tags\n",
    "===========\n",
    "\n",
    "* heading\n",
    "`<h1></h1> ... <h6></h6>`\n",
    "\n",
    "* paragraph\n",
    "`<p></p>` \n",
    "\n",
    "* line break\n",
    "`<br>` \n",
    "\n",
    "* link with attribute\n",
    "\n",
    "`<a href=\"http://www.example.com/\">An example link</a>`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scraping with Python:\n",
    "=====================\n",
    "\n",
    "* example of a beautifully simple webpage:\n",
    "\n",
    "http://www.crummy.com/software/BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scraping with Python:\n",
    "=====================\n",
    "\n",
    "* good news: \n",
    "    - some browsers help\n",
    "    - look for: inspect element\n",
    "    - need only basic html\n",
    "    \n",
    "** Try 'Ctrl-Shift I' in Chrome **\n",
    "\n",
    "** Try 'Command-Option I' in Safari **\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scraping with Python\n",
    "==================\n",
    "\n",
    "* different useful libraries:\n",
    "    - urllib\n",
    "    - beautifulsoup\n",
    "    - pattern\n",
    "    - soupy\n",
    "    - LXML\n",
    "    - ...\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The following cell just defines a url as a string and then reads the data from that url using the `urllib` library. If you uncomment the print command you see that we got the whole HTML content of the page into the string variable source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"\n",
      "\"http://www.w3.org/TR/REC-html40/transitional.dtd\">\n",
      "<html>\n",
      "<head>\n",
      "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n",
      "<title>Beautiful Soup: We called him Tortoise because he taught us.</title>\n",
      "<link rev=\"made\" href=\"mailto:leonardr@segfault.org\">\n",
      "<link rel=\"stylesheet\" type=\"text/css\" href=\"/nb/themes/Default/nb.css\">\n",
      "<meta name=\"Description\" content=\"Beautiful Soup: a library designed for screen-scraping HTML and XML.\">\n",
      "<meta name=\"generator\" content=\"Markov Approximation 1.4 (module: leonardr)\">\n",
      "<meta name=\"author\" content=\"Leonard Richardson\">\n",
      "</head>\n",
      "<body bgcolor=\"white\" text=\"black\" link=\"blue\" vlink=\"660066\" alink=\"red\">\n",
      "<img align=\"right\" src=\"10.1.jpg\" width=\"250\"><br />\n",
      "\n",
      "<p>You didn't write that awful page. You're just trying to get some\n",
      "data out of it. Beautiful Soup is here to help. Since 2004, it's been\n",
      "saving programmers hours or days of work on quick-turnaround\n",
      "screen scraping projects.</p>\n",
      "\n",
      "<div align=\"center\">\n",
      "\n",
      "<a href=\"bs4/download/\"><h1>Beautiful Soup</h1></a>\n",
      "\n",
      "<p>\"A tremendous boon.\" -- Python411 Podcast</p>\n",
      "\n",
      "<p>[ <a href=\"#Download\">Download</a> | <a\n",
      "href=\"bs4/doc/\">Documentation</a> | <a href=\"#HallOfFame\">Hall of Fame</a> | <a href=\"https://code.launchpad.net/beautifulsoup\">Source</a> | <a href=\"https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup\">Discussion group</a> ]</p>\n",
      "\n",
      "<small>If Beautiful Soup has saved you a lot of time and money, the\n",
      "best way to pay me back is to check out <a\n",
      "href=\"http://www.candlemarkandgleam.com/shop/constellation-games/\"><i>Constellation\n",
      "Games</i>, my sci-fi novel about alien video games</a>.<br />You can\n",
      "<a\n",
      "href=\"http://constellation.crummy.com/Constellation%20Games%20excerpt.html\">read\n",
      "the first two chapters for free</a>, and the full novel starts at 5\n",
      "USD. Thanks!</small> </div>\n",
      "\n",
      "<p><i>If you have questions, send them to <a\n",
      "href=\"https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup\">the discussion\n",
      "group</a>. If you find a bug, <a href=\"https://bugs.launchpad.net/beautifulsoup/\">file it</a>.</i></p>\n",
      "\n",
      "<p>Beautiful Soup is a Python library designed for quick turnaround\n",
      "projects like screen-scraping. Three features make it powerful:\n",
      "\n",
      "<ol>\n",
      "\n",
      "<li>Beautiful Soup provides a few simple methods and Pythonic idioms\n",
      "for navigating, searching, and modifying a parse tree: a toolkit for\n",
      "dissecting a document and extracting what you need. It doesn't take\n",
      "much code to write an application\n",
      "\n",
      "<li>Beautiful Soup automatically converts incoming documents to\n",
      "Unicode and outgoing documents to UTF-8. You don't have to think\n",
      "about encodings, unless the document doesn't specify an encoding and\n",
      "Beautiful Soup can't detect one. Then you just have to specify the\n",
      "original encoding.\n",
      "\n",
      "<li>Beautiful Soup sits on top of popular Python parsers like <a\n",
      "href=\"http://lxml.de/\">lxml</a> and <a\n",
      "href=\"http://code.google.com/p/html5lib/\">html5lib</a>, allowing you\n",
      "to try out different parsing strategies or trade speed for\n",
      "flexibility.\n",
      "\n",
      "</ol>\n",
      "\n",
      "<p>Beautiful Soup parses anything you give it, and does the tree\n",
      "traversal stuff for you. You can tell it \"Find all the links\", or\n",
      "\"Find all the links of class <tt>externalLink</tt>\", or \"Find all the\n",
      "links whose urls match \"foo.com\", or \"Find the table heading that's\n",
      "got bold text, then give me that text.\"\n",
      "\n",
      "<p>Valuable data that was once locked up in poorly-designed websites\n",
      "is now within your reach. Projects that would have taken hours take\n",
      "only minutes with Beautiful Soup.\n",
      "\n",
      "<p>Interested? <a href=\"bs4/doc/\">Read more.</a>\n",
      "\n",
      "<a name=\"Download\"><h2>Download Beautiful Soup</h2></a>\n",
      "\n",
      "<p>The current release is <a href=\"bs4/download/\">Beautiful Soup\n",
      "4.4.0</a> (July 3, 2015). You can install Beautiful Soup 4 with\n",
      "<code>pip install beautifulsoup4</code>. It's also available as the\n",
      "<code>python-beautifulsoup4</code> package in recent versions of\n",
      "Debian, Ubuntu, and Fedora .\n",
      "\n",
      "<p>Beautiful Soup 4 works on both Python 2 (2.6+) and Python 3.\n",
      "\n",
      "<p>Beautiful Soup is licensed under the MIT license, so you can also\n",
      "download the tarball, drop the <code>bs4/</code> directory into almost\n",
      "any Python application (or into your library path) and start using it\n",
      "immediately. (If you want to do this under Python 3, you will need to\n",
      "manually convert the code using <code>2to3</code>.)\n",
      "\n",
      "<h3>Beautiful Soup 3</h3>\n",
      "\n",
      "<p>Beautiful Soup 3 was the official release line of Beautiful Soup\n",
      "from May 2006 to March 2012. It is considered stable, and only\n",
      "critical bugs will be fixed. <a\n",
      "href=\"http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html\">Here's\n",
      "the Beautiful Soup 3 documentation.</a>\n",
      "\n",
      "<p>Beautiful Soup 3 works only under Python 2.x. It is licensed under\n",
      "the same license as Python itself.\n",
      "\n",
      "<p>The current release of Beautiful Soup 3 is <a\n",
      "href=\"download/3.x/BeautifulSoup-3.2.1.tar.gz\">3.2.1</a> (February 16,\n",
      "2012). You can install Beautiful Soup 3 with <code>pip install\n",
      "BeautifulSoup</code>. It's also available as\n",
      "<code>python-beautifulsoup</code> in Debian and Ubuntu, and as\n",
      "<code>python-BeautifulSoup</code> in Fedora.\n",
      "\n",
      "<p>You can also download the tarball and use\n",
      "<code>BeautifulSoup.py</code> in your project directly.\n",
      "\n",
      "\n",
      "<a name=\"HallOfFame\"><h2>Hall of Fame</h2></a>\n",
      "\n",
      "<p>Over the years, Beautiful Soup has been used in hundreds of\n",
      "different projects. There's no way I can list them all, but I want to\n",
      "highlight a few high-profile projects. Beautiful Soup isn't what makes\n",
      "these projects interesting, but it did make their completion easier:\n",
      "\n",
      "<ul>\n",
      "\n",
      "<li><a\n",
      " href=\"http://www.nytimes.com/2007/10/25/arts/design/25vide.html\">\"Movable\n",
      " Type\"</a>, a work of digital art on display in the lobby of the New\n",
      " York Times building, uses Beautiful Soup to scrape news feeds.\n",
      "\n",
      "<li>Reddit uses Beautiful Soup to <a\n",
      "href=\"https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py\">parse\n",
      "a page that's been linked to and find a representative image</a>.\n",
      "\n",
      "<li>Alexander Harrowell uses Beautiful Soup to <a\n",
      " href=\"http://www.harrowell.org.uk/viktormap.html\">track the business\n",
      " activities</a> of an arms merchant.\n",
      "\n",
      "<li>The developers of Python itself used Beautiful Soup to <a\n",
      "href=\"http://svn.python.org/view/tracker/importer/\">migrate the Python\n",
      "bug tracker from Sourceforge to Roundup</a>.\n",
      "\n",
      "<li>The <a href=\"http://www2.ljworld.com/\">Lawrence Journal-World</a>\n",
      "uses Beautiful Soup to <A\n",
      "href=\"http://www.b-list.org/weblog/2010/nov/02/news-done-broke/\">gather\n",
      "statewide election results</a>.\n",
      "\n",
      "<li>The <a href=\"http://esrl.noaa.gov/gsd/fab/\">NOAA's Forecast\n",
      "Applications Branch</a> uses Beautiful Soup in <a\n",
      "href=\"http://laps.noaa.gov/topograbber/\">TopoGrabber</a>, a script for\n",
      "downloading \"high resolution USGS datasets.\"\n",
      "\n",
      "</ul>\n",
      "\n",
      "<p>If you've used Beautiful Soup in a project you'd like me to know\n",
      "about, please do send email to me or <a\n",
      "href=\"http://groups.google.com/group/beautifulsoup/\">the discussion\n",
      "group</a>.\n",
      "\n",
      "<h2>Development</h2>\n",
      "\n",
      "<p>Development happens at <a\n",
      "href=\"https://launchpad.net/beautifulsoup\">Launchpad</a>. You can <a\n",
      "href=\"https://code.launchpad.net/beautifulsoup/\">get the source\n",
      "code</a> or <a href=\"https://bugs.launchpad.net/beautifulsoup/\">file\n",
      "bugs</a>.<hr><table><tr><td valign=\"top\">\n",
      "<p>This document (<a href=\"/source/software/BeautifulSoup/index.bhtml\">source</a>) is part of Crummy, the webspace of <a href=\"/self/\">Leonard Richardson</a> (<a href=\"/self/contact.html\">contact information</a>). It was last modified on Friday, July 03 2015, 15:38:09 Nowhere Standard Time and last built on Tuesday, September 08 2015, 19:00:01 Nowhere Standard Time.</p><p><table class=\"licenseText\"><tr><td><a href=\"http://creativecommons.org/licenses/by-sa/2.0/\"><img border=\"0\" src=\"/nb//resources/img/somerights20.jpg\"></a></td><td valign=\"top\">Crummy is &copy; 1996-2015 Leonard Richardson. Unless otherwise noted, all text licensed under a <a href=\"http://creativecommons.org/licenses/by-sa/2.0/\">Creative Commons License</a>.</td></tr></table></span><!--<rdf:RDF xmlns=\"http://web.resource.org/cc/\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"><Work rdf:about=\"http://www.crummy.com/\"><dc:title>Crummy: The Site</dc:title><dc:rights><Agent><dc:title>Crummy: the Site</dc:title></Agent></dc:rights><dc:format>text/html</dc:format><license rdf:resource=http://creativecommons.org/licenses/by-sa/2.0//></Work><License rdf:about=\"http://creativecommons.org/licenses/by-sa/2.0/\"></License></rdf:RDF>--></p></td><td valign=top><p><b>Document tree:</b>\n",
      "<dl><dd><a href=\"http://www.crummy.com/\">http://www.crummy.com/</a><dl><dd><a href=\"http://www.crummy.com/software/\">software/</a><dl><dd><a href=\"http://www.crummy.com/software/BeautifulSoup/\">BeautifulSoup/</a></dl>\n",
      "</dl>\n",
      "</dl>\n",
      "\n",
      "\n",
      "Site Search:\n",
      "\n",
      "<form method=\"get\" action=\"/search/\">\n",
      "        <input type=\"text\" name=\"q\" maxlength=\"255\" value=\"\"></input>\n",
      "        </form>\n",
      "        </td>\n",
      "\n",
      "</tr>\n",
      "\n",
      "</table>\n",
      "</body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url = 'http://www.crummy.com/software/BeautifulSoup'\n",
    "source = urllib2.urlopen(url).read()\n",
    "print source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Quiz :\n",
    "======\n",
    "\n",
    "* Is the word 'Alice' mentioned on the beautiful soup homepage?\n",
    "* How often does the word 'Soup' occur on the site?\n",
    "    - hint: use `.count()`\n",
    "* At what index occurs the substring 'alien video games' ?\n",
    "    - hint: use `.find()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "41\n",
      "1639\n",
      "alien video games</a\n",
      "alien video games\n"
     ]
    }
   ],
   "source": [
    "## is 'Alice' in source?\n",
    "print 'Alice' in source\n",
    "\n",
    "## count occurences of 'Soup'\n",
    "print source.count('Soup')\n",
    "\n",
    "## find index of 'alien video games'\n",
    "position =  source.find('alien video games')\n",
    "print position\n",
    "\n",
    "## quick test to see the substring in the source variable\n",
    "## you can access strings like lists\n",
    "print source[position:position + 20]\n",
    "\n",
    "## or the tidier version:\n",
    "print source[position:position + len('alien video games')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Beautiful Soup\n",
    "==============\n",
    "\n",
    "* designed to make your life easier\n",
    "* many good functions for parsing html code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Some examples\n",
    "=============\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a href=\"/index.asp\"><img alt=\"downtowla.com home\" border=\"0\" height=\"63\" src=\"/images/global/spacer.gif\" width=\"333\"/></a>,\n",
       " <a href=\"1_07_maps.asp\">Maps</a>,\n",
       " <a href=\"0_02_calendar.asp\">Events</a>,\n",
       " <a href=\"4_0_connectDirectory.asp\">Search</a>,\n",
       " <a href=\"5_02_safeAndCleanTeams.asp\">Safe &amp; Clean</a>,\n",
       " <a href=\"5_0_aboutUs.asp\">About</a>,\n",
       " <a href=\"0_03_askUs.asp\">Contact</a>,\n",
       " <a href=\"newsletter\">Newsletter</a>,\n",
       " <a href=\"http://twitter.com/downtown_la\" target=\"_blank\">Twitter</a>,\n",
       " <a href=\"http://www.facebook.com/DowntownLA.com\" target=\"_blank\">Facebook</a>,\n",
       " <a href=\"http://pinterest.com/DowntownLAcom/\" target=\"_blank\">Pinterest</a>,\n",
       " <a href=\"http://www.foursquare.com/downtown_la\" target=\"_blank\">Foursquare</a>,\n",
       " <a href=\"3_01_downtownSpecials.asp\">Downtown Specials</a>,\n",
       " <a href=\"3_02_maps.asp\">Downtown Maps</a>,\n",
       " <a href=\"3_03_artsAndEntertainment.asp\">Arts &amp; Entertainment</a>,\n",
       " <a href=\"6_0_families.asp\">Kids &amp; Families</a>,\n",
       " <a href=\"3_04_dining.asp\">Dining</a>,\n",
       " <a href=\"3_05_nightlife.asp\">Nightlife</a>,\n",
       " <a href=\"3_06_hotels.asp\">Hotels</a>,\n",
       " <a href=\"3_07_sports.asp\">Sports</a>,\n",
       " <a href=\"3_08_shopping.asp\">Shopping</a>,\n",
       " <a href=\"3_09_parking.asp\">Parking Here</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1100\">First &amp; Hope American Bistro</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.firstandhope.com/\" target=\"_blank\">http://www.firstandhope.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=631\">McCormick &amp; Schmick's</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.mccormickandschmicks.com/Locations/southern-california-los-angeles/los-angeles-california/FifthSt.aspx\" target=\"_blank\">http://www.mccormickandschmicks.com/Locations/southern-california-los-angeles/los-angeles-california/FifthSt.aspx</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=709\">Nick &amp; Stef's Steakhouse LA</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.patinagroup.com/restaurant.php?restaurants_id=6\" target=\"_blank\">http://www.patinagroup.com/restaurant.php?restaurants_id=6</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://mo-chica.com/sample-page\" target=\"_blank\">http://mo-chica.com/sample-page</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=120\">Bar &amp; Kitchen</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://barandkitchenla.com/\" target=\"_blank\">http://barandkitchenla.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=143\">Big Wangs Sports Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://bigwangs.com/bar.html\" target=\"_blank\">http://bigwangs.com/bar.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=163\">Border Grill</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://bordergrill.com\" target=\"_blank\">http://bordergrill.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=228\">Casey's Bar &amp; Grill</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/caseysirishpub\" target=\"_blank\">http://213nightlife.com/caseysirishpub</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=252\">Chaya Downtown</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.thechaya.com/downtown-la/\" target=\"_blank\">http://www.thechaya.com/downtown-la/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1868\">City Tavern Downtown Los Angeles</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.citytaverndtla.com/menus/hoppy-hour/\" target=\"_blank\">http://www.citytaverndtla.com/menus/hoppy-hour/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=330\">Cucina Rustica</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.cucinarusticala.com/\" target=\"_blank\">http://www.cucinarusticala.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1008\">Drago Centro</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://dragocentro.com/menus.php\" target=\"_blank\">http://dragocentro.com/menus.php</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1043\">Engine Co. No. 28</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.engineco.com/\" target=\"_blank\">http://www.engineco.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1092\">FIGat7th</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.figat7th.com/special-promotions/\" target=\"_blank\">http://www.figat7th.com/special-promotions/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1155\">Golden Gopher</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/goldengopher\" target=\"_blank\">http://213nightlife.com/goldengopher</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1094\">Hotel Figueroa</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.figueroahotel.com/\" target=\"_blank\">http://www.figueroahotel.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1866\">Local Table</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.localtable-la.com\" target=\"_blank\">http://www.localtable-la.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=612\">Maria's Italian Kitchen</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://mariasdowntownla.com/\" target=\"_blank\">http://mariasdowntownla.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=624\">Mas Malo</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://masmalorestaurant.com/\" target=\"_blank\">http://masmalorestaurant.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=682\">Morton's The Steakhouse</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.mortons.com/losangeles/\" target=\"_blank\">http://www.mortons.com/losangeles/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=717\">Noe Restaurant &amp; Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.noerestaurant.com/losangeles/default.html\" target=\"_blank\">http://www.noerestaurant.com/losangeles/default.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=392\">Salvage Bar &amp; Lounge</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"www.facebook.com/SalvageLA\" target=\"_blank\">www.facebook.com/SalvageLA</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=431\">Seven Grand</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/sevengrand\" target=\"_blank\">http://213nightlife.com/sevengrand</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=432\">Seven Restaurant Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.sevenrestaurantbar.com/\" target=\"_blank\">http://www.sevenrestaurantbar.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=886\">Suede Bar &amp; Lounge</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://suedebarla.com/\" target=\"_blank\">http://suedebarla.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=906\">Takami Sushi &amp; Robata Restaurant</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.takamisushi.com/index.html\" target=\"_blank\">http://www.takamisushi.com/index.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=926\">The Cabbage Patch</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.cabbagepatchla.com/portfolio/downtown/\" target=\"_blank\">http://www.cabbagepatchla.com/portfolio/downtown/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=983\">The Standard, Downtown LA</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://standardhotels.com/downtown-la/food-drink/24-7-restaurant\" target=\"_blank\">http://standardhotels.com/downtown-la/food-drink/24-7-restaurant</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1450\">Westin Bonaventure Hotel and Suites</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"https://www.facebook.com/events/451752961575476\" target=\"_blank\">https://www.facebook.com/events/451752961575476</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://http://www.cervetecadtla.com/\">www.cervetecaDTLA.com</a>,\n",
       " <a href=\"http://www.cervetecadtla.com/menus/happyhour-2/\" target=\"_blank\">http://www.cervetecadtla.com/menus/happyhour-2/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1933\">Cerveteca</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.cervetecadtla.com/menus/happyhour-2/\" target=\"_blank\">http://www.cervetecadtla.com/menus/happyhour-2/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1358\">Tony's Saloon</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/tonyssaloon\" target=\"_blank\">http://213nightlife.com/tonyssaloon</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1804\">Ebanos Crossing</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.ebanoscrossing.com\" target=\"_blank\">http://www.ebanoscrossing.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1804\">Ebanos Crossing</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.ebanoscrossing.com\" target=\"_blank\">http://www.ebanoscrossing.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=850\">Redwood Bar &amp; Grill</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.theredwoodbar.com/2012/calendar.html\" target=\"_blank\">http://www.theredwoodbar.com/2012/calendar.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=793\">Philippe the Original</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.philippes.com/\" target=\"_blank\">http://www.philippes.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1923\">Pressed Juicery</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"www.pressedjuicery.com\" target=\"_blank\">www.pressedjuicery.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1517\">Artisan House</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://artisanhouse.net\" target=\"_blank\">http://artisanhouse.net</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=175\">Broadway Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/broadwaybar\" target=\"_blank\">http://213nightlife.com/broadwaybar</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1834\">Bugis Street Brasserie</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.milleniumhotels.com\" target=\"_blank\">http://www.milleniumhotels.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=308\">Cole's French Dip</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/colesfrenchdip\" target=\"_blank\">http://213nightlife.com/colesfrenchdip</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1336\">La Cita</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://site.lacitabar.com\" target=\"_blank\">http://site.lacitabar.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1336\">La Cita</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://site.lacitabar.com\" target=\"_blank\">http://site.lacitabar.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1336\">La Cita</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://site.lacitabar.com\" target=\"_blank\">http://site.lacitabar.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1336\">La Cita</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://site.lacitabar.com\" target=\"_blank\">http://site.lacitabar.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1336\">La Cita</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://site.lacitabar.com\" target=\"_blank\">http://site.lacitabar.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=536\">Las Perlas</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/lasperlas\" target=\"_blank\">http://213nightlife.com/lasperlas</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1948\">Mrs. Fish</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.mrsfish.com\" target=\"_blank\">http://www.mrsfish.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1809\">Peking Tavern</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.pekingtavern.com\" target=\"_blank\">http://www.pekingtavern.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=792\">Pete's Cafe &amp; Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.petescafe.com/index.html\" target=\"_blank\">http://www.petescafe.com/index.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=473\">Spring Street Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://springstla.com/\" target=\"_blank\">http://springstla.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=919\">The Association</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.theassociation-la.com/\" target=\"_blank\">http://www.theassociation-la.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=938\">The Crocker Club</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://crockerclub.com/\" target=\"_blank\">http://crockerclub.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1903\">The DFM</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"www.theowlsbrew.com\" target=\"_blank\">www.theowlsbrew.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=941\">The Edison</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.edisondowntown.com/\" target=\"_blank\">http://www.edisondowntown.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=943\">The Falls</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://thefallslounge.com/\" target=\"_blank\">http://thefallslounge.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=948\">The Gorbals</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.thegorbalsla.com/\" target=\"_blank\">http://www.thegorbalsla.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=33\">Aburiya Toranoko</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.toranokola.com/menu/happy-hour.html\" target=\"_blank\">http://www.toranokola.com/menu/happy-hour.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1068\">Far Bar</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://farbarla.com/\" target=\"_blank\">http://farbarla.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1518\">Izakaya Fu-Ga</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://izakayafu-ga.com/\" target=\"_blank\">http://izakayafu-ga.com/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1862\">Justice Urban Tavern</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.justicela.com/images/menus/social-hour-menu.pdf\" target=\"_blank\">http://www.justicela.com/images/menus/social-hour-menu.pdf</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=541\">Lazy Ox Canteen</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.lazyoxcanteen.com/menus/dinner-2/\" target=\"_blank\">http://www.lazyoxcanteen.com/menus/dinner-2/</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=942\">The Escondite</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://theescondite.com/index.html\" target=\"_blank\">http://theescondite.com/index.html</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=87\">Arashi Sushi</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://arashisushi.com\" target=\"_blank\">http://arashisushi.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=166\">BottleRock</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://bottlerockla.com\" target=\"_blank\">http://bottlerockla.com</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=212\">Cana Rum Bar at the Doheny</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://213nightlife.com/canarumbar\" target=\"_blank\">http://213nightlife.com/canarumbar</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1051\">ESPN Zone at L.A. Live</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.lalive.com/concierge/detail/espn-zone\" target=\"_blank\">http://www.lalive.com/concierge/detail/espn-zone</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1070\">Farm of Beverly Hills</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.thefarmofbeverlyhills.com/la_live.shtml\" target=\"_blank\">http://www.thefarmofbeverlyhills.com/la_live.shtml</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1111\">Fleming's Prime Steakhouse</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.flemingssteakhouse.com/locations/ca/los-angeles\" target=\"_blank\">http://www.flemingssteakhouse.com/locations/ca/los-angeles</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=1305\">Katsuya</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://www.lalive.com/concierge/detail/katsuya-at-la-live\" target=\"_blank\">http://www.lalive.com/concierge/detail/katsuya-at-la-live</a>,\n",
       " <a href=\"0_0_directory_businessDetails.asp?BID=365\">Rock'n Fish</a>,\n",
       " <a href=\"HappyHoursRSS.xml\" target=\"_blank\"><img alt=\"RSS feed\" border=\"0\" src=\"images/global/icon_rss_small.gif\"/></a>,\n",
       " <a href=\"http://rocknfishrestaurants.com/htm/LA/home.shtml\" target=\"_blank\">http://rocknfishrestaurants.com/htm/LA/home.shtml</a>]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## get bs4 object\n",
    "soup = bs4.BeautifulSoup(source)\n",
    " \n",
    "## compare the two print statements\n",
    "#print soup\n",
    "#print soup.prettify()\n",
    "\n",
    "## show how to find all a tags\n",
    "soup.findAll('a')\n",
    "\n",
    "## ***Why does this not work? ***\n",
    "#soup.findAll('Soup')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The last command only returns an empty list, because `Soup` is not an HTML tag. It is just a string that occours in the webpage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Some examples\n",
    "============="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bs4/download/',\n",
       " '#Download',\n",
       " 'bs4/doc/',\n",
       " '#HallOfFame',\n",
       " 'https://code.launchpad.net/beautifulsoup',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'http://www.candlemarkandgleam.com/shop/constellation-games/',\n",
       " 'http://constellation.crummy.com/Constellation%20Games%20excerpt.html',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " 'http://lxml.de/',\n",
       " 'http://code.google.com/p/html5lib/',\n",
       " 'bs4/doc/',\n",
       " None,\n",
       " 'bs4/download/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html',\n",
       " 'download/3.x/BeautifulSoup-3.2.1.tar.gz',\n",
       " None,\n",
       " 'http://www.nytimes.com/2007/10/25/arts/design/25vide.html',\n",
       " 'https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py',\n",
       " 'http://www.harrowell.org.uk/viktormap.html',\n",
       " 'http://svn.python.org/view/tracker/importer/',\n",
       " 'http://www2.ljworld.com/',\n",
       " 'http://www.b-list.org/weblog/2010/nov/02/news-done-broke/',\n",
       " 'http://esrl.noaa.gov/gsd/fab/',\n",
       " 'http://laps.noaa.gov/topograbber/',\n",
       " 'http://groups.google.com/group/beautifulsoup/',\n",
       " 'https://launchpad.net/beautifulsoup',\n",
       " 'https://code.launchpad.net/beautifulsoup/',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " '/source/software/BeautifulSoup/index.bhtml',\n",
       " '/self/',\n",
       " '/self/contact.html',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://www.crummy.com/',\n",
       " 'http://www.crummy.com/software/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## get attribute value from an element:\n",
    "## find tag: this only returns the first occurrence, not all tags in the string\n",
    "first_tag = soup.find('a')\n",
    "\n",
    "## get attribute `href`\n",
    "first_tag.get('href')\n",
    "\n",
    "## get all links in the page\n",
    "link_list = [l.get('href') for l in soup.findAll('a')]\n",
    "link_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object has no attribute '__getitem__'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-24-db28b6c7151c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;31m# if it starts with 'http' we are happy\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0ml\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mlink_list\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m     \u001b[1;32mif\u001b[0m \u001b[0ml\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'http'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      9\u001b[0m         \u001b[0mexternal_links\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ml\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: 'NoneType' object has no attribute '__getitem__'"
     ]
    }
   ],
   "source": [
    "## filter all external links\n",
    "# create an empty list to collect the valid links\n",
    "external_links = []\n",
    "\n",
    "# write a loop to filter the links\n",
    "# if it starts with 'http' we are happy\n",
    "for l in link_list:\n",
    "    if l[:4] == 'http':\n",
    "        external_links.append(l)\n",
    "\n",
    "# this throws an error! It says something about 'NoneType'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bs4/download/',\n",
       " '#Download',\n",
       " 'bs4/doc/',\n",
       " '#HallOfFame',\n",
       " 'https://code.launchpad.net/beautifulsoup',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'http://www.candlemarkandgleam.com/shop/constellation-games/',\n",
       " 'http://constellation.crummy.com/Constellation%20Games%20excerpt.html',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " 'http://lxml.de/',\n",
       " 'http://code.google.com/p/html5lib/',\n",
       " 'bs4/doc/',\n",
       " None,\n",
       " 'bs4/download/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html',\n",
       " 'download/3.x/BeautifulSoup-3.2.1.tar.gz',\n",
       " None,\n",
       " 'http://www.nytimes.com/2007/10/25/arts/design/25vide.html',\n",
       " 'https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py',\n",
       " 'http://www.harrowell.org.uk/viktormap.html',\n",
       " 'http://svn.python.org/view/tracker/importer/',\n",
       " 'http://www2.ljworld.com/',\n",
       " 'http://www.b-list.org/weblog/2010/nov/02/news-done-broke/',\n",
       " 'http://esrl.noaa.gov/gsd/fab/',\n",
       " 'http://laps.noaa.gov/topograbber/',\n",
       " 'http://groups.google.com/group/beautifulsoup/',\n",
       " 'https://launchpad.net/beautifulsoup',\n",
       " 'https://code.launchpad.net/beautifulsoup/',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " '/source/software/BeautifulSoup/index.bhtml',\n",
       " '/self/',\n",
       " '/self/contact.html',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://www.crummy.com/',\n",
       " 'http://www.crummy.com/software/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets investigate. Have a close look at the link_list:\n",
    "link_list\n",
    "\n",
    "# Seems that there are None elements!\n",
    "# Let's verify\n",
    "#print sum([l is None for l in link_list])\n",
    "\n",
    "# So there are two elements in the list that are None!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://code.launchpad.net/beautifulsoup',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'http://www.candlemarkandgleam.com/shop/constellation-games/',\n",
       " 'http://constellation.crummy.com/Constellation%20Games%20excerpt.html',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " 'http://lxml.de/',\n",
       " 'http://code.google.com/p/html5lib/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html',\n",
       " 'http://www.nytimes.com/2007/10/25/arts/design/25vide.html',\n",
       " 'https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py',\n",
       " 'http://www.harrowell.org.uk/viktormap.html',\n",
       " 'http://svn.python.org/view/tracker/importer/',\n",
       " 'http://www2.ljworld.com/',\n",
       " 'http://www.b-list.org/weblog/2010/nov/02/news-done-broke/',\n",
       " 'http://esrl.noaa.gov/gsd/fab/',\n",
       " 'http://laps.noaa.gov/topograbber/',\n",
       " 'http://groups.google.com/group/beautifulsoup/',\n",
       " 'https://launchpad.net/beautifulsoup',\n",
       " 'https://code.launchpad.net/beautifulsoup/',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://www.crummy.com/',\n",
       " 'http://www.crummy.com/software/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's filter those objects out in the for loop\n",
    "external_links = []\n",
    "\n",
    "# write a loop to filter the links\n",
    "# if it is not None and starts with 'http' we are happy\n",
    "for l in link_list:\n",
    "    if l is not None and l[:4] == 'http':\n",
    "        external_links.append(l)\n",
    "        \n",
    "external_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Note: The above `if` condition works because of lazy evaluation in Python. The `and` statement becomes `False` if the first part is `False`, so there is no need to ever evaluate the second part. Thus a `None` entry in the list gets never asked about its first four characters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://code.launchpad.net/beautifulsoup',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'http://www.candlemarkandgleam.com/shop/constellation-games/',\n",
       " 'http://constellation.crummy.com/Constellation%20Games%20excerpt.html',\n",
       " 'https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " 'http://lxml.de/',\n",
       " 'http://code.google.com/p/html5lib/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html',\n",
       " 'http://www.nytimes.com/2007/10/25/arts/design/25vide.html',\n",
       " 'https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py',\n",
       " 'http://www.harrowell.org.uk/viktormap.html',\n",
       " 'http://svn.python.org/view/tracker/importer/',\n",
       " 'http://www2.ljworld.com/',\n",
       " 'http://www.b-list.org/weblog/2010/nov/02/news-done-broke/',\n",
       " 'http://esrl.noaa.gov/gsd/fab/',\n",
       " 'http://laps.noaa.gov/topograbber/',\n",
       " 'http://groups.google.com/group/beautifulsoup/',\n",
       " 'https://launchpad.net/beautifulsoup',\n",
       " 'https://code.launchpad.net/beautifulsoup/',\n",
       " 'https://bugs.launchpad.net/beautifulsoup/',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://creativecommons.org/licenses/by-sa/2.0/',\n",
       " 'http://www.crummy.com/',\n",
       " 'http://www.crummy.com/software/',\n",
       " 'http://www.crummy.com/software/BeautifulSoup/']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and we can put this in a list comprehension as well, it almost reads like \n",
    "# a sentence.\n",
    "\n",
    "[l for l in link_list if l is not None and l.startswith('http')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Parsing the Tree\n",
    "================\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<body><h3> Test </h3><p>Hello world!</p></body>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# redifining `s` without any line breaks\n",
    "s = \"\"\"<!DOCTYPE html><html><head><title>This is a title</title></head><body><h3> Test </h3><p>Hello world!</p></body></html>\"\"\"\n",
    "## get bs4 object\n",
    "tree = bs4.BeautifulSoup(s)\n",
    "\n",
    "## get html root node\n",
    "root_node = tree.html\n",
    "\n",
    "## get head from root using contents\n",
    "head = root_node.contents[0]\n",
    "\n",
    "## get body from root\n",
    "body = root_node.contents[1]\n",
    "\n",
    "## could directly access body\n",
    "tree.body"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Quiz:\n",
    "=====\n",
    "\n",
    "* Find the `h3` tag by parsing the tree starting at `body`\n",
    "* Create a list of all __Hall of Fame__ entries listed on the Beautiful Soup webpage\n",
    "    - hint: it is the only unordered list in the page (tag `ul`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h3> Test </h3>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## get h3 tag from body\n",
    "body.contents[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "## use ul as entry point\n",
    "entry_point = soup.find('ul')\n",
    "\n",
    "## get hall of fame list from entry point\n",
    "## skip the first entry \n",
    "hall_of_fame_list = entry_point.contents[1:]\n",
    "\n",
    "## reformat into a list containing strings\n",
    "tmp = []\n",
    "for li in hall_of_fame_list:\n",
    "    tmp.append(li.contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`tmp` now is actually a list of lists containing the hall of fame entries. \n",
    "Here is some advanced Python on how to print really just one entry per list item.\n",
    "\n",
    "The cool things about this are: \n",
    "* The use of `\"\"` to just access the `join` function of strings.\n",
    "* The `join` function itself\n",
    "* that you can actually have two nested for loops in a list comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a href=\"http://www.nytimes.com/2007/10/25/arts/design/25vide.html\">\"Movable\n",
      " Type\"</a>, a work of digital art on display in the lobby of the New\n",
      " York Times building, uses Beautiful Soup to scrape news feeds.\n",
      "\n",
      "\n",
      "Reddit uses Beautiful Soup to <a href=\"https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py\">parse\n",
      "a page that's been linked to and find a representative image</a>.\n",
      "\n",
      "\n",
      "Alexander Harrowell uses Beautiful Soup to <a href=\"http://www.harrowell.org.uk/viktormap.html\">track the business\n",
      " activities</a> of an arms merchant.\n",
      "\n",
      "\n",
      "The developers of Python itself used Beautiful Soup to <a href=\"http://svn.python.org/view/tracker/importer/\">migrate the Python\n",
      "bug tracker from Sourceforge to Roundup</a>.\n",
      "\n",
      "\n",
      "The <a href=\"http://www2.ljworld.com/\">Lawrence Journal-World</a>\n",
      "uses Beautiful Soup to <a href=\"http://www.b-list.org/weblog/2010/nov/02/news-done-broke/\">gather\n",
      "statewide election results</a>.\n",
      "\n",
      "\n",
      "The <a href=\"http://esrl.noaa.gov/gsd/fab/\">NOAA's Forecast\n",
      "Applications Branch</a> uses Beautiful Soup in <a href=\"http://laps.noaa.gov/topograbber/\">TopoGrabber</a>, a script for\n",
      "downloading \"high resolution USGS datasets.\"\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test =  [\"\".join(str(a) for a in sublist) for sublist in tmp]\n",
    "print '\\n'.join(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Advanced Example\n",
    "===============\n",
    "\n",
    "Idea by [Jesse Steinweg-Woods](https://jessesw.com/Data-Science-Skills/)\n",
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scraping data science skills\n",
    "=============================\n",
    "\n",
    "- What skills are in demand for data scientists?\n",
    "- Should we have a lecture on Spark or only on MapReduce?\n",
    "\n",
    "We want to scrape the information from job advertisements for data scientists from indeed.com\n",
    "Let's scrape and find out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Fixed url for job postings containing data scientist\n",
    "url = 'http://www.indeed.com/jobs?q=data+scientist&l='\n",
    "# read the website\n",
    "source = urllib2.urlopen(url).read()\n",
    "# parse html code\n",
    "bs_tree = bs4.BeautifulSoup(source)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search yielded 22,926 hits.\n",
      "22926\n"
     ]
    }
   ],
   "source": [
    "# see how many job postings we found\n",
    "job_count_string = bs_tree.find(id = 'searchCount').contents[0]\n",
    "job_count_string = job_count_string.split()[-1]\n",
    "print(\"Search yielded %s hits.\" % (job_count_string))\n",
    "\n",
    "# not that job_count so far is still a string, \n",
    "# not an integer, and the , separator prevents \n",
    "# us from just casting it to int\n",
    "\n",
    "job_count_digits = [int(d) for d in job_count_string if d.isdigit()]\n",
    "job_count = np.sum([digit*(10**exponent) for digit, exponent in \n",
    "                    zip(job_count_digits[::-1], range(len(job_count_digits)))])\n",
    "\n",
    "print job_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2293\n",
      "2283\n",
      "We found a lot of jobs:  240\n"
     ]
    }
   ],
   "source": [
    "# The website is only listing 10 results per page, \n",
    "# so we need to scrape them page after page\n",
    "num_pages = int(np.ceil(job_count/10.0))\n",
    "\n",
    "base_url = 'http://www.indeed.com'\n",
    "job_links = []\n",
    "for i in range(20): #do range(num_pages) if you want them all\n",
    "    if i%10==0:\n",
    "        print num_pages-i\n",
    "    url = 'http://www.indeed.com/jobs?q=data+scientist&start=' + str(i*10)\n",
    "    html_page = urllib2.urlopen(url).read() \n",
    "    bs_tree = bs4.BeautifulSoup(html_page)\n",
    "    job_link_area = bs_tree.find(id = 'resultsCol')\n",
    "    job_postings = job_link_area.findAll(\"div\")\n",
    "    job_postings = [jp for jp in job_postings if not jp.get('class') is None \n",
    "                    and ''.join(jp.get('class')) ==\"rowresult\"]\n",
    "    job_ids = [jp.get('data-jk') for jp in job_postings]\n",
    "    \n",
    "    # go after each link\n",
    "    for id in job_ids:\n",
    "        job_links.append(base_url + '/rc/clk?jk=' + id)\n",
    "\n",
    "    time.sleep(1)\n",
    "\n",
    "print \"We found a lot of jobs: \", len(job_links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Some precautions to enable us to restart our search\n",
    "========================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Save the scraped links\n",
    "#with open('scraped_links.pkl', 'wb') as f:\n",
    "#    cPickle.dump(job_links, f)\n",
    "    \n",
    "# Read canned scraped links\n",
    "with open('scraped_links.pkl', 'r') as f:\n",
    "    job_links = cPickle.load(f)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "skill_set = {'mapreduce': 0, 'spark': 0}\n",
    "\n",
    "## write initialization into a file, so we can restart later\n",
    "#with open('scraped_links_restart.pkl', 'wb') as f:\n",
    "#    cPickle.dump((skill_set, 0),f)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Python Dictonaries\n",
    "==================\n",
    "\n",
    "* build in data type\n",
    "* uses key: value pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 1, 'b': 2}\n",
      "['a', 'b']\n",
      "[1, 2]\n",
      "a 1\n",
      "b 2\n",
      "a 1\n",
      "b 2\n"
     ]
    }
   ],
   "source": [
    "a = {'a': 1, 'b':2}\n",
    "print a\n",
    "\n",
    "#show keys\n",
    "print a.keys()\n",
    "\n",
    "#show values\n",
    "print a.values()\n",
    "\n",
    "#show for loop over all entries\n",
    "# option 1 using zip\n",
    "# this works also for iterating over any\n",
    "# other two lists\n",
    "for k,v in zip(a.keys(), a.values()):\n",
    "    print k,v\n",
    "\n",
    "# option 2 using the dictionary `iteritems()` function\n",
    "for k,v in a.iteritems():\n",
    "    print k,v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How many websites still to go?  0\n"
     ]
    }
   ],
   "source": [
    "# This code below does the trick, but could be optimized for speed if necessary\n",
    "# e.g. skills are typically listed at the end of the webpage\n",
    "# might not need to split/join the whole webpage, as we already know\n",
    "# which words we are looking for \n",
    "# and could stop after the first occurance of each word\n",
    "\n",
    "with open('scraped_links_restart.pkl', 'r') as f:\n",
    "    skill_set, index = cPickle.load(f)\n",
    "    print \"How many websites still to go? \", len(job_links) - index\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "counter = 0\n",
    "\n",
    "for link in job_links[index:]:\n",
    "    counter +=1  \n",
    "    \n",
    "    try:\n",
    "        html_page = urllib2.urlopen(link).read()\n",
    "    except urllib2.HTTPError:\n",
    "        print \"HTTPError:\"\n",
    "        continue\n",
    "    except urllib2.URLError:\n",
    "        print \"URLError:\"\n",
    "        continue\n",
    "    except socket.error as error:\n",
    "        print \"Connection closed\"\n",
    "        continue\n",
    "\n",
    "    html_text = re.sub(\"[^a-z.+3]\",\" \", html_page.lower()) # replace all but the listed characters\n",
    "        \n",
    "    for key in skill_set.keys():\n",
    "        if key in html_text:  \n",
    "            skill_set[key] +=1\n",
    "            \n",
    "    if counter % 5 == 0:\n",
    "        print len(job_links) - counter - index\n",
    "        print skill_set\n",
    "        with open('scraped_links_restart.pkl','wb') as f:\n",
    "            cPickle.dump((skill_set, index+counter),f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'spark': 31, 'mapreduce': 16}\n"
     ]
    }
   ],
   "source": [
    "print skill_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnwAAAIKCAYAAAC5lpKKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYlPX+//EXIiMouStGuJB6QiUQhSMWaoqlabhvx31B\nBcm90lbtaBkVZogLammuaehXTT3lkpm55Y6mVEfNcM0NUhRZnN8fHecXBzU6yox+eD6ui+tyPvcs\n76Hr4np233Pf42S1Wq0CAACAsQo5egAAAADkL4IPAADAcAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAA\nDEfwAbgrPXr0kI+Pj+3H19dXISEhGj58uI4cOfKXn2/Xrl168cUX73qu69eva8qUKWrevLn8/PxU\nr149DRgwQLt37/5Lz+Pj46PFixff9Tz2sHnzZvXs2VN169ZV7dq11bp1a82ePVvZ2dm2+0yePFkh\nISG3fY7Ro0erc+fOkqQTJ07Ix8dH3377ba5tAB4shR09AIAH35NPPqmhQ4dKkjIyMnTmzBnNnj1b\nHTt21MKFC+Xj45Pn50pISFBycvJdzzRq1Cjt3r1bERERql69un777TclJCSoV69emjVrloKDg/P0\nPEuWLJGXl9ddz5PfNm3apMjISHXt2lXh4eFycXHR7t279eGHH+qnn37S22+/nafniYqKUkZGxm23\nOzk53auRAdgRwQfgrpUsWVJ+fn451po2bap27dpp7Nix+vTTT//S893t9eBPnDihL774QvHx8WrU\nqJFtvUmTJurQoYNmzJiR5+D77/d1v5o1a5aeeeYZvfbaa7a1+vXry93dXdHR0Ro6dKg8PDz+9Hkq\nVqx4x+1cqx94MHFIF0C+cHV1Vb9+/bRv374ch3aXLVumNm3ayN/fXwEBAerbt69t++jRo7V8+XLt\n379fPj4+OnXqlCRp48aN6tKliwICAuTv768uXbpoz549t33tixcvSpJu3LiRY71QoUIaPny4wsLC\ncqzPnTtXzZo1U+3atdWmTRt9/fXXtm0+Pj45gvXIkSMKDw9XQECAgoODNX78eF2/ft22vUePHoqJ\niVF0dLTq16+vOnXq6IUXXtCVK1ds98nKylJsbKwaN26sgIAA/eMf/9C+ffts2zMyMvTuu+8qJCRE\n/v7+6tGjh5KSku74+7506VKu9ytJzz33nEaMGKFChW795/6rr75SzZo1NWPGDEl/7bDtxo0b1bZt\nW/n7+6tBgwa5fhcA7h8EH4B8c3MvWmJioiRpzZo1eu2119SiRQt99NFHGjNmjI4ePWrbKxUVFaVG\njRqpevXqWrJkicqWLau9e/dq0KBBCggIUHx8vKKjo3XlyhW98MILtwwc6fdIK1u2rEaPHq0PPvhA\ne/fuVWZmpiSpQYMGatu2re2+s2bN0rvvvquWLVtq2rRpCggI0PPPP6/vv//edp+bhzHPnj2rbt26\n6dq1a/rggw/00ksvafXq1XrppZdyvP6iRYt0/Phxvffee3rppZe0du1aTZ8+3bb9rbfe0uzZs9Wr\nVy9NnTpV5cqVU3h4uM6cOSNJevnll/XZZ58pKipKsbGxKlKkiHr16qWzZ8/e9nf95JNPau3atRoy\nZIjWrl2rS5cuSZLKli2r/v37q1y5crkes3//fo0YMUK9e/fWgAEDcr3fOzl+/LiGDBmi4OBgzZo1\nS0OHDtWyZcs0efLkP30sAPvjkC6AfFOmTBlJ0oULFyT9fqj1j3ERGBiolJQURUdHS/r9cGKpUqWU\nkpJiO5R69OhRtWrVSqNGjbI9r7OzswYPHqzTp0/rkUceyfW6FotFM2bM0IgRIxQfH6/4+Hi5ubnp\niSeeUI8ePWwheuPGDc2cOVPdu3fXkCFDJP1+GPTf//63du/erVq1auV43k8++UTOzs6aOXOmihYt\nKkl6+OGH1adPHyUlJdk+q+ju7q64uDjbXrWdO3dq06ZNeuGFF5SSkqIlS5Zo9OjR6tGjh+330LZt\nW+3bt09Vq1bV6tWrNXHiRLVo0UKSFBISombNmumjjz7SK6+8csvf9YgRI3Tx4kWtWrVKa9eulZOT\nk2rUqKFWrVqpa9euslgsOe7/888/a+DAgWrZsmWuYL3TYdubMXjw4EFlZmaqb9++KleunIKCgmSx\nWHKcIALg/kHwAbCbm6GXkpKiI0eO6OjRo9q4caMkKTMzUy4uLrke0759e7Vv315XrlzR0aNHdfTo\nUW3YsMH2mNupWbOm/vWvf2nXrl36+uuvtW3bNn311VfasGGDRowYoQEDBujYsWNKTU3VU089leOx\n8+bNu+Vz7ty5U3Xq1FGRIkWUlZUlSapbt65cXV21Y8cOW/DVqlUrxyFUDw8P7d+/X9Lve9Wys7Nz\nvKaLi4tWrVolSVq4cKGk3yPv5mtIv4fojh07bvt+ixQpovfee0/Dhg3Thg0btGXLFn333Xd65513\n9Pnnn2vevHm2SE1PT1d4eLiuXbuml19++bbPeSs3Y9DPz08Wi0WdOnXSc889p8aNGyssLIyTOoD7\nFMEHIN+cO3dOkmyHE8+ePatXXnlFW7Zskaurq3x8fOTu7i6r1XrbvUppaWl67bXX9OWXX8rZ2VnV\nq1e37dX7sxMInJycFBQUpKCgIElScnKyRo8erdjYWHXo0EEpKSmSpNKlS+fp/aSkpOjAgQO59vw5\nOTnZ3qv0++cX/6hQoUK2w8+pqal3fM2bM/3973/Pte3mHtM7eeSRR9SzZ0/17NlTGRkZmjVrlmJj\nY5WQkKCePXtKkq5cuSIfHx9duHBB06ZN+58ug1OxYkXNnj1b06dP15w5czRz5kx5enpqzJgxOU6U\nAXB/IPgA5JvvvvtOklSnTh1J0osvvqiUlBQtX75cjz32mJycnLRw4ULbdd5uZfz48dqzZ4/mzp2r\ngIAAOTs7a9OmTVq3bt1tHzNhwgQlJiZq0aJFOdYrVqyoUaNGqVOnTjp58qQeeughSbJ93u2mQ4cO\nqXDhwvrb3/6WY7148eJq1qxZjs+7Sb+HZ9myZf/kt/G7P75msWLFbOt79uxRmTJlVLx4cVksllyz\nW61WFS586z/Z+/bt08CBAzV37lw99thjtnWLxaJBgwZp1apV+vnnn23r7u7umjFjhubOnau4uDi1\na9dOVatWzdP8f1S3bl3NnDlT165d0+bNmzV9+nSNGDFC27dvv+XeWgCOw0kbAPJFRkaGZs+eraCg\nINulPvbv36/WrVvLx8fHduhv69atcnJysu2t+++zSffv36/Q0FAFBgbK2dnZ9hgp91m4N1WuXFl7\n9+695Zm8x44dk4uLiypVqiRvb28VL148x1m5kvTaa6/d8rBuQECAjh49qho1aqhWrVqqVauWypcv\nrw8++EDHjx/P0+/F19fXFq03ZWRkaMiQIfrXv/6lOnXqKCMjQ1lZWbbXqFWrllauXKkvvvjils9Z\npUoVpaena8GCBbm2XblyRRcuXMgRdK6uripatKj69u2rhx9+WOPGjcvxmDsdlr25bdGiRQoNDVVW\nVpbc3Nz0zDPPKDw8XGlpaTnOSAZwf2APH4C7dunSJe3fv19Wq1VZWVk6ceKE5s+fr19//VUffvih\n7X6+vr5avHixKleuLFdXV61cuVK7du2S1WpVWlqaihQpohIlSuiXX37Rtm3bFBAQIF9fX1sIlSlT\nRl999ZVWr14tSbp69eot52nfvr2WLl2q8PBw9ezZU0FBQSpUqJB2796tjz/+WAMGDFCJEiUkSeHh\n4YqNjZW7u7sCAgL0xRdf6MiRI5owYUKu5+3du7eWL1+uQYMGqUuXLsrKylJcXJwuXryY4+LSdzrU\nXK5cObVv314xMTG6ceOGHn30US1cuFDZ2dlq166dypcvr6eeekpDhgzR4MGD5eXlpbVr12rhwoWK\niYm55XOWLFlSgwcP1vvvv6/z58+rVatWKlu2rE6cOKHZs2erfPnyateuXa7HWSwWjRo1Ss8//7zW\nrFljO0nkTvPf3Pb3v/9dEyZM0IgRI9SlSxddvXpV06ZNU926dVWqVKnbPh6AY9g1+NasWaPJkyfr\nzJkzeuSRRzRs2DA1bdpUBw4cUKdOnXJ87iUyMjLXYRMA96etW7fa9rpZLBZ5eHgoICBAMTExqly5\nsu1+EyZM0BtvvKEXX3xRxYoVU4sWLfTZZ5/p6aefVmJiop566il16tRJ69evV0REhD755BONHj1a\n165d09ixY+Xs7KwmTZpo+fLlat68uRITE/X444/nmqdIkSKaN2+eZs6cqfXr1+uTTz6RJFWvXl1v\nvPGG2rRpY7vvgAED5OLiovnz52vGjBl67LHHNHPmzByHRm/y8vLS/Pnz9d5772n48OGyWCwKDAzU\npEmTbAEp5d5D5uTklGPtjTfeUMmSJTVjxgxduXJFfn5+tjCTpEmTJumDDz7QpEmTdPnyZXl7e+v9\n999Xy5Ytb/vfIDw8XJUrV9bChQs1duxYXblyReXLl1fTpk01ePBgubm53XK2pk2b6oknnlB0dLQa\nNWqUa9b//vfN21WrVlVcXJwmT56sqKgoWSwWNWzY8C+fBALAPpysdrps+rFjx9SuXTvNnj1btWvX\n1rZt2zRgwAB98803Wrdunb766qsc16kCAADAvWG3PXze3t7aunWr3NzclJWVpXPnzsnd3V0uLi46\ndOjQX/quTQAAAOSdXQ/purm5KTk5Wc2aNZPVatWbb74pd3d3HT58WEWKFFFoaKhu3Lih5s2b2w6X\nAAAA4O7Y/SxdT09PHThwQLNnz9aECRO0fft2lS5dWk2aNNHq1as1d+5c7dixg6/nAQAAuEfs9hm+\nWxk9erTc3d1t36N509q1azVx4sTbXoLgj9LT03Xw4EGVK1fOdskGAACAgiQ7O1vnzp2Tr69vrou/\nS3Y8pLtp0ybNmTNHs2fPtq1lZGRIkt555x0NHjzYdhHS9PT0Ww57KwcPHlS3bt3u/cAAAAAPmAUL\nFigwMDDXut2Cr1atWjp48KBWrFihsLAwbd68Wd98840WL16syMhIOTk5aeTIkTp58qTi4+PVuXPn\nPD3vza9sWrBggSpUqJCfbwEAAOC+dObMGXXr1s3WRf/NbsFXtmxZTZs2TRMmTNA///lPeXt7a+rU\nqapatari4+M1btw4BQcHy9XVVV26dLF95+OfuXkYt0KFCvLy8srPtwAAAHBfu93H2+x6lm5gYKCW\nLl2aa93b21sff/yxPUcBAAAoMPguXQAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4\ngg8AAMBwBB8AAIDhCD4AAADDEXwAAACGI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIPAADAcAQf\nAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEIPgAA\nAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADDEXwAAACG\nI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIPAADAcAQfAACA4Qo7egDkr6tXryopKcnRY+AB5+Pj\no6JFizp6DADA/4jgM1xSUpL6vTZfD5X2cvQoeEBdvnhCH43vrjp16jh6FADA/4jgKwAeKu2lEh5V\nHT0GAABwED7DBwAAYDiCDwAAwHAEHwAAgOEIPgAAAMPZNfjWrFmjZ599VgEBAXruuee0fv16SVJq\naqqioqIUGBioxo0bKyEhwZ5jAQAAGM1uZ+keO3ZMr776qmbPnq3atWtr27ZtGjBggL755huNGTNG\n7u7u2rp1q5KSktS/f39Vr15d/v7+9hoPAADAWHYLPm9vb23dulVubm7KysrSuXPn5O7uLhcXF23Y\nsEFffvmlLBaL/Pz8FBYWpuXLlxN8AAAA94Bdr8Pn5uam5ORkNWvWTFarVW+++aZ++eUXFS5cWF5e\n///CwFWqVNG6devsORoAAICx7H7hZU9PTx04cEA7d+5UZGSkwsPD5erqmuM+rq6uSk9Pt/doAAAA\nRrJ78Dk7O0uSgoOD1axZMx08eFDXr1/PcZ/09HS+txMAAOAesdtZups2bVKfPn1yrGVkZKhSpUrK\nzMzU6dOnbevHjh1TtWrV7DUaAACA0ewWfLVq1dLBgwe1YsUK3bhxQ5s2bdI333yjLl26KDQ0VDEx\nMUpPT1diYqJWrVqlsLAwe40GAABgNLsFX9myZTVt2jTNnTtXQUFBmjx5sqZOnSpvb2+NGzdOWVlZ\natSokYYOHapRo0bJz8/PXqMBAAAYza6f4QsMDNTSpUtzrZcoUUKTJk2y5ygAAAAFBl+tBgAAYDiC\nDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8A\nAIDhCD4AAADDEXwAAACGI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIPAADAcAQfAACA4Qg+AAAA\nwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj\n+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADDEXwAAACGI/gAAAAMR/AB\nAAAYjuADAAAwHMEHAABgOIIPAADAcAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGM6uwbdr\n1y517NhRgYGBevrpp7V48WJJ0oEDB1SjRg0FBATYfmbMmGHP0QAAAIxV2F4vlJqaqkGDBmnMmDFq\n2bKlDh06pD59+qhSpUpKTk5Wo0aNNH36dHuNAwAAUGDYLfhOnz6txo0bq2XLlpKkmjVrql69etqz\nZ4/Onz8vHx8fe40CAABQoNjtkK6Pj4+io6Ntt1NTU7Vr1y75+Pjo8OHD2rNnj0JDQ9W4cWNFR0cr\nIyPDXqMBAAAYzSEnbVy+fFkRERHy9fVVkyZNVKpUKTVp0kSrV6/W3LlztWPHDk2ePNkRowEAABjH\n7sGXnJysLl26qFSpUoqLi5OTk5OmTZum3r17y9XVVRUrVlRERITWrVtn79EAAACMZNfg+/7779W5\nc2c1bNhQU6dOlcVi0W+//aYJEyYoLS3Ndr/09HS5urraczQAAABj2e2kjfPnzys8PFz9+vVTeHi4\nbd3d3V0bN25UoUKFNHLkSJ08eVLx8fHq3LmzvUYDAAAwmt2CLyEhQZcuXdKUKVM0ZcoU23qvXr0U\nHx+vcePGKTg4WK6ururSpYt69uxpr9EAAACMZrfgi4iIUERExG23f/zxx/YaBQAAoEDhq9UAAAAM\nR/ABAAAYjuADAAAwHMEHAABgOIIPAADAcAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGI7g\nAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcA\nAGA4gg8AAMBwBB8AAIDhCD4AAADDEXwAAACGI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIPAADA\ncAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEI\nPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADD2TX4\ndu3apY4dOyowMFBPP/20Fi9eLElKTU1VVFSUAgMD1bhxYyUkJNhzLAAAAKMVttcLpaamatCgQRoz\nZoxatmypQ4cOqU+fPqpUqZIWLVokd3d3bd26VUlJSerfv7+qV68uf39/e40HAABgLLvt4Tt9+rQa\nN26sli1bSpJq1qypevXqac+ePdqwYYMGDx4si8UiPz8/hYWFafny5fYaDQAAwGh2Cz4fHx9FR0fb\nbqempmrXrl2yWq0qXLiwvLy8bNuqVKmio0eP2ms0AAAAoznkpI3Lly8rIiJCvr6+Cg4Olqura47t\nrq6uSk9Pd8RoAAAAxrF78CUnJ6tLly4qVaqU4uLi5ObmpuvXr+e4T3p6uooWLWrv0QAAAIxk1+D7\n/vvv1blzZzVs2FBTp06VxWJR5cqVlZmZqdOnT9vud+zYMVWrVs2eowEAABjLbsF3/vx5hYeHq2/f\nvho1apRt3d3dXaGhoYqJiVF6eroSExO1atUqhYWF2Ws0AAAAo9ntsiwJCQm6dOmSpkyZoilTptjW\ne/XqpXHjxmnMmDFq1KiRihYtqlGjRsnPz89eowEAABjNbsEXERGhiIiI226fNGmSvUYBAAAoUPhq\nNQAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBweQq+0NBQXbp0Kdf62bNn\nFRwcfM+HAgAAwL1z2wsvr1mzRuvXr5cknTx5Uq+//rosFkuO+5w6dUouLi75OyEAAADuym338AUH\nB6tIkSK2yHNxcZHFYrH9FClSRL6+vpo6dardhgUAAMBfd9s9fKVLl9aECRMkSY888oj69eunokWL\n2m0wAAAA3Bt5+i7dwYMHKyUlRQcOHFBWVpasVmuO7SEhIfkyHAAAAO5enoJv2bJlGjNmjDIzM2+5\nPSkp6Z4OBQAAgHsnT8EXGxurzp07a9iwYXJ3d8/vmQAAAHAP5emyLJcuXVLv3r2JPQAAgAdQnoKv\nfv362rJlS37PAgAAgHyQp0O6vr6+euutt7Rx40Z5e3vbrr1ntVrl5OSkESNG5OuQAAAA+N/lKfh2\n7Nghf39/paWl6eDBg/k9EwAAAO6hPAXfvHnz8nsOAAAA5JM8Bd/OnTvvuD0oKOieDAMAAIB7L0/B\n16NHj1s/uHBhlSxZUt9+++09HQoAAAD3Tp6Cb//+/TluZ2VlKTk5WW+99ZY6d+6cL4MBAADg3sjT\nZVmKFCmS46dYsWLy8fHRq6++qokTJ+b3jAAAALgLeQq+27l27ZouXrx4r2YBAABAPsjTId2YmBg5\nOTnlWLt8+bLWrl2rhg0b5stgAAAAuDfyFHz79u3LtWaxWNSxY0f17dv3ng8FAACAe4fr8AEAABgu\nT8EnSadOndLcuXP173//W1arVd7e3urcubOqV6+en/MBAADgLuXppI2dO3eqRYsW2rNnj6pVqyZv\nb2/t3btX7du3165du/J7RgAAANyFPO3hi46OVteuXfXSSy/lWo+JidGiRYvyZTgAAADcvTzt4fvx\nxx/VqVOnXOudOnXSoUOH7vlQAAAAuHfyFHwVKlTQjz/+mGv9p59+UsmSJe/5UAAAALh38nRIt1u3\nbnr99df166+/ys/PT9Lvl2qZMmWKevXqla8DAgAA4O7kKfh69eqlq1evKi4uTikpKZKkcuXKadCg\nQerZs2e+DggAAIC7c8fgy87O1tq1a9WgQQNFRkYqIiJCFy9e1KpVq1SmTBm1aNEi1zdwAAAA4P5y\n28/wXb16VX379tXIkSP1ww8/SJKcnJxUpkwZHTlyRKNHj9bAgQN1/fp1uw0LAACAv+62wRcfH68z\nZ85o1apVqlu3bo5t//znP7Vs2TL99NNPmjlzZr4PCQAAgP/dbYNvzZo1euWVV/Too4/ecvvf/vY3\nvfTSS/r888/zbTgAAADcvdsG36+//qpq1ard8cGPP/64zpw5c8+HAgAAwL1z2+CrUKGCjh8/fscH\nnzhxQmXKlLnnQwEAAODeue1Zus8884zi4uIUGBgoi8WSa/v169f14YcfqlGjRvk6IADgwXf16lUl\nJSU5egw84Hx8fFS0aFFHj/FAum3wDRw4UJ06dVK7du3UvXt3+fn56aGHHlJqaqr279+v+fPnKzs7\nW88//7w95wUAPICSkpLU77X5eqi0l6NHwQPq8sUT+mh8d9WpU8fRozyQbht87u7u+vTTTxUTE6P3\n3ntPaWlptm0lSpRQWFiYoqKiVKpUKbsMCgB4sD1U2kslPKo6egygQLrjhZeLFy+uN998U6+++qqS\nk5OVmpqqUqVKqVKlSnJ2drbXjAAAALgLefpqNYvFoqpV+b8yAACAB9Ftz9IFAACAGQg+AAAAwzkk\n+BITE9WgQQPb7QMHDqhGjRoKCAiw/cyYMcMRowEAABgnT5/hu1esVquWLl2qd955Ry4uLrb1w4cP\nq1GjRpo+fbo9xwEAACgQ7LqHb/r06Zo3b54iIyNltVpt64cOHZKPj489RwEAACgw7LqHr0OHDoqM\njNSOHTtyrB8+fFhFihRRaGiobty4oebNm2v48OG3/IYPAAAA/DV23cNXrly5W66XLl1aTZo00erV\nqzV37lzt2LFDkydPtudoAAAAxrovztKdNm2aevfuLVdXV1WsWFERERFat26do8cCAAAwgsODLzU1\nVRMmTMjx1W3p6elydXV14FQAAADmcHjwPfTQQ9q4caPi4uKUlZWl48ePKz4+Xu3atXP0aAAAAEZw\nWPA5OTn9PkChQoqPj9cPP/yg4OBgdevWTc8++6x69uzpqNEAAACMYtezdG+qV6+etm3bZrvt7e2t\njz/+2BGjAAAAGM/hh3QBAACQvwg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiC\nDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8A\nAIDhCD4AAADDEXwAAACGI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIPAADAcAQfAACA4Qg+AAAA\nwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj\n+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADDEXwAAACGI/gAAAAMR/AB\nAAAYjuADAAAwHMEHAABgOIcEX2Jioho0aGC7nZqaqqioKAUGBqpx48ZKSEhwxFgAAABGKmzPF7Na\nrVq6dKneeecdubi42NZff/11ubu7a+vWrUpKSlL//v1VvXp1+fv723M8AAAAI9l1D9/06dM1b948\nRUZGymoyK8kKAAAUHUlEQVS1SpLS0tK0YcMGDR48WBaLRX5+fgoLC9Py5cvtORoAAICx7Bp8HTp0\n0IoVK+Tr62tbO378uAoXLiwvLy/bWpUqVXT06FF7jgYAAGAsuwZfuXLlcq1dvXpVrq6uOdZcXV2V\nnp5ur7EAAACM5vCzdN3c3HT9+vUca+np6SpatKiDJgIAADCLw4OvcuXKyszM1OnTp21rx44dU7Vq\n1Rw4FQAAgDkcHnzu7u4KDQ1VTEyM0tPTlZiYqFWrViksLMzRowEAABjBYcHn5ORk+/e4ceOUlZWl\nRo0aaejQoRo1apT8/PwcNRoAAIBR7Hodvpvq1aunbdu22W6XKFFCkyZNcsQoAAAAxnP4IV0AAADk\nL4IPAADAcAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAE\nHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4A\nAADDEXwAAACGI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIPAADAcAQfAACA4Qg+AAAAwxF8AAAA\nhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH\n8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADDEXwAAACGI/gAAAAMR/ABAAAY7r4K\nvo8++ki+vr4KCAiw/ezevdvRYwEAADzQCjt6gD86fPiwRo4cqT59+jh6FAAAAGPcV3v4Dh8+LB8f\nH0ePAQAAYJT7JviuXbumY8eO6ZNPPlFISIhatGihpUuXOnosAACAB959c0j3woULqlu3rrp27aon\nnnhC+/btU2RkpMqVK6eGDRs6ejwAAIAH1n0TfF5eXpo3b57tdmBgoFq3bq3169cTfAAAAHfhvjmk\ne/DgQcXHx+dYS09Pl6urq4MmAgAAMMN9E3zu7u6aOnWqvvzyS924cUPbtm3TmjVr1LZtW0ePBgAA\n8EC7bw7pVqlSRbGxsYqJidHo0aP18MMPKzo6WjVq1HD0aAAAAA+0+yb4JKlRo0Zq1KiRo8cAAAAw\nyn1zSBcAAAD5g+ADAAAwHMEHAABgOIIPAADAcAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAA\nGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAc\nwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADDEXwAAACGI/gAAAAMR/ABAAAYjuADAAAwHMEHAABgOIIP\nAADAcAQfAACA4Qg+AAAAwxF8AAAAhiP4AAAADEfwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAA\ngOEIPgAAAMMRfAAAAIYj+AAAAAxH8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDhCD4AAADD\nEXwAAACGu6+C79ChQ+rQoYMCAgLUpk0b7d+/39EjAQAAPPDum+C7fv26IiIi1KFDB+3atUs9evRQ\nZGSkrl696ujRAAAAHmj3TfBt375dzs7O6tKli5ydndW+fXuVKVNGmzZtcvRoAAAAD7T7JviOHTum\nqlWr5ljz9vbW0aNHHTQRAACAGe6b4Lt69arc3NxyrLm5uSk9Pd1BEwEAAJihsKMHuKlo0aK54u7a\ntWsqVqzYHR+XnZ0tSTpz5ky+zfYgO3funFLP/qjs65cdPQoeUFdSTuvcuVo6ceKEo0fBA4y/Rbhb\n/C26s5sddLOL/tt9E3yPPvqo5s+fn2Pt2LFjatWq1R0fd+7cOUlSt27d8m02E6T/4ugJ8CAbMmSj\no0eAIfhbhLvB36I/d+7cOVWuXDnX+n0TfMHBwcrIyND8+fPVuXNnrVixQhcvXlRISMgdH+fr66sF\nCxaoXLlycnZ2ttO0AAAA94/s7GydO3dOvr6+t9zuZLVarXae6bZ++OEHjRkzRj/++KOqVKmisWPH\nys/Pz9FjAQAAPNDuq+ADAADAvXffnKULAACA/EHwAQAAGI7gAwAAMBzBBwAAYDiCDwAAwHAEHwAA\ngOEIPgAAHGDHjh0aOXKkevTooQsXLig2Nva2X4sF3C2CDwXODz/8kGstPT1d0dHRDpgGQEG0bNky\nvfDCC/L29tb3338vJycnrVu3jr9DyDcEHwqcPn36KCkpyXZ727ZtCgsL0+bNmx04FYCCJD4+XjNn\nztTzzz8vZ2dnlS5dWrNmzdLq1asdPRoMRfChwBk9erT69eunHTt26NVXX1VkZKQ6dOig5cuXO3o0\nAAVESkqKqlWrlmOtdOnSysrKctBEMF1hRw8A2FurVq3k5uam8PBw1axZU59//rkqVqzo6LEAFCAB\nAQGaPHmyhg0bZlubO3euateu7cCpYDK+SxcFxoIFC3Lc3rp1q7777jtFRUXJxcVFktStWzdHjAag\ngDl16pQiIiJ05swZXblyRR4eHnJ1ddWMGTP4H1DkC4IPBUaPHj3+9D7z5s2zwyQAIN24cUOJiYk6\ndeqUypcvr9q1a6twYQ68IX8QfChwVq5cqWeeeUaurq6OHgVAAZWSkqLx48crMjJSVatW1ZQpU3T8\n+HG98cYbcnd3d/R4MBAnbaDAGTdunJydnR09BoAC7NVXX5X0+4kaktS6dWtZrVaNGTPGkWPBYOzh\nQ4Hz8ssv65FHHlGrVq1UtmxZOTk52ba5ubk5cDIABUVQUJC2bNkii8ViW7t+/boaNmyoHTt2OHAy\nmIoPC6DAWbt2rdLS0hQXF5dj3cnJSYcPH3bQVAAKEjc3N506dUpVqlSxrf36668qVqyY44aC0Qg+\nFDgrVqxw9AgACrhOnTqpf//+6tmzpypUqKAzZ85o/vz56tKli6NHg6E4pIsC6dq1a0pNTdWNGzck\nSVlZWTpy5IgaN27s4MkAFAQ3btzQ4sWLtXr1al24cEEeHh4KCwtT+/btHT0aDEXwocBZvHix3n77\nbV2/fj3HevXq1fX55587aCoAAPIPh3RR4EybNk3jx4+XxWLR119/rcGDBys6OlpNmzZ19GgACoj/\nvi7oH08emzt3rr3HQQFA8KHA+e233xQWFqYzZ85o2rRp8vT01NixY9W3b1+FhYU5ejwABUCzZs1y\n3E5JSdHKlSvVtm1bB00E0xF8KHA8PDxsn5k5ffq0MjIyVLx4cSUnJzt6NAAFRPfu3XOttW7dWkOH\nDlVkZKQDJoLpCD4UOM2aNVOvXr00Z84cBQcH68UXX5TFYlHVqlUdPRqAAszDw0O//PKLo8eAoThp\nAwXOjRs3tHLlSjVt2lSZmZmaOHGi0tLSNGTIkBzXxAKA/LJgwYIctzMzM7V582ZlZWXpk08+cdBU\nMBnBhwLr5MmTOn/+vCpUqCAPDw9HjwOgAPnvkzacnZ1VpUoVDRgwQJ6eng6aCiYj+FDgnD59WsOG\nDdP+/fvl5uama9euqX79+po4caJKlSrl6PEAALjnCD4UOP369ZOHh4dGjx6t4sWL69KlS3r33XeV\nlpam2NhYR48HwGD/93//l+MSLLfSpk0bO02DgoTgQ4ETEBCgHTt25PjS8vT0dIWEhGjXrl0OnAyA\n6fr16ycnJyelpaVp79698vf3l5eXl86cOaM9e/aoYcOGio+Pd/SYMBBn6aLAqVixog4fPix/f3/b\nWnJysipVquTAqQAUBB999JEkaciQIYqOjlbr1q1t27744gt99tlnjhoNhmMPHwqcd955R8uWLVPH\njh1VqVIlnT17VkuWLFFgYKB8fHxktVrl5OSkiIgIR48KwFABAQHavXu3ChUqZFvLyspSUFCQ9u7d\n68DJYCr28KHASUlJUWhoqC5evKiLFy8qOztbDRo0kCQdP37cwdMBKAi8vLy0atUqtWrVyra2bNky\neXt7O3AqmIw9fChwkpOTFRcXp+joaH399dcaPny4SpQoodjYWPn5+Tl6PAAFwLZt2xQVFaUqVaqo\nfPnyOnXqlM6ePauZM2fydwj5guBDgRMeHq4KFSpo3Lhxat68uTp06KCHHnpICQkJSkhIcPR4AAqI\nixcvauPGjTp//rw8PDzUuHFjlShRwtFjwVAc0kWBc+jQIcXHx+vYsWM6efKkunfvLjc3N0VHRzt6\nNAAFSIkSJVShQgVZrVY9++yzOn36NMGHfEPwocApVKiQrly5og0bNiggIEBubm5KTk6Wu7u7o0cD\nUED8/PPPGjhwoLKysnTx4kXVq1dPYWFhmjRpkkJDQx09HgxU6M/vApildevWatu2raZMmaKuXbsq\nKSlJPXr0ULt27Rw9GoACYuzYseratas2bNigwoULq2LFipo4caImTZrk6NFgKD7DhwJp27Ztslgs\nqlu3rs6ePas9e/bo2WefdfRYAAqIoKAgbd++Xc7OzgoKCtLOnTtltVpVt25d7dmzx9HjwUAc0kWB\nVL9+fdu/PTw8iD0AduXh4aEDBw6odu3atrXDhw/L09PTgVPBZAQfAAB2NmTIEPXv319t2rRRRkaG\nYmNjtWTJEr388suOHg2G4pAuAAAOkJiYqISEBJ0+fVrly5dXmzZtFBQU5OixYCiCDwAAO4uIiND7\n77/P1QFgN5ylCwCAnR08eNDRI6CA4TN8AADYWUhIiLp166bGjRurXLlyObZ169bNQVPBZAQfAAB2\ndvLkSRUvXly7d+/OtY3gQ37gM3wAAACGYw8fAAB2ZrVatXjxYq1evVrnzp2Tp6en2rRpo1atWjl6\nNBiK4AMAwM5iY2O1YsUK9ezZUxUqVNCpU6c0adIkXbx4Ub1793b0eDAQh3QBALCz+vXr69NPP1Xl\nypVta0ePHlWvXr20efNmB04GU3FZFgAAHKBMmTI5bj/88MMOmgQFAcEHAICdRUREaPjw4Tpy5Igy\nMzOVnJysV155RS1bttTZs2dtP8C9wiFdAADszMfH50/v4+TkpMOHD9thGhQEBB8AAIDhOEsXAAAH\nSE5O1tmzZ3Vzv0tWVpaOHDmi7t27O3gymIjgAwDAzuLi4jRlyhS5urpK+j32MjMz1bBhQ4IP+YLg\nAwDAzhYtWqR58+YpMzNTy5cv1/jx4xUdHS0vLy9HjwZDcZYuAAB2dv36dQUGBqp69eo6ePCgXFxc\nNHz4cC1ZssTRo8FQBB8AAHb28MMP65dfflHZsmV14cIFXblyRYULF+ZSLMg3HNIFAMDOOnbsqH/8\n4x9atmyZnnnmGfXv318uLi7y9/d39GgwFJdlAQDAAXbt2qXHH39c2dnZmj17tq5cuaJevXqpQoUK\njh4NBiL4AACws1WrVumtt97SpUuXcqxzsWXkF4IPAAA7CwkJUVRUlEJCQuTk5JRjG2fqIj/wGT4A\nAOwsOztbnTp1krOzs6NHQQHBHj4AAOwsPj5eKSkp6t27t4oXL55jm5ubm4OmgskIPgAA7CwhIUFj\nxoxRdnZ2jnU+w4f8QvABAGBnwcHBGjlypOrVq6dChXJeEpfP8CE/8Bk+AADsrFChQmrXrh2f4YPd\nsIcPAAA7mzNnjk6dOqU+ffqoRIkSOc7U5TN8yA8EHwAAdlavXj2lpqbmWuczfMgvBB8AAHZ24sSJ\n227jM3zIDwQfAACA4Qr9+V0AAADwICP4AAAADEfwAQAAGI7gA4D/yMrK0vTp09WsWTM9/vjjevLJ\nJzVq1CidOnVKkjR58mR17tz5lo/dsWOHfHx8lJGRoRMnTsjHx0fHjh2TJPn4+Ojbb7+12/sAgP9G\n8AHAf0ycOFHLly/X66+/ri+//FJTp07V+fPn1b17d6Wnp9/xsXXq1NGWLVtksVjsNC0A5B3BBwD/\nsXTpUg0ZMkQhISHy9PSUv7+/PvzwQ/3666/atGnTHR/r4uKiMmXK2GlSAPhrCD4A+I9ChQpp69at\nOb7Q3t3dXatXr1ZISEiO+2ZnZ2vQoEEKCwvTb7/9luOQ7p3s3LlT7dq1k7+/v5566ilNmTIlX94L\nAPwR36ULAP/Rp08fTZw4UV9//bUaNmyoevXqqUGDBqpcubLtPje/Auv111/X0aNHtWDBAhUvXjxP\nz5+dna3nn39eXbt2VVxcnH766ScNHTpUNWrUUJMmTfLlPQGARPABgM2AAQNUqVIlLVq0SCtXrtSy\nZctUuHBh9ejRQ6NGjZIkWa1Wvffee9q+fbsWLVr0lw7jXr58WampqSpbtqw8PT3l6empOXPmyNPT\nM7/eEgBIIvgAIIfmzZurefPmSktL0/bt27V8+XLNnj1bnp6etu85TUxMlLe3t0qXLv2XnrtkyZLq\n3r27xo0bp2nTpumpp55Sq1atVL58+Xx6NwDwOz7DBwCSkpKS9Pbbb9tuFytWTKGhoZo8ebJCQ0O1\nZcsWSVKRIkW0YMECpaamasaMGX/5dV577TV98cUX6t27t44fP65evXrp008/vWfvAwBuheADAEk3\nbtzQ3LlzlZiYmGtbsWLFbHvzHn30UdWpU0cjR47UjBkz9Msvv+T5NU6ePKk33nhDHh4eCg8P17x5\n89SpUyetWbPmnr0PALgVgg8AJNWsWVNPP/20oqKitGzZMiUnJ+vQoUOKj4/X+vXr1bNnT1mtVtv9\n27Vrpxo1aujNN9/M82uULFlSX375pcaPH6+ff/5ZiYmJ2rVrl3x9ffPjLQGADcEHAP8RExOjLl26\n6OOPP1ZYWJi6d++unTt3as6cOfLx8ZGTk5PtLF3p9zN1t23bZttD98dtf/z3TcWKFdO0adN05MgR\ntW3bVgMHDlS9evU0bNiw/H9zAAo0J+sf/5cVAAAAxmEPHwAAgOEIPgAAAMMRfAAAAIYj+AAAAAxH\n8AEAABiO4AMAADAcwQcAAGA4gg8AAMBwBB8AAIDh/h/zJjwXQMuFbwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x8c9c7d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pseries = pd.Series(skill_set)\n",
    "pseries.sort(ascending=False)\n",
    "\n",
    "pseries.plot(kind = 'bar')\n",
    "## set the title to Score Comparison\n",
    "plt.title('Data Science Skills')\n",
    "plt.xlabel('Skills')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some more regular expressions:\n",
    "================="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another Example\n",
    "================\n",
    "\n",
    "Designed by Katharine Jarmul\n",
    "----------------------------\n",
    "\n",
    "https://github.com/kjam/python-web-scraping-tutorial\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Scraping Happy Hours\n",
    "====================\n",
    "\n",
    "Scrape the happy hour list of LA for personal preferences\n",
    "\n",
    "http://www.downtownla.com/3_10_happyHours.asp?action=ALL\n",
    "\n",
    "This example is part of her talk about data scraping at PyCon2014. She is a really good speaker and I enjoyed watching her talk. Check it out: http://www.youtube.com/watch?v=p1iX0uxM1w8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "stuff_i_like = ['burger', 'sushi', 'sweet potato fries', 'BBQ','beer']\n",
    "found_happy_hours = []\n",
    "my_happy_hours = []\n",
    "# First, I'm going to identify the areas of the page I want to look at\n",
    "url = 'http://www.downtownla.com/3_10_happyHours.asp?action=ALL'\n",
    "source = urllib2.urlopen(url).read()\n",
    "tables = bs4.BeautifulSoup(source)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The scraper found 63 happy hours!\n"
     ]
    }
   ],
   "source": [
    "# Then, I'm going to sort out the *exact* parts of the page\n",
    "# that match what I'm looking for...\n",
    "for t in tables.findAll('p', {'class': 'calendar_EventTitle'}):\n",
    "    text = t.text\n",
    "    for s in t.findNextSiblings():\n",
    "        text += '\\n' + s.text\n",
    "    found_happy_hours.append(text)\n",
    "\n",
    "print \"The scraper found %d happy hours!\" % len(found_happy_hours)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "YAY! I found some burger!\n",
      "YAY! I found some burger!\n",
      "YAY! I found some sushi!\n",
      "YAY! I found some sushi!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "YAY! I found some beer!\n",
      "I think you might like 12 of them, yipeeeee!\n"
     ]
    }
   ],
   "source": [
    "# Now I'm going to loop through the food I like\n",
    "# and see if any of the happy hour descriptions match\n",
    "for food in stuff_i_like:\n",
    "    for hh in found_happy_hours:\n",
    "        # checking for text AND making sure I don't have duplicates\n",
    "        if food in hh and hh not in my_happy_hours:\n",
    "            print \"YAY! I found some %s!\" % food\n",
    "            my_happy_hours.append(hh)\n",
    "\n",
    "print \"I think you might like %d of them, yipeeeee!\" % len(my_happy_hours)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Now, let's make a mail message we can read:\n",
    "message = 'Hey Katharine,\\n\\n\\n'\n",
    "message += 'OMG, I found some stuff for you in Downtown, take a look.\\n\\n'\n",
    "message += '==============================\\n'.join(my_happy_hours)\n",
    "message = message.encode('utf-8')\n",
    "# To read more about encoding:\n",
    "# http://diveintopython.org/xml_processing/unicode.html\n",
    "message = message.replace('\\t', '').replace('\\r', '')\n",
    "message += '\\n\\nXOXO,\\n Your Py Script'\n",
    "\n",
    "#print message"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Getting Data with an API\n",
    "=========================\n",
    "\n",
    "* API: application programming interface\n",
    "* some sites try to make your life easier\n",
    "* Twitter, New York Times, ImDB, rotten Tomatoes, Yelp, ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Rotten Tomatoes\n",
    "===============\n",
    "\n",
    "![The Wizard of Oz](images/wiz_oz.png \"The wizard of Oz\")\n",
    "\n",
    "http://www.rottentomatoes.com/top/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "API keys\n",
    "=========\n",
    "\n",
    "* required for data access\n",
    "* identifies application (you)\n",
    "* monitors usage\n",
    "* limits rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Rotten Tomatoes Key\n",
    "===================\n",
    "\n",
    "http://developer.rottentomatoes.com/member/register"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "api_key = rottenTomatoes_key()\n",
    "\n",
    "url = 'http://api.rottentomatoes.com/api/public/v1.0/lists/dvds/top_rentals.json?apikey=' + api_key\n",
    "data = urllib2.urlopen(url).read()\n",
    "#print data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "JSON\n",
    "======\n",
    "\n",
    "* JavaScript Object Notation\n",
    "* human readable\n",
    "* transmit attribute-value pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 1, 'b': 2}\n",
      "{\"a\": 1, \"b\": 2}\n",
      "{u'a': 1, u'b': 2}\n"
     ]
    }
   ],
   "source": [
    "a = {'a': 1, 'b':2}\n",
    "s = json.dumps(a)\n",
    "a2 = json.loads(s)\n",
    "\n",
    "## a is a dictionary\n",
    "print a\n",
    "## vs s is a string containing a in JSON encoding\n",
    "print s\n",
    "## reading back the keys are now in unicode\n",
    "print a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting keys:\n",
      "[u'movies', u'link_template', u'links']\n",
      "\n",
      "Movie keys:\n",
      "[u'ratings', u'links', u'title', u'critics_consensus', u'release_dates', u'abridged_cast', u'synopsis', u'mpaa_rating', u'year', u'alternate_ids', u'posters', u'runtime', u'id']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## create dictionary from JSON \n",
    "dataDict = json.loads(data)\n",
    "\n",
    "## explore dictionary\n",
    "print \"Starting keys:\"\n",
    "print dataDict.keys()\n",
    "print\n",
    "\n",
    "## there is a key named `movies` containing a list of movies as a value\n",
    "movies = dataDict['movies']\n",
    "\n",
    "## each element of the list `movies` is a dictionary\n",
    "print \"Movie keys:\"\n",
    "print movies[0].keys()\n",
    "print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movie rating keys:\n",
      "[u'critics_score', u'audience_score', u'critics_rating', u'audience_rating']\n",
      "\n",
      "Movie critics score rating:\n",
      "30\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## one of the keys is called `ratings`\n",
    "## the value is yet another dictionary\n",
    "print \"Movie rating keys:\"\n",
    "print movies[0]['ratings'].keys()\n",
    "print\n",
    "\n",
    "## so we made it all the way to find the critics score\n",
    "print \"Movie critics score rating:\"\n",
    "print movies[0]['ratings']['critics_score']\n",
    "print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Quiz\n",
    "=====\n",
    "\n",
    "* build a list with critics scores\n",
    "* build a list with audience scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# critics scores list\n",
    "critics_scores = [m['ratings']['critics_score'] for m in movies]\n",
    "\n",
    "# audience scores list\n",
    "audience_scores = [m['ratings']['audience_score'] for m in movies]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The following code shows how to create a pandas data frame with the data we gathered from the webpage.\n",
    "Beware of the `set_index()` function in pandas. Per default it does not change the actual data frame! You need to either reassign the output or set the `inplace` argument to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>critics</th>\n",
       "      <th>audience</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Insurgent</th>\n",
       "      <td>30</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mad Max: Fury Road</th>\n",
       "      <td>97</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Home</th>\n",
       "      <td>45</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    critics  audience\n",
       "Insurgent                30        62\n",
       "Mad Max: Fury Road       97        88\n",
       "Home                     45        65"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## create pandas data frame with critics and audience score\n",
    "scores = pd.DataFrame(data=[critics_scores, audience_scores]).transpose()\n",
    "scores.columns = ['critics', 'audience']\n",
    "\n",
    "## also create a list with all movie titles\n",
    "movie_titles = [m['title'] for m in movies]\n",
    "\n",
    "## set index of dataFrame BEWARE of inplace!\n",
    "scores.set_index([movie_titles])\n",
    "\n",
    "## the line above does not changes scores!\n",
    "## You need to either reassign\n",
    "\n",
    "scores = scores.set_index([movie_titles])\n",
    "\n",
    "## or set the inplace argument to True\n",
    "scores.set_index([movie_titles], inplace=True)\n",
    "scores.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x141dd4b0>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoIAAAJICAYAAAD1g4XHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8jvXjx/H32GZmzOYwGZnGt33FnCbHzDHCcvxWDqGv\n5DRTKksOk/NoTokcQl+HSg4l0pcRKed8I3FTjWoMGzN2soP794eH+9dCDbt3zX29no/HHg/73Nd1\n3+/bZnu7rs/1uZysVqtVAAAAMJ1CRgcAAACAMSiCAAAAJkURBAAAMCmKIAAAgElRBAEAAEyKIggA\nAGBSFEEA+WLXrl3q3bu36tatq1q1aqljx45aunSpsrOzjY72l86ePasJEyaoVatWqlmzptq0aaPI\nyEhdvnzZ6Gj3pUWLFpoxY4bRMQAYzIl1BAHY286dOzVo0CD16NFDTZs2lYuLi7777jstXrxY7dq1\n0+TJk42OeFuHDx/WSy+9JD8/P/Xo0UMPPfSQfvrpJy1YsECenp5asWKFPD09jY55TywWi7y8vOTj\n42N0FAAGoggCsLvnn39epUqV0qxZs3KML1u2TJGRkdqxY0eBKyTp6enq0KGD/Pz8tHDhQhUq9P8n\nUE6fPq2OHTuqR48eCg8PNzAlANwfTg0DsLvExERdv379lvEOHTpo+PDhOUrW0aNH1bdvX9WuXVtN\nmzbVtGnTlJmZaXt8z549eu6552yPv/322zkeb9GihWbOnKnOnTurVq1a+vzzzyVJ//vf/9S9e3fV\nrFlTTZs21dy5c/VX/w/etm2bYmNjFR4eniOfJPn5+enVV19V1apVbWMJCQl644039MQTT6h27dp6\n6aWX9Msvv9gef+edd9SjRw+tXr1aLVq0UO3atRUaGqqrV69qzpw5atiwoRo1aqR33nnnln1WrFih\nxo0bKygoSK+++qqSkpJs22RkZGjGjBlq3bq1qlevroYNG2rkyJFKTk6WJMXGxiogIEDLly9X06ZN\nVb9+ff36669q0aKFoqKibM+zcOFCtWzZUjVq1FDbtm21cuXKHO/5xx9/1AsvvKB69eqpYcOGGjt2\nrO01pBtlPyoqSpGRkWrYsKHq1Kmj1157Lcc2AAoeZ6MDAHB8jRs31gcffKCwsDB16NBB9erVk5eX\nl0qXLq3+/fvbtouNjdXzzz+v2rVra9asWbp48aKmTJmi69ev64033tD27ds1ePBgde7cWUOHDtUv\nv/yiWbNmKTY2NsfRxiVLlui1116Tr6+vatSoIYvFoj59+uiJJ57Q3Llzdfr0ac2YMUOpqakaMWLE\nbTPv2bNHPj4+OcreH/Xu3dv25+TkZD333HNycXHRyJEj5erqqgULFqhHjx767LPPVK5cOUk3Tseu\nWrVKY8eOVXx8vN566y1169ZN/v7+ioqKUnR0tN599101atRIdevWlSSdPHlScXFxGjVqlDIzMzVt\n2jSFhoZq+fLlkqRJkyYpOjpar7/+unx9fXXkyBHNnDlTZcuW1SuvvGLLuHDhQkVERCg1NVWVKlWS\nJDk5OUmSPv30U82dO1ejRo2Sv7+/du3apQkTJqhixYpq2rSpjh49qu7du6tx48Z6++23FR8fr5kz\nZ+rkyZNatWqVrSh/+OGHevzxxzV9+nTFxsZq8uTJKleunF577bW7+4YBkG8oggDsbvjw4bp06ZI2\nbtyoLVu2yMnJSf/85z/19NNPq0ePHnJ1dZUk/ec//5GHh4cWLFggFxcXSVJqaqo2b94sSZozZ44a\nN26sKVOmSLpRMD09PRUeHi6LxaKAgABJUo0aNdSnTx/b60+ePFmVKlXS3Llz5eTkpCeeeEJFixbV\nuHHj9OKLL8rb2/uWzBcuXNBDDz2Uq/e3du1anTt3Tlu2bFH58uUlSQ0aNFDLli21ZMkSvfnmm7b3\n8vbbb6tKlSqSpM8//1wWi0UzZsyQm5ubGjRooLVr1+ro0aO2IpicnKwlS5YoMDBQklSyZEkNGDBA\nP/zwg2rUqKHLly/rzTffVPv27SVJ9erV03fffadDhw7lyNi1a1e1atXqtvkPHTokX19fPfvss5Kk\noKAgFSlSRO7u7pKk+fPnq2LFipo/f76tPPr5+alXr1766quv1LJlS0mSh4eH5s6dayuGBw4c0M6d\nOymCQAHGqWEAdlekSBFNnz5d0dHRevPNN9W0aVOdOnVKU6dOVffu3ZWamirpxunbRo0a2UqgJPXq\n1UsrV65USkqKTpw4obZt2+Z47qeeekpOTk767rvvbGOVK1fOsc2BAwfUqFEjZWdnKysrS1lZWWrS\npImysrJuKUw3FSpUKNdXNH/33XeqUaOGrQRKN0pRkyZNcuQqWrSorQRKUqlSpeTn5yc3NzfbaxYv\nXlxXr161bePr62srgZLUtGlTOTs723LPnj1b7du3V1xcnL799lstW7ZMv/zyS47T5bf7O/mjevXq\n6dSpU+rWrZvef/99nTp1SoMHD1ZQUJDt/bVu3dpWAqUbZbFMmTI53t9jjz2W4zS6j4+P0tLS/uZv\nD4CRKIIA8o2vr6969+6tBQsWaN++fQoLC9OPP/6oNWvWSJKuXLly26NzknT16lVZrVaVKlUqx7ir\nq6s8PDyUkpJiG/vzNpcvX9YHH3yg6tWr2z6aNWsmJycnxcfH3/b1ypcvr7i4uDu+l8uXLysjI8OW\nu3Tp0rds4+3tnSNXsWLFbtnmZgm8kzJlyuT43MnJSSVLltSVK1ckSQcPHlT79u3VvHlzDR8+XN98\n881tn/PPfyd/FBISosmTJ+v69euaPn26nnrqKXXv3l2//vqrpBt/97d7f6VKlcoxB/DPr1uoUKHb\nzg0FUHBQBAHY1ffff6/69evrxIkTOcZdXV01ePBgPfLIIzp9+rSkG0fRLl26lGO7y5cva8+ePfLw\n8JCTk5MuXryY4/Fr167p6tWrf7mMS4kSJdSrVy+tXbs2x8cnn3yiNm3a3HafRo0aKSEhQT/99NNt\nH4+MjFSLFi2UnZ2tEiVK3LZQXrx4USVLlrxjrtz483qF169f1+XLl+Xt7a2rV69q0KBBCggI0Fdf\nfaV9+/Zp8eLFf3n07066dOmidevWadeuXYqIiNDp06c1fvx4STf+/hISEm7ZJyEh4b7fHwBjUQQB\n2JWfn5/S09NvuQpVujH/7eLFi/L395ck1apVS3v27MlxWvPzzz/X4MGD5erqqkcffdQ2X/Cmm5/X\nqlXrjhlq166t06dP67HHHrN9FC5cWLNmzbqleN4UHBysihUratq0abecIv7pp5+0efNmtWnTRoUL\nF1bdunV19OhRnTlzJsd7++abb/4yV2789ttvtqIsSTt27FBWVpYaNGigmJgYXb16VX379rXNZ0xP\nT9ehQ4fu6kjciBEjFBYWJunGEcju3bvrySef1IULFyRJderU0ZYtW3I858GDB5WQkHDf7w+AsbhY\nBIBdlSxZUkOHDtXbb7+thIQEPf300ypdurRiY2O1dOlSlS1bVl26dJEk9enTR+vWrdPgwYPVq1cv\nnT9/XnPmzFHv3r3l6uqq0NBQhYaG6o033lD79u116tQpzZ49W08++aQeffTRO2YYOHCgevbsqZEj\nR6pdu3ZKSkrSzJkz5e7ufsejZy4uLpo0aZIGDBignj17qmfPnipdurSOHTumRYsWqWLFiho+fLik\nGxdifPDBB+rXr5/CwsLk6uqqhQsX2t7TX/nzEja3+3zIkCF65ZVXdPXqVU2fPl1t27aVv7+/rly5\nInd3d82ePVv//ve/dfXqVS1ZskQZGRm2eZe5Ub9+fY0aNUqzZ89WgwYN9Ntvv+mLL75Qr169bH9/\n3bt3ty0KHh8fr1mzZikwMFDNmjW7Y3YABV++F8EjR45oyJAh2rVrlyQpKSlJb775pvbt26fixYtr\nyJAh6tatm237qKgorVmzRtnZ2erYsaNGjhx5y5peAAq2F198UZUqVdKqVas0btw4JScnq2zZsmrV\nqpWGDh2qokWLSpIefvhhffDBB5o2bZrCwsJUqlQp9e3bV4MGDZIktWrVSnPnztXcuXO1adMmlS5d\nWr169VJoaOhfvn7NmjX1/vvva+bMmQoNDVWxYsX0xBNP6PXXX1fhwoXvuN/jjz+uDz/8UIsWLVJU\nVJQSExNVvnx5devWTQMHDrTN+fPw8NCKFSsUGRmpMWPGyMnJSUFBQZo6dartSN0fL7S4ycnJ6Zbx\nP3/+0EMPqXv37hozZoysVqtCQkL0+uuvS7pxynb27NmaNm2aBg4cqIceekjPP/+8vL29NWLECNs8\nwtu99h917dpVV69e1Ycffqj3339fpUqVUs+ePTV06FBJN67CXrZsmaKiojR06FAVL15cTz75pF59\n9dUcP49v917+7rUBGCvf7ixitVq1du1aTZ06VS4uLtqzZ48kKSwsTEWLFtWECRNksVjUv39/LVy4\nUDVr1tSKFSu0evVqLVmyRJI0YMAAPfXUU3rxxRfzIzIAGOqdd97R+vXrtX37dqOjAHBQ+XZo7b33\n3tPy5cs1aNAg2+mDlJQUbdu2TUOHDpWrq6sCAwMVEhKiTz/9VJL02WefqW/fvipdurRKly6tAQMG\naP369fkVGQAAwKHlWxHs1q2bPvvsM1WvXt029uuvv8rZ2VkVKlSwjfn5+SkmJkaSdOrUqRxrbvn5\n+enUqVP5FRkADMWpVQD2lm9F8M9rYUk3Vtn/87pTbm5uunbtmiQpLS0tx+NFixbV9evXbWt3AYAj\nCw0N1bZt24yOAcCBGXrVcNGiRW2l76b09HTbbY3c3NyUnp5ueywtLU3Ozs6221H9lfT0dB09elRl\nypT5y8ngAAAAjio7O1vx8fGqXr36bRebN7QIVqpUSZmZmYqLi7NdWXfq1CnbmmL+/v6KiYmx3V7p\nj4/9naNHj6pnz572CQ4AAPAAWblype22kX9kaBH08PBQy5YtFRUVpYkTJ+rkyZPauHGjFi1aJEl6\n+umn9f7776thw4YqXLiwFixYoI4dO+bquW+eil65cqXKlStnt/cAAABQUJ07d049e/a87RQ9yaAi\n+MfJzxMmTFBERISCg4Pl7u6u8PBw2xHAHj16KCEhQd26dVNGRoY6duyoF154IVevcfN0cLly5XJc\njAIAAGA2d5oml2/rCOa32NhYtWzZUtu2baMIAgAAU/q7PsQtOgAAAEyKIggAAGBSFEEAAACToggC\nAACYFEUQAADApCiCAAAAJkURBAAAMClD7ywC46Smpspisdzz/gEBAbZ7QgP55X6/b++Gvb/Hz507\nx73QARiOImhSFotF/UavUHHvu19s++qlWL0/sZfq1Kljh2TAnd3P9+3dsMf3+MGDBzVixAht375d\nCQkJeuqpp7R7924VLVpUERER8vLy0ssvv5xnrwcAuUERNLHi3hXk6eNvdAzgrjyo37dBQUHavn27\nJCk9PV1paWm6eWOnt956y8hoAEyMOYIAcJ/279+vrl27qnbt2urQoYO++eYbvfHGG3rllVfUokUL\ndezYUXv37lWDBg0kSV27dpUkNWnSRMePH9cbb7yhyMhISdKVK1f0+uuvq169emrUqJGmT58uScrI\nyNDIkSPVoEEDPfHEEwoLC9Ply5eNecMAHAZHBAHgPly8eFEDBw7UmDFj1KlTJ+3YsUNDhw5Vo0aN\ndPjwYa1fv15FixbVjz/+aNtn3bp1atmypb799lsVLVpUTk5OcnJykiRFREQoPT1d27dvV2Zmpnr3\n7q1KlSrJyclJMTEx2rFjhyRp6NChWr58uYYOHWrE2wZu637m8TL33BgUQQC4Dzt27JCfn586d+4s\nSWrevLk++OADLV26VA0bNlSZMmVu2efmKeE/y8jIUHR0tNasWaPixYtLkubPny9XV1ft379fp0+f\n1rp169SsWTMtXLjQVh6BguJe5/Ey99w4FEEAuA8XL15U2bJlc4wFBgbKzc1NJUuWvKvnunLlijIz\nM1WuXDnbWMWKFSVJISEhSk5O1rp16zRp0iT94x//0FtvvaXAwMD7fxNAHnpQ5/GaFXMEAeA++Pj4\n6MKFCznGFixYoIyMjLt+Lm9vb7m4uOjcuXO2sW+//VabNm3Sb7/9poYNG+qTTz7R7t27FRQUpPDw\n8PvOD8DcKIIAcB+Cg4N15swZbdiwQdnZ2dq+fbuWLFmilJSUO+7j6uoqSbp69aqkG6eKrVarChUq\npHbt2mnOnDlKTk5WfHy8pk6dqrS0NG3dulXDhw/XxYsXVbx4cbm7u8vLyytf3iMAx8WpYQAPlKuX\nYgvUa5QsWVILFizQlClTNH78eFWsWFHvvvuu1q5de8scvpufly1bVsHBwWrTpo3mz5+f42KRMWPG\naNKkSWrTpo2cnJz03HPPqVu3bsrMzNSvv/6qDh06KD09XTVq1NCUKVPy7k0DMCWKIIAHRkBAgN6f\n2CvfXiu3atWqpY8//jjHWFBQUI7P69evrz179tg+X7Bgge3PN5eVkSQPD4/bFjwXFxeNHz9e48eP\nz3UuAPg7FEEADwx3d3euKgSAPMQcQQAAAJOiCAIAAJgURRAAAMCkKIIAAAAmRREEAAAwKYogAACA\nSbF8DIAHRmpqqiwWS768VkBAgNzd3fPltQDAKBRBAA8Mi8WiQUtfVwlf+95a7cqZRM1/YTprFgJw\neBRBwAHc75GyB+noVwlfL3lVLmN0jDz3/fffa/jw4dq+fbsOHjyoESNGaPv27UbHAuDgKIKAA7BY\nLOo3eoWKe1e4632vXorV+xN7cfSrAAkKCqIEAsgXXCwCOIji3hXk6eN/1x/3Uh6R06ZNm9SlSxfV\nr19f9evXV0REhCSpRYsW2rFjh227yMhIjRw5UpJ07do1jR07VvXq1VOzZs20bds223b79u3Lcf/h\nLVu2qEOHDqpXr5769u2r06dPS5JiY2MVFBSkRYsWqUmTJmrUqFGO+xTHxcVp4MCBqlu3rpo2bapl\ny5b97XMCMBeOCALAfYiNjdXo0aP1n//8RzVq1NAvv/yiZ555Rm3btpUkOTk53Xa/d955RydOnNB/\n//tfWa1WDR48+LbbHjlyRKNGjdKiRYtUo0YNrVixQgMGDNAXX3whSUpOTtaZM2f01Vdf6dixY+rV\nq5fatWunmjVratiwYQoICNDu3bt1/vx59ejRQ//4xz/k4eFx2+fctGmTnJ35tQCYCUcEAeA++Pj4\naOPGjapRo4YSExOVmJgoT09PnT9//i/327RpkwYMGCBvb2+VKlVKoaGhslqtt2y3Zs0aderUSbVq\n1VLhwoXVp08fZWVlae/evbZt+vfvLxcXF9WsWVOPPPKITp8+rd9//11HjhzRiBEjVKRIET388MP6\n4IMPFBAQcMfn3LdvX57//QAo2PivHwDcB2dnZ61evVpr166Vu7u7qlWrpszMzNuWuj+Kj49XuXLl\nbJ/7+vredru4uDjt379fn376qW0sKytLcXFxqlSpkiTJ29s7Rx6r1aqLFy/K3d1dHh4etsf8/f3/\n9jkBmAtFEADuw8aNG7V582Z99tlnKlWqlCSpVatWkqTChQsrIyPDtu3ly5dtf/bx8dGZM2dUrVo1\nSbrjEcSyZcuqX79+CgsLs439/vvvKlu2rOLj4++Yq1y5ckpNTVVycrKtDH7xxRcqXrz4Xz4nAHPh\n1DCAB8qVM4lKPBVv148rZxJznSclJUXOzs5ycXFRRkaGFi1apNjYWGVlZcnPz09fffWVrl+/rmPH\njumrr76y7de5c2fNnz9fFy5cUGJioubNm3fb5+/UqZNWr16tY8eOyWq1auvWrWrXrt3fHr0rV66c\ngoKCFBUVpYyMDJ0+fVpTpkyRs7PzPT8nAMfDEUEAD4yAgADNf2F6vr1WbnTu3Fl79uxRixYtVKpU\nKbVt21bPPPOMYmJi9Oqrr9quDK5WrZq6dOmixMQbJXPQoEFKSUlRSEiIXFxc9K9//Utnz561Pe/N\nC0fq1aunkSNHasSIETp79qx8fX01a9Ys+fn5KTY29o4Xo0jSjBkzNH78eD3xxBNyd3dXaGioGjZs\nKEl3fE4A5kIRBPDAcHd3L3DrHRYpUkSzZ8++4+OrV6++7XjhwoUVHh6u8PBw29iwYcMkSfXr19ee\nPXts4+3bt1f79u1veY4KFSro+PHjOcbWrl1r+3PZsmU1d+7c277+nZ4TgLlwahgAAMCkKIIAAAAm\nRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIsH4O7lp2VccuSFbkVEBAgd3f3PE4EAADuBUUQdy3t\nygXN2btVJX73uqv9rpxJ1PwXphe4deAAADAriiDuSQlfL3lVLmN0DAAAcB+YIwgAAGBSFEEAAACT\noggCAACYFEUQAADApCiCAAAAJsVVw4DJsS4kAJgXRRAwOdaFBADzoggCYF1IADAp5ggCAACYFEUQ\nAADApCiCAAAAJkURBAAAMCmKIAAAgElRBAEAAEyKIggAAGBSFEEAAACToggCAACYFEUQAADApCiC\nAAAAJkURBAAAMCmKIAAAgElRBAEAAEyKIggAAGBSFEEAAACTKhBFcPv27erQoYPq1Kmjtm3bauPG\njZKkpKQkDRkyREFBQWrevLnWrFljcFIAAADH4Wx0gLS0NA0bNkxRUVF68skndfDgQfXt21e1a9dW\nZGSkPDw8tHv3blksFvXv319Vq1ZVzZo1jY4NAADwwDP8iKCTk5OKFSumrKwsWa1WOTk5ycXFRYUL\nF9a2bds0dOhQubq6KjAwUCEhIfr000+NjgwAAOAQDC+Cbm5uioyM1MiRI1W9enX16tVLY8eO1aVL\nl+Ts7KwKFSrYtvXz81NMTIyBaQEAAByH4UUwNjZWw4cP18SJE3X48GG99957mjRpklJSUuTm5pZj\nWzc3N6WnpxuUFAAAwLEYPkcwOjpa1apVU0hIiCQpODhYzZo105w5c3Tt2rUc26anp8vd3d2ImAAA\nAA7H8COCbm5utxS+woULq3r16srMzFRcXJxt/NSpU6pSpUp+RwQAAHBIhhfBZs2aKSYmRuvWrZPV\natX+/fsVHR2tDh06qGXLloqKilJ6erqOHDmijRs32o4cAgAA4P4YXgTLlSun9957Tx9++KHq1aun\nCRMmKDIyUo899pgmTJigrKwsBQcHa9iwYQoPD1dgYKDRkQEAAByC4XMEJSkoKEiffPLJLeOenp6a\nNWuWAYkAAAAcn+FHBAEAAGAMiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQo\nggAAACZFEQQAADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZF\nEQQAADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKScjQ4AAABuLzU1VRaL\n5Z73DwgIkLu7ex4mgqOhCAIAUEBZLBYNWvq6Svh63fW+V84kav4L01WnTh07JIOjoAgCAFCAlfD1\nklflMkbHgINijiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZFEQQA\nADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZFEQQAADApiiAA\nAIBJORsdAADyQ2pqqiwWyz3vHxAQIHd39zxMBADGowgCMAWLxaJBS19XCV+vu973yplEzX9huurU\nqWOHZABgHIogANMo4eslr8pljI4BAAUGcwQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQo\nggAAACZFEQQAADApiiAAAIBJUQQBAABMijuLAHdwP/em5b60AJB72VkZOn78+D3vz8/ce0cRBO7g\nXu9Ny31pAeDupF25oDl7t6rE79wLPL9RBIG/wL1pASB/8PPWGMwRBAAAMCmKIAAAgElRBAEAAEyK\nIggAAGBSFEEAAACToggCAACYFEUQAADApCiCAAAAJkURBAAAMCnuLJKHuDctAAB4kFAE8xD3pgUA\nAA8SimAe416JAADgQVEg5gieO3dOAwYMUN26dRUcHKzly5dLkpKSkjRkyBAFBQWpefPmWrNmjcFJ\nAQAAHIfhRwStVqsGDx6shg0bat68eTp16pR69uyp6tWra+nSpfLw8NDu3btlsVjUv39/Va1aVTVr\n1jQ6NgAAwAPP8CJ4+PBhxcfH67XXXpOTk5OqVKmijz/+WC4uLtq2bZv++9//ytXVVYGBgQoJCdGn\nn35KEQQAAMgDhp8a/vHHH1W1alVNmzZNTZo0UZs2bfT9998rKSlJzs7OqlChgm1bPz8/xcTEGJgW\nAADAcRheBJOSkrRv3z55eXlpx44dmjp1qiZMmKDU1FS5ubnl2NbNzU3p6ekGJQUAAHAshp8adnV1\nlaenp1566SVJUu3atfXkk09qzpw5unbtWo5t09PTWWsPAAAYypHWDTa8CD7yyCPKzs7W9evXVajQ\njQOU2dnZeuyxx/Tdd98pLi5ODz30kCTp1KlTqlKlipFxAQCAyVksFvUbvULFvSv8/cZ/cPVSrN6f\n2KtArRtseBFs3Lix3NzcNHfuXA0ZMkSHDx9WdHS0li1bpjNnzigqKkoTJ07UyZMntXHjRi1atMjo\nyAAAwOSKe1eQp4+/0THum+FFsEiRIlq+fLnGjx+vRo0aycPDQ2PGjFFgYKAmTJigiIgIBQcHy93d\nXeHh4QoMDDQ6MgAAgEMwvAhK0sMPP6zFixffMu7p6alZs2YZkAgAAMDxGX7VMAAAAIxBEQQAADAp\niiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZFEQQAADApiiAAAIBJ\nUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmFSui+D27dt14cIFSdJHH32k3r17a9q0acrIyLBbOAAA\nANhProrgu+++q+HDh+vs2bPav3+/3nrrLVWuXFk7d+7U1KlT7Z0RAAAAdpCrIvjJJ59o9uzZqlWr\nljZs2KCgoCC99dZbmjJlijZv3mzvjAAAALAD59xslJiYqKpVq0qSduzYob59+0qSSpQoofT0dLuF\nA4A/S00tMxh1AAAgAElEQVRNlcViuev9jh8/boc0APBgy1UR9Pf31yeffKLSpUsrISFBrVq1Ulpa\nmhYtWqRHH33U3hkBwMZisajf6BUq7l3hrvY7f+o7PRxip1AA8IDKVREcOXKkQkNDlZSUpF69esnP\nz09jx45VdHS05s+fb++MAJBDce8K8vTxv6t9ki/FSvrNPoEA4AGVqyJYr1497d69W8nJyfL09JQk\nDRgwQCNGjJCHh4ddAwIAAMA+cr18zOXLl7Vq1SqFh4crISFBhw8f1tmzZ+2ZDQAAAHaUqyJ47Ngx\nPfnkk9q5c6c2bdqk1NRU7d69W926ddPu3bvtnREAAAB2kKsiOGXKFPXp00cfffSRXFxc5OTkpIkT\nJ6pv376Kioqyd0YAAADYQa6K4I8//qiOHTveMt6tWzf9/PPPeR4KAAAA9perIujp6akzZ87cMn7s\n2DF5e3vneSgAAADYX66KYPfu3TV27Fj997//ldVq1YkTJ7Ry5UpFRETo2WeftXdGAAAA2EGulo95\n6aWXVKxYMU2ZMkXp6ekKDQ1V6dKlNWjQIPXp08feGQEAAGAHuSqCX375pTp06KCePXsqJSVF2dnZ\nKlGihL2zAQAAwI5ydWp4zJgxSkhIkCQVK1aMEggAAOAAclUEq1evrp07d9o7CwAAAPJRrk4Nu7q6\natq0aZo3b558fX3l5uZme8zJyUkfffSR3QICAADAPnJVBKtXr67q1avf9jEnJ6c8DQQAAID8kasi\nOHToUHvnAAAAQD7L1RxBSdq8ebO6dOmiWrVqqXr16goJCdHq1avtmQ0AAAB2lKsjgh999JGmTp2q\nXr16aciQIbp+/boOHTqkyZMny8nJSf/617/snRMAAAB5LFdFcPHixRozZoy6du1qG2vdurX8/f31\n/vvvUwQBAAAeQLk6NZyQkKCgoKBbxoOCghQbG5vnoQAAAGB/uSqCVapUUXR09C3j0dHR8vPzy+tM\nAAAAyAe5OjX88ssva+DAgTp8+LACAwMlSd9//72++uorzZ49264BAQAAYB+5OiLYpEkTLV26VFar\nVevXr9cXX3yhIkWK6JNPPlGrVq3snREAAAB2kKsjgpIUGBio0aNHy8fHR5K0e/duTgsDAAA8wHJ1\nRPCHH35Qs2bNtGzZMtvY2LFj9dRTT+nkyZP2ygYAAAA7ylURnDRpktq3b6/hw4fbxrZs2aJWrVpp\nwoQJdgsHAAAA+8lVEbRYLOrTp49cXFz+f8dChdS7d2/98MMPdgsHAAAA+8lVESxbtqwOHTp0y/ix\nY8fk5eWV56EAAABgf7m6WKRv374aN26cfvrpJ9WoUUPSjRK4cuVKhYaG2jUgAAAA7CNXRbBHjx4q\nUqSIVq1apZUrV8rFxUV+fn4aP3682rVrZ++MAAAAsIO/PDWcnp6uLVu2KCUlRV27dtXatWs1fPhw\n1alTR/7+/nrkkUfyKycAAADy2B2L4NmzZ9WuXTsNHz5cCQkJkqTIyEhNmjRJhQsXVnZ2trp3764j\nR47kW1gAAADknTueGp49e7YqV66sDRs2yMPDQ5cuXdLy5cvVunVrvfPOO5KkBQsWaM6cOVq8eHG+\nBQYAAEDeuOMRwW+++UZhYWHy8PCQJO3atUtZWVnq1KmTbZsnnnjitlcTAwAAoOC7YxG8cuWKypQp\nY/t83759cnZ2VoMGDWxjxYsX1/Xr1+2bEAAAAHZxx1PD5cuXV0xMjMqXL6/s7Gx9/fXXqlu3rooV\nK2bb5sCBA6pQoUK+BAXuRWpqqiwWyz3te/z48TxOAwBAwXLHIti5c2dNnDhRYWFh2r17txISEjR6\n9Gjb40eOHNHMmTP13HPP5UtQ4F5YLBb1G71Cxb3v/j8s5099p4dD7BAKAIAC4o5F8MUXX9SVK1f0\n1ltvqVChQho+fLjatm0rSZo8ebL+85//qHXr1urfv3++hQXuRXHvCvL08b/r/ZIvxUr6Le8DAQBQ\nQNyxCDo7O2vEiBEaMWLELY916dJFnTp1UrVq1ewaDgAAAPaTqzuL/FlAQEBe5wAAAEA++8s7iwAA\nAMBxUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZFEQQAADApiiAAAIBJUQQB\nAABMiiIIAABgUgWqCCYkJKhhw4basWOHJCkpKUlDhgxRUFCQmjdvrjVr1hgbEAAAwIE4Gx3gj0aN\nGqWkpCQ5OTlJksaMGSMPDw/t3r1bFotF/fv3V9WqVVWzZk2DkwIAADz4CswRwQ8//FDu7u4qV66c\nJCklJUXbtm3T0KFD5erqqsDAQIWEhOjTTz81OCkAAIBjKBBF8NSpU1q2bJnGjRtnG/v111/l7Oys\nChUq2Mb8/PwUExNjQEIAAADHY/ip4aysLIWHh2vMmDHy9PS0jaempsrNzS3Htm5ubkpPT8/viAAA\nAPctOytDx48fv+f9AwIC5O7unoeJCkARnDdvngICAtSkSRNZrVZJktVqVdGiRXXt2rUc26anp+f5\nXwAAAEB+SLtyQXP2blWJ373uet8rZxI1/4XpqlOnTp5mMrwIbt68WfHx8dq8ebMkKTk5Wa+88or6\n9++vzMxMxcXF6aGHHpJ04xRylSpVjIwLAABwz0r4esmrchmjY9gUiCL4Ry1atFBERISCg4NlsVgU\nFRWliRMn6uTJk9q4caMWLVpkUFIAAADHUiAuFrmTCRMmKCsrS8HBwRo2bJjCw8MVGBhodCwAAACH\nYPgRwT/bvn277c+enp6aNWuWgWkAAAAcV4E+IggAAAD7oQgCAACYFEUQAADApCiCAAAAJkURBAAA\nMCmKIAAAgElRBAEAAEyKIggAAGBSFEEAAACToggCAACYVIG7xRwA4N6lpqbKYrHc8/4BAQFyd3fP\nw0QACjKKIAA4EIvFokFLX1cJX6+73vfKmUTNf2G66tSpY4dkAAoiiiAAOJgSvl7yqlzG6BgAHgDM\nEQQAADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZFEQQAADAp\niiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk3I2OgAAAHcrNTVVFovlnvcPCAiQu7t7HiYCHkwUQQDA\nA8disWjQ0tdVwtfrrve9ciZR81+Yrjp16tghGfBgoQgCAB5IJXy95FW5jNExgAcacwQBAABMiiOC\nAADY2b3OaTx+/Lgd0gD/jyIIAICdWSwW9Ru9QsW9K9zVfudPfaeHQ+wUChBFEACAfFHcu4I8ffzv\nap/kS7GSfrNPIEDMEQQAADAtjgj+yf2sTcVcDgAA8CChCP7Jvc7jkJjLAQAAHiwUwdu4l3kcEnM5\nAADAg4U5ggAAACZFEQQAADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQo\nggAAACZFEQQAADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZF\nEQQAADApiiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQoggAAACZFEQQAADAp\niiAAAIBJUQQBAABMiiIIAABgUhRBAAAAk6IIAgAAmBRFEAAAwKQKRBE8ePCg/vWvfykoKEitW7fW\nxx9/LElKSkrSkCFDFBQUpObNm2vNmjUGJwUAAHAczkYHSEpK0uDBgxUREaH27dvr2LFjeuGFF/Tw\nww/rww8/lIeHh3bv3i2LxaL+/furatWqqlmzptGxAQAAHniGHxGMi4tT8+bN1b59e0lStWrVVL9+\nfR06dEjbtm3T0KFD5erqqsDAQIWEhOjTTz81ODEAAIBjMLwIBgQEKDIy0vZ5UlKSDh48KKvVKmdn\nZ1WoUMH2mJ+fn2JiYoyICQAA4HAML4J/dPXqVQ0cOFDVq1dXgwYN5ObmluNxNzc3paenG5QOAADA\nsRSYIvj777/rueeek5eXl+bOnauiRYvq2rVrObZJT0+Xu7u7QQkBAAAcS4Eogj/++KOeffZZNW3a\nVPPmzZOrq6sqVaqkzMxMxcXF2bY7deqUqlSpYmBSAAAAx2F4EUxISNCLL76of//73woPD7eNe3h4\nqGXLloqKilJ6erqOHDmijRs3KiQkxMC0AAAAjsPw5WPWrFmjxMREvfvuu3r33Xdt43369NGECRMU\nERGh4OBgubu7Kzw8XIGBgQamBQAAcByGF8GBAwdq4MCBd3x81qxZ+ZgGAJCfUlNTZbFY7nq/48eP\n2yENYD6GF0EAgHlZLBb1G71Cxb0r/P3Gf3D+1Hd6mJlCwH2jCAIADFXcu4I8ffzvap/kS7GSfrNP\nIMBEDL9YBAAAAMagCAIAAJgURRAAAMCkKIIAAAAmRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIU\nQQAAAJOiCAIAAJgURRAAAMCkKIIAAAAmRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIUQQAAAJOi\nCAIAAJgURRAAAMCkKIIAAAAmRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIUQQAAAJOiCAIAAJgU\nRRAAAMCkKIIAAAAmRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIUQQAAAJOiCAIAAJgURRAAAMCk\nKIIAAAAmRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIUQQAAAJOiCAIAAJgURRAAAMCkKIIAAAAm\nRREEAAAwKYogAACASVEEAQAATIoiCAAAYFIUQQAAAJOiCAIAAJgURRAAAMCkKIIAAAAmRREEAAAw\nKWejAwAAbpWamiqLxXLX+x0/ftwOaQA4KoogABRAFotF/UavUHHvCne13/lT3+nhEDuFAuBwKIIA\nUEAV964gTx//u9on+VKspN/sEwiAw2GOIAAAgElRBAEAAEyKIggAAGBSFEEAAACToggCAACYFEUQ\nAADApCiCAAAAJkURBAAAMCmKIAAAgElRBAEAAEyKIggAAGBSFEEAAACTeiCK4LFjx9StWzfVrl1b\nnTp10uHDh42OBAAA8MAr8EXw2rVrGjhwoLp166aDBw/q+eef16BBg5Sammp0NAAAgAdagS+Ce/fu\nVeHChfXcc8+pcOHC6tq1q0qVKqWdO3caHQ0AAOCBVuCL4KlTp+Tv759jrHLlyoqJiTEoEQAAgGMo\n8EUwNTVVRYsWzTFWtGhRpaenG5QIAADAMTgbHeDvuLu731L60tLSVKxYsb/cLzs7W5J07ty5u3q9\n+Ph4JZ0/qexrV+8uqKSUS7+p0M/xyrqacVf7XT2fpPja8YqNjb3r17xXvM+/x/v8a0a8T+ne3+u9\nvk/pwfqa8j7/Hu/Tfniff82I93mzB93sRX/mZLVarXedJh99/fXXGj9+vKKjo21jISEhGjZsmFq1\nanXH/Q4ePKiePXvmR0QAAIACbeXKlQoKCrplvMAfEWzQoIEyMjK0YsUKPfvss/rss8906dIlNWnS\n5C/3q169ulauXKkyZcqocOHC+ZQWAACg4MjOzlZ8fLyqV69+28cL/BFBSTpx4oQiIiJ08uRJ+fn5\nady4cQoMDDQ6FgAAwAPtgSiCAAAAyHsF/qphAAAA2AdFEAAAwKQoggAAACZFEQQAADApiiAAAIBJ\nUQQBAChgMjIy9Ouvv+r69eu6fv260XHgwAr8gtIFmcViUUBAwC3j+/btU/369Q1IlLdGjhz5t9tM\nmTIlH5Lkn3Xr1mn16tWKj4/XqlWrFBUVpYiIiL+9peGDJi0tTUlJSbZfMJmZmYqJiVHz5s0NTpa3\nTp8+rfXr1+v8+fMaOXKkoqOj1bVrV6Nj5YmMjAzNmzdP//vf//TPf/5TL730kry9vW2PP//881q+\nfLmBCfPenj17dOHCBd1c9ezm9214eLjByfJOcnKyJk6cqA0bNsjFxUUbNmxQv379tGDBAvn7+xsd\nL0/t2bNHDRs21MWLFzVnzhx5eXlp4MCBcnNzMzrafftjN3ByctKfV+pzcnLS8ePH8zvWbXFE8C5l\nZ2crLS1Nqamp6tGjh9LS0nJ8XLhwQQMHDjQ6Zp6oXLmyKleuLEn66quvVLp0adWqVUvlypXTrl27\nHOIf6x8tXLhQS5cu1bPPPqvLly/L3d1dFy5c0Pjx442Olqc+/vhjNWjQQM2aNVOLFi3UokULtWnT\nRjNmzDA6Wp7auXOnnn32WV24cEFbtmzRtWvXNGvWLL333ntGR8sT06ZN0549e9S6dWudPHlS3bp1\n05kzZ2yPHz161MB0eS8iIkKhoaGaPXu2oqKiNHfuXI0ZM0YpKSlGR8tTkydPVkZGhrZu3SpXV1dV\nqFBBbdq0cbifQ9OnT9fIkSNltVo1evRoxcTE6PDhwxo3bpzR0fLErl27tGvXLoWGhqpZs2b64osv\ndOTIEW3dulVPPfWUwsLCjI74/6y4K3FxcdZatWpZH3300Tt+vPTSS0bHzFOdO3e2Hj16NMfYiRMn\nrE899ZRBieyjefPm1tjYWKvVarUGBQVZrVarNTEx0fr4448bGSvPBQcHWzds2GD98ssvrW+88Yb1\nzJkz1rCwMOuGDRuMjpanOnToYN23b5/Vav3/r+dPP/1kbdq0qZGx8kzjxo2t8fHxts8nTJhgbdu2\nrTUpKclqtVqttWrVMiqaXTz++OPWn3/+2XrgwAHrsGHDrFar1bpw4ULrzJkzDU6Wtxo2bGhNTk62\nWq3//32bkZFhrVu3rpGx8lyrVq2s586dsyYlJVmrVatm/e2336zJycnWevXqGR0tTzVs2NB69erV\nHGMpKSnW+vXrG5ToVhwRvEvlypXT1q1bFR0dLU9PT23btk3R0dG2j6+//loLFiwwOmaeOnXqlKpW\nrZpjrFKlSjp37pxBiewjPT1dpUqVyjFWtGhROTs71gyKK1euKCQkRDVr1tTx48dVvnx5jRs3TkuW\nLDE6Wp46d+7cLTdYr1y5slJTUw1KlLcyMzNVokQJ2+ejRo1S1apVFRYWpqysLAOT2Y+/v7/8/f11\n7NgxSVKfPn20efNmg1PlLTc3NyUnJ+cYu3z5co6vtSO4fPmyfHx8tHv3bvn6+qpixYpycXG55RTq\ng85qtSoxMTHH2Llz5wrU7xWK4D0oXbq0KlSooH379snX11cVKlSwfZQtW9boeHmuVq1amjx5su2H\nU1JSksaNG+cQ8yD/qHHjxho3bpwuX74s6cYv2qioKDVo0MDgZHnLx8dHFy9elI+Pj+Li4pSRkaES\nJUro999/NzpangoICNDHH3+cY+zLL7/Uo48+alCivFW3bl1NmzZNFy9elHRjztG0adOUnJysoUOH\nOtwvVF9fX/3www/y8vJSamqqLl68qPT0dNv7dxRdunTRwIEDFR0drezsbO3bt0/Dhw/X008/bXS0\nPPXPf/5TkyZN0nvvvadWrVopMTFRb731lgIDA42OlqeeeeYZ9evXT8uXL9fWrVu1bNkyvfjii+rT\np4/R0Wy41/B9OHbsmKKiomxXdt3k5OSkbdu2GZgsb509e1Yvv/yyjhw5Ind3d6Wmpurxxx/XzJkz\nbzmC9iC7fPmyRowYoV27dslqtcrZ2Vl169bVjBkzHOp9zpo1S9HR0Vq2bJkmTJggSXJ1ddVvv/12\nS3F6kJ04cUL9+vVTqVKl9Msvv6hWrVqKiYnR4sWLVa1aNaPj3bdz587p9ddfV7FixXLMe7x8+bKG\nDh2qAwcOyGKxGJgwb23atEmjR4/Wxo0btXr1am3evFnOzs6qWrWqZs+ebXS8PJOdna2lS5dqzZo1\niouLU9myZdWpUycNGDCgQB1Ful9nz57VzJkz5ebmplGjRunEiROaN2+eIiIiVL58eaPj5Rmr1aqP\nPvpImzZtUnx8vHx8fNSlSxd16tTJ6Gg2FMH70LVrV1WqVEnt2rVT4cKFczzmaFdfStLvv/+u+Ph4\nlStXzqH+of5ZfHy87QdwuXLljI6T565fv64NGzaoVatWyszM1IwZM5SSkqKwsDD5+fkZHS9PJScn\na+fOnTp79qzKli2rZs2aydPT0+hYeSojI0Ourq45xqxWq/73v/+pTp06BqWyj9jYWPn4+MjZ2Vkb\nN25USkqKOnfurCJFihgdDXdp/fr1Cg4OznGlu6O73b/VgoAieB9q166tvXv3muKH0N69e3X+/HmH\nXrbh+vXr2rVrl86ePXvLul09e/Y0KBXuR0ZGhi5dunTL19OR/yPjyLKysnTx4kVlZ2fnGHeEr+fc\nuXP/dpvQ0NB8SJI/+vXrp0OHDunRRx9V8+bN1axZM4eZtvFHmZmZeu+997R69WpdvXpVGzduVHh4\nuGbNmqUyZcoYHU8S6wjel2rVqun06dMO+c37RxEREdq4caM8PT2VmZmpIkWKKDY2Vs8884zR0fLU\na6+9pt27d6tKlSpycnLK8ZgjFUGzTGn47LPPNGHChFsm3hek9buQexs2bFBERITS0tJyjDvK1/PC\nhQtycnJSQkKCvv76a7Vu3VoVKlTQuXPntGXLFrVu3droiHnq/fffV1pamvbu3WtbZiU7O1vNmjXT\n2LFjjY6XZ2bMmKGjR48qKipKgwcPVunSpVW+fHlFRERo3rx5RseTxBHB+3Jz0c+WLVuqZMmStnEn\nJyeNGDHCwGR5q379+lq1apUSExO1YsUKzZo1S4sWLVJKSopefvllo+PlmTp16uiLL75wyNPBf2SW\nKQ2tW7fW888/r44dO94yt8rRFgg3g+DgYA0aNEht27aVi4tLjscc6evZu3dvDRw4UI0aNbKNHTx4\nUJGRkfrkk08MTGYfMTEx2r9/v/bu3asdO3bI29tb27dvNzpWnnniiSf02WefydvbW/Xq1dOBAweU\nlpam4OBg7d+/3+h4kjgieF+Sk5PVokUL2+XhN1cP//PRJEfg7++vxMTEHMs2hISEOFQR9PX1VaFC\njn8hfUxMjFatWuXwUxouXbqkXr16meJrKt04BZWUlKSSJUs61EUFN2VmZuq5554zOobd/fDDD7es\nyBAYGKiff/7ZoET2MXz4cB04cECFChVSUFCQGjdurFdffVUVK1Y0OlqecnJyuuU/3FlZWQXqhgyO\n99MiH02dOtXoCPni5rINNWrUsC3b4OLi4nDLNowbN079+vVT+/btVbx48RyPOdKpYbNMaWjZsqU2\nbtzocMtu/NmVK1c0btw4bd26VZmZmXJzc1OHDh00evToAvXL5n61atVKGzZscPiv5z/+8Q8tXrxY\nAwYMkHRj7vL8+fNVvXp1g5PlrbNnzyo1NVWNGjVS7dq1VatWLYcrgZL05JNPavjw4XrttddktVoV\nGxurt99+Wy1btjQ6mg2nhu/Tn+9NO2PGDI0dO9ahTlWYZdmG0NBQ7d+/X1WrVr3lKJIj3bPVLFMa\n+vXrp2+//Vbly5eXl5eXbdzJyUlr1qwxMFneGjZsmFJTUzV8+HD5+PjYluUoX768bXmgB9nNe0Nf\nu3ZNP//8sypWrJhjcWVH+3paLBYNGjRIaWlpKlOmjC5cuKCSJUtq4cKFqlSpktHx8lRKSooOHjyo\nvXv3at++fbpw4YLq1q3rUL9X0tPTFRkZqfXr1ys9PV0uLi4KCQnRqFGjCkxPoAjeh4ULF+rzzz/X\nv//9b02cOFE7duzQ0KFD5ePjo8jISKPj5SkzLNtQu3Ztbdu2zeGXM3jjjTdyfP7HKQ1TpkwxKFXe\nW7du3R0f69KlSz4msa+6detq586d8vDwsI1duXJFLVu21IEDBwxMljf+6uso3fj+7dy5cz6lyR8Z\nGRk6dOiQbbmuOnXq3HJ60VH8/vvv2rNnj3bv3q09e/bo4Ycfdsi5kDenkHl6eha4ryVF8D60aNFC\ny5cvl6+vr20S6OXLl9WmTRvt27fP6Hh5KiEhQdHR0Tp//rxKly6tVq1aycfHx+hYeerpp5/W4sWL\nHfLuMHBcrVu31sKFC1W5cmXbWFxcnF544QV9+eWXBiazj5iYGNvPoT/f+vJB9s0336hJkybasWPH\nHeeZBwcH53Mq+xk7dqx2796txMRENW7cWM2aNVNwcLDDLN6/fv36v71uoKAsKs0cwftglnvTHjly\nRP369ZO/v7/Kly+vb775RlFRUVq0aJHq1q1rdLw8065dO/Xu3VshISE5TplKjjFHMDw8XJGRkQoL\nC7P9gJJyHhF0hFMyLVq00Pbt2+94a0AnJyft2bMnn1PZz9NPP60XX3xR/fr1s90DfMmSJapTp44+\n//xz29c2JCTE6Kj35cKFCwoLC7PdZi4xMVFVq1bVggULHOI/pVOnTtXGjRs1fvz4O27jSFfTFi1a\nVBMmTFC9evUc7nemJG3cuPH/2rvzgJjz/w/gz9FBjlWhlMJvk21tjkmJnKVDxyTZHK1jW1cbkpt2\nv1uLRctuIaGwpL6bEF1CKEeoyH7dhRwdOgyhlKaa3x++zdeoFI0+M595Pf5Z85lZPYcx85r38XqD\nw+GgrKwMV69eRf/+/UXtgDIyMjBixAipKQRpRLAZli5dCgUFBaxYsQJWVla4cOECNmzYAD6fjz/+\n+IPpeBIzadIkuLi4iNbqAMDBgwcRGRmJyMhIBpNJ1tSpUxu8jw1rBHfs2IE5c+Zgy5Yt9d7P4XBY\n0bD28uXLMDY2/uCoPJvOyf7Q6/Zdsv4a9vT0RMeOHeHt7Q0VFRWUlZVh3bp1KCkpaVIzZllXU1PD\nuh3w586dw+HDh1FQUIDOnTvD0dERlpaWTMeSKE9PT4wePRpjx44VXTt27BgOHDiAXbt2MZjsf6gQ\nbAZ5OZvW2NgYaWlpYm9CVVVVMDExwdWrVxlMRgiRF4MHD0ZycrLYTujy8nIMHz4cly9fZjCZ5Ny/\nfx/3799H//79xUY5MzMz8dNPP7FqU8zhw4exfv16TJw4EVpaWsjNzcXBgwexZMkSuLi4MB1PYrhc\nLq5cuSLVn5/sG49tQbU7udh+Nq2GhgYyMjJgbGwsuvaf//yHFcc6vS8iIkLsGyqPx8P333/PdCyJ\nym4XSWoAACAASURBVM/Px7Zt25Cbm4uqqirRdQ6Hg9DQUAaTSUZDU8K12DY1DMjH67Z169Z49uyZ\n2PtOSUmJ2CYZWXbw4EH88ssv6NixI968eYM9e/agX79+2L59OwIDA1k1ig283WwZEhKCfv36ia5Z\nW1uzrhDU0dGp08YqKipKbE0v06gQbIbaxaC1srOzAQBKSkpQU1ND//79pWZ7eHN4eHhgzpw5GDdu\nHLp164a8vDwcOXIEvr6+TEeTqJ07d2L//v2YOXOm6Bvq7t278ebNG1FPLzZYsmQJFBUVYWFhIbY2\nhy2N0NmwzvFjyMvrdty4cXB3d8e8efNE70NBQUFiU26yLCQkBH5+fuDxeAgNDcWOHTugpqaGEydO\nYPXq1azbGV1cXIxvvvlG7No333yDZ8+eMZTo8/D29sbcuXOxZ88eaGhoID8/H4WFhQgJCWE6mghN\nDTfDpEmT8M8//0BDQwNaWlooLCxEQUEBtLW18ebNGwgEAmzbto0VGyrOnTuHmJgY0TfysWPHio0Q\nsoGlpSWCg4Px5Zdfiq7dv38fbm5uOHv2LIPJJMvIyAgXLlxgVbPhpqqqqkJ2djZ69+7NdBSJkZfX\nbVVVFTZv3ozo6GjR+5CTkxNmzZrFis0GXC4XGRkZ4HA4qKyshLGxMb788ksEBQWxcvZlypQpsLW1\nFduIFx4ejqNHjyI8PJzBZJL37NkzJCUlidoBmZubo2PHjkzHEpH9fz0M0tPTg7m5uehbt1AoxN69\ne5GdnY1Vq1bh77//hp+fHys2VAwfPhzDhw8X3RYKhQgNDcW0adMYTCVZL1++rNPZXldXFxUVFQwl\n+jx69+4NPp+Pbt26MR3lszp58iTWrFmDwsJCvPt9t3379qxZUwbIz+tWUVERixYtwqJFi5iO8tnU\njsorKyuDw+EgMDCQlUUg8Laf6ffff4+IiAjRCO/Tp0+xc+dOpqNJXPv27TF06FBUV1eLdhKXlZVJ\nzd8tjQg2g6mpKS5cuCDWHLKqqgpDhw5FamoqampqYGxsjIyMDAZTfroHDx5gxYoVyMrKgpmZGfz8\n/NC+fXtkZ2fD29sbd+7cwT///MN0TImZM2cO9PX1sWjRIrRq1QrV1dUICAhAZmYmgoODmY7XbLXf\nsu/cuYO0tDQ4OTmJndAAsKNNTi0bGxs4Ozujbdu2yMjIgKurKzZv3oyZM2eyqh8b21+3np6eYrdr\ni6V32x+xYTkAl8sV2zxQ25uWzZ49e4ZTp06JRnhHjhxZ5z1J1kVHR2P16tUoLS0Vu87hcHD79m2G\nUomjEcFmUFVVxYULF8RGyi5duiRaF1hQUCBVw78fy9fXF5qamvjxxx+xZ88ebN26FYMHD8aiRYtg\naGiImJgYpiNKlLe3N9zc3BAZGQlNTU0UFRWhU6dO2L59O9PRJOLd5sIaGhq4cOFCncewqRAsKirC\n7NmzkZeXh+joaJiYmOD333/HwoULWVUIsv11q6+vL9b3shaHw8Hz589ZcwqFUCjEvXv3RL+uqakR\n3a7Vq1cvJqJJ1PvPicvlinpdFhUVoaioiBXPs1ZgYCA8PT0xduxYqV3CQCOCzXDixAksW7YMw4YN\nQ9euXZGfn48LFy5g3bp16N27N6ZPn46ZM2fK7O69gQMHIjk5GR06dEBxcTFcXFxQXl6OefPmNbl3\nmayprKzElStXwOfzoa2tjX79+kntP97mqKmpgVAohIKCAp49ewZVVVXW9SiztrZGVFQUVFRUYGZm\nhosXL4LD4WDQoEGsG2mRl9dtrZs3byIsLAzx8fFQU1PDmTNnmI7UbAYGBh+8X5pGkJpDXp5nrYED\nByI9PV2q31+pEGyG0tJSFBcXIz4+HsXFxdDS0oKDgwN0dHTw5MkTFBYWYsCAAUzH/GTvT1UYGhpi\nw4YNsLW1ZTCV5DWlKDAxMWmBJC3j5s2bmDdvHgICAtC/f39s3LgRCQkJ2LFjB6u+ifv6+uLBgwfY\ntGkTlixZAj09PbRu3RpJSUmIjY1lOp5EPHv2DNnZ2WIbtwIDAzFp0iR07tyZwWSSV1VVhYSEBISH\nh+M///kPxowZg2+//RZmZmas2fFO2Kd2sOjd9jHShgrBZhg1ahTi4uJY08fqfe8XgkZGRjK73vFD\nTE1NRR8kL168qHc6/9KlSy0d67NxcXGBk5MTXF1dRc+bjbv1KioqsGvXLkyZMgUvX76Ej48PysrK\nsHLlSpn+glbr0aNH+O677zBy5Ej89ttvAN6+ft3c3MDn8xEWFlZnE4ksKioqQkREBCIjI6GmpoZJ\nkyZhy5YtiIuLY12xKw/enxquD5u+kM6YMQMpKSnQ1taGmpqa6DqHw5GaBuFUCDaDvb09Nm3axKoX\n7bvkcfGyPDzH9/9egbdTxSYmJrhy5QpDqcjH8vT0hI6ODpYtWyZ2XSgUwtfXF69evcKff/7JUDrJ\nMTQ0hK2tLVxdXcHlcgEAw4YNQ3R0NKtOcJIXjU0NA283tLFFVFRUvdc5HI7U9IZk7yKSFqCjo4Nv\nv/0Wffr0EftmypZdbBUVFfj2229Fi7Rfv34tdt6wNH2jIU2nra2N8+fPY9iwYaJrqamprGkns3Ll\nykYfs27duhZI8nldvnwZa9eurXOdw+FgwYIF4PF4DKSSPFtbW5w9exbl5eUYP348qzb6yCM2FXlN\n4ezsDODtOt78/Hx0795dtD5bWlAh2AyGhoYwNDSsc50t61Vqp5sawpbnKW88PT0xd+5cjBgxApqa\nmigsLMT58+fh7+/PdDSJqD266cGDB0hKSoKLiwt0dHRQUFCAAwcOwMrKiuGEkiEQCBo8uUhVVZU1\nfQQ3bNiAFy9e4MiRI/jzzz/h6+uLly9f4vHjx6wcEWxoeUpOTg4rpvrfdffuXcTExKCwsBCdOnWC\nvb19vZ+psqy0tBRr1qxBTEwMlJSUEBMTgxkzZmDHjh3Q09NjOh4AmhomRIw8TA0DQFZWFo4dO4an\nT5+ia9euGDNmjNjJFGzg7OyM1atXix1jlZWVBS8vLxw9epTBZJIxadIkLF26tN6Ti65cuQJfX1/W\nbIp5V0ZGBiIjI3Hs2DH07NkT9vb2mDVrFtOxJKa+tdgCgQCmpqasWqN95swZeHl5wcLCQnQ0YlJS\nEv744w9YWloyHU9ivL29UVFRgcWLF8PJyQmXLl3Cn3/+iRs3bmDv3r1MxwNAI4LNUl8LldpRstDQ\n0JaOQz5RY727AHYtXgbeni7y7jFr1dXVyMrKYtXRaw8ePIC+vr7YtR49eqCgoIChRJLl6uoKb29v\nBAYGij3PO3fuwNvbm7UtnoyMjGBkZISffvoJ0dHROHDggMwXgjk5OZg8eTKqqqrw+vVrDB48WOz+\nN2/e1Hktyzp/f38EBASITfWfOXMGGzduZFUhmJycjMTERNHovYKCAry8vDBkyBCGk/0PFYLNYGNj\nI3a7pKQEMTExUrMAlDSNg4PDB2+zra/Vu0evAf87oYFtR68NGDAAa9euxZIlS9C+fXu8ePEC69ev\nh6mpKdPRJMLR0RH379/H2LFj0b17d6ipqYHP5yMvLw9ubm6YMmUK0xE/qw4dOmDKlCmseJ66uroI\nDg7Gq1evMGvWLGzevFmsgbaysnKTNlnIkpycHLHDGIC3m4AWLlzIUKLPo02bNigtLRVbxlFSUiJV\nJ6jQ1LCE5eTkYMGCBQ3uFCKEafJy9Fp+fj68vLxw7do1tG3bFq9fv8agQYPg7+/PqrVleXl5SElJ\nwfPnz9GlSxeYmZmha9euTMcin4jP54u9PouLi6GiosK6NmXOzs7w8PAQG/07efIkgoKCWPX5GRgY\niFOnTmHu3LlYtmwZtm3bhsDAQAwcOBBeXl5MxwNAhaDEVVZWwszMjFUjK7Nnz4ajoyMsLS3Rpk0b\npuOQZuJyucjIyEBeXh68vLxw8OBBPHnyBAsXLkRERATT8SQuJycHxcXF0NTURLdu3VBYWAhNTU2m\nYxFSr5s3b2L9+vXYt28foqKi8NNPP6FNmzZ1plFlXUpKCjw8PDBkyBB069YNeXl5SE1NxdatW2Fm\nZsZ0PImprq7GX3/9JXqf1dDQgJOTE+bMmSM1p/9QIdgM7zffFQgEOHfuHKqqqqRmEagk7Nu3D3Fx\nccjKyoKlpSV4PB6GDRsm1UfmkIbJ09Fr70pLS0NYWBhOnz6NGzduMB2HkHpNmTIFgwcPxty5czFq\n1CgsXrwYqqqq+P333xEXF8d0PIm6e/cu4uPj8ezZM2hra8POzg7du3dnOpbcoUKwGd5fjK2goICe\nPXti9uzZ0NbWZijV55OTk4P4+HicOHECRUVFGDNmDBwdHdGvXz+mo5GPIA9Hr9UqLy9HTEwMwsLC\nkJOTg9GjR2PcuHFiPRQJkSampqZITU3F7du3MXnyZKSlpUFZWbneRvCyKDExkTUtnD5k+fLl8PPz\ng6enZ51Wa0KhUKr6DUvHuKSM2rdvH9MRWpSuri54PB7atGmDQ4cOITo6GufOnYOysjJ+/fVXGBkZ\nMR2RNMGKFSuwa9cucDgc+Pj4iI5eW716NdPRJCYnJwfh4eGIiopCjx49kJubi9jYWNb1YSPs07Zt\nWzx58gTHjh2DsbExlJWVcevWLbHjyWTZsmXLxAralStXsqLB+/tq23Hp6+uDw+Hg/TE3aerDSyOC\nzVBSUoJ9+/Zh/vz5uHbtGpYvXw41NTWsW7cOPXr0YDqexBQWFiIhIQEJCQm4ffs2hg8fDkdHR5ib\nm0NZWRlhYWHYvn07zp8/z3RUQjB79mxcvXoVtra2mDhxIr755htWHkmWnJzc6IcJm9aUyYs9e/Zg\n06ZNEAgECAkJQbt27eDm5gYvLy9WtASSt6NLfX19RZ0LpBWNCDbDL7/8gvLycgiFQvj4+GDYsGFo\n27Yt/vWvf7Gqj6C5uTm4XC6cnZ0xZsyYOl3vR4wYgUuXLjGUjjSVvBy9lp6ejpEjR2LkyJGsa7nx\nrlWrVjX6mNOnT7dAEiJJ33//PczNzaGkpARtbW08f/4cf/31Fy3BkVEJCQnw9vZmOsYHUSHYDFev\nXkViYiKePHmCrKws7NmzBx06dKi3078s8/DwgJubW4PHWXXv3h2BgYEtnIp8rMOHD6Ndu3YYPXo0\nq0bG3nfu3DnExMRg69at8PHxwbhx41BZWcl0LImjIo+9VFVVcfz4cRQUFGDGjBl4/fo105HIJ7K2\ntsbcuXNhbW2NLl26iI3iS8uIPRWCzSAQCAC8naLp06cPOnbsCD6fz7oWK6GhoXB3d2c6BmmmQ4cO\nITY2FsePH4eenh6cnJxY2RKoffv2cHV1haurK65du4YDBw5AIBBgzpw54PF4sLOzQ5cuXZiO2Wxn\nzpxp9DHS8kFDmu7atWuYPXs2evfujevXr+Pbb7+Fh4cHVq5cCRcXF6bjNVtNTY3otSsUClFVVVXn\ntcym121KSgoAYNu2bXXuk5Yvc7RGsBlWrlyJ7OxsPHz4UHRkzKJFi2BoaNikaRtZsWrVKlRWVsLe\n3h6dO3cW+0bDtqPX5EFNTQ0uXbqE2NhYpKSkwMzMDI6OjhgyZIhULWCWpLKyMsTFxWH//v3IzMzE\nzZs3mY7UbBYWFo0+Rlo+aEjTTZgwAT/88APGjBkjWj93+fJlrFy5EomJiUzHazZ5ed2uWbMGP//8\nM/755x8MGDCA6TgfRIVgM1RWViI6Ohpt2rQBj8fDo0ePkJiYiOnTp0NJSYnpeBLzoXVWd+7cacEk\nRNLevHmDpKQkBAYG4uXLlzh79izTkT67W7duoU+fPkzHIKReJiYmSE1NRatWrUSFoFAohLGxMa5c\nucJ0PNJEXC4XFy5cwNChQ5GRkcF0nA+iqeFmUFZWFhuq79GjB2bOnMlgos+Dij12evToEWJjYxEf\nH4+ysjLY29szHalFsLEIvHTpEgoLC0UtKgQCAbKzs7F8+XKGk5GP1b17dyQlJWH06NGia5cuXULP\nnj2ZC0U+2tdff40hQ4agoqICgwcPrnM/h8PBxYsXGUhWFxWCzZCSkoLVq1fj0aNHYj2COBwObt++\nzWAyyfrQ1n4TE5MWTEKaq6CgAPHx8YiPj0dubi6srKzg4+ODQYMGseakmIKCArk6a9fHxwdxcXHo\n2LEjBAIBWrdujdzcXEyYMIHpaOQTrFixAu7u7jA1NUVFRQWWL1+OpKQkBAQEMB2NfITg4GDcuXMH\nM2bMwObNm+v0EZQmNDXcDPb29hgxYgR4PF6dMwN79+7NUCrJMzU1FVs7VlpaiurqavTp0weHDh1i\nMBn5GK6urrh9+7boNTtixAgoKyszHUviak9m+PHHH+tdoM02pqam+Pe//43nz58jLCwMAQEBCAkJ\nQVlZmdQcak8+Tn5+PuLi4pCfnw8NDQ3Y2dnRiKCMSk1NhampKdMxPogKwWbgcrlIS0tj1XrApqht\ndFpWVoalS5cyHYc0kYGBAVRUVNC6det675emqYrmMDU1xZQpUxAcHIwVK1bU+5jvvvuuhVN9PrWF\n7/PnzzFx4kScOHEClZWV4PF4OH78ONPxCKnXixcv6vSkBd6eCsSmE4Dy8vKwY8cOPHr0CDU1NaLr\nHA5HavoN09RwM5iamuLKlSv1zv+zmZKSEubMmQMzMzMqBGXI3r17mY7QIpYtW4aYmBhUV1fj2LFj\n9T6GTYVgt27dcP36dfTt2xevX78Gn8+HkpIS+Hw+09HIJzAwMKhzJBmHw4GCggLU1NQwYsQIrFy5\nUqpPqmgKc3PzOpsoBAIBxo4dK/WbKz7GihUrIBQKYWlpKTZzKE0dGqgQbIb27dtj5syZ4HK5YudA\nStNh0p9Leno62rZty3QM8hGkfXpCUsaPH4/x48fjhx9+wO7du5mO89nNmDED06ZNQ1xcHMaPH4/J\nkydDUVERQ4cOZToa+QSLFy9GYmIiPD09oaWlhcLCQgQFBcHQ0BBDhgxBaGgofvvtN5k8BSgnJweT\nJ09GVVUVXr9+XWcQ5c2bN9DX12co3edx8+ZNnD17VqoLd5oaboYPnaYxb968Fkzyeb3/j1UgEKC8\nvBxLly6Fm5sbQ6kIaVxWVhZiY2NRWFiITp06wd7eHoaGhkzHkrjc3FxoampCQUEB8fHxKC0thbOz\nc4PLAIj0srKyQkREhNjpP8+fP8eECROQmJiIly9fwsrKCqmpqQym/HS3bt3Cq1evMGvWLOzcuVNs\n5FNZWVm0hIUtnJycsHXrVnTr1o3pKA2iEcFPEB4eDgBio4Bs9v7opoKCAnR1daGpqclQIkIad+bM\nGXh5ecHCwgJaWlrIzc3Fd999hz/++AOWlpZMx5MoHR0d0a95PB6DSUhzvXjxos616upqPHv2DABk\nvkiqbd+UnJwMdXV1htN8fiNHjoSbmxscHR3r1AzSskSFCsFP0NC6o3dJy19wc9y4cQOGhoZyM6VI\n2MXf3x8BAQFix1WdOXMGGzduZEUhWN9asnexrY2VvHBwcMDs2bPh4eEBLS0t5OfnY/v27bCzs8PL\nly+xdu1amX5Prh1I+RA2fH7WysjIgKamZr0juNLyPGlqmDSIy+Xi6tWrotubN2+Gp6cng4mIJAQG\nBsLDw0OsbyCfz8cvv/yCrVu3MphMsgYOHIj09HSx51ldXQ0TExNWLEbn8XjIz8+HtbU1HBwcoKGh\nUacoZFMbK3khEAgQFBSEmJgYFBcXQ0tLC+PGjcOMGTNw69YtREVFYeHChVBVVWU66ieZOnVqo4/Z\nt29fCyQhtagQJA16vxCsPe6IyDYHBwe0b98eGzZsgK6uLo4ePYpVq1bByMgIQUFBTMeTGGdnZ3h4\neIiN/p08eRJBQUGIiopiMJnkZGZmIjY2FgkJCejZsyd4PB5sbGxkfvqQEFmXnJxcZ2ewsrIyNDU1\n8eWXXzKUqn5UCJIGUSHITpWVlfD398ehQ4fQt29fZGVl4eeff4aNjQ3T0SQqJSUFHh4eGDJkCLp1\n64a8vDykpqZi69atMDMzYzqeRAmFQqSnpyM2NhZnz56FiYkJHB0dMWLECKajkY/0bt+52o9noVAo\nVX3nSOMsLCzqXKuurgafz8dXX32FkJAQqVkjSYUgaRAVgux15swZ/PTTTygvL8eAAQOwdu1aVm7+\nuXv3LuLj4/Hs2TNoa2vDzs4O3bt3ZzrWZ5Weng5fX19kZ2fTGkEZNHXq1Ab7zknLmjLy6V6/fo21\na9eisrISv//+O9NxANBmEfIBQqEQ9+7dE/26pqZGdLtWr169mIhGmmHx4sVITk7G8uXLYW9vjw0b\nNsDOzg6LFi1i3QeNvr6+XByzlpmZiaNHjyI+Ph41NTWwt7eHv78/07HIJ5CFvnPk07Vt2xaLFi2C\ng4MD01FEqBAkDaqoqKjzYn33Nu1KlE1FRUWIjo4WtRzx9fWFlZUVfv75Z9YVgmz2+PFjxMfHIz4+\nHk+fPsWYMWOwfv16GBsbMx2NNEP37t3x4sULKgRZTF1dHeXl5UzHEKGpYUIIAKC0tJQ+fGSIgYEB\n1NTUYG1tjeHDh0NRUbHO4vR3W+cQ2eDv74+EhASp7jtHmufhw4eYOXMmTp48yXQUAFQIEiI3li9f\nDj8/P9HtpKQkmJubi27b2toiISGBiWjkE9S3GP19p0+fboEkRJI+1F6F2qrIjveXUQFAVVUV8vLy\nEBgYiFGjRmHBggUMJKuLCkFC5ERjm3/ev58NUlNTERkZiaKiIgQEBCA8PBxz586FgoIC09EIISxm\nYGBQ55qSkhI0NTVhZ2eH+fPnQ0lJiYFkddEaQUIIK0VFRcHf3x8TJ05EUlISOBwOEhMTUVpaCm9v\nb6bjEdKgiIgIHD58GAUFBejcuTN4PB6+//57pmORj3Dnzh2mIzRZq8YfQgghsmfHjh0ICQnBvHnz\noKCgAHV1dezcuRPx8fFMRyOkQTt37sSuXbvg7OyM1atXY/z48QgLC8OOHTuYjkZYikYESaMKCgrQ\ntWvXJl8nRBqUlJTUaW+krq6OqqoqhhIR0riIiAgEBweLnT5hamoKNzc3zJkzh8FkhK1oRJA0ytbW\n9qOuE+lU2xfy3r17uHv3rqgvZO1tti0X5nK52LJli9jzCg0NxYABAxhMRciHvXz5Erq6umLXdHV1\nUVFRwVAiwna0WYQ0Kj8/H9ra2k2+TqRTfYuX3ydL61oak5+fD3d3dxQUFKC0tBSamppo06YNgoOD\n63zQyrK4uLh6m9M2dJ1Itzlz5kBfXx+LFi1Cq1atUF1djYCAAGRmZiI4OJjpeISFqBAkjbp//z70\n9PTqXD958iQsLS0ZSERI01RXV+P69evIz8+HhoYGBgwYIHZsFxs4ODggLi6uydeJdHv06BHc3NxE\nX16KiorQqVMnbN++nfXHIxJmUCFIGsXlcrFkyRJRM9OXL19i9erVOHXqFDIyMhhOR0j90tLS6jRY\nBt62cFBTU0OPHj0YSEVI4yorK3HlyhXw+XxoaWmhf//+rPsCQ6QHFYKkUenp6VixYgV69eoFR0dH\n+Pn5oU+fPvDx8YGWlhbT8Qipl4WFBfLz86GoqAhVVVWUlJSgqqoKioqKqKqqQvfu3bFt27Z6R7tl\niZeXFwICAupcnzFjBnbt2sVAIiJpz549w/z58xEeHs50FMJCVAiSJuHz+XB2dkZhYSHGjRuHdevW\nMR2JkA/asGEDysrKsHz5cqioqKCiogL+/v5o06YN3N3dsX37dvzzzz/Yu3cv01E/Wl5eHsLCwgC8\nPW1i2rRpok0xHA4HpaWlOH78OFJTU5mMSSSkoKAAo0aNYtUaXiI9aNcwaVRKSgomTJgAHR0drFu3\nDufPn8e8efNQVFTEdDRCGnTkyBF4e3tDRUUFANCmTRssXrwYBw8ehIqKCubNm4cbN24wnPLTdOvW\nDQDw/Plz0X9LSkpQUlIiulbfKCEhhLyPFh2QRnl4eGDBggVwc3MDh8PB6NGjsWrVKtjb24sdUUaI\nNFFSUkJmZib69u0rupaVlSVaN1haWioqEmXR8uXLAbzdDT516lQ6No8Q8kmoECSNioqKEltH9cUX\nX2Djxo04efIkg6kI+bA5c+ZgxowZGDduHLS0tJCXl4fo6GjMnz8fubm5+PHHHzFu3DimYzbbtm3b\nMHnyZCoEZdy9e/cavK+4uLgFkxB5Q2sESZNcunQJhYWFonVIAoEA2dnZolEJQqTRxYsXERMTg+Li\nYmhpacHJyQkDBw7EgwcPcOvWLdjb2zMdsdlmz54NW1tbODg4SM0h9uTjyVufTyI9qBAkjfLx8UFc\nXBw6duwIgUCA1q1bIzc3FxMmTMCqVauYjkdIkwiFQiQnJyM8PBw7d+5kOo7E2Nra4sGDB2jVqhU6\ndOggmvrmcDi4ePEiw+kIIdKOpoZJo44dO4bIyEg8f/4cYWFhCAgIQEhICEpLS5mORkijXr16hQMH\nDuDf//43iouLWdcE3dfXl+kIhBAZRoUgaRI9PT08f/4ct27dAgBMnz4dPB4PCxcuZDgZIfXLzMxE\nWFgY4uLi8ObNG6xYsQLjxo1Dhw4dmI4mUaampnWuVVVVITs7m4E0hBBZQ+1jSKO6deuG69evQ01N\nDa9fvwafz0dFRQX4fD7T0QipIyEhAVOmTMGUKVPQunVrREZGQl1dHfb29qwrAoG3Rz2OGjUKX3/9\nNQwMDGBgYABDQ0O4uroyHY0QIgNoRJA0asaMGZg2bRri4uIwfvx4TJ48GYqKihg6dCjT0QipY+HC\nhXBxccH27dvRvn17puN8dhs2bMDkyZPRtm1bZGRkwNXVFZs3b8bMmTOZjkYIkQG0WYQ0SW5uLjQ1\nNaGgoID4+HiUlpbC2dkZrVu3ZjoaIWIOHDiAyMhI5OTkYNy4cXBxccG0adMQHR2NTp06MR1P4rhc\nLjIyMpCXlwcvLy8cPHgQT548wcKFCxEREcF0PPIJUlNTERkZiaKiIgQEBCA8PBxz586lFkHks6Cp\nYdIkOjo6UFJSQqtWrcDj8TB58mQqAolUcnFxwYEDB7Bnzx5UVFRg0qRJ4PP5iI6OFp26wSZdcQID\npQAADnVJREFUunRBWVkZtLS0kJOTg5qaGnTt2hX3799nOhr5BFFRUViyZAn+7//+Dzdv3gSHw0Fi\nYiL8/PyYjkZYikYESYMMDAzA4XDQ0EuEw+Hg9u3bLZyKkI9TXl6OhIQEREZG4saNGxgyZAhCQkKY\njiUxvr6+ePDgATZt2oQlS5ZAT08PrVu3RlJSEmJjY5mORz6SjY0NNm3aBAMDA5iYmCA9PR2FhYVw\ndnZGSkoK0/EIC9EaQdIgfX195Ofnw9raGg4ODtDQ0GiwKCREWqmoqMDZ2RnOzs7IysrCgQMHmI4k\nUStWrMCuXbvA4XDg4+MDHx8flJWVYfXq1UxHI5+gpKQEvXr1Erumrq6OqqoqhhIRtqMRQfJBmZmZ\niI2NRUJCAnr27AkejwcbGxuZPqOVEEKklbu7O7766it4eXlh0KBBSE9Px65du5CWloYdO3YwHY+w\nEBWCpEmEQiHS09MRGxuLs2fPwsTEBI6OjhgxYgTT0QiRS56envVer13OweFwsGnTphZORZorPz8f\n7u7uKCgoQGlpKTQ1NdGmTRsEBwdDV1eX6XiEhagQJB8tPT0dvr6+yM7OpjWChDBky5Ytja7hnTdv\nXgunIpJQXV2N69evIz8/HxoaGhgwYAAUFWklF/k8qBAkTZKZmYmjR48iPj4eNTU1sLe3B4/HQ+/e\nvZmORkiDqA0HkUWFhYXIy8tDdXW12HUTExOGEhE2o68YpEGPHz9GfHw84uPj8fTpU4wZMwbr16+H\nsbEx09EIaVRUVBT8/f0xceJEJCUlidpwlJaWwtvbm+l4EnHr1i3cuXMHpqam6Nq1K9atW4eLFy/C\nyMgI3t7etJZXBu3evRu///472rVrByUlJbH7Ll26xFAqwmY0IkgaZGBgADU1NVhbW2P48OFQVFQE\nh8MRe8zIkSMZSkfIh7G9DcfBgwexatUq6Ovr48mTJxg1ahSysrLg5OSEuLg46Ovr085hGTRkyBD4\n+fnR+mvSYmhEkDRIW1sbAHDu3DmcO3eu3secPn26JSMR0mRsb8MREhKC8PBw9O3bFxcvXoSbmxsS\nExOhq6sLGxsbjB07lgpBGaSkpIThw4czHYPIESoESYOoyCOyjMvlYsuWLfDy8hJdCw0NxYABAxhM\nJTmFhYXo27cvgLejSMrKyqJdpV26dEFFRQWT8cgncnJyQkhICGbOnIlWrejwL/L5USFICGGlX375\nBe7u7vj7779RVlYGc3NzURsONnh/mcb7Rz7Sqh/ZMnjwYABv/95evHiBLVu2oF27dqL7ORwOLl68\nyFQ8wmJUCBJCWElbWxuHDx9mdRuO8vJyAP8r+t6/TWQH9XwkTKHNIoQQ1mJzGw4DA4NGH3Pnzp0W\nSEIkwdfXF76+vkzHIHKICkFCCCuxvQ1Hbm5uo4/R0dFpgSREErhcLq5evcp0DCKH2DNHQiQuMDCw\n0cfQyQVEWoWEhCA4OJi1bTioyCOESAIVgqRBRUVF4HA4ePr0Kc6ePQsrKyvo6OigoKAAJ06cgJWV\nFdMRCWkQteEgskQgEDT65Zu+eJPPgQpB0qBVq1YBAKZNm4YdO3bAzMxMdN+ECRPg5+fHVDRCGkVt\nOIgsEQqFyMrKYjoGkUO0RpA0isvl4vLly2Lns1ZWVsLU1JTWtBCp834bDiUlJWrDQaQerREkTKER\nQdKo3r17Y+fOnZgzZw4AoKamBtu2bYOhoSHDyQipS17acCQnJ9fpJfg+OgKSENIYKgRJo3799Vf8\n+OOP+Ouvv9ClSxcUFRVBVVWVNY15CbskJCTIRRuO2qUbH0KnA8mOgQMHMh2ByCmaGiZNUllZiYyM\nDBQXF6Nr164wMjISmyomRFrQFBshhDQdjQiSJiksLISioiK6du0KAEhLS8P9+/cxZcoUhpMRIt+O\nHDnS4H1OTk4tmIQQIouoECSNCgwMxNatW9GmTRsAQFVVFQQCAUaOHEmFIJE68taGIzQ0VGytYElJ\nCfLz8zFq1CgqBAkhjaJCkDTq77//xr59+yAQCHDkyBGsWbMGfn5+1NCWSCV5a8MRFRVV59rhw4eR\nlpbGQBpCiKyhNYKkUcbGxrh8+TKePn2K6dOnIz4+HmVlZXBxccHRo0eZjkeIGFoj+LYYNjExweXL\nl5mOQgiRcjQiSBqlpaWFx48fo3v37uDz+SgtLYWSkhIKCwuZjkaI3CsvLxe7LRAIkJiYiE6dOjGU\niBAiS6gQJI1ycXHB5MmTERUVBWtra8yaNQtKSkro378/09EIqUPe2nBwudw61zp06AAfHx8G0hBC\nZA1NDZMmuXz5MgwNDcHhcPDXX3+hrKwMbm5uUFdXZzoaIXItNzdX7LaCggI6deoEZWVlhhIRQmQJ\nFYKEECLjysvL8eLFC9TU1AB4u7P//v37MDc3ZzgZIUTa0dQwaZCFhQU4HA4a+q7A4XBw6tSpFk5F\nCHnX/v37sXbtWrx580bsur6+PhWChJBGUSFIGlRWVoaqqipYW1vD3NwcysrKDRaFhBBmbNu2DWvW\nrIGysjKSk5Mxf/58+Pn5wdLSkulohBAZQFPDpEECgQDnzp1DbGwsLl++DHNzczg6OsLY2JjpaISQ\n/zIyMkJGRgYKCgrg7u6OI0eO4Pnz5/jhhx9w+PBhpuMRQqQcjQiSBikpKcHCwgIWFhYoLS1FYmIi\ngoKCkJOTAzs7Ozg6OkJPT4/pmITINU1NTfD5fGhqauLJkyeorKzEF198gZycHKajEUJkAI0Iko92\n8+ZNLFu2DNnZ2bh9+zbTcQiRawEBATh58iT27NmD1atXAwCUlZXx+PFj7N+/n+F0hBBpR4UgaZKS\nkhKcOHECcXFxuHHjBkaNGgUej0eL0QlhWE1NDWJiYmBpaQmBQIA///wTZWVl8PT0RM+ePZmORwiR\nclQIkgaVlZXh1KlTiI+PR2pqKkxMTODg4ABLS0u0a9eO6XiEyD2hUIgXL15AVVWV6SiEEBlFhSBp\n0IABA6CiogIbGxvY2tpCXV0dHA5H7DG9evViKB0h8u3u3buYNWsWCgoK0Lt3b2zZsgU9evRgOhYh\nRMZQIUgaZGBg8MH7ORwOrREkhCEzZsxA79694ezsjH379uHp06cICgpiOhYhRMZQIUgIITJo4MCB\nSE1NhaKiIl68eAE7OzukpKQwHYsQImNaMR2AEELIxxMKhVBUfNsBrGPHjqisrGQ4ESFEFlEhSAgh\nMogmcwghkkANpQkhRAYJhULcu3dP9OuamhrR7Vq0mYsQ0hhaI0gIITKINnMRQiSBCkFCCCGEEDlF\nawQJIYQQQuQUFYKEEEIIIXKKCkFCCCGEEDlFhSAhhBBCiJyiQpAQQvB2F66BgQHu3r1b575r167B\nwMAAU6dOldjPOn/+vER+L0IIaQ7qI0gIIf+lpKSEkydPQl9fX+z6iRMnwOFwwOFwJPJzUlJS8MUX\nX0jk9yKEkOagEUFCCPmvQYMG4eTJk3WuJyYmYsCAARI7zaNTp05QUlKSyO9FCCHNQYUgIYT8l6Wl\nJW7duoXCwkLRtczMTJSVlcHIyEjssQ8fPoS7uztMTExgZmaG3377TXTer6urKzZs2CD2+JUrV2Lx\n4sUAxKeGBQIB/Pz8MHToUJiYmMDd3R05OTmi/+/EiROws7NDv379YG1tjf3793+W504IkU9UCBJC\nyH/p6Ojgq6++EhsVTExMhKWlJVq1+t/bZUlJCVxdXdGhQwdERETgjz/+wOnTp7Fu3ToAAI/Hw/Hj\nx0WPFwgEOHXqFHg8Xp2f6e/vj7S0NAQGBmL//v3o0qULpk+fjjdv3oDP52PRokWYNm0ajh8/Dg8P\nD/j6+iIzM/Mz/ikQQuQJFYKEEPIOS0vLOoWgjY2N2LRwbGwsFBQUsHbtWujp6WHIkCHw8fFBZGQk\nXr16BRsbGxQUFODGjRsA3q4J5HA4GDZsmNjPqqioQFhYGHx9fcHlcvHll1/i119/RXV1NY4fP47C\nwkJUVVVBQ0MDWlpacHJywt69e6GpqdkyfxiEENajQpAQQv6Lw+HAysoK6enpKC0txaNHj1BQUABT\nU1PR/QBw//59fP3112Lr/IyMjFBdXY0HDx5AXV0dQ4cOxbFjxwAACQkJsLGxgaKi+P68x48fo7Ky\nEtOmTQOXywWXy8XAgQPx9OlTPHz4EH369IGFhQU8PDxgaWmJNWvWoH379lBVVW2hPxFCCNvRrmFC\nCHnHV199BW1tbZw+fRpFRUV1poUBoHXr1nU2jlRXV4v918HBAZs2bcKCBQtw+vRpbN26tc7Pqn1s\naGioWHEnFArRoUMHAEBQUBBu376NU6dOISkpCREREQgMDMSoUaMk9pwJIfKLRgQJIeQ9lpaWSEpK\nwsmTJ2FtbS26Xlv86enp4fbt26LNIQBw9epVtGrVCj169AAAjB49Gnw+H6GhoVBRUcGgQYPq/Jzu\n3btDUVERfD4furq60NXVhZaWFjZu3IjMzEzcvn0ba9aswddff4158+bh0KFDGDx4sNj6Q0IIaQ4q\nBAkh5D1WVlZITk7GgwcPMHTo0Dr383g8tGrVCt7e3rh//z4uXLiAVatWwdbWFurq6gCAtm3bwsLC\nAlu3boW9vX29P6ddu3aYPHkyVq1ahfPnz+Phw4f417/+hdTUVPTq1Qvt27fH/v37sWnTJuTk5CAt\nLQ137tyBoaHhZ33+hBD5QVPDhBDynv79+6Ndu3YYMmSIaF3fuw2lVVRUsGvXLvz2229wdnZGhw4d\n4OjoCC8vL7Hfx8HBAUePHoWDg0ODP2vp0qVo1aoVli9fjtevX8PQ0BC7d+9G586dAQCbNm1CQEAA\ndu/ejY4dO8LFxQWurq6f6ZkTQuQNRyipDqmEEEIIIUSm0NQwIYQQQoicokKQEEIIIUROUSFICCGE\nECKnqBAkhBBCCJFTVAgSQgghhMgpKgQJIYQQQuQUFYKEEEIIIXKKCkFCCCGEEDlFhSAhhBBCiJz6\nf0lbcvNs3XGZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x141b8ab0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## create a bar plot with the data\n",
    "## notice that we are using the data frame itself and call its plot function\n",
    "scores.plot(kind = 'bar')\n",
    "\n",
    "## set the title to Score Comparison\n",
    "plt.title('Score Comparison')\n",
    "\n",
    "## set the x label\n",
    "plt.xlabel('Movies')\n",
    "\n",
    "## set the y label\n",
    "plt.ylabel('Scores')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "## show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Twitter Example:\n",
    "================\n",
    "\n",
    "* API a bit more complicated\n",
    "* libraries make life easier\n",
    "* python-twitter\n",
    "\n",
    "https://github.com/bear/python-twitter\n",
    "\n",
    "What we are going to do is scrape Joe's twitter account, and then filter it for the interesting tweets. Defining interesting as tweets that have be re-tweeted at least 10 times. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import twitter\n",
    "\n",
    "## define the necessary keys\n",
    "cKey = twitterAPI_key()\n",
    "cSecret = twitterAPI_secret()\n",
    "aKey = twitterAPI_access_token_key()\n",
    "aSecret = twitterAPI_access_token_secret()\n",
    "\n",
    "## create the api object with the twitter-python library\n",
    "api = twitter.Api(consumer_key=cKey, consumer_secret=cSecret, access_token_key=aKey, access_token_secret=aSecret)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>favorited</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>id</th>\n",
       "      <th>in_reply_to_screen_name</th>\n",
       "      <th>in_reply_to_status_id</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>lang</th>\n",
       "      <th>media</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>retweeted</th>\n",
       "      <th>retweeted_status</th>\n",
       "      <th>source</th>\n",
       "      <th>text</th>\n",
       "      <th>truncated</th>\n",
       "      <th>urls</th>\n",
       "      <th>user</th>\n",
       "      <th>user_mentions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mon Sep 07 22:17:04 +0000 2015</td>\n",
       "      <td>2</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>641012335738421248</td>\n",
       "      <td>lallen1618</td>\n",
       "      <td>6.409978e+17</td>\n",
       "      <td>2355874964</td>\n",
       "      <td>en</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com\" rel=\"nofollow\"&gt;Tw...</td>\n",
       "      <td>@lallen1618 @wzchen @AriBFriedman Good luck wi...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{u'id': 184543773, u'profile_sidebar_fill_colo...</td>\n",
       "      <td>[{u'screen_name': u'lallen1618', u'id': 235587...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mon Sep 07 18:51:12 +0000 2015</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>640960524784640001</td>\n",
       "      <td>albertocairo</td>\n",
       "      <td>6.409594e+17</td>\n",
       "      <td>34255829</td>\n",
       "      <td>en</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>@albertocairo @Sulliview @NateSilver538 But a ...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{u'id': 184543773, u'profile_sidebar_fill_colo...</td>\n",
       "      <td>[{u'screen_name': u'albertocairo', u'id': 3425...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       created_at  favorite_count favorited hashtags  \\\n",
       "0  Mon Sep 07 22:17:04 +0000 2015               2     False      NaN   \n",
       "1  Mon Sep 07 18:51:12 +0000 2015             NaN     False      NaN   \n",
       "\n",
       "                   id in_reply_to_screen_name  in_reply_to_status_id  \\\n",
       "0  641012335738421248              lallen1618           6.409978e+17   \n",
       "1  640960524784640001            albertocairo           6.409594e+17   \n",
       "\n",
       "   in_reply_to_user_id lang media  retweet_count retweeted retweeted_status  \\\n",
       "0           2355874964   en   NaN            NaN     False              NaN   \n",
       "1             34255829   en   NaN            NaN     False              NaN   \n",
       "\n",
       "                                              source  \\\n",
       "0  <a href=\"http://twitter.com\" rel=\"nofollow\">Tw...   \n",
       "1  <a href=\"http://twitter.com/download/iphone\" r...   \n",
       "\n",
       "                                                text truncated urls  \\\n",
       "0  @lallen1618 @wzchen @AriBFriedman Good luck wi...     False  NaN   \n",
       "1  @albertocairo @Sulliview @NateSilver538 But a ...     False  NaN   \n",
       "\n",
       "                                                user  \\\n",
       "0  {u'id': 184543773, u'profile_sidebar_fill_colo...   \n",
       "1  {u'id': 184543773, u'profile_sidebar_fill_colo...   \n",
       "\n",
       "                                       user_mentions  \n",
       "0  [{u'screen_name': u'lallen1618', u'id': 235587...  \n",
       "1  [{u'screen_name': u'albertocairo', u'id': 3425...  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## get the user timeline with screen_name = 'stat110'\n",
    "twitter_statuses = api.GetUserTimeline(screen_name = 'stat110')\n",
    "\n",
    "## create a data frame\n",
    "## first get a list of panda Series or dict\n",
    "pdSeriesList = [pd.Series(t.AsDict()) for t in twitter_statuses]\n",
    "\n",
    "## then create the data frame\n",
    "data = pd.DataFrame(pdSeriesList)\n",
    "\n",
    "data.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "######\n",
      "RT @NateSilver538: .@Sulliview: The fundamental sin is cherry-picking evidence to fit a narrative. Can happen in both \"reported\" and \"data-…\n",
      "######\n",
      "RT @Agent_Analytics: Watch out #datascientist 's. Always read the #job decscription. #data #DataScience #BigData #analytics #jokes http://t…\n",
      "######\n",
      "RT @rasbt: just felt like writing this morning ;) \"Python, Machine Learning &amp; Language Wars. A Highly Subjective Point of View\" http://t.co…\n",
      "######\n",
      "RT @ericcolson: Linear Algebra: took this course by G Strang years ago. Excellent and free! via @ajinkyakale \n",
      "http://t.co/UAqeHMzs4a http:/…\n"
     ]
    }
   ],
   "source": [
    "## filter tweets with enough retweet_count\n",
    "maybe_interesting = data[data.retweet_count>20]\n",
    "\n",
    "## get the text of these tweets\n",
    "tweet_text = maybe_interesting.text\n",
    "\n",
    "## print them out\n",
    "text = tweet_text.values\n",
    "\n",
    "for t in text:\n",
    "    print '######'\n",
    "    print t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Extracting columns:\n",
    "===================\n",
    "\n",
    "__Warning:__ The returned column `tweet_text` is a `view` on the data\n",
    "    \n",
    "* it is not a copy\n",
    "* you change the Series => you change the DataFrame\n",
    "\n",
    "Below is another example of such a view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python27\\lib\\site-packages\\ipykernel\\__main__.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "C:\\Python27\\lib\\site-packages\\pandas\\core\\series.py:664: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self.loc[key] = value\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------\n",
      "This is view:\n",
      "3   NaN\n",
      "5   NaN\n",
      "6   NaN\n",
      "7   NaN\n",
      "Name: favorite_count, dtype: float64\n",
      "-----------------\n",
      "This is view after changing view[8]\n",
      "3     NaN\n",
      "5     NaN\n",
      "6     NaN\n",
      "7     NaN\n",
      "8    9999\n",
      "Name: favorite_count, dtype: float64\n",
      "-----------------\n",
      "This is maybe_interesting after changing view[8]\n",
      "It changed too!\n",
      "3     NaN\n",
      "5     NaN\n",
      "6     NaN\n",
      "7     NaN\n",
      "8    9999\n",
      "Name: favorite_count, dtype: float64\n",
      "-----------------\n",
      "This is independent_data after changed at 10:\n",
      "3      NaN\n",
      "5      NaN\n",
      "6      NaN\n",
      "7      NaN\n",
      "8     9999\n",
      "10     999\n",
      "Name: favorite_count, dtype: float64\n",
      "-----------------\n",
      "This is maybe_interesting after changing independent_data:\n",
      "It did not change because we only changed a copy of it\n",
      "3     NaN\n",
      "5     NaN\n",
      "6     NaN\n",
      "7     NaN\n",
      "8    9999\n",
      "Name: favorite_count, dtype: float64\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python27\\lib\\site-packages\\IPython\\core\\interactiveshell.py:3066: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  exec(code_obj, self.user_global_ns, self.user_ns)\n"
     ]
    }
   ],
   "source": [
    "## create a view for favorite_count on maybe_interesting\n",
    "view = maybe_interesting['favorite_count']\n",
    "print '-----------------'\n",
    "print \"This is view:\"\n",
    "print view\n",
    "## change a value\n",
    "view[8] = 9999\n",
    "\n",
    "## look at original frame\n",
    "print '-----------------'\n",
    "print \"This is view after changing view[8]\"\n",
    "print view\n",
    "\n",
    "print '-----------------'\n",
    "print \"This is maybe_interesting after changing view[8]\"\n",
    "print \"It changed too!\"\n",
    "print maybe_interesting['favorite_count']\n",
    "\n",
    "## to avoid this you can use copy\n",
    "independent_data = maybe_interesting['favorite_count'].copy()\n",
    "independent_data[10] = 999\n",
    "print '-----------------'\n",
    "print \"This is independent_data after changed at 10:\"\n",
    "print independent_data\n",
    "print '-----------------'\n",
    "print \"This is maybe_interesting after changing independent_data:\"\n",
    "print \"It did not change because we only changed a copy of it\"\n",
    "print maybe_interesting['favorite_count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What we covered today:\n",
    "============\n",
    "\n",
    "* Pandas data frames\n",
    "* Guidelines for friendly scraping\n",
    "* Scraping html sites\n",
    "* Scraping with Api's\n",
    "* Basic data cleanup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Further material\n",
    "================\n",
    "\n",
    "* I highly recommend Katharine Jarmul's scraping tutorials\n",
    "* For example [this one](https://www.youtube.com/watch?v=p1iX0uxM1w8)\n",
    "* Pandas has extensive [documentation](http://pandas.pydata.org/pandas-docs/stable/)\n",
    "* Especially the [tem minutes to pandas chapter](http://pandas.pydata.org/pandas-docs/stable/10min.html)\n",
    "\n",
    "* [Greg Reda](http://www.gregreda.com/2013/10/26/using-pandas-on-the-movielens-dataset/) did a lot more pandas examples for the movie lens data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}