how_to_build_facebook_scraper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Scrape Data From Facebook Page Posts for Statistical Analysis\n",
    "\n",
    "By [Max Woolf (@minimaxir)](http://minimaxir.com/)\n",
    "\n",
    "This notebook describes how to build a Facebook Scraper using the latest version of Facebook's Graph API (v2.4). This is the accompanyment to my blog post [How to Scrape Data From Facebook Page Posts for Statistical Analysis](http://minimaxir.com/2015/07/facebook-scraper/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import some Python dependencies\n",
    "\n",
    "import urllib2\n",
    "import json\n",
    "import datetime\n",
    "import csv\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing Facebook page data requires an access token.\n",
    "\n",
    "Since the user access token expires within an hour, we need to create a dummy application *for the sole purpose of scraping* and use the app ID and app secret generated there [as described here](https://developers.facebook.com/docs/facebook-login/access-tokens#apptokens), both of which never expire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Since the code output in this notebook leaks the app_secret,\n",
    "# it has been reset by the time you read this.\n",
    "\n",
    "app_id = \"272535582777707\"\n",
    "app_secret = \"59e7ab31b01d3a5a90ec15a7a45a5e3b\" # DO NOT SHARE WITH ANYONE!\n",
    "\n",
    "access_token = app_id + \"|\" + app_secret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can access public Facebook data without limit. Let's do our analysis on the [New York Times Facebook page](https://www.facebook.com/nytimes), which is popular enough to yield good data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "page_id = 'nytimes'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a quick program to ping NYT's Facebook page to verify that the `access_token` works and the `page_id` is valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"id\": \"5281959998\", \n",
      "    \"name\": \"The New York Times\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "def testFacebookPageData(page_id, access_token):\n",
    "    \n",
    "    # construct the URL string\n",
    "    base = \"https://graph.facebook.com/v2.4\"\n",
    "    node = \"/\" + page_id\n",
    "    parameters = \"/?access_token=%s\" % access_token\n",
    "    url = base + node + parameters\n",
    "    \n",
    "    # retrieve data\n",
    "    req = urllib2.Request(url)\n",
    "    response = urllib2.urlopen(req)\n",
    "    data = json.loads(response.read())\n",
    "    \n",
    "    print json.dumps(data, indent=4, sort_keys=True)\n",
    "    \n",
    "\n",
    "testFacebookPageData(page_id, access_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When scraping large amounts of data from public APIs, there's a high probability that you'll hit an [HTTP Error 500 (Internal Error)](http://www.checkupdown.com/status/E500.html) at some point. There is no way to avoid that on our end. \n",
    "\n",
    "Instead, we'll use a helper function to catch the error and try again after a few seconds, which usually works. This helper function also consolidates the data retrival code, so it kills two birds with one stone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def request_until_succeed(url):\n",
    "    req = urllib2.Request(url)\n",
    "    success = False\n",
    "    while success is False:\n",
    "        try: \n",
    "            response = urllib2.urlopen(req)\n",
    "            if response.getcode() == 200:\n",
    "                success = True\n",
    "        except Exception, e:\n",
    "            print e\n",
    "            time.sleep(5)\n",
    "            \n",
    "            print \"Error for URL %s: %s\" % (url, datetime.datetime.now())\n",
    "\n",
    "    return response.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is the Facebook Page metadata however; we need to change the endpoint to the /feed endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"data\": [\n",
      "        {\n",
      "            \"created_time\": \"2015-07-20T01:25:01+0000\", \n",
      "            \"id\": \"5281959998_10150628157724999\", \n",
      "            \"message\": \"The planned megalopolis, a metropolitan area that would be about 6 times the size of New York\\u2019s, is meant to revamp northern China\\u2019s economy and become a laboratory for modern urban growth.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T22:55:01+0000\", \n",
      "            \"id\": \"5281959998_10150628161129999\", \n",
      "            \"message\": \"\\\"It\\u2019s safe to say that federal agencies are not where we want them to be across the board,\\\" said President Barack Obama's top cybersecurity adviser. \\\"We clearly need to be moving faster.\\\"\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T22:25:01+0000\", \n",
      "            \"id\": \"5281959998_10150626434639999\", \n",
      "            \"message\": \"Showcase your summer tomatoes in this elegant crostata.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T21:55:08+0000\", \n",
      "            \"id\": \"5281959998_10150628170209999\", \n",
      "            \"message\": \"The task: Create a technologically sophisticated barbecue smoker that could outperform the best product on the market and be sold for less than $1,500.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T21:25:00+0000\", \n",
      "            \"id\": \"5281959998_10150626449129999\", \n",
      "            \"message\": \"Achieving pastel hair can be time-consuming and toxic \\u2014 but for some, so very worth it.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T20:53:05+0000\", \n",
      "            \"id\": \"5281959998_10150626425084999\", \n",
      "            \"message\": \"Attention, meat lovers: This simple barbecue sauce goes beautifully with pork and chicken.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T20:25:07+0000\", \n",
      "            \"id\": \"5281959998_10150628132119999\", \n",
      "            \"message\": \"He passed the police officer exam in 2011. He went through orientation and started undergoing the required background checks in 2013. Then, the process stopped cold. No emails. No calls. No explanations. Silence.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T19:55:32+0000\", \n",
      "            \"id\": \"5281959998_10150628116259999\", \n",
      "            \"message\": \"The election is 16 months away, but knowing what we know now, what should we expect the economic backdrop to be when Americans choose their next president?\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T19:25:07+0000\", \n",
      "            \"id\": \"5281959998_10150628097394999\", \n",
      "            \"message\": \"\\\"By focusing so intently on physical fitness, the corps is avoiding the real barrier to integration \\u2014 the hypermasculine culture at its heart.\\\" Read on in The New York Times Opinion.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T19:05:01+0000\", \n",
      "            \"id\": \"5281959998_10150628071729999\", \n",
      "            \"message\": \"U2's \\u201cInnocence and Experience\\u201d tour merges past and present, peace and war, audience and band, punk and statesman, grass-roots activism and corporate philanthropy.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T18:55:05+0000\", \n",
      "            \"id\": \"5281959998_10150628073894999\", \n",
      "            \"message\": \"\\\"I always believe in apologizing if you\\u2019ve done something wrong, but if you read my statement, you\\u2019ll see I said nothing wrong,\\\" Donald J. Trump said in an interview.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T18:25:21+0000\", \n",
      "            \"id\": \"5281959998_10150628056964999\", \n",
      "            \"message\": \"Booing, like opera, can be divided into several genres.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T17:55:08+0000\", \n",
      "            \"id\": \"5281959998_10150628040459999\", \n",
      "            \"message\": \"\\\"Nearly at once, [the Confederate flag and Atticus Finch] have fallen from grace in ways that were unimaginable just months ago. They are forcing a reckoning with ourselves and our history, a reassessment of who we were and of what we might become.\\\" Read on in The New York Times Opinion.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T17:25:00+0000\", \n",
      "            \"id\": \"5281959998_10150627982469999\", \n",
      "            \"message\": \"It's National Ice Cream Day. How about cooling off with a treat?\", \n",
      "            \"story\": \"The New York Times added 4 new photos.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T16:55:07+0000\", \n",
      "            \"id\": \"5281959998_10150628000024999\", \n",
      "            \"message\": \"Bystanders watched people wave flags celebrating Pan-Africanism, the Confederacy and the Nazi Party. And they watched as black demonstrators raised clenched fists, and white demonstrators performed Nazi salutes.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T16:25:08+0000\", \n",
      "            \"id\": \"5281959998_10150627989069999\", \n",
      "            \"message\": \"\\\"Because in the sunset of his presidency, Barack Obama's bolder side is rising. He\\u2019s a lame duck who doesn\\u2019t give a damn.\\\" Read on in The New York Times Opinion.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T15:55:06+0000\", \n",
      "            \"id\": \"5281959998_10150627979424999\", \n",
      "            \"message\": \"The flyby of Pluto was a triumph of human ingenuity and the capstone of a mission that unfolded nearly flawlessly. Yet it almost didn't happen.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T15:25:04+0000\", \n",
      "            \"id\": \"5281959998_10150627970394999\", \n",
      "            \"message\": \"After 6 months apart, Caroline Dove planned to reunite with her boyfriend of more than 2 years. But before she could make the trip, there came a final, portentous message.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T14:55:08+0000\", \n",
      "            \"id\": \"5281959998_10150627962014999\", \n",
      "            \"message\": \"Hillary Clinton has made the struggles of her mother a central part of her 2016 campaign\\u2019s message. But her father, whom she rarely talks about publicly, exerted an equally powerful, if sometimes bruising, influence on the woman who wants to become the first female president.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T14:25:09+0000\", \n",
      "            \"id\": \"5281959998_10150627952769999\", \n",
      "            \"message\": \"Quotation of the Day: \\\"When your contract is over, they send you home, saying they\\u2019ve transferred the money. You get home, and there is nothing there.\\\" \\u2014 Yuriy Cheng, a Ukrainian seaman, describing the owner of the Dona Liberta, a ship that is a case study of misconduct at sea.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T12:55:01+0000\", \n",
      "            \"id\": \"5281959998_10150626434214999\", \n",
      "            \"message\": \"Summer on a stick. (via The New York Times Food)\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T09:55:00+0000\", \n",
      "            \"id\": \"5281959998_10150627665974999\", \n",
      "            \"message\": \"The surge of migrants into Europe from war-ravaged and impoverished parts of the Middle East, Afghanistan and Africa has shifted in recent months. Migrants are now pushing by land across the western Balkans, in numbers roughly equal to those entering the Continent through Italy.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T03:55:00+0000\", \n",
      "            \"id\": \"5281959998_10150626450789999\", \n",
      "            \"message\": \"When your big toe isn't your biggest toe.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T02:55:00+0000\", \n",
      "            \"id\": \"5281959998_10150626440069999\", \n",
      "            \"message\": \"\\\"Progress is occurring, as courts accept that in marriage and other matters, gender can't be reduced to chromosomes or surgeries,\\\" writes J. Courtney Sullivan in The New York Times Opinion.\"\n",
      "        }, \n",
      "        {\n",
      "            \"created_time\": \"2015-07-19T01:55:01+0000\", \n",
      "            \"id\": \"5281959998_10150627562209999\", \n",
      "            \"message\": \"Experimenting with neon lavender, sea-foam green and soft periwinkle.\"\n",
      "        }\n",
      "    ], \n",
      "    \"paging\": {\n",
      "        \"next\": \"https://graph.facebook.com/v2.4/5281959998/feed?access_token=272535582777707|59e7ab31b01d3a5a90ec15a7a45a5e3b&limit=25&until=1437270901&__paging_token=enc_AdB73LgZAUngYJIdoZCGUgWvKdL9zs23TBqdfeK90PnPs9MqO7xeze7ANGK2zMxZAveZAvwa1nHzTObmzuKiHY7MVVow\", \n",
      "        \"previous\": \"https://graph.facebook.com/v2.4/5281959998/feed?since=1437355501&access_token=272535582777707|59e7ab31b01d3a5a90ec15a7a45a5e3b&limit=25&__paging_token=enc_AdC4YOxNofFbJWmap6PZC6S0iyiWG8A1FpsYTMrBG62tmT6HfNuhc6rcxL6fMk8ZAxx0EQcFy52SJ2fJ1TbIL47EQx&__previous=1\"\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "def testFacebookPageFeedData(page_id, access_token):\n",
    "    \n",
    "    # construct the URL string\n",
    "    base = \"https://graph.facebook.com/v2.4\"\n",
    "    node = \"/\" + page_id + \"/feed\" # changed\n",
    "    parameters = \"/?access_token=%s\" % access_token\n",
    "    url = base + node + parameters\n",
    "    \n",
    "    # retrieve data\n",
    "    data = json.loads(request_until_succeed(url))\n",
    "    \n",
    "    print json.dumps(data, indent=4, sort_keys=True)\n",
    "    \n",
    "\n",
    "testFacebookPageFeedData(page_id, access_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In v2.4, the default behavior is to return very, very little metadata for statuses in order to reduce bandwidth, with the expectation the user will request the necessary fields.\n",
    "\n",
    "We don't need data on every NYT status. Yet. Let's reduce the requested fields to exactly what we need, and the number of stories returned to 1 so we can process it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"comments\": {\n",
      "        \"data\": [\n",
      "            {\n",
      "                \"can_remove\": false, \n",
      "                \"created_time\": \"2015-07-20T01:28:02+0000\", \n",
      "                \"from\": {\n",
      "                    \"id\": \"859569687424896\", \n",
      "                    \"name\": \"Chris Gagne\"\n",
      "                }, \n",
      "                \"id\": \"10150628157724999_10150628249759999\", \n",
      "                \"like_count\": 9, \n",
      "                \"message\": \"Aaaaaaaand there goes the rest of Beijing's clean air, whatever was left of it.\", \n",
      "                \"user_likes\": false\n",
      "            }\n",
      "        ], \n",
      "        \"paging\": {\n",
      "            \"cursors\": {\n",
      "                \"after\": \"MzE=\", \n",
      "                \"before\": \"MzE=\"\n",
      "            }, \n",
      "            \"next\": \"https://graph.facebook.com/v2.0/5281959998_10150628157724999/comments?order=chronological&limit=1&summary=true&access_token=272535582777707|59e7ab31b01d3a5a90ec15a7a45a5e3b&after=MzE%3D\"\n",
      "        }, \n",
      "        \"summary\": {\n",
      "            \"order\": \"ranked\", \n",
      "            \"total_count\": 31\n",
      "        }\n",
      "    }, \n",
      "    \"created_time\": \"2015-07-20T01:25:01+0000\", \n",
      "    \"id\": \"5281959998_10150628157724999\", \n",
      "    \"likes\": {\n",
      "        \"data\": [\n",
      "            {\n",
      "                \"id\": \"1001217933243627\", \n",
      "                \"name\": \"Josh Smith\"\n",
      "            }\n",
      "        ], \n",
      "        \"paging\": {\n",
      "            \"cursors\": {\n",
      "                \"after\": \"MTAwMTIxNzkzMzI0MzYyNw==\", \n",
      "                \"before\": \"MTAwMTIxNzkzMzI0MzYyNw==\"\n",
      "            }, \n",
      "            \"next\": \"https://graph.facebook.com/v2.0/5281959998_10150628157724999/likes?limit=1&summary=true&access_token=272535582777707|59e7ab31b01d3a5a90ec15a7a45a5e3b&after=MTAwMTIxNzkzMzI0MzYyNw%3D%3D\"\n",
      "        }, \n",
      "        \"summary\": {\n",
      "            \"total_count\": 278\n",
      "        }\n",
      "    }, \n",
      "    \"link\": \"http://nyti.ms/1Jr6LhU\", \n",
      "    \"message\": \"The planned megalopolis, a metropolitan area that would be about 6 times the size of New York\\u2019s, is meant to revamp northern China\\u2019s economy and become a laboratory for modern urban growth.\", \n",
      "    \"name\": \"China Molds a Supercity Around Beijing, Promising to Change Lives\", \n",
      "    \"shares\": {\n",
      "        \"count\": 50\n",
      "    }, \n",
      "    \"type\": \"link\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "def getFacebookPageFeedData(page_id, access_token, num_statuses):\n",
    "    \n",
    "    # construct the URL string\n",
    "    base = \"https://graph.facebook.com\"\n",
    "    node = \"/\" + page_id + \"/feed\" \n",
    "    parameters = \"/?fields=message,link,created_time,type,name,id,likes.limit(1).summary(true),comments.limit(1).summary(true),shares&limit=%s&access_token=%s\" % (num_statuses, access_token) # changed\n",
    "    url = base + node + parameters\n",
    "    \n",
    "    # retrieve data\n",
    "    data = json.loads(request_until_succeed(url))\n",
    "    \n",
    "    return data\n",
    "    \n",
    "\n",
    "test_status = getFacebookPageFeedData(page_id, access_token, 1)[\"data\"][0]\n",
    "print json.dumps(test_status, indent=4, sort_keys=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a sample Facebook page status, we can write a function to process each field individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(u'5281959998_10150628157724999', 'The planned megalopolis, a metropolitan area that would be about 6 times the size of New York\\xe2\\x80\\x99s, is meant to revamp northern China\\xe2\\x80\\x99s economy and become a laboratory for modern urban growth.', 'China Molds a Supercity Around Beijing, Promising to Change Lives', u'link', u'http://nyti.ms/1Jr6LhU', '2015-07-19 20:25:01', 278, 31, 50)\n"
     ]
    }
   ],
   "source": [
    "def processFacebookPageFeedStatus(status):\n",
    "    \n",
    "    # The status is now a Python dictionary, so for top-level items,\n",
    "    # we can simply call the key.\n",
    "    \n",
    "    # Additionally, some items may not always exist,\n",
    "    # so must check for existence first\n",
    "    \n",
    "    status_id = status['id']\n",
    "    status_message = '' if 'message' not in status.keys() else status['message'].encode('utf-8')\n",
    "    link_name = '' if 'name' not in status.keys() else status['name'].encode('utf-8')\n",
    "    status_type = status['type']\n",
    "    status_link = '' if 'link' not in status.keys() else status['link']\n",
    "    \n",
    "    \n",
    "    # Time needs special care since a) it's in UTC and\n",
    "    # b) it's not easy to use in statistical programs.\n",
    "    \n",
    "    status_published = datetime.datetime.strptime(status['created_time'],'%Y-%m-%dT%H:%M:%S+0000')\n",
    "    status_published = status_published + datetime.timedelta(hours=-5) # EST\n",
    "    status_published = status_published.strftime('%Y-%m-%d %H:%M:%S') # best time format for spreadsheet programs\n",
    "    \n",
    "    # Nested items require chaining dictionary keys.\n",
    "    \n",
    "    num_likes = 0 if 'likes' not in status.keys() else status['likes']['summary']['total_count']\n",
    "    num_comments = 0 if 'comments' not in status.keys() else status['comments']['summary']['total_count']\n",
    "    num_shares = 0 if 'shares' not in status.keys() else status['shares']['count']\n",
    "    \n",
    "    # return a tuple of all processed data\n",
    "    return (status_id, status_message, link_name, status_type, status_link,\n",
    "           status_published, num_likes, num_comments, num_shares)\n",
    "\n",
    "processed_test_status = processFacebookPageFeedStatus(test_status)\n",
    "print processed_test_status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, we're almost done! Now we just need to:\n",
    "\n",
    "1. Query each page of Facebook Page Statuses (100 statuses per page) using `getFacebookPageFeedData`.\n",
    "2. Process all statuses on that page using `processFacebookPageFeedStatus` and writing the output to a CSV file.\n",
    "3. Navigate to the next page, and repeat until no more statuses\n",
    "\n",
    "This block implements both the writing to CSV and page navigation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping nytimes Facebook Page: 2015-07-19 18:36:33.051000\n",
      "\n",
      "1000 Statuses Processed: 2015-07-19 18:36:59.366000\n",
      "2000 Statuses Processed: 2015-07-19 18:37:28.289000\n",
      "3000 Statuses Processed: 2015-07-19 18:37:56.487000\n",
      "4000 Statuses Processed: 2015-07-19 18:38:30.355000\n",
      "5000 Statuses Processed: 2015-07-19 18:38:58.661000\n",
      "6000 Statuses Processed: 2015-07-19 18:39:26.990000\n",
      "7000 Statuses Processed: 2015-07-19 18:39:55.906000\n",
      "8000 Statuses Processed: 2015-07-19 18:40:20.628000\n",
      "9000 Statuses Processed: 2015-07-19 18:40:44.801000\n",
      "10000 Statuses Processed: 2015-07-19 18:41:11.759000\n",
      "11000 Statuses Processed: 2015-07-19 18:41:38.739000\n",
      "12000 Statuses Processed: 2015-07-19 18:42:05.562000\n",
      "13000 Statuses Processed: 2015-07-19 18:42:32.696000\n",
      "14000 Statuses Processed: 2015-07-19 18:42:59.939000\n",
      "15000 Statuses Processed: 2015-07-19 18:43:26.889000\n",
      "16000 Statuses Processed: 2015-07-19 18:43:53.106000\n",
      "17000 Statuses Processed: 2015-07-19 18:44:19.457000\n",
      "18000 Statuses Processed: 2015-07-19 18:44:45.637000\n",
      "19000 Statuses Processed: 2015-07-19 18:45:11.255000\n",
      "20000 Statuses Processed: 2015-07-19 18:45:34.447000\n",
      "21000 Statuses Processed: 2015-07-19 18:45:58.425000\n",
      "22000 Statuses Processed: 2015-07-19 18:46:23.920000\n",
      "23000 Statuses Processed: 2015-07-19 18:46:49.274000\n",
      "24000 Statuses Processed: 2015-07-19 18:47:15.616000\n",
      "25000 Statuses Processed: 2015-07-19 18:47:39.930000\n",
      "26000 Statuses Processed: 2015-07-19 18:48:08.076000\n",
      "HTTP Error 502: Error parsing server response\n",
      "Error for URL https://graph.facebook.com/v2.0/5281959998/feed?fields=message,link,created_time,type,name,id,likes.limit%281%29.summary%28true%29,comments.limit%281%29.summary%28true%29,shares&limit=100&__paging_token=enc_AdBLHCQ9lOKXuEx1TEXyLWs7FEQ8RN7yGjUH0LXbw5iUpDXvcZCUIXJa2ZC2s6sBHC8EyrGl6Oafb9OqZBgBFzmuRZB9&access_token=272535582777707|59e7ab31b01d3a5a90ec15a7a45a5e3b&until=1340213557: 2015-07-19 18:48:23.256000\n",
      "27000 Statuses Processed: 2015-07-19 18:48:38.748000\n",
      "28000 Statuses Processed: 2015-07-19 18:49:03.033000\n",
      "29000 Statuses Processed: 2015-07-19 18:49:26.957000\n",
      "30000 Statuses Processed: 2015-07-19 18:49:51.405000\n",
      "31000 Statuses Processed: 2015-07-19 18:50:15.830000\n",
      "32000 Statuses Processed: 2015-07-19 18:50:37.641000\n",
      "33000 Statuses Processed: 2015-07-19 18:50:57.574000\n",
      "\n",
      "Done!\n",
      "33296 Statuses Processed in 0:14:28.200000\n"
     ]
    }
   ],
   "source": [
    "def scrapeFacebookPageFeedStatus(page_id, access_token):\n",
    "    with open('%s_facebook_statuses.csv' % page_id, 'wb') as file:\n",
    "        w = csv.writer(file)\n",
    "        w.writerow([\"status_id\", \"status_message\", \"link_name\", \"status_type\", \"status_link\",\n",
    "           \"status_published\", \"num_likes\", \"num_comments\", \"num_shares\"])\n",
    "        \n",
    "        has_next_page = True\n",
    "        num_processed = 0   # keep a count on how many we've processed\n",
    "        scrape_starttime = datetime.datetime.now()\n",
    "        \n",
    "        print \"Scraping %s Facebook Page: %s\\n\" % (page_id, scrape_starttime)\n",
    "        \n",
    "        statuses = getFacebookPageFeedData(page_id, access_token, 100)\n",
    "        \n",
    "        while has_next_page:\n",
    "            for status in statuses['data']:\n",
    "                w.writerow(processFacebookPageFeedStatus(status))\n",
    "                \n",
    "                # output progress occasionally to make sure code is not stalling\n",
    "                num_processed += 1\n",
    "                if num_processed % 1000 == 0:\n",
    "                    print \"%s Statuses Processed: %s\" % (num_processed, datetime.datetime.now())\n",
    "                    \n",
    "            # if there is no next page, we're done.\n",
    "            if 'paging' in statuses.keys():\n",
    "                statuses = json.loads(request_until_succeed(statuses['paging']['next']))\n",
    "            else:\n",
    "                has_next_page = False\n",
    "                \n",
    "        \n",
    "        print \"\\nDone!\\n%s Statuses Processed in %s\" % (num_processed, datetime.datetime.now() - scrape_starttime)\n",
    "\n",
    "\n",
    "scrapeFacebookPageFeedStatus(page_id, access_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CSV can be opened in all major statistical programs. Have fun! :)\n",
    "\n",
    "You can download the [NYTimes data here](https://dl.dropboxusercontent.com/u/2017402/nytimes_facebook_statuses.zip). [4.6MB]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}