last_fm_recommender.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Last.fm Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Last.fm data come from the [Music Technology Group](http://www.dtic.upf.edu/~ocelma/MusicRecommendationDataset/index.html) at the [Universitat Pompeu Fabra](http://mtg.upf.edu/) in Barcela, Spain. The data were scraped by Òscar Celma using the Last.fm API, and they are available free of charge. So, thank you Òscar!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Last.fm data are broken into two parts: the activity data and the profile data. The activity data is comprised of about 360,000 individual user's Last.fm artist listening information. It details how many times a Last.fm user played songs by various artists. The profile data contains each user's country of residence. We'll use `read.table` from `pandas` to read in the tab-delimited files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.spatial.distance import cosine\n",
    "import numpy as np\n",
    "from scipy.sparse import csr_matrix\n",
    "\n",
    "# display results to 3 decimal points, not in scientific notation\n",
    "pd.set_option('display.float_format', lambda x: '%.3f' % x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user_data = pd.read_table('/users/nickbecker/Downloads/lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv',\n",
    "                          header = None, nrows = 2e7,\n",
    "                          names = ['users', 'musicbrainz-artist-id', 'artist-name', 'plays'],\n",
    "                          usecols = ['users', 'artist-name', 'plays'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user_profiles = pd.read_table('/users/nickbecker/Downloads/lastfm-dataset-360K/usersha1-profile.tsv',\n",
    "                          header = None,\n",
    "                          names = ['users', 'gender', 'age', 'country', 'signup'],\n",
    "                          usecols = ['users', 'country'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick look at the two datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>artist-name</th>\n",
       "      <th>plays</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>betty blowtorch</td>\n",
       "      <td>2137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>die Ärzte</td>\n",
       "      <td>1099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>melissa etheridge</td>\n",
       "      <td>897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>elvenking</td>\n",
       "      <td>717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>juliette &amp; the licks</td>\n",
       "      <td>706</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      users           artist-name  plays\n",
       "0  00000c289a1829a808ac09c00daf10bc3c4e223b       betty blowtorch   2137\n",
       "1  00000c289a1829a808ac09c00daf10bc3c4e223b             die Ärzte   1099\n",
       "2  00000c289a1829a808ac09c00daf10bc3c4e223b     melissa etheridge    897\n",
       "3  00000c289a1829a808ac09c00daf10bc3c4e223b             elvenking    717\n",
       "4  00000c289a1829a808ac09c00daf10bc3c4e223b  juliette & the licks    706"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00001411dc427966b17297bf4d69e7e193135d89</td>\n",
       "      <td>Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00004d2ac9316e22dc007ab2243d6fcb239e707d</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>000063d3fe1cf2ba248b9e3c3f0334845a27a6bf</td>\n",
       "      <td>Mexico</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00007a47085b9aab8af55f52ec8846ac479ac4fe</td>\n",
       "      <td>United States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      users        country\n",
       "0  00000c289a1829a808ac09c00daf10bc3c4e223b        Germany\n",
       "1  00001411dc427966b17297bf4d69e7e193135d89         Canada\n",
       "2  00004d2ac9316e22dc007ab2243d6fcb239e707d        Germany\n",
       "3  000063d3fe1cf2ba248b9e3c3f0334845a27a6bf         Mexico\n",
       "4  00007a47085b9aab8af55f52ec8846ac479ac4fe  United States"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_profiles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the datasets loaded in memory, we can start doing some data cleaning and eventually make recommendations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering to Only Popular Artists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we're going to be doing item-based collaborative filtering, our recommendations will be based on user patterns in listening to artists. Lesser known artists will have views from fewer viewers, making the pattern more noisy. This would probably result in bad recommendations (or at least ones highly sensitive to an individual person who _**loves**_ one artist. To avoid this problem, we'll just look at the popular artists.\n",
    "\n",
    "To find out which artists are popular, we need to know the total play count of every artist. Since our user play count data has one row per artist per user, we need to aggregate it up to the artist level. With `pandas`, we can group by the artist name and then calculate the sum of the `plays` column for every artist. If the `artist-name` variable is missing, our future reshaping and analysis won't work. So I'll start by removing rows where the artist name is missing just to be safe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if user_data['artist-name'].isnull().sum() > 0:\n",
    "    user_data = user_data.dropna(axis = 0, subset = ['artist-name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "artist_plays = (user_data.\n",
    "     groupby(by = ['artist-name'])['plays'].\n",
    "     sum().\n",
    "     reset_index().\n",
    "     rename(columns = {'plays': 'total_artist_plays'})\n",
    "     [['artist-name', 'total_artist_plays']]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artist-name</th>\n",
       "      <th>total_artist_plays</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>04)]</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>58725ab=&gt;</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>80lİ yillarin tÜrkÇe sÖzlÜ aŞk Şarkilari</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>amy winehouse</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 artist-name  total_artist_plays\n",
       "0                                       04)]                   6\n",
       "1                                         2                 1606\n",
       "2                                  58725ab=>                  23\n",
       "3   80lİ yillarin tÜrkÇe sÖzlÜ aŞk Şarkilari                  70\n",
       "4                              amy winehouse                  23"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artist_plays.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can merge the total play count data into the user activity data, giving us exactly what we need to filter out the lesser known artists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "user_data_with_artist_plays = user_data.merge(artist_plays, left_on = 'artist-name', right_on = 'artist-name', how = 'left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>artist-name</th>\n",
       "      <th>plays</th>\n",
       "      <th>total_artist_plays</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>betty blowtorch</td>\n",
       "      <td>2137</td>\n",
       "      <td>25651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>die Ärzte</td>\n",
       "      <td>1099</td>\n",
       "      <td>3704875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>melissa etheridge</td>\n",
       "      <td>897</td>\n",
       "      <td>180391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>elvenking</td>\n",
       "      <td>717</td>\n",
       "      <td>410725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>juliette &amp; the licks</td>\n",
       "      <td>706</td>\n",
       "      <td>90498</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      users           artist-name  plays  \\\n",
       "0  00000c289a1829a808ac09c00daf10bc3c4e223b       betty blowtorch   2137   \n",
       "1  00000c289a1829a808ac09c00daf10bc3c4e223b             die Ärzte   1099   \n",
       "2  00000c289a1829a808ac09c00daf10bc3c4e223b     melissa etheridge    897   \n",
       "3  00000c289a1829a808ac09c00daf10bc3c4e223b             elvenking    717   \n",
       "4  00000c289a1829a808ac09c00daf10bc3c4e223b  juliette & the licks    706   \n",
       "\n",
       "   total_artist_plays  \n",
       "0               25651  \n",
       "1             3704875  \n",
       "2              180391  \n",
       "3              410725  \n",
       "4               90498  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_data_with_artist_plays.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Picking a threshold for popular artists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With nearly 300,000 different artists, it's almost a guarantee most artists have been played only a few times. Let's look at the descriptive statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count     292364.000\n",
      "mean       12907.037\n",
      "std       185981.313\n",
      "min            1.000\n",
      "25%           53.000\n",
      "50%          208.000\n",
      "75%         1048.000\n",
      "max     30466827.000\n",
      "Name: total_artist_plays, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print artist_plays['total_artist_plays'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the median artist has only been played about 200 times. Let's take a look at the top of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.900     6138.000\n",
      "0.910     7410.000\n",
      "0.920     9102.960\n",
      "0.930    11475.590\n",
      "0.940    14898.440\n",
      "0.950    19964.250\n",
      "0.960    28419.880\n",
      "0.970    43541.330\n",
      "0.980    79403.440\n",
      "0.990   198482.590\n",
      "Name: total_artist_plays, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print artist_plays['total_artist_plays'].quantile(np.arange(.9, 1, .01)), "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So about 1% of artists have roughly 200,000 or more plays, 2% have 80,000 or more, and 3% have 40,000 or more. Since we have so many artists, we'll limit it to the top 3%. This is arbitrary threshold for popularity, but it gives us about 9000 different artists. It seems to me that including more than that might push us into the problem of noisy listening patterns we're trying to avoid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>artist-name</th>\n",
       "      <th>plays</th>\n",
       "      <th>total_artist_plays</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>die Ärzte</td>\n",
       "      <td>1099</td>\n",
       "      <td>3704875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>melissa etheridge</td>\n",
       "      <td>897</td>\n",
       "      <td>180391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>elvenking</td>\n",
       "      <td>717</td>\n",
       "      <td>410725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>juliette &amp; the licks</td>\n",
       "      <td>706</td>\n",
       "      <td>90498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>red hot chili peppers</td>\n",
       "      <td>691</td>\n",
       "      <td>13547741</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      users            artist-name  plays  \\\n",
       "1  00000c289a1829a808ac09c00daf10bc3c4e223b              die Ärzte   1099   \n",
       "2  00000c289a1829a808ac09c00daf10bc3c4e223b      melissa etheridge    897   \n",
       "3  00000c289a1829a808ac09c00daf10bc3c4e223b              elvenking    717   \n",
       "4  00000c289a1829a808ac09c00daf10bc3c4e223b   juliette & the licks    706   \n",
       "5  00000c289a1829a808ac09c00daf10bc3c4e223b  red hot chili peppers    691   \n",
       "\n",
       "   total_artist_plays  \n",
       "1             3704875  \n",
       "2              180391  \n",
       "3              410725  \n",
       "4               90498  \n",
       "5            13547741  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "popularity_threshold = 40000\n",
    "user_data_popular_artists = user_data_with_artist_plays.query('total_artist_plays >= @popularity_threshold')\n",
    "user_data_popular_artists.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering to US Users Only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since I'm in Washington, D.C., I'll limit the user data to just those from the United States. This will reduce the number of users and artists considerably, improving the computational efficiency significantly.\n",
    "\n",
    "First, I'll merge in the user profile data that has the user's country. Then I'll filter the data to only users in the United States."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>users</th>\n",
       "      <th>artist-name</th>\n",
       "      <th>plays</th>\n",
       "      <th>total_artist_plays</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>00007a47085b9aab8af55f52ec8846ac479ac4fe</td>\n",
       "      <td>devendra banhart</td>\n",
       "      <td>456</td>\n",
       "      <td>2366807</td>\n",
       "      <td>United States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>00007a47085b9aab8af55f52ec8846ac479ac4fe</td>\n",
       "      <td>boards of canada</td>\n",
       "      <td>407</td>\n",
       "      <td>6115545</td>\n",
       "      <td>United States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>00007a47085b9aab8af55f52ec8846ac479ac4fe</td>\n",
       "      <td>cocorosie</td>\n",
       "      <td>386</td>\n",
       "      <td>2194862</td>\n",
       "      <td>United States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>00007a47085b9aab8af55f52ec8846ac479ac4fe</td>\n",
       "      <td>aphex twin</td>\n",
       "      <td>213</td>\n",
       "      <td>4248296</td>\n",
       "      <td>United States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>00007a47085b9aab8af55f52ec8846ac479ac4fe</td>\n",
       "      <td>animal collective</td>\n",
       "      <td>203</td>\n",
       "      <td>3495537</td>\n",
       "      <td>United States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        users        artist-name  plays  \\\n",
       "156  00007a47085b9aab8af55f52ec8846ac479ac4fe   devendra banhart    456   \n",
       "157  00007a47085b9aab8af55f52ec8846ac479ac4fe   boards of canada    407   \n",
       "158  00007a47085b9aab8af55f52ec8846ac479ac4fe          cocorosie    386   \n",
       "159  00007a47085b9aab8af55f52ec8846ac479ac4fe         aphex twin    213   \n",
       "160  00007a47085b9aab8af55f52ec8846ac479ac4fe  animal collective    203   \n",
       "\n",
       "     total_artist_plays        country  \n",
       "156             2366807  United States  \n",
       "157             6115545  United States  \n",
       "158             2194862  United States  \n",
       "159             4248296  United States  \n",
       "160             3495537  United States  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined = user_data_popular_artists.merge(user_profiles, left_on = 'users', right_on = 'users', how = 'left')\n",
    "usa_data = combined.query('country == \\'United States\\'')\n",
    "usa_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before doing any analysis, we should make sure dataset is internally consistent. Every user should only have a play count variable once for each artist. So we'll check for instances where rows have the same `users` and `artist-name` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial dataframe shape (2788019, 5)\n",
      "New dataframe shape (2788013, 5)\n",
      "Removed 6 rows\n"
     ]
    }
   ],
   "source": [
    "if not usa_data[usa_data.duplicated(['users', 'artist-name'])].empty:\n",
    "    initial_rows = usa_data.shape[0]\n",
    "    \n",
    "    print 'Initial dataframe shape {0}'.format(usa_data.shape)\n",
    "    usa_data = usa_data.drop_duplicates(['users', 'artist-name'])\n",
    "    current_rows = usa_data.shape[0]\n",
    "    print 'New dataframe shape {0}'.format(usa_data.shape)\n",
    "    print 'Removed {0} rows'.format(initial_rows - current_rows)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implemeting the Nearest Neighbor Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reshaping the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For K-Nearest Neighbors, we want the data to be in an `(artist, user)` array, where each row is an artist and each column is a different user. To reshape the dataframe, we'll `pivot` the dataframe to the wide format with artists as rows and users as columns. Then we'll fill the missing observations with `0`s since we're going to be performing linear algebra operations (calculating distances between vectors). Finally, we transform the values of the dataframe into a scipy sparse matrix for more efficient calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "wide_artist_data = usa_data.pivot(index = 'artist-name', columns = 'users', values = 'plays').fillna(0)\n",
    "wide_artist_data_sparse = csr_matrix(wide_artist_data.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# save the model for future \n",
    "from scipy.sparse import csr_matrix\n",
    "\n",
    "def save_sparse_csr(filename,array):\n",
    "    np.savez(filename,data = array.data ,indices=array.indices,\n",
    "             indptr =array.indptr, shape=array.shape )\n",
    "\n",
    "def load_sparse_csr(filename):\n",
    "    loader = np.load(filename)\n",
    "    return csr_matrix((loader['data'], loader['indices'], loader['indptr']),\n",
    "                         shape = loader['shape'])\n",
    "\n",
    "save_sparse_csr('/users/nickbecker/Python_Projects/lastfm_sparse_artist_matrix.npz', wide_artist_data_sparse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to implement the model. We'll initialize the `NearestNeighbors` class as `model_knn` and `fit` our sparse matrix to the instance. By specifying the `metric = cosine`, the model will measure similarity bectween artist vectors by using cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NearestNeighbors(algorithm='brute', leaf_size=30, metric='cosine',\n",
       "         metric_params=None, n_jobs=1, n_neighbors=5, p=2, radius=1.0)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import NearestNeighbors\n",
    "\n",
    "model_knn = NearestNeighbors(metric = 'cosine', algorithm = 'brute')\n",
    "model_knn.fit(wide_artist_data_sparse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Making Recommendations\n",
    "\n",
    "And we're finally ready to make some recommendations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8374\n",
      "Recommendations for tony bennett:\n",
      "\n",
      "1: frank sinatra, with distance of 0.226917809755:\n",
      "2: keiko matsui, with distance of 0.39397227453:\n",
      "3: andy williams, with distance of 0.477163884119:\n",
      "4: chic, with distance of 0.488077997533:\n",
      "5: cherry poppin daddies, with distance of 0.4908909547:\n"
     ]
    }
   ],
   "source": [
    "query_index = np.random.choice(wide_data.shape[0])\n",
    "print query_index\n",
    "distances, indices = model_knn.kneighbors(wide_artist_data.iloc[query_index, :].reshape(1, -1), n_neighbors = 6)\n",
    "\n",
    "for i in range(0, len(distances.flatten())):\n",
    "    if i == 0:\n",
    "        print 'Recommendations for {0}:\\n'.format(wide_artist_data.index[query_index])\n",
    "    else:\n",
    "        print '{0}: {1}, with distance of {2}:'.format(i, wide_artist_data.index[indices.flatten()[i]], distances.flatten()[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty good! Frank Sinatra and Andy Williams are obviously good recommendations. I'd never heard of [Keiko Matsui](https://www.youtube.com/watch?v=XzqsWxau_gI) or [Cherry Poppin Daddies](https://www.youtube.com/watch?v=1IqH3uliwJY), but they both seem like good recommendations after listening to their music. [Chic](https://www.youtube.com/watch?v=eKl6EZShaaw), though, doesn't seem as similar to me as the other artists (they sound a little more disco)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why would our model recommend Chic? Since we're doing item-based collaborative filtering with K-Nearest Neighbors on the actual play count data, outliers can have a big influence. If a few users listened to Tony Bennett and Chic a _**ton**_, our distance metric between vectors will be heavily influenced by those individual observations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So is this good? Maybe. Depending on our goal, we might want **super-fans** to have disproportionate weight in the distance calculation. But could we represent the data differently to avoid this feature?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary Play Count Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, we used the actual play counts as values in our artist vectors. Another approach would be convert each vector into a binary (1 or 0): either a user played the song or they did not. We can do this by applying the `sign` function in `numpy` to each column in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wide_artist_data_zero_one = wide_artist_data.apply(np.sign)\n",
    "wide_artist_data_zero_one_sparse = csr_matrix(wide_artist_data_zero_one.values)\n",
    "\n",
    "save_sparse_csr('/users/nickbecker/Python_Projects/lastfm_sparse_artist_matrix_binary.npz', wide_artist_data_zero_one_sparse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NearestNeighbors(algorithm='brute', leaf_size=30, metric='cosine',\n",
       "         metric_params=None, n_jobs=1, n_neighbors=5, p=2, radius=1.0)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_nn_binary = NearestNeighbors(metric='cosine', algorithm='brute')\n",
    "model_nn_binary.fit(wide_artist_data_zero_one_sparse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a quick comparison. Which recommendations for Tony Bennett look better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommendations with binary play data for tony bennett:\n",
      "\n",
      "1: nat king cole, with distance of 0.771590841231:\n",
      "2: dean martin, with distance of 0.791135426625:\n",
      "3: frank sinatra, with distance of 0.815388695965:\n",
      "4: bobby darin, with distance of 0.818033228367:\n",
      "5: doris day, with distance of 0.81859043384:\n"
     ]
    }
   ],
   "source": [
    "query_index = np.random.choice(wide_artist_data_zero_one.shape[0])\n",
    "#print query_index\n",
    "distances, indices = model_nn_binary.kneighbors(wide_artist_data_zero_one.iloc[query_index, :].reshape(1, -1), n_neighbors = 6)\n",
    "\n",
    "for i in range(0, len(distances.flatten())):\n",
    "    if i == 0:\n",
    "        print 'Recommendations with binary play data for {0}:\\n'.format(wide_artist_data_zero_one.index[query_index])\n",
    "    else:\n",
    "        print '{0}: {1}, with distance of {2}:'.format(i, wide_artist_data_zero_one.index[indices.flatten()[i]], distances.flatten()[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "These are great, too. At least for Tony Bennett, the binary data representation recommendations looks just as good. Someone who likes Tony Bennett might also like Nat King Cole or Frank Sinatra. The distances are higher, but that's due to squashing the data by using the sign function. Again, it's not obvious which method is better. Since ultimately it's the listeners's future actions that indicate which recommender system is better, it's a perfect candidate for A/B Testing. For now, I'll stick with the binary data representation model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommending Artists with Fuzzy Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously we picked query artists at random. But really, we want to make recommendations for a specific artist on command. Since some artists's names are ambiguous or commonly mispelled, we'll include fuzzy matching part in the process so we don't need exact name matches.\n",
    "\n",
    "So we can do this anytime we want, we'll define a function `print_artist_recommendations` to do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import string\n",
    "from fuzzywuzzy import fuzz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_artist_recommendations(query_artist, artist_plays_matrix, knn_model, k):\n",
    "    \"\"\"\n",
    "    Inputs:\n",
    "    query_artist: query artist name\n",
    "    artist_plays_matrix: artist play count dataframe (not the sparse one, the pandas dataframe)\n",
    "    knn_model: our previously fitted sklearn knn model\n",
    "    k: the number of nearest neighbors.\n",
    "    \n",
    "    Prints: Artist recommendations for the query artist\n",
    "    Returns: None\n",
    "    \"\"\"\n",
    "    query_index = None\n",
    "    ratio_tuples = []\n",
    "    \n",
    "    for i in artist_plays_matrix.index:\n",
    "        ratio = fuzz.ratio(i.lower(), query_artist.lower())\n",
    "        if ratio >= 75:\n",
    "            current_query_index = artist_plays_matrix.index.tolist().index(i)\n",
    "            ratio_tuples.append((i, ratio, current_query_index))\n",
    "    \n",
    "    print 'Possible matches: {0}\\n'.format([(x[0], x[1]) for x in ratio_tuples])\n",
    "    \n",
    "    try:\n",
    "        query_index = max(ratio_tuples, key = lambda x: x[1])[2] # get the index of the best artist match in the data\n",
    "    except:\n",
    "        print 'Your artist didn\\'t match any artists in the data. Try again'\n",
    "        return None\n",
    "    \n",
    "    distances, indices = knn_model.kneighbors(artist_plays_matrix.iloc[query_index, :].reshape(1, -1), n_neighbors = k + 1)\n",
    "\n",
    "    for i in range(0, len(distances.flatten())):\n",
    "        if i == 0:\n",
    "            print 'Recommendations for {0}:\\n'.format(artist_plays_matrix.index[query_index])\n",
    "        else:\n",
    "            print '{0}: {1}, with distance of {2}:'.format(i, artist_plays_matrix.index[indices.flatten()[i]], distances.flatten()[i])\n",
    "\n",
    "    return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to try a few sample bands and get some recommendations for music I might like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible matches: [('red hot chili peppers', 100)]\n",
      "\n",
      "Recommendations for red hot chili peppers:\n",
      "\n",
      "1: incubus, with distance of 0.686632912166:\n",
      "2: the beatles, with distance of 0.693856742888:\n",
      "3: sublime, with distance of 0.70540037526:\n",
      "4: foo fighters, with distance of 0.71155686859:\n",
      "5: coldplay, with distance of 0.716691422348:\n",
      "6: led zeppelin, with distance of 0.722488787624:\n",
      "7: nirvana, with distance of 0.724943983169:\n",
      "8: green day, with distance of 0.734603813118:\n",
      "9: radiohead, with distance of 0.737372302802:\n",
      "10: rage against the machine, with distance of 0.740136491957:\n"
     ]
    }
   ],
   "source": [
    "print_artist_recommendations('red hot chili peppers', wide_artist_data_zero_one, model_nn_binary, k = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible matches: [('arctic monkeys', 100)]\n",
      "\n",
      "Recommendations for arctic monkeys:\n",
      "\n",
      "1: the strokes, with distance of 0.746696592481:\n",
      "2: the kooks, with distance of 0.767492571954:\n",
      "3: bloc party, with distance of 0.772120302741:\n",
      "4: franz ferdinand, with distance of 0.774566073856:\n",
      "5: the killers, with distance of 0.807176759929:\n",
      "6: radiohead, with distance of 0.812762633074:\n",
      "7: the fratellis, with distance of 0.814611330462:\n",
      "8: kings of leon, with distance of 0.815408152181:\n",
      "9: the beatles, with distance of 0.815680085574:\n",
      "10: the white stripes, with distance of 0.81607343278:\n"
     ]
    }
   ],
   "source": [
    "print_artist_recommendations('arctic monkeys', wide_artist_data_zero_one, model_nn_binary, k = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible matches: [('u2', 100)]\n",
      "\n",
      "Recommendations for u2:\n",
      "\n",
      "1: r.e.m., with distance of 0.690057376797:\n",
      "2: coldplay, with distance of 0.697303504846:\n",
      "3: the beatles, with distance of 0.726085401263:\n",
      "4: the police, with distance of 0.756131589948:\n",
      "5: radiohead, with distance of 0.77692434746:\n",
      "6: pearl jam, with distance of 0.778566201394:\n",
      "7: the rolling stones, with distance of 0.782771546531:\n",
      "8: led zeppelin, with distance of 0.788269370325:\n",
      "9: dave matthews band, with distance of 0.788520439902:\n",
      "10: bruce springsteen, with distance of 0.789619233997:\n"
     ]
    }
   ],
   "source": [
    "print_artist_recommendations('u2', wide_artist_data_zero_one, model_nn_binary, k = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible matches: [('dispatch', 100)]\n",
      "\n",
      "Recommendations for dispatch:\n",
      "\n",
      "1: state radio, with distance of 0.665125909027:\n",
      "2: o.a.r., with distance of 0.749067403207:\n",
      "3: jack johnson, with distance of 0.778821492779:\n",
      "4: dave matthews band, with distance of 0.792342999987:\n",
      "5: guster, with distance of 0.821963512261:\n",
      "6: ben harper, with distance of 0.824748164332:\n",
      "7: slightly stoopid, with distance of 0.837663503717:\n",
      "8: the john butler trio, with distance of 0.841162765581:\n",
      "9: donavon frankenreiter, with distance of 0.841397926422:\n",
      "10: sublime, with distance of 0.846795058358:\n"
     ]
    }
   ],
   "source": [
    "print_artist_recommendations('dispatch', wide_artist_data_zero_one, model_nn_binary, k = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible matches: [('pearl jam', 100)]\n",
      "\n",
      "Recommendations for pearl jam:\n",
      "\n",
      "1: stone temple pilots, with distance of 0.706407198885:\n",
      "2: alice in chains, with distance of 0.714291500864:\n",
      "3: nirvana, with distance of 0.724837631157:\n",
      "4: soundgarden, with distance of 0.728456327455:\n",
      "5: foo fighters, with distance of 0.741056843206:\n",
      "6: the smashing pumpkins, with distance of 0.751413106905:\n",
      "7: red hot chili peppers, with distance of 0.759106030065:\n",
      "8: led zeppelin, with distance of 0.76954203648:\n",
      "9: u2, with distance of 0.778566201394:\n",
      "10: radiohead, with distance of 0.793734608023:\n"
     ]
    }
   ],
   "source": [
    "print_artist_recommendations('pearl jam', wide_artist_data_zero_one, model_nn_binary, k = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be brief, these are fantastic. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future Ideas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scaling up to Massive Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we're using K-Nearest Neighbors, we have to calculate the distance of each artist vector in our `wide_artist_data_sparse` array to the query artist vector every time we make a query. If our data is fairly small (like in this post), this isn't an issue.\n",
    "\n",
    "If we had the entirety of Last.fm's user data, we'd be bottlenecked like crazy at query time. Fortunately, there's been great work done on Approximate Nearest Neighbor Search techniques such as [locality sensitive hashing](http://www.mit.edu/~andoni/LSH/). These techniques sacrifice the **guarantee** of finding the nearest neighbors for  increases in computational efficiency, and work extremely well with high dimensional data. The [Machine Learning: Clustering & Retrieval](https://www.coursera.org/learn/ml-clustering-and-retrieval) course on Coursera has a great walk-through of LSH for those curious."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Recommending less popular artists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While our recommendation engine is doing a great job, it's only recommending popular artists (by design). A really cool alternative recommender might recommend us unknown artists given a query artist so we can discover new music. \n",
    "\n",
    "Recommending lesser known artists is a huge challenge that does fit well with collaborative filtering, so we might want to incorporate feature based recommendations into such a system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible matches: [('ana johnsson', 75), ('eric johnson', 75), ('jack johnson', 100), ('jan johansson', 80)]\n",
      "\n",
      "Recommendations for jack johnson:\n",
      "\n",
      "1: john mayer, with distance of 0.654393490687:\n",
      "2: coldplay, with distance of 0.680812261515:\n",
      "3: dave matthews band, with distance of 0.70400482133:\n",
      "4: jason mraz, with distance of 0.71803418231:\n",
      "5: death cab for cutie, with distance of 0.72085237553:\n",
      "6: the beatles, with distance of 0.731239048976:\n",
      "7: red hot chili peppers, with distance of 0.743804519069:\n",
      "8: ben harper, with distance of 0.747046316392:\n",
      "9: dispatch, with distance of 0.778821492779:\n",
      "10: the shins, with distance of 0.781652817882:\n"
     ]
    }
   ],
   "source": [
    "print_artist_recommendations('jack johnson', wide_artist_data_zero_one, model_nn_binary, k = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}