Sprint2Day4N2_Linear_Regression_Modelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression using the Spotify Charts Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you build a linear model to predict a song's popularity using the metrics provided as features?\n",
    "\n",
    "The relevant metrics are 'popularity', 'danceability', 'energy','loudness','speechiness', 'acousticness', 'instrumentalness',\n",
    " 'liveness', 'valence', and 'tempo'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:06.394295Z",
     "start_time": "2020-06-02T06:43:06.339326Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:06.762084Z",
     "start_time": "2020-06-02T06:43:06.409291Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/spotify_daily_charts_tracks.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:06.832044Z",
     "start_time": "2020-06-02T06:43:06.776078Z"
    }
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:06.923991Z",
     "start_time": "2020-06-02T06:43:06.846036Z"
    }
   },
   "outputs": [],
   "source": [
    "# remove null\n",
    "df = df[~df['track_name'].isnull()]\n",
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:07.139868Z",
     "start_time": "2020-06-02T06:43:06.934985Z"
    }
   },
   "outputs": [],
   "source": [
    "#filter unnecessary fields\n",
    "df =df[['popularity', 'danceability', 'energy',\n",
    "       'loudness','speechiness', 'acousticness', 'instrumentalness',\n",
    "       'liveness', 'valence', 'tempo']]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Explore the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:07.456686Z",
     "start_time": "2020-06-02T06:43:07.152860Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Make a table of distribution stats of song metrics using df.describe\n",
    "\n",
    "df[['popularity', 'danceability', 'energy',\n",
    "       'loudness','speechiness', 'acousticness', 'instrumentalness',\n",
    "       'liveness', 'valence', 'tempo']].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:18.433395Z",
     "start_time": "2020-06-02T06:43:07.465680Z"
    }
   },
   "outputs": [],
   "source": [
    "# Visualize histograms of each song metric\n",
    "for col in ['popularity', 'danceability', 'energy','loudness', 'speechiness', 'acousticness', 'instrumentalness',\n",
    "       'liveness', 'valence', 'tempo']:\n",
    "    sns.distplot(df[col])\n",
    "    plt.title(col)\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize loudness and tempo. \n",
    ">Q: Whats the best norm to use for each?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:18.553326Z",
     "start_time": "2020-06-02T06:43:18.449388Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "df['loudness'] = scaler.fit_transform(df[['loudness']])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:18.698253Z",
     "start_time": "2020-06-02T06:43:18.563321Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "df['tempo'] = scaler.fit_transform(df[['tempo']])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the new loudness and tempo distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:21.019914Z",
     "start_time": "2020-06-02T06:43:18.708241Z"
    }
   },
   "outputs": [],
   "source": [
    "# Visualize histograms of each song metric\n",
    "for col in ['loudness', 'tempo']:\n",
    "    sns.distplot(df[col])\n",
    "    plt.title(col)\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Examine Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce variability, we could limit our analysis to only those songs that are sufficiently popular"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:21.063889Z",
     "start_time": "2020-06-02T06:43:21.027910Z"
    }
   },
   "outputs": [],
   "source": [
    "#filter songs with above median popularity\n",
    "df= df[df['popularity']>=61]\n",
    "df = df.reset_index(drop=True)\n",
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick 3 features that you think would give you a good fit.\n",
    ">Q: *Hypothesis*: Why do you think these 3 could be a good predictor for popularity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the relationship between the features and the response using scatterplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:23.227646Z",
     "start_time": "2020-06-02T06:43:21.073883Z"
    }
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, 3, figsize=(16, 8), sharey=True)\n",
    "df.plot(kind='scatter', x='danceability', y='popularity', ax=axs[0], grid=True)\n",
    "df.plot(kind='scatter', x='energy', y='popularity', ax=axs[1], grid=True)\n",
    "df.plot(kind='scatter', x='loudness', y='popularity', ax=axs[2], grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Fit the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the following steps for each of your selected features:\n",
    "\n",
    "a. Determine best fit line coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:23.313599Z",
     "start_time": "2020-06-02T06:43:23.232644Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "feature_cols = ['danceability']\n",
    "X = df[feature_cols]\n",
    "y = df['popularity']\n",
    "\n",
    "model = LinearRegression(fit_intercept=True)\n",
    "model.fit(X,y)\n",
    "\n",
    "print('Model slope: %0.4f' % model.coef_[0])\n",
    "print('Model intercept: %0.4f' % model.intercept_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Obtain the R2 for the fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:23.459514Z",
     "start_time": "2020-06-02T06:43:23.321594Z"
    }
   },
   "outputs": [],
   "source": [
    "print('Model R2: %0.4f' % model.score(X,y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Q: Interpret the model coefficients. What does the R2 value tell you about the fitted model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An increase of 0.1 in danceability will result to an increase of popularity by 0.7 points. \n",
    "But based from the R2, this is a very bad fit and so we hold back from this interpretation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Compute for RMSE and MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:23.600436Z",
     "start_time": "2020-06-02T06:43:23.465511Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "#define RMSE function\n",
    "def RMSE(model, X, y):\n",
    "    predicted = model.predict(X)\n",
    "    rmse = np.sqrt(mean_squared_error(y, predicted))\n",
    "    return rmse\n",
    "  \n",
    "#define MAE function\n",
    "def MAE(model, X, y):\n",
    "    predicted = model.predict(X)\n",
    "    mae = mean_absolute_error(y, predicted)\n",
    "    return mae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:23.750602Z",
     "start_time": "2020-06-02T06:43:23.606430Z"
    }
   },
   "outputs": [],
   "source": [
    "print('Model RMSE: %0.4f' % RMSE(model,X,y))\n",
    "print('Model MAE: %0.4f' % MAE(model,X,y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Q: What does the RMSE and MAE tell you about the model performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model's predicted popularity is more or less expected to be off by 8.22 points (conservative) or 6.9 points (equal weighting) for all songs considered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Check for outliers and determine if removing them could result to a better fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:43:24.085559Z",
     "start_time": "2020-06-02T06:43:23.755595Z"
    }
   },
   "outputs": [],
   "source": [
    "q1 = df['popularity'].quantile(0.25)\n",
    "q3 =  df['popularity'].quantile(0.75)\n",
    "IQR = q3 -q1\n",
    "\n",
    "outliers = df[(df['popularity']<(q1-1.5*IQR))&(df['popularity']>(q3+1.5*IQR))]['popularity']\n",
    "outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Using multiple features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Will using all 3 of your chosen features result into a better fit? Repeat the procedure in 4 and see if the metrics improve.\n",
    "If it did improve, do you think its enough to make the model more credible?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:46:19.835110Z",
     "start_time": "2020-06-02T06:46:19.795131Z"
    }
   },
   "outputs": [],
   "source": [
    "# create X and y\n",
    "feature_cols = ['danceability', 'energy', 'loudness']\n",
    "X = df[feature_cols]\n",
    "y = df['popularity']\n",
    "\n",
    "lm = LinearRegression()\n",
    "lm.fit(X, y)\n",
    "\n",
    "# print intercept and coefficients\n",
    "print(lm.intercept_)\n",
    "# pair the feature names with the coefficients\n",
    "print(list(zip(feature_cols, lm.coef_)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:46:21.678286Z",
     "start_time": "2020-06-02T06:46:21.651310Z"
    }
   },
   "outputs": [],
   "source": [
    "# calculate the R-squared\n",
    "lm.score(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Using k-fold cross validation\n",
    "We could further investigate on the models predictive performance using k-fold cross validation.\n",
    "What does folding reveal about the linear model you built?\n",
    "\n",
    "- For the model you built in (5), try the validation procedure for k=5 and k=10\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.to_numpy()\n",
    "Y = y.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-02T06:47:08.607913Z",
     "start_time": "2020-06-02T06:47:08.499974Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "kf = KFold(n_splits=5, random_state=None, shuffle=False)\n",
    "\n",
    "\n",
    "for train_index, test_index in kf.split(X):\n",
    "    #print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n",
    "    X_train, X_test = X[train_index], X[test_index]\n",
    "    Y_train, Y_test = Y[train_index], Y[test_index]\n",
    "    #print(np.shape(X_test), np.shape(Y_test))\n",
    "    \n",
    "    #fit using training data\n",
    "    lin_model = LinearRegression()\n",
    "    lin_model.fit(X_train, Y_train)\n",
    "    \n",
    "    #evaluate fit of train data\n",
    "    print('train: R2=%0.2f '% lin_model.score(X_train, Y_train))\n",
    "\n",
    "    #evaluate using test data\n",
    "    print('test: RMSE=%0.2f, R2=%0.2f' % (RMSE(lin_model, X_test, Y_test), lin_model.score(X_test,Y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> All training sets show poor fit. Some test sets produced relatively better fits, but this is only local to the fold since RMSE remains high."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}