NCAA_GamePlay_Kaggle2019.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NCAA GamePlay-Kaggle2019.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/todnewman/coe_training/blob/master/NCAA_GamePlay_Kaggle2019.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "zKmslqfjSpby",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## NCAA Basketball Gameplay and Win Probability Predictor\n",
        "\n",
        "This Notebook contains everything needed to predict the NCAA tournament.  The prediction is based upon over 10 years worth of NCAA tournament game results.  Features in the data consist of Kenpom.com features (I have to pay him $20 a year to update my data) and a few extras I have blended in."
      ]
    },
    {
      "metadata": {
        "id": "plAv8RYsSpcR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 270
        },
        "outputId": "652a50d4-1d31-4af8-c881-1c5a456d6337"
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import math\n",
        "import matplotlib\n",
        "import sklearn as sk\n",
        "import numpy as np\n",
        "from sklearn import preprocessing\n",
        "from sklearn import metrics\n",
        "import csv\n",
        "from itertools import tee\n",
        "import sys\n",
        "from random import random\n",
        "!pip install mlxtend\n",
        "%matplotlib inline\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: mlxtend in /usr/local/lib/python3.6/dist-packages (0.14.0)\n",
            "Requirement already satisfied: scipy>=0.17 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.1.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from mlxtend) (40.8.0)\n",
            "Requirement already satisfied: pandas>=0.17.1 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (0.22.0)\n",
            "Requirement already satisfied: scikit-learn>=0.18 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (0.20.3)\n",
            "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.14.6)\n",
            "Requirement already satisfied: matplotlib>=1.5.1 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (3.0.3)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.1->mlxtend) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.1->mlxtend) (2.5.3)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (1.0.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (2.3.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2->pandas>=0.17.1->mlxtend) (1.11.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "nWiAAUVWSpcf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Location of Data\n",
        "\n",
        "Change the below to wherever the input files live (this .ipynb file and the actual_xx.csv files).  This is also where data outputs will go.\n",
        "\n",
        "* SampleSubmissionStage2_0.csv:  Here is the output file formatted for Kaggle submission.\n",
        "* pred_xx_out.csv: This is the predictive dataframe for each round (in case you're interested in looking the data over.\n",
        "* NCAA_game_pred-xx-lasso.csv:  This is the results of the games played the number represents the win margin for the team in the first column.  The last two columns are the mean value of the win margin and the standard deviation.\n",
        "* There is also a logfile, which should be very useful.  It will go wherever you tell it to go in the main block."
      ]
    },
    {
      "metadata": {
        "id": "yc1Gq95RVEvo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Mount YOUR Google Drive\n",
        "\n",
        "This will bring up a link you need to click to allow this script access to save files in your google drive."
      ]
    },
    {
      "metadata": {
        "id": "2q7uKy7vVFd3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 810
        },
        "outputId": "29792725-050c-455c-c8c7-992bca8dccf9"
      },
      "cell_type": "code",
      "source": [
        "#Here's where you can mount your own Google Drive and copy results from /content into it...\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd /content/drive/'My Drive'/data\n",
        "%ls\n",
        "\n"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n",
            "/content/drive/My Drive/data\n",
            "1_FBB_2017_matrix.png              PlayByPlay_2011.zip\n",
            "1_rain_confusion_matrix.png        PlayByPlay_2012.zip\n",
            "2_FBB_2017_matrix.png              PlayByPlay_2013.zip\n",
            "2_rain_confusion_matrix.png        PlayByPlay_2014.zip\n",
            "Cities.csv                         PlayByPlay_2015.zip\n",
            "Conferences.csv                    PlayByPlay_2016.zip\n",
            "ConferenceTourneyGames.csv         PlayByPlay_2017.zip\n",
            "corr_mat.png                       PlayByPlay_2018.zip\n",
            "DataFiles.zip                      Players_2010.csv\n",
            "Events_2010.csv                    Players_2011.csv\n",
            "Events_2011.csv                    Players_2012.csv\n",
            "Events_2012.csv                    Players_2013.csv\n",
            "Events_2013.csv                    Players_2014.csv\n",
            "Events_2014.csv                    Players_2015.csv\n",
            "Events_2015.csv                    Players_2016.csv\n",
            "Events_2016.csv                    Players_2017.csv\n",
            "Events_2017.csv                    Players_2018.csv\n",
            "Events_2018.csv                    rain_model\n",
            "FBB_pitchers_predicted_out.csv     rain_seq_model\n",
            "FBB_predicted_hitters_out.csv      RegularSeasonCompactResults.csv\n",
            "GameCities.csv                     RegularSeasonDetailedResults.csv\n",
            "hitter_data.csv                    results_16.csv\n",
            "lightning_sensor_data.csv          results_2.csv\n",
            "lightning_sensor_data.gsheet       results_2.gsheet\n",
            "lightning_sensor_data_predict.csv  results_32.csv\n",
            "MasseyOrdinals.csv                 results_4.csv\n",
            "MasseyOrdinals.zip                 results_64.csv\n",
            "ncaa_logfile.gdoc                  results_8.csv\n",
            "ncaa_logfile.txt                   SampleSubmissionStage1.csv\n",
            "NCAATourneyCompactResults.csv      Seasons.csv\n",
            "NCAATourneyDetailedResults.csv     SecondaryTourneyCompactResults.csv\n",
            "NCAATourneySeedRoundSlots.csv      SecondaryTourneyTeams.csv\n",
            "NCAATourneySeeds.csv               TeamCoaches.csv\n",
            "NCAATourneySlots.csv               TeamConferences.csv\n",
            "pitcher_data.csv                   Teams.csv\n",
            "pitcher_data.gsheet                TeamSpellings.csv\n",
            "PlayByPlay_2010.zip\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Y2DpLXY6SpdB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Dataset Download Area\n",
        "\n",
        "Here we grab data from Github.  It gets saved in /content."
      ]
    },
    {
      "metadata": {
        "id": "YX_ojNxkSpdE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 981
        },
        "outputId": "3f334dca-8843-4a63-f4eb-5250f3860c53"
      },
      "cell_type": "code",
      "source": [
        "!wget -O /content/kenpom_2019.csv 'https://raw.githubusercontent.com/todnewman/data/master/kenpom_2019.csv'\n",
        "!wget -O /content/historical_tourney_data_2019.csv 'https://raw.githubusercontent.com/todnewman/data/master/historical_tourney_data_2019.csv'\n",
        "!wget -O /content/SampleSubmissionStage2-temp.csv  'https://raw.githubusercontent.com/todnewman/data/master/SampleSubmissionStage2-temp.csv'\n",
        "!wget -O /content/Teams.csv 'https://raw.githubusercontent.com/todnewman/data/master/Teams.csv'\n",
        "!wget -O /content/pytools.py 'https://raw.githubusercontent.com/todnewman/data/master/pytools.py'\n",
        "\n",
        "%cd /content\n",
        "%ls\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-03-29 02:50:35--  https://raw.githubusercontent.com/todnewman/data/master/kenpom_2019.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1993149 (1.9M) [text/plain]\n",
            "Saving to: ‘/content/kenpom_2019.csv’\n",
            "\n",
            "\r/content/kenpom_201   0%[                    ]       0  --.-KB/s               \r/content/kenpom_201 100%[===================>]   1.90M  --.-KB/s    in 0.08s   \n",
            "\n",
            "2019-03-29 02:50:35 (23.3 MB/s) - ‘/content/kenpom_2019.csv’ saved [1993149/1993149]\n",
            "\n",
            "--2019-03-29 02:50:36--  https://raw.githubusercontent.com/todnewman/data/master/historical_tourney_data_2019.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 563356 (550K) [text/plain]\n",
            "Saving to: ‘/content/historical_tourney_data_2019.csv’\n",
            "\n",
            "/content/historical 100%[===================>] 550.15K  --.-KB/s    in 0.05s   \n",
            "\n",
            "2019-03-29 02:50:36 (10.5 MB/s) - ‘/content/historical_tourney_data_2019.csv’ saved [563356/563356]\n",
            "\n",
            "--2019-03-29 02:50:37--  https://raw.githubusercontent.com/todnewman/data/master/SampleSubmissionStage2-temp.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 43290 (42K) [text/plain]\n",
            "Saving to: ‘/content/SampleSubmissionStage2-temp.csv’\n",
            "\n",
            "/content/SampleSubm 100%[===================>]  42.28K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2019-03-29 02:50:37 (2.10 MB/s) - ‘/content/SampleSubmissionStage2-temp.csv’ saved [43290/43290]\n",
            "\n",
            "--2019-03-29 02:50:38--  https://raw.githubusercontent.com/todnewman/data/master/Teams.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 9359 (9.1K) [text/plain]\n",
            "Saving to: ‘/content/Teams.csv’\n",
            "\n",
            "/content/Teams.csv  100%[===================>]   9.14K  --.-KB/s    in 0s      \n",
            "\n",
            "2019-03-29 02:50:38 (116 MB/s) - ‘/content/Teams.csv’ saved [9359/9359]\n",
            "\n",
            "--2019-03-29 02:50:39--  https://raw.githubusercontent.com/todnewman/data/master/pytools.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 404 Not Found\n",
            "2019-03-29 02:50:39 ERROR 404: Not Found.\n",
            "\n",
            "/content\n",
            "\u001b[0m\u001b[01;34mdrive\u001b[0m/                            pytools.py                       Teams.csv\n",
            "historical_tourney_data_2019.csv  \u001b[01;34msample_data\u001b[0m/\n",
            "kenpom_2019.csv                   SampleSubmissionStage2-temp.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EK69iL8cSpdO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Prepare the dataframe - Feature Engineering\n",
        "\n",
        "Here we have a few convenience functions that we call to get the Pandas dataframe ready for Machine Learning algorithms.  \n",
        "\n",
        "prep_dataframe is intended to allow one spot for the developer to identify which features (columns) to bring in to the ML algorithm.\n",
        "\n",
        "split_data takes a dataframe that has been prepared and splits it into training and test data sets.  This is also the location where I will do unbalanced dataset operations when this is necessary using the inbalanced learning algorithms\n",
        "\n",
        "split_pred_data is a simple function to enable prediction on a predictive dataset\n",
        "\n",
        "Mutual Information functions are for future use.\n",
        "\n",
        "Correlation Matrix is used to identify the features with the most positive and negative correlations with the target."
      ]
    },
    {
      "metadata": {
        "id": "sy_3PKW_SpdQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "'''Prep the dataset we wish to apply the model to.  There won't be a Target vector.'''\n",
        "def prep_dataframe(df, flag, target, drop_seed):\n",
        "    \n",
        "    Name = 0\n",
        "    \n",
        "    if flag == 'predict':        \n",
        "        df[target] = 0 \n",
        "        if drop_seed:\n",
        "            df = df.drop(['Season.1', 'KeyField.1', 'SEED.1', 'SEED.2', 'TARGET16.1', 'Season.2',\n",
        "                  'KeyField.2','TARGET16.2'], axis=1) # eliminate seed vector\n",
        "        else:\n",
        "            df = df.drop(['Season.1', 'KeyField.1', 'TARGET16.1', 'Season.2',\n",
        "                  'KeyField.2','TARGET16.2'], axis=1) # eliminate seed vector\n",
        "\n",
        "\n",
        "        Name1 = df.pop('TeamName.1')\n",
        "        Name2 = df.pop('TeamName.2')\n",
        "        return df, Name1, Name2\n",
        "    else:\n",
        "        \n",
        "        if drop_seed:\n",
        "            df = df.drop(['Unnamed: 0','TeamName.1', 'TeamName.2', 'Season.1', 'KeyField.1', 'SEED.1', 'SEED.2', 'TARGET16.1', 'Score.1','Season.2',\n",
        "                  'KeyField.2','TARGET16.2','Score.2', 'score_delta'], axis=1) # eliminate seed vector\n",
        "\n",
        "        else:\n",
        "            df = df.drop(['Unnamed: 0', 'TeamName.1', 'TeamName.2', 'Season.1', 'KeyField.1', 'TARGET16.1', 'Score.1','Season.2',\n",
        "                  'KeyField.2','TARGET16.2','Score.2', 'score_delta'], axis=1) # eliminate seed vector\n",
        "\n",
        "            \n",
        "        df = df.fillna(0)\n",
        "        Name = 0\n",
        "        \n",
        "        return df, Name"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pPt-rbPvSpdV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def split_data(df_s, size, target, os_flag):\n",
        "    ''' Split the data into training and testing sets.  Column headers below are custom for the \"Douglas data set\".\n",
        "    NaN's are filled with zero to maintain consistent data shapes.\n",
        "    '''\n",
        "    from sklearn.model_selection import train_test_split\n",
        "    from sklearn import preprocessing\n",
        "    from imblearn.combine import SMOTEENN\n",
        "    from imblearn.combine import SMOTETomek \n",
        "    from imblearn.over_sampling import ADASYN, SMOTE, RandomOverSampler\n",
        "\n",
        "    y = df_s.pop(target) # Remove the 'target' column that has the labels\n",
        "    X = df_s.fillna(0)\n",
        "    y = np.array(y).astype(int)\n",
        "    X = np.array(X)\n",
        "    \n",
        "    # Scale the data so it works better with the ML algorithms... Also run oversampling algorithms\n",
        "    \n",
        "    \n",
        "    if os_flag:\n",
        "        sm = RandomOverSampler(random_state=41)    \n",
        "        X, y = sm.fit_sample(X, y)\n",
        "        \n",
        "    if minmax_flag:\n",
        "        X = preprocessing.MinMaxScaler().fit_transform(X)\n",
        "    else:\n",
        "        X = preprocessing.MaxAbsScaler().fit_transform(X)\n",
        "    \n",
        "    X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=size) # Keeping the test set small to evaluate timing\n",
        "\n",
        "    return X_train, X_test, y_train, y_test, X, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ht3Vj-ZvSpdZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def split_pred_data(df_s, size, target):\n",
        "    ''' Split the data into training and testing sets.  Column headers below are custom for the \"Douglas data set\".\n",
        "    NaN's are filled with zero to maintain consistent data shapes.\n",
        "    '''\n",
        "    from sklearn.model_selection import train_test_split\n",
        "    from sklearn import preprocessing\n",
        "    df_s[target] = 0 # Just so we have a target vector in the DF, even though we won't use it\n",
        "\n",
        "    y = df_s.pop(target) # Remove the 'target' column that has the labels\n",
        "    X = df_s.fillna(0)\n",
        "    y = np.array(y).astype(int)\n",
        "    X = np.array(X)\n",
        "    \n",
        "    # Scale the data so it works better with the ML algorithms\n",
        "    if minmax_flag:\n",
        "        X = preprocessing.MinMaxScaler().fit_transform(X)\n",
        "    else:\n",
        "        X = preprocessing.MaxAbsScaler().fit_transform(X)\n",
        "    \n",
        "    return X, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lVTSQyuwSpde",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Feature Selection\n",
        "\n",
        "This module is designed to select the most informative features.  Right now it's using a basic correlation matrix and then selecting the most positive and negative correlated features using a variable called \"half_range\".  Setting half_range to 4 will take the top 25% of the ordered matrix as well as the bottom 25%.  \n",
        "For the small data in this NCAA dataset this function is probably less useful than it would be on a larger dataset.  So I created a global called all_feat that will accept all features."
      ]
    },
    {
      "metadata": {
        "id": "Hg2uR5qpSpdg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def feature_select(target, df_hist, mi_flag): \n",
        "    \n",
        "    half_range = 3.0  # This will determine how many of the most positive and negative correlated features are used\n",
        "    if all_feat:\n",
        "        half_range = 2.0\n",
        "    # Open a dataframe just to create the correlation matrix                             \n",
        "    df_new = pd.DataFrame()\n",
        "    df_copy = pd.DataFrame()\n",
        "    df_copy['Target'] = df_hist[target]\n",
        "    \n",
        "    # Convert Objects to Values\n",
        "    cols = df_hist.columns\n",
        "    df_hist[cols] = df_hist[cols].apply(pd.to_numeric, errors='coerce')\n",
        "    \n",
        "    df_new = df_hist.corr().sort_values(target, ascending = False) \n",
        "\n",
        "    size = len(df_new)\n",
        "    \n",
        "    ratio = (1/half_range) # will let us pick the highest and lowest correlated features by this ratio\n",
        "    \n",
        "    upper_limit = int(size*ratio)\n",
        "    lower_limit = int(size*(1-ratio))\n",
        "    upper_range = df_new[target].index[1:upper_limit].tolist()\n",
        "    lower_range = df_new[target].index[lower_limit:size].tolist()\n",
        "    \n",
        "    '''\n",
        "    Below allows us to switch between mutual information-based and correlation-based feature selectors\n",
        "    '''\n",
        "    if mi_flag:\n",
        "        total_range = get_mutual_information(df_hist, target, upper_limit)\n",
        "    else:\n",
        "        total_range = (upper_range + lower_range)\n",
        "        \n",
        "    target_vector = df_copy['Target']\n",
        "    df_hist = df_hist[total_range] # limit the dataframe used for training to just the most informative features\n",
        "    df_hist['Target'] = target_vector\n",
        "\n",
        "\n",
        "    return (df_hist, 'Target', total_range)\n",
        "\n",
        "\n",
        "def get_mutual_information(df, target, range):\n",
        "    from sklearn.feature_selection import mutual_info_classif\n",
        "    # Split data:\n",
        "    df_copy = pd.DataFrame()\n",
        "    df_copy = df\n",
        "    \n",
        "    y = df_copy.pop(target)\n",
        "    X = df_copy.fillna(0)\n",
        "    X = np.array(X)\n",
        "\n",
        "    df_n = pd.DataFrame()\n",
        "    df_n['MI'] = mutual_info_classif(X, y)\n",
        "    df_n['Keys'] = df_copy.keys()\n",
        "    df_n = df_n.sort_values(['MI'], ascending=False)\n",
        "    features = np.array(df_n['Keys'][1:range].tolist())\n",
        "    \n",
        "    return features"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "K7ciOQ1jSpdu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Process data from the Kaggle input into Predictive DataFrame Format\n",
        "\n",
        "This pulls data from the Kaggle example submission format and from that it grabs the teams competing in each match.  It then calls grab_kp_data, a function that takes the kaggle inputs, the 2019 statistics for each team, and creates the file to be used for inference by the model in the next round."
      ]
    },
    {
      "metadata": {
        "id": "be4HI_dySpdx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def process_data(df_team, df_input, round_t, curr_year, verbose):\n",
        "    '''\n",
        "    Pass in results of the prediction and output the file used for prediction.\n",
        "    \n",
        "    Returns df_pred, the official predictive dataframe.  We will predict the results of the games using the model\n",
        "    trained on years of tournament games.\n",
        "    '''\n",
        "     # Lists that we'll use    \n",
        "    team1 = []\n",
        "    team2 = []\n",
        "    \n",
        "    team_dict = {}\n",
        "    \n",
        "    for i, id in enumerate(df_team['TeamID'].unique()):\n",
        "        team_dict[id] = df_team['TeamName'].iloc[i]\n",
        "    \n",
        "    # Below, we go through the Kaggle submission file and we separate out the year and the two team\n",
        "    # ID values.  Then we go into the dictionary to pull out the names from the IDs.\n",
        "        \n",
        "    input_val = df_input['ID'].astype(str)\n",
        "    \n",
        "    for i, val in enumerate(input_val):\n",
        "        year, id1, id2 = val.split(sep='_')\n",
        "        name1 = team_dict[int(id1)]\n",
        "        name2 = team_dict[int(id2)]\n",
        "        team1.append(name1)\n",
        "        team2.append(name2)\n",
        "            \n",
        "    # Dataframes that we'll use\n",
        "\n",
        "    df_prediction = pd.DataFrame()   \n",
        "       \n",
        "    # Output Filename\n",
        "    pred_file = ('pred_%s_out.csv' % round_t)\n",
        "\n",
        "    #\n",
        "    # Create the DataFrame with the info we need to be able to \n",
        "    df_prediction['WTeamID'] = team1\n",
        "    df_prediction['LTeamID'] = team2\n",
        "    df_prediction['Season'] = curr_year\n",
        "    \n",
        "    if verbose:\n",
        "        print(\"This is the dataframe we end up routing to the play_games function.\")\n",
        "        print(df_prediction)\n",
        "        \n",
        "    #\n",
        "    # Send info on the teams from the Kaggle example file over to the function\n",
        "    #   that creates the predictive dataframe for each match (team1 vs. team2)\n",
        "    df_pred = grab_kp_data(df_prediction, curr_year)\n",
        "    df_pred.to_csv(pred_file)\n",
        "    return df_pred\n",
        "\n",
        "def grab_kp_data(df_winlose, curr_year):\n",
        "    '''\n",
        "    Here we take the actual basketball data for the teams playing each other and build out the\n",
        "    predictive dataframe.  I have a strange practice of flipping the teams so that each team is\n",
        "    in the Team1 spot once.  This seems to make a difference.\n",
        "    \n",
        "    df_winlose:  this is the prediction dataframe that captures the teamnames from the\n",
        "                 Kaggle example submission file ID's.\n",
        "    '''\n",
        "    game = pd.DataFrame()\n",
        "    combined_pred_df = pd.DataFrame()\n",
        "    \n",
        "    filename_kp_data = 'kenpom_2019.csv'\n",
        "    \n",
        "    ncaa_data = pd.read_csv(filename_kp_data, parse_dates=True) # Open Up Latest Kenpom File\n",
        "    \n",
        "    ncaa_data = ncaa_data[ncaa_data['Season']==curr_year]  # We only want the current year's data\n",
        "        \n",
        "    for i, row in df_winlose.iterrows():\n",
        "        key_team1 =  (\"%s%s\" % (row['Season'], row['WTeamID'])) # Key to use to enter the NCAA KP Data\n",
        "        key_team2 = (\"%s%s\" % (row['Season'], row['LTeamID'])) # Key for the second team\n",
        "                \n",
        "        team1_filter = ncaa_data['KeyField']==(key_team1) # Filter to grab the first team's data from the KP Data\n",
        "        team2_filter = ncaa_data['KeyField']==(key_team2) # Filter for the second team\n",
        "               \n",
        "        team1=ncaa_data[team1_filter]  # Record from current year for Team 1\n",
        "        team2=ncaa_data[team2_filter] # Record from current year for Team 2\n",
        "        team2.index = team1.index.copy()  # So they're not on separate lines            \n",
        "        team1 = team1.add_suffix(\".1\") # Done so we don't have columns with the same name\n",
        "        team2 = team2.add_suffix(\".2\")\n",
        "        \n",
        "        game = pd.concat([team1, team2], axis=1)  # This record will contain both teams' stats\n",
        "        \n",
        "        combined_pred_df = combined_pred_df.append(game)\n",
        "        \n",
        "    return (combined_pred_df)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oUL3i9euSpd1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# CLASSIFICATION Function\n",
        "\n",
        "This predicts the probability of a win for the team in the first column (team 1) over the team in the second column (team 2).  Right now it is set up to use the ensemble stacking classifier (through call to find_best_ensemble), but you could substitute any other classifier with a 'model=XX'."
      ]
    },
    {
      "metadata": {
        "id": "zTjnOYNgSpeA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def run_classification(df_hist, df_predict, round_t):\n",
        "    from sklearn import linear_model\n",
        "    from sklearn.neural_network import MLPClassifier \n",
        "    \n",
        "    # Global Variables\n",
        "    os_flag = False  # Oversampling\n",
        "    drop_seed = False # Should we drop the SEED feature?    \n",
        "    \n",
        "    target = 'win'  # Column header that contains our binary target\n",
        "\n",
        "    best_features = []\n",
        "\n",
        "    df_test_train = df_hist\n",
        "    \n",
        "    #C_values = [0.001, 0.01, 0.05, 0.1, 1, 10]  # For Logistic Regression CV    \n",
        "    #model = linear_model.LogisticRegressionCV(Cs=C_values, cv=5, penalty='l2', scoring='roc_auc', fit_intercept=False)\n",
        "    #model = MLPClassifier(solver='lbfgs', early_stopping = True, hidden_layer_sizes=(64,32,8))\n",
        "    \n",
        "    '''\n",
        "    First, process the training data set.\n",
        "    '''\n",
        "    \n",
        "    df_train, Name = prep_dataframe(df_test_train, 'train', target, drop_seed)\n",
        "    df_train, target, best_features = feature_select(target, df_train, mi_flag)\n",
        "    \n",
        "    #\n",
        "    # Split data: determine the percentage of data that goes into the test set\n",
        "    X_train, X_test, y_train, y_test, X, y = split_data(df_train, 0.20, target, os_flag)\n",
        "    \n",
        "    '''\n",
        "    Call the ensemble function to return the stacking model that we will use.\n",
        "    '''\n",
        "    model = find_best_ensemble(X,y)\n",
        "    \n",
        "    #    \n",
        "    # Fit the training set data.    \n",
        "    model.fit(X_train, y_train)\n",
        "    \n",
        "    expected = y_test # This is the values of the Target that we 'know' from the historical data\n",
        "    predicted = model.predict(X_test) # These are the Targets we predict (to compare to the expected values later)\n",
        "    print(metrics.classification_report(expected, predicted))\n",
        "    print(metrics.confusion_matrix(expected, predicted))\n",
        "    #pytools.roc(model, X_train, y_train, target, 'model_kaggle.png')\n",
        "    \n",
        "    '''\n",
        "    Now we use the trained model to predict how this seasons games will go.\n",
        "    '''        \n",
        "    \n",
        "    df_pred, Name1, Name2 = prep_dataframe(df_predict, 'predict', target, drop_seed)\n",
        "    df_pred = df_pred[best_features]  # Align predictive data with the training data features\n",
        "    X_predict, y_bogus = split_pred_data(df_pred, 0, target)  # Do scaling of data to match training data\n",
        "    \n",
        "    prob_in_class = model.predict_proba(X_predict)[:,1] # Grab the probability of team 1 beating team 2\n",
        "    \n",
        "    return (Name1, Name2, prob_in_class, model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oHHjR_pmSpeH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Build the Stacking Classifier\n",
        "\n",
        "Here we show how to use sklearn classifiers and mlxtend's nice stacking classifier (http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/) to return a stacking classifier model to the run_classifier function.\n",
        "\n",
        "This is a much simpler way of doing stacking classifiers than the way I've been doing it in the past.\n",
        "\n",
        "My strategy for picking classifiers for a stacker is generally to select models that are dissimilar from one another, i.e., Gaussian Processes using an RBF kernel is very different from KNN.\n",
        "\n",
        "In this case, the use of the stacker drops the log-loss score significantly over straight Multi-layer Perceptron or optimized Logistic Regression."
      ]
    },
    {
      "metadata": {
        "id": "TTFuvH_6SpeI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def find_best_ensemble(X,y):\n",
        "    '''\n",
        "    Currently Im not doing anything to actually find the optimal set of models for a stacker.  But I intend to do\n",
        "    this soon, so hence the name of this function.  The objective of this function right now is to return\n",
        "    indicators of the strength of the stacker defined and then to return the stacker model.\n",
        "    \n",
        "    Returns:  model\n",
        "    '''\n",
        "    from sklearn import model_selection\n",
        "    from sklearn.linear_model import LogisticRegressionCV\n",
        "    from sklearn.linear_model import LogisticRegression\n",
        "    from sklearn.ensemble import RandomForestClassifier\n",
        "    from mlxtend.classifier import StackingCVClassifier\n",
        "    from sklearn.neural_network import MLPClassifier \n",
        "    from sklearn.ensemble import AdaBoostClassifier\n",
        "    from sklearn.gaussian_process import GaussianProcessClassifier\n",
        "    from sklearn.neighbors import KNeighborsClassifier\n",
        "    from sklearn.gaussian_process.kernels import RBF\n",
        "    import numpy as np\n",
        "    \n",
        "    kernel_GP = 1.0 * RBF(1.0) # Kernel for the Gaussian Process Classifier\n",
        "    C_values = [0.001, 0.01, 0.05, 0.1, 1, 10] # For the Logistic Regression C value\n",
        "    \n",
        "    #\n",
        "    # Here are a couple of different models that could be substituted for the stacking classifier\n",
        "    #clf1 = AdaBoostClassifier(n_estimators=285, learning_rate=0.001)\n",
        "    #clf3 = MLPClassifier(solver='lbfgs', early_stopping = True, hidden_layer_sizes=(16,32,8))\n",
        "    \n",
        "    #\n",
        "    # Classifiers to use in a Stacking Classifier\n",
        "    clf3 = KNeighborsClassifier(n_neighbors = 3)\n",
        "    clf2 = RandomForestClassifier(n_estimators=144, criterion='gini', bootstrap=False, min_samples_leaf = 7, \n",
        "               min_samples_split = 5, max_features = 10, max_depth = None)\n",
        "    clf1 = GaussianProcessClassifier(kernel=kernel_GP, random_state = 0)    \n",
        "    clf4 = LogisticRegressionCV(Cs=C_values, cv=5, penalty='l2', scoring='roc_auc', fit_intercept=False)\n",
        "    #\n",
        "    # Meta-Classifier below for classifying across the Above Classifiers\n",
        "    lr   = LogisticRegressionCV(Cs=C_values, cv=3, scoring = 'roc_auc')\n",
        "    #\n",
        "    # Set up the Stacking Classifier\n",
        "    sclf = StackingCVClassifier(classifiers=[clf3, clf1, clf2, clf4],\n",
        "                          use_probas=True,\n",
        "                          meta_classifier=lr)\n",
        "\n",
        "    print('5-fold cross validation:\\n')\n",
        "\n",
        "    for clf, label in zip([clf3, clf1, clf2, clf4, sclf], \n",
        "                      ['KNN', \n",
        "                       'Random Forest', \n",
        "                       'Gaussian Process',\n",
        "                       'Logistic Regression CV',\n",
        "                       'StackingCVClassifier']):\n",
        "\n",
        "        scores = model_selection.cross_val_score(clf, X, y, \n",
        "                                              cv=5, scoring='roc_auc')\n",
        "        print(\"Accuracy: %0.2f (+/- %0.2f) [%s]\" \n",
        "              % (scores.mean(), scores.std(), label))\n",
        "        \n",
        "    return clf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "spGYdyHHSpeN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Handle the Results of the Played Games\n",
        "\n",
        "play_games: sets up the data to run the classification function.  Will play any number of games desired and then take the mean and std of the resulting probability of winning.  Set up the number of iterations with n_iter in the main.  Calls run_classification which does the bulk of the machine learning.  In this case, it is set up to build a stacking classifier.\n",
        "\n",
        "determine_win: takes the mean probability of winning and builds the dataframe for the Kaggle output.\n"
      ]
    },
    {
      "metadata": {
        "id": "tGZ5de1uSpeO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def play_games(pred_df, round_t, df_hist_train, n_iter, verbose ):\n",
        "    '''\n",
        "    This is the executive for the gameplaying feature.\n",
        "    '''\n",
        "    r2 = []\n",
        "    \n",
        "    combined_df = pd.DataFrame()\n",
        "    first_time = True\n",
        "          \n",
        "    for n in range (0, n_iter):\n",
        "        Team1, Team2, Prediction, modelinfo = run_classification(df_hist_train, pred_df, round_t)\n",
        "        if first_time:\n",
        "            combined_df['Team1'] = Team1.values\n",
        "            combined_df['Team2'] = Team2.values\n",
        "        first_time = False\n",
        "        combined_df[n] = np.round(Prediction,3)\n",
        "    \n",
        "    combined_df['mean'] = combined_df.mean(axis = 1)\n",
        "    combined_df['std'] = combined_df.std(axis = 1)\n",
        "    combined_df.to_csv('NCAA_game_pred-%s-lasso.csv' % round_t)\n",
        "    \n",
        "    if verbose:\n",
        "        print (\"Here are the results from regression from the play_games function for %s iterations\" % n_iter)\n",
        "        print (combined_df[['Team1', 'Team2', 'mean', 'std']].sort_values(['mean'], ascending = False))\n",
        "    return combined_df, modelinfo\n",
        "\n",
        "\n",
        "def determine_win(df, round_t, verbose):\n",
        "    id_val = []  # List to hold the integer ID for each team\n",
        "    team_dict = {}\n",
        "    \n",
        "    teamfile = 'Teams.csv'\n",
        "    outfile = ('SampleSubmissionStage2_%s.csv' % round_t) # Save off output in Kaggle Format\n",
        "    df_team = pd.read_csv(teamfile) # to create dictionary relating teams and IDs\n",
        "    \n",
        "    df_kaggle = pd.DataFrame() # Final Kaggle Output Format\n",
        "    \n",
        "    #\n",
        "    # Build the team dictionary\n",
        "    for i, name in enumerate(df_team['TeamName'].unique()):\n",
        "        team_dict[name] = df_team['TeamID'].iloc[i]\n",
        "    \n",
        "    # Need to map into the dictionary the right way.\n",
        "    # Due to multiple spellings.\n",
        "    \n",
        "    id1 = df['Team1'].map(team_dict)\n",
        "    id2 = df['Team2'].map(team_dict)\n",
        "    \n",
        "    df['id1'] = id1\n",
        "    df['id2'] = id2\n",
        "    for i, row in df.iterrows():\n",
        "        id_val.append(('2019_%s_%s' % (row['id1'], row['id2'])))\n",
        "    \n",
        "    df['prob1'] = df['mean']\n",
        "    \n",
        "    #df_kaggle['ID'] = \n",
        "    if verbose:\n",
        "        print(\"Winning Team  Losing Team  Win Value1  Win Value2\")\n",
        "        print(df[['Team1', 'id1', 'Team2', 'id2','prob1']][0:35])        \n",
        "    \n",
        "    df_kaggle['ID'] = id_val\n",
        "    df_kaggle['Pred'] = df['prob1']\n",
        "    df_kaggle.to_csv(outfile)\n",
        "    \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3LGxkAj1SpeR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# MAIN function\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "scrolled": false,
        "id": "VkRe0-SUSpeT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2218
        },
        "outputId": "b55c22ec-1a49-4fc7-c934-0f6d1575b0bf"
      },
      "cell_type": "code",
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from sklearn.neural_network import MLPRegressor\n",
        "from sklearn.metrics import r2_score\n",
        "from sklearn.linear_model import LinearRegression, BayesianRidge, LassoCV, RidgeCV, LassoLarsCV, MultiTaskLassoCV, ElasticNetCV\n",
        "from sklearn.svm import SVR\n",
        "from sklearn.model_selection import KFold\n",
        "import pytools\n",
        "import logging\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "# Input Files\n",
        "filename_train = 'historical_tourney_data_2019.csv' # This is the historical dataset from years of playoffs results\n",
        "input_filename = 'SampleSubmissionStage2-temp.csv'  # Kaggle's example format file\n",
        "teamfile = 'Teams.csv' # List mapping Team Names and Team Integer ID's\n",
        "\n",
        "# Logging setup.  This will go in the path chosen at the top of this notebook.\n",
        "logname = 'ncaa_logfile.txt'\n",
        "\n",
        "logger = logging.getLogger('NCAA Predictor')\n",
        "logger.setLevel(logging.DEBUG)\n",
        "fh = logging.FileHandler(logname)\n",
        "fh.setLevel(logging.DEBUG)\n",
        "formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')\n",
        "fh.setFormatter(formatter)\n",
        "logger.addHandler(fh)\n",
        "\n",
        "logger.info(\"NCAA tourney output RUN Starts here\")\n",
        "\n",
        "# Global Variables\n",
        "global mi_flag     # Use mutual information instead of correlation.  \n",
        "global all_feat    # use all numerical features\n",
        "global os_flag     # Use random oversampling to address imbalance in the target\n",
        "global minmax_flag # Use the MinMax Scaler instead of MaxAbs\n",
        "\n",
        "mi_flag = False\n",
        "os_flag = False\n",
        "minmax_flag = True\n",
        "all_feat = True\n",
        "\n",
        "curr_year = 2019 # change every year\n",
        "n_iter = 5\n",
        "round_t = 0\n",
        "\n",
        "logger.info(\"mi_flag %s * os_flag %s * minmax_flag %s * all_feat %s\\n\" % \n",
        "             ( mi_flag, os_flag, minmax_flag,  all_feat))\n",
        "\n",
        "df_hist_train = pd.read_csv(filename_train) # Historical NCAA Tournament match data\n",
        "df_input      = pd.read_csv(input_filename) # Dataframe from the Kaggle Example Submission File    \n",
        "df_team       = pd.read_csv(teamfile)\n",
        "   \n",
        "verbose = False\n",
        "pred_df = process_data(df_team, df_input, round_t, curr_year, verbose)        # Dataframe of games to predict\n",
        "combined, model = play_games(pred_df, round_t, df_hist_train, n_iter, verbose)  # Run games for this year's teams\n",
        "    \n",
        "logger.info(\"Model Info: %s\" % model)\n",
        "\n",
        "verbose = True\n",
        "determine_win(combined, round_t, verbose)   # Determine the results and format for Kaggle\n",
        "    "
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "5-fold cross validation:\n",
            "\n",
            "Accuracy: 0.64 (+/- 0.03) [KNN]\n",
            "Accuracy: 0.76 (+/- 0.03) [Random Forest]\n",
            "Accuracy: 0.75 (+/- 0.02) [Gaussian Process]\n",
            "Accuracy: 0.77 (+/- 0.02) [Logistic Regression CV]\n",
            "Accuracy: 0.77 (+/- 0.03) [StackingCVClassifier]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.70      0.75      0.72        72\n",
            "           1       0.78      0.74      0.76        87\n",
            "\n",
            "   micro avg       0.74      0.74      0.74       159\n",
            "   macro avg       0.74      0.74      0.74       159\n",
            "weighted avg       0.74      0.74      0.74       159\n",
            "\n",
            "[[54 18]\n",
            " [23 64]]\n",
            "5-fold cross validation:\n",
            "\n",
            "Accuracy: 0.64 (+/- 0.03) [KNN]\n",
            "Accuracy: 0.76 (+/- 0.03) [Random Forest]\n",
            "Accuracy: 0.75 (+/- 0.03) [Gaussian Process]\n",
            "Accuracy: 0.77 (+/- 0.02) [Logistic Regression CV]\n",
            "Accuracy: 0.77 (+/- 0.03) [StackingCVClassifier]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.69      0.60      0.64        84\n",
            "           1       0.61      0.71      0.65        75\n",
            "\n",
            "   micro avg       0.65      0.65      0.65       159\n",
            "   macro avg       0.65      0.65      0.65       159\n",
            "weighted avg       0.65      0.65      0.65       159\n",
            "\n",
            "[[50 34]\n",
            " [22 53]]\n",
            "5-fold cross validation:\n",
            "\n",
            "Accuracy: 0.64 (+/- 0.03) [KNN]\n",
            "Accuracy: 0.76 (+/- 0.03) [Random Forest]\n",
            "Accuracy: 0.75 (+/- 0.03) [Gaussian Process]\n",
            "Accuracy: 0.77 (+/- 0.02) [Logistic Regression CV]\n",
            "Accuracy: 0.76 (+/- 0.03) [StackingCVClassifier]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.62      0.72      0.67        74\n",
            "           1       0.72      0.62      0.67        85\n",
            "\n",
            "   micro avg       0.67      0.67      0.67       159\n",
            "   macro avg       0.67      0.67      0.67       159\n",
            "weighted avg       0.67      0.67      0.67       159\n",
            "\n",
            "[[53 21]\n",
            " [32 53]]\n",
            "5-fold cross validation:\n",
            "\n",
            "Accuracy: 0.64 (+/- 0.03) [KNN]\n",
            "Accuracy: 0.76 (+/- 0.03) [Random Forest]\n",
            "Accuracy: 0.75 (+/- 0.03) [Gaussian Process]\n",
            "Accuracy: 0.77 (+/- 0.02) [Logistic Regression CV]\n",
            "Accuracy: 0.76 (+/- 0.03) [StackingCVClassifier]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.74      0.73      0.73        74\n",
            "           1       0.77      0.78      0.77        85\n",
            "\n",
            "   micro avg       0.75      0.75      0.75       159\n",
            "   macro avg       0.75      0.75      0.75       159\n",
            "weighted avg       0.75      0.75      0.75       159\n",
            "\n",
            "[[54 20]\n",
            " [19 66]]\n",
            "5-fold cross validation:\n",
            "\n",
            "Accuracy: 0.64 (+/- 0.03) [KNN]\n",
            "Accuracy: 0.76 (+/- 0.03) [Random Forest]\n",
            "Accuracy: 0.76 (+/- 0.03) [Gaussian Process]\n",
            "Accuracy: 0.77 (+/- 0.02) [Logistic Regression CV]\n",
            "Accuracy: 0.76 (+/- 0.03) [StackingCVClassifier]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.72      0.71      0.72        80\n",
            "           1       0.71      0.72      0.72        79\n",
            "\n",
            "   micro avg       0.72      0.72      0.72       159\n",
            "   macro avg       0.72      0.72      0.72       159\n",
            "weighted avg       0.72      0.72      0.72       159\n",
            "\n",
            "[[57 23]\n",
            " [22 57]]\n",
            "Winning Team  Losing Team  Win Value1  Win Value2\n",
            "                Team1   id1                Team2   id2   prob1\n",
            "0   Abilene Christian  1101          Arizona St.  1113  0.5060\n",
            "1   Abilene Christian  1101               Auburn  1120  0.2306\n",
            "2   Abilene Christian  1101               Baylor  1124  0.3310\n",
            "3   Abilene Christian  1101              Belmont  1125  0.3668\n",
            "4   Abilene Christian  1101              Bradley  1133  0.5818\n",
            "5   Abilene Christian  1101              Buffalo  1138  0.2318\n",
            "6   Abilene Christian  1101           Cincinnati  1153  0.2650\n",
            "7   Abilene Christian  1101              Colgate  1159  0.4064\n",
            "8   Abilene Christian  1101                 Duke  1181  0.1030\n",
            "9   Abilene Christian  1101  Fairleigh Dickinson  1192  0.5700\n",
            "10  Abilene Christian  1101              Florida  1196  0.3312\n",
            "11  Abilene Christian  1101          Florida St.  1199  0.1718\n",
            "12  Abilene Christian  1101         Gardner Webb  1205  0.5224\n",
            "13  Abilene Christian  1101          Georgia St.  1209  0.3880\n",
            "14  Abilene Christian  1101              Gonzaga  1211  0.1062\n",
            "15  Abilene Christian  1101              Houston  1222  0.1378\n",
            "16  Abilene Christian  1101                 Iona  1233  0.5578\n",
            "17  Abilene Christian  1101                 Iowa  1234  0.3652\n",
            "18  Abilene Christian  1101             Iowa St.  1235  0.3272\n",
            "19  Abilene Christian  1101               Kansas  1242  0.1816\n",
            "20  Abilene Christian  1101           Kansas St.  1243  0.2088\n",
            "21  Abilene Christian  1101             Kentucky  1246  0.1598\n",
            "22  Abilene Christian  1101              Liberty  1251  0.5020\n",
            "23  Abilene Christian  1101           Louisville  1257  0.3530\n",
            "24  Abilene Christian  1101                  LSU  1261  0.2448\n",
            "25  Abilene Christian  1101            Marquette  1266  0.2682\n",
            "26  Abilene Christian  1101             Maryland  1268  0.2856\n",
            "27  Abilene Christian  1101             Michigan  1276  0.1168\n",
            "28  Abilene Christian  1101         Michigan St.  1277  0.1788\n",
            "29  Abilene Christian  1101            Minnesota  1278  0.5072\n",
            "30  Abilene Christian  1101          Mississippi  1279  0.3126\n",
            "31  Abilene Christian  1101      Mississippi St.  1280  0.2268\n",
            "32  Abilene Christian  1101              Montana  1285  0.3542\n",
            "33  Abilene Christian  1101           Murray St.  1293  0.2368\n",
            "34  Abilene Christian  1101     North Dakota St.  1295  0.5610\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "eN_xFIAWSpeZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 149
        },
        "outputId": "0c5c436c-6193-4566-da4c-9d0e28ab65c9"
      },
      "cell_type": "code",
      "source": [
        "%ls /content\n",
        "%cp SampleSubmissionStage2_0.csv /content/drive/'My Drive'/."
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[0m\u001b[01;34mdrive\u001b[0m/                            \u001b[01;34m__pycache__\u001b[0m/\n",
            "historical_tourney_data_2019.csv  pytools.py\n",
            "kenpom_2019.csv                   \u001b[01;34msample_data\u001b[0m/\n",
            "NCAA_game_pred-0-lasso.csv        SampleSubmissionStage2_0.csv\n",
            "ncaa_logfile.txt                  SampleSubmissionStage2-temp.csv\n",
            "pred_0_out.csv                    Teams.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "37E2AXCQZCqy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}