ChallengeFINAL_blanchet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style=\"color:rgb(36,115,172);text-align:center;\"> Data Challenge </h1>\n",
    "\n",
    "<h2 style=\"text-align:center;\"> Project of Data Mining course - SJTU ParisTech </h2>\n",
    "\n",
    "<h3 style=\"text-align:center;\"> Prediction of link between papers </h3>\n",
    "\n",
    "<i>Florian Blanchet (Télécom Bretagne, France) <br/> florian.blanchet@telecom-bretagne.eu</i>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---------------------------------------------------------------------------------\n",
    "========================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary : \n",
    "1 - Introduction <br/>\n",
    "2 - Import libraries<br/>\n",
    "3 - Import files<br/>\n",
    "4 - Feature creation<br/>\n",
    "5 - Feature saling<br/>\n",
    "6 - Evaluation<br/>\n",
    "7 - Training script<br/>\n",
    "8 - Testing script<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Introduction \n",
    "<p>The task of this data challenge is to identify whether a citation exists for a given pair of research papers. We will find that there is resemblance of this task with that of link prediction. We will be given a training dataset with the ground truth of whether a link is \"true\" or \"false\" among pairs of papers and a test dataset for which we will have to classify the pair of papers.  </p>\n",
    "<p>The report is linearly organized, the functions presented will be at the end launched in the same order in a 'Main' block. It appears that I have modified some of them that is why they are sometimes with a number in the name.  </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Import libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first we have to import libraries that we will be needed after. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Tools :\n",
    "import random\n",
    "import numpy as np\n",
    "# To create features :\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "from sklearn import preprocessing\n",
    "import nltk\n",
    "# To import files : \n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Classifiers :\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import svm\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "# To evaluate performances :\n",
    "from sklearn.cross_validation import KFold\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import roc_curve, auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now download tools from nltk library to clean our documents by removing for instances stopwords or to just keep the root of a word. It allow us to have better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/florianblanchet/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/florianblanchet/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk.download('punkt') # for tokenization\n",
    "nltk.download('stopwords')\n",
    "stpwds = set(nltk.corpus.stopwords.words(\"english\"))\n",
    "stemmer = nltk.stem.PorterStemmer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------------------------------------------------------------------------\n",
    "========================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - Import files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part I describe the files that will be used after. I import them and save data in several list or dictionnaries. The function download_files() allow to download all of them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### testing_set.txt :\n",
    "There are 32,648 node pairs in testing_set. One pair per row, as: source node ID, target node ID. These pairs are the ones we have to predict the label. It will be used at the end to create the submission file. <br/>\n",
    "Data saved in the 'testing_set' list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### training_set.txt :  \n",
    "This file is composed of 615,512 labeled node pairs (1 if there is an edge between the two nodes, 0 else). One pair and label per row, as: source node ID, target node ID, and 1 or 0. The IDs match the papers in the node_information.csv file (see below). <br/>\n",
    "Data saved in the 'training_set' list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### node_information.csv :  \n",
    "For each paper out of 27,770, contains the paper :<br/>\n",
    "(1) paper unique ID (integer)<br/>\n",
    "(2) publication year (between 1993 and 2003) (integer)<br/> \n",
    "(3) paper title (string)<br/> \n",
    "(4) authors (strings separated by ,)<br/> \n",
    "(5) name of journal (not available for all papers) (string)<br/> \n",
    "(6) abstract (string). Abstracts are already in lowercase, common English stopwords have been removed, and punctuation marks have been removed except for intra-word dashes. <br/> \n",
    "Data saved in the 'node_info' list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### download_files() :  \n",
    "<p>This function import files presented above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def download_files():\n",
    "    print \"------------------------------------ \"\n",
    "    print \"#########  DOWNLOAD FILES  ######### \"\n",
    "    with open(\"testing_set.txt\", \"r\") as f:\n",
    "        reader = csv.reader(f)\n",
    "        testing_set  = list(reader)\n",
    "        testing_set = [element[0].split(\" \") for element in testing_set]\n",
    "    with open(\"training_set.txt\", \"r\") as f:\n",
    "        reader = csv.reader(f)\n",
    "        training_set  = list(reader)\n",
    "        training_set = [element[0].split(\" \") for element in training_set]\n",
    "    with open(\"node_information.csv\", \"r\") as f:\n",
    "        reader = csv.reader(f)\n",
    "        node_info  = list(reader)\n",
    "    return testing_set,training_set,node_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reduction of training_set & actualization of node_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training_set contains too much samples and we can't train on all the dataset. So that we reduce the dataset to 5% for instance to make our tests before training on all the dataset at the end to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Actualization of training_Set and creation of valid_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Valid_ids is a list of IDs we keep in the new training_set reduced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#to test code we select sample\n",
    "def actualisation_set(training_set,percentage):\n",
    "    print \"---------------------------------------\"\n",
    "    print \"#####  REDUCTION OF TRAINING_SET ###### \"\n",
    "    to_keep = random.sample(range(len(training_set)), k=int(round(len(training_set)*percentage))) # ID à garder de training_set, 30,776 IDs\n",
    "    training_set = [training_set[i] for i in to_keep]\n",
    "    valid_ids=set()   # Type : set  ??\n",
    "    for element in training_set:\n",
    "        valid_ids.add(element[0]) # Ajoute ID source\n",
    "        valid_ids.add(element[1]) # Ajoute ID target\n",
    "    # Valid_ids = tous les noeuds de notre graph qu'on garde, pas de redondance d'ID\n",
    "    print \"We keep \",len(to_keep),\" elements of training_set\"\n",
    "    print \"Valid_ids' length : \",len(valid_ids)\n",
    "    return training_set, valid_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Actualization of node_info\n",
    "We just keep information about IDs selected before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Sort node_info to keep information of valid_ids' IDs\n",
    "def actualisation_node_info(node_info):\n",
    "    print \"-----------------------------------------\"\n",
    "    print \"######  ACTUALISATION OF NODE_INFO ###### \"\n",
    "    tmp=[element for element in node_info if element[0] in valid_ids ]\n",
    "    node_info=tmp\n",
    "    del tmp\n",
    "    print \"Number of nodes in node_info : \",len(node_info)\n",
    "    return node_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creation of ID_pos, data, cite and dates : \n",
    "This dictionnary helps us to find the index of a given ID in node_info. Moreiver we use this function to initialize a dictionnary 'cite', to create a list of dates and to create a list 'data' containing the abstract (1), abstract+title(2), abstract+title+journal(3) or abstract+title+author+journal(4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def create_ID_pos(node_info):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CREATE ID_POS, data, cite and dates ###### \"\n",
    "    IDs = []\n",
    "    ID_pos={}\n",
    "    data = []\n",
    "    dates = []\n",
    "    cite={}\n",
    "    for element in node_info:\n",
    "        ID_pos[element[0]]=len(IDs) # Donne la position de element[0] dans node_info\n",
    "        IDs.append(element[0]) # liste des ID dans node_info\n",
    "        data.append(element[5])\n",
    "        dates.append(int(element[1]))\n",
    "        cite[element[0]]=0 # initialise le dictionnaire des citations\n",
    "    print \"Nombre de nodes dans ID_pos : \",len(IDs)\n",
    "    return ID_pos, data, dates, cite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def create_ID_pos2(node_info):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CREATE ID_POS, data, cite and dates ###### \"\n",
    "    IDs = []\n",
    "    ID_pos={}\n",
    "    data = []\n",
    "    dates = []\n",
    "    cite={}\n",
    "    for element in node_info:\n",
    "        ID_pos[element[0]]=len(IDs) # Donne la position de element[0] dans node_info\n",
    "        IDs.append(element[0]) # liste des ID dans node_info\n",
    "        data.append(' '.join([element[5],element[3]]))\n",
    "        dates.append(int(element[1]))\n",
    "        cite[element[0]]=0 # initialise le dictionnaire des citations\n",
    "    #print(IDs)\n",
    "    #print(node_info[ID_pos[ID]])\n",
    "    print \"Nombre de nodes dans ID_pos : \",len(IDs)\n",
    "    return ID_pos, data, dates, cite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def create_ID_pos3(node_info):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CREATE ID_POS, data, cite and dates ###### \"\n",
    "    IDs = []\n",
    "    ID_pos={}\n",
    "    data = []\n",
    "    dates = []\n",
    "    cite={}\n",
    "    for element in node_info:\n",
    "        ID_pos[element[0]]=len(IDs) # Donne la position de element[0] dans node_info\n",
    "        IDs.append(element[0]) # liste des ID dans node_info\n",
    "        data.append(' '.join([element[5],element[2],element[3]]))\n",
    "        dates.append(int(element[1]))\n",
    "        cite[element[0]]=0 # initialise le dictionnaire des citations\n",
    "    #print(IDs)\n",
    "    #print(node_info[ID_pos[ID]])\n",
    "    print \"Nombre de nodes dans ID_pos : \",len(IDs)\n",
    "    return ID_pos, data, dates, cite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def create_ID_pos4(node_info):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CREATE ID_POS, data, cite and dates ###### \"\n",
    "    IDs = []\n",
    "    ID_pos={}\n",
    "    data = []\n",
    "    dates = []\n",
    "    cite={}\n",
    "    for element in node_info:\n",
    "        ID_pos[element[0]]=len(IDs) # Donne la position de element[0] dans node_info\n",
    "        IDs.append(element[0]) # liste des ID dans node_info\n",
    "        data.append(' '.join([element[5],element[2],element[3],element[3]]))\n",
    "        dates.append(int(element[1]))\n",
    "        cite[element[0]]=0 # initialise le dictionnaire des citations\n",
    "    #print(IDs)\n",
    "    #print(node_info[ID_pos[ID]])\n",
    "    print \"Nombre de nodes dans ID_pos : \",len(IDs)\n",
    "    return ID_pos, data, dates, cite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------------------------------------------------------------------------\n",
    "========================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - Feature creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity using Tfidf : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My_preprocessor clean a text to remove stopwords (such as 'of', 'a', 'an'..) and keep the root of a word (such as 'creation' -> 'creat')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def my_preprocessor(text):\n",
    "    words=text.split(\" \") # Already in lower, we convert text(string) into a list of words in unicode\n",
    "    without_stpwords = [token for token in words if token not in stpwds]  # Remove stopwords\n",
    "    L=[]\n",
    "    for token in without_stpwords:  # Handle exception if appears when keeping roots (ex : 'aed')\n",
    "        try :\n",
    "            L.append(stemmer.stem(token))\n",
    "        except : \n",
    "            print \"Didn't arrive to find root of : \", token\n",
    "            L.append(token)\n",
    "            pass\n",
    "    words2 = L\n",
    "    doc=' '.join(words2)\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the tfidf matrix with data list created before. It contains a list of texts containing abstract or abstract+title or abstract+title+journal ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_tfidf(data,my_preprocessor):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CREATE TFIDF MATRIX  ###### \"\n",
    "    m = TfidfVectorizer(preprocessor=my_preprocessor)\n",
    "    tfidf_matrix = m.fit_transform(data)\n",
    "    tfidf_matrix = tfidf_matrix.toarray()\n",
    "    print \"Taille de tfidf_matrix : \",tfidf_matrix.shape\n",
    "    return tfidf_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function compute the cosine simalirity between to papers. It takes into arguments two elements of data list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cosine_similarity(vector1, vector2):\n",
    "    top=vector1.dot(vector2)\n",
    "    bottom=np.linalg.norm(vector1)*np.linalg.norm(vector2)\n",
    "    if bottom==0: \n",
    "        return 0.0\n",
    "    return top/bottom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of times paper cite or cited or both"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To define is it cites or is cited I took in training_set element [IDA,IDB,Label] <-> IDA cites IDB if label is '1'. \n",
    "<br/>The first function gives us a dictionnary to know the number of times an ID was cited or cite in training_set. \n",
    "<br/> The second one tells us the number of times an ID cite and is cited.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Number of times where an ID is cited or cites another (citer) \n",
    "def cite_cited1(training_set,cite):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CREATE DICTIONNARY CITE ###### \"\n",
    "    compteur=0\n",
    "    for i in range(len(training_set)):\n",
    "        if (training_set[i][2] == '1'):\n",
    "            compteur+=1\n",
    "            cite[training_set[i][1]]+=1\n",
    "            cite[training_set[i][0]]+=1\n",
    "    print \"Number of '1' labels in training_set : \",compteur, \" of \",len(training_set)\n",
    "    return cite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Number of times where an ID is cited and an ID cites another (citer) \n",
    "def cite_cited2(training_set,cite0):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CHECK PAPERS CITED AND CITER ###### \"\n",
    "    cited = cite0\n",
    "    citer = cite0\n",
    "    for i in range(len(training_set)):\n",
    "        if (training_set[i][2] == '1'):\n",
    "            cited[training_set[i][1]]+=1\n",
    "            citer[training_set[i][0]]+=1  # Si il a deja cité quelqu'un\n",
    "    return cited, citer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarité entre les auteurs, les titres et les abstracts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions allow to create features depending on what we have put on 'data' to compute the cosine simalirity and if we want to use 'cite' or 'cited and citer' in argument. They return features as lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Citer/cited in argument and cosine similarity of abstracts.\n",
    "Compute number of common words between titles and authors too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# similarité des titres et auteurs \n",
    "def extract_autor_title_abstract_similarity2(training_set,citer,cited,ID_pos,node_info,tfidf_matrix):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \"\n",
    "    print \"Number of features to handle : \", len(training_set)\n",
    "\n",
    "    overlap_title = []  # number of overlapping words in title\n",
    "    comm_auth = []      # number of common authors\n",
    "    citer_source = []\n",
    "    cited_target = []\n",
    "    similarity_abstract = []\n",
    "\n",
    "    for i in range(len(training_set)):\n",
    "        source = training_set[i][0]\n",
    "        target = training_set[i][1]\n",
    "        citer_source.append(citer[source])\n",
    "        cited_target.append(cited[target])\n",
    "\n",
    "        source_info = node_info[ID_pos[source]]\n",
    "        target_info = node_info[ID_pos[target]]\n",
    "\n",
    "        # convert to lowercase and tokenize\n",
    "        source_title = source_info[2].lower().split(\" \") # renvoie liste des mots du titre en minuscule : dbrane boundstate wavefunctions -> ['dbrane', 'boundstate', 'wavefunctions']\n",
    "        target_title = target_info[2].lower().split(\" \")\n",
    "        # remove stopwords\n",
    "        source_title = [token for token in source_title if token not in stpwds]\n",
    "        target_title = [token for token in target_title if token not in stpwds]\n",
    "        source_title = [stemmer.stem(token) for token in source_title] # Garde la racine des mots\n",
    "        target_title = [stemmer.stem(token) for token in target_title]\n",
    "\n",
    "        # Prend auteur de source et target\n",
    "        source_auth = source_info[3].split(\",\") # met sous forme de liste\n",
    "        target_auth = target_info[3].split(\",\")\n",
    "        \n",
    "        overlap_title.append(len(set(source_title).intersection(set(target_title)))) # Nombre de mots en commun dans titre\n",
    "        comm_auth.append(len(set(source_auth).intersection(set(target_auth)))) # Nombre de mots en commun dans auteur\n",
    "\n",
    "        \n",
    "        similarity  = cosine_similarity(tfidf_matrix[ID_pos[source]],tfidf_matrix[ID_pos[target]])\n",
    "        similarity_abstract.append(similarity)\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "\n",
    "    print \"-----------------------------------\"\n",
    "    print \"nombre de similitudes sur titres : \",sum(i for i in overlap_title)\n",
    "    print \"nombre de similitudes sur auteurs : \",sum(i for i in comm_auth)\n",
    "    return overlap_title,comm_auth,citer_source,cited_target,similarity_abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Cite in argument and cosine similarity of abstracts.\n",
    "Compute number of common words between titles and authors too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# similarité des titres et auteurs \n",
    "def extract_autor_title_abstract_similarity3(training_set,cite,ID_pos,node_info,tfidf_matrix):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \"\n",
    "    print \"Number of features to handle : \", len(training_set)\n",
    "\n",
    "    overlap_title = []  # number of overlapping words in title\n",
    "    comm_auth = []      # number of common authors\n",
    "    cite_source = []\n",
    "    cite_target = []\n",
    "    similarity_abstract = []\n",
    "\n",
    "    for i in range(len(training_set)):\n",
    "        source = training_set[i][0]\n",
    "        target = training_set[i][1]\n",
    "        cite_source.append(cite[source])\n",
    "        cite_target.append(cite[target])\n",
    "\n",
    "        source_info = node_info[ID_pos[source]]\n",
    "        target_info = node_info[ID_pos[target]]\n",
    "\n",
    "        # convert to lowercase and tokenize\n",
    "        source_title = source_info[2].lower().split(\" \") # renvoie liste des mots du titre en minuscule : dbrane boundstate wavefunctions -> ['dbrane', 'boundstate', 'wavefunctions']\n",
    "        target_title = target_info[2].lower().split(\" \")\n",
    "        # remove stopwords\n",
    "        source_title = [token for token in source_title if token not in stpwds]\n",
    "        target_title = [token for token in target_title if token not in stpwds]\n",
    "        source_title = [stemmer.stem(token) for token in source_title] # Garde la racine des mots\n",
    "        target_title = [stemmer.stem(token) for token in target_title]\n",
    "\n",
    "        # Prend auteur de source et target\n",
    "        source_auth = source_info[3].split(\",\") # met sous forme de liste\n",
    "        target_auth = target_info[3].split(\",\")\n",
    "        \n",
    "        overlap_title.append(len(set(source_title).intersection(set(target_title)))) # Nombre de mots en commun dans titre\n",
    "        comm_auth.append(len(set(source_auth).intersection(set(target_auth)))) # Nombre de mots en commun dans auteur\n",
    "\n",
    "        \n",
    "        similarity  = cosine_similarity(tfidf_matrix[ID_pos[source]],tfidf_matrix[ID_pos[target]])\n",
    "        similarity_abstract.append(similarity)\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "\n",
    "    print \"-----------------------------------\"\n",
    "    print \"nombre de similitudes sur titres : \",sum(i for i in overlap_title)\n",
    "    print \"nombre de similitudes sur auteurs : \",sum(i for i in comm_auth)\n",
    "    return overlap_title,comm_auth,cite_source,cite_target,similarity_abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Citer/cited in argument and cosine similarity of abstracts+titles.\n",
    "Compute number of common words between authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# similarité des titres et auteurs \n",
    "def extract_autor_title_abstract_similarity4(training_set,citer, cited,ID_pos,node_info,tfidf_matrix):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \"\n",
    "    print \"Number of features to handle : \", len(training_set)\n",
    "\n",
    "    comm_auth = []      # number of common authors\n",
    "    cite_source = []\n",
    "    cite_target = []\n",
    "    similarity_ = []\n",
    "\n",
    "    for i in range(len(training_set)):\n",
    "        source = training_set[i][0]\n",
    "        target = training_set[i][1]\n",
    "        cite_source.append(citer[source])\n",
    "        cite_target.append(cited[target])\n",
    "\n",
    "        source_info = node_info[ID_pos[source]]\n",
    "        target_info = node_info[ID_pos[target]]\n",
    "\n",
    "        # Prend auteur de source et target\n",
    "        source_auth = source_info[3].split(\",\") # met sous forme de liste\n",
    "        target_auth = target_info[3].split(\",\")\n",
    "        \n",
    "        comm_auth.append(len(set(source_auth).intersection(set(target_auth)))) # Nombre de mots en commun dans auteur\n",
    "\n",
    "        similarity  = cosine_similarity(tfidf_matrix[ID_pos[source]],tfidf_matrix[ID_pos[target]])\n",
    "        similarity_.append(similarity)\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "\n",
    "    print \"-----------------------------------\"\n",
    "    print \"nombre de similitudes sur auteurs : \",sum(i for i in comm_auth)\n",
    "    return comm_auth,cite_source,cite_target,similarity_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Cite in argument and cosine similarity of abstracts+titles.\n",
    "Compute number of common words between authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# similarité des titres et auteurs \n",
    "def extract_autor_title_abstract_similarity5(training_set,cite,ID_pos,node_info,tfidf_matrix):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \"\n",
    "    print \"Number of features to handle : \", len(training_set)\n",
    "\n",
    "    comm_auth = []      # number of common authors\n",
    "    cite_source = []\n",
    "    cite_target = []\n",
    "    similarity_ = []\n",
    "\n",
    "    for i in range(len(training_set)):\n",
    "        source = training_set[i][0]\n",
    "        target = training_set[i][1]\n",
    "        cite_source.append(cite[source])\n",
    "        cite_target.append(cite[target])\n",
    "\n",
    "        source_info = node_info[ID_pos[source]]\n",
    "        target_info = node_info[ID_pos[target]]\n",
    "\n",
    "        # Prend auteur de source et target\n",
    "        source_auth = source_info[3].split(\",\") # met sous forme de liste\n",
    "        target_auth = target_info[3].split(\",\")\n",
    "        \n",
    "        comm_auth.append(len(set(source_auth).intersection(set(target_auth)))) # Nombre de mots en commun dans auteur\n",
    "\n",
    "        \n",
    "        similarity  = cosine_similarity(tfidf_matrix[ID_pos[source]],tfidf_matrix[ID_pos[target]])\n",
    "        similarity_.append(similarity)\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "\n",
    "    print \"-----------------------------------\"\n",
    "    print \"nombre de similitudes sur auteurs : \",sum(i for i in comm_auth)\n",
    "    return comm_auth,cite_source,cite_target,similarity_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Citer/cited in argument and cosine similarity of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# similarité des titres et auteurs \n",
    "def extract_autor_title_abstract_similarity6(training_set,citer, cited,ID_pos,node_info,tfidf_matrix):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \"\n",
    "    print \"Number of features to handle : \", len(training_set)\n",
    "\n",
    "    cite_source = []\n",
    "    cite_target = []\n",
    "    similarity_ = []\n",
    "\n",
    "    for i in range(len(training_set)):\n",
    "        source = training_set[i][0]\n",
    "        target = training_set[i][1]\n",
    "        cite_source.append(citer[source])\n",
    "        cite_target.append(cited[target])\n",
    "        \n",
    "        source_info = node_info[ID_pos[source]]\n",
    "        target_info = node_info[ID_pos[target]]\n",
    "\n",
    "        # Prend auteur de source et target\n",
    "        source_auth = source_info[5].split(\",\") # met sous forme de liste\n",
    "        target_auth = target_info[5].split(\",\")\n",
    "   \n",
    "        similarity  = cosine_similarity(tfidf_matrix[ID_pos[source]],tfidf_matrix[ID_pos[target]])\n",
    "        similarity_.append(similarity)\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "    return cite_source,cite_target,similarity_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Citer/cited in argument and cosine similarity of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# similarité des titres et auteurs \n",
    "def extract_autor_title_abstract_similarity7(training_set,cite,ID_pos,node_info,tfidf_matrix):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \"\n",
    "    print \"Number of features to handle : \", len(training_set)\n",
    "\n",
    "    cite_source = []\n",
    "    cite_target = []\n",
    "    similarity_ = []\n",
    "\n",
    "    for i in range(len(training_set)):\n",
    "        source = training_set[i][0]\n",
    "        target = training_set[i][1]\n",
    "        cite_source.append(cite[source])\n",
    "        cite_target.append(cite[target])\n",
    "        \n",
    "        similarity  = cosine_similarity(tfidf_matrix[ID_pos[source]],tfidf_matrix[ID_pos[target]])\n",
    "        similarity_.append(similarity)\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "    return cite_source,cite_target,similarity_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create dates features :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# similarité des dates\n",
    "def extract_date(sett,dates,ID_pos):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  EXTRACTION OF DATE FEATURE ###### \"\n",
    "\n",
    "    date_source = []\n",
    "    date_target = []\n",
    "    diff = []\n",
    "    for i in range(len(sett)):\n",
    "        source = sett[i][0]\n",
    "        target = sett[i][1]\n",
    "        date_source.append(dates[ID_pos[source]])\n",
    "        date_target.append(dates[ID_pos[target]])\n",
    "        diff.append((abs(int(dates[ID_pos[source]])-int(dates[ID_pos[target]]))))\n",
    "        if i % 1000 == 0:\n",
    "            print i, \" examples processsed\"\n",
    "    return date_source, date_target,diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 - Feature scaling before training or testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List of avaibles features : <br/>\n",
    "overlap_abstract <br/>\n",
    "overlap_title<br/>\n",
    "comm_auth<br/>\n",
    "similarity_abstract<br/>\n",
    "similarity<br/>\n",
    "cite_source<br/>\n",
    "cite_target<br/>\n",
    "date_source<br/>\n",
    "date_target<br/>\n",
    "diff<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#features = [overlap_abstract,overlap_title,comm_auth,cite_source,cite_target,diff]\n",
    "def scaling(features):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  SCALING FEATURES ###### \"\n",
    "    featuress = np.array(features).T # transposée \n",
    "    featuress = preprocessing.scale(featuress)\n",
    "    return featuress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We put labels in a separated list before training a classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# convert labels into integers then into column array\n",
    "def convert_labels(training_set):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CONVERT LABELS ###### \"\n",
    "    labels = [int(element[2]) for element in training_set]\n",
    "    labels = list(labels)\n",
    "    labels_array = np.array(labels)\n",
    "    return labels_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------------------------------------------------------------------------\n",
    "========================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 - Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I decided to test with 3 classifiers. It appears that the MLP was the best so that I kept it. You can see results with other classifiers beelow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def classification(training_set,labels_array,training_features):\n",
    "    print \"-----------------------------------\"\n",
    "    print \"#########  CLASSIFICATION ###### \"\n",
    "    kf = KFold(len(training_set), n_folds=10) # Cross validation sur 10 sous ensembles de training_features\n",
    "    sumf1=0\n",
    "    for train_index, test_index in kf: # train_index, test_indexs = liste d'index sur interval\n",
    "        X_train, X_test = training_features[train_index], training_features[test_index]\n",
    "        y_train, y_test = labels_array[train_index], labels_array[test_index]\n",
    "        # initialize basic SVM :\n",
    "        #classifier = svm.LinearSVC()\n",
    "        #classifier = RandomForestClassifier(n_estimators=200, min_samples_split=2)\n",
    "        classifier =  MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(10,5), random_state=1)\n",
    "\n",
    "        # train\n",
    "        classifier.fit(X_train, y_train)\n",
    "        labels_predicted = classifier.predict(X_test)\n",
    "        #y_score = classifier.predict_proba(X_test)\n",
    "        sumf1+=f1_score(labels_predicted,y_test)\n",
    "\n",
    "        # Evaluation of the prediction\n",
    "        #print classification_report(y_test, labels_predicted)\n",
    "        #print \"The accuracy score is {:.2%}\".format(accuracy_score(y_test, labels_predicted))\n",
    "        # Compute ROC curve and area under the curve\n",
    "        #fpr, tpr, thresholds = roc_curve(y_test, y_score[:, 1])\n",
    "        #print thresholds\n",
    "        #roc_auc = auc(fpr, tpr)\n",
    "        #print \"Area under the ROC curve : %f\" % roc_auc\n",
    "        #print \"---------------------------------------------\"\n",
    "\n",
    "    print \"Resultat final : \"\n",
    "    print sumf1/10.0 # moyenne sur les 10 sous ensembles de training_set\n",
    "    return classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluation metric : <br/>\n",
    "𝐹1 = 2 (𝑝∗𝑟)/(𝑝+𝑟) <br/><br/>\n",
    "Where : 𝑝=𝑡𝑝/(𝑡𝑝+𝑓𝑝) : precision <br/> and r=𝑡𝑝/(𝑡𝑝+𝑓𝑛) : recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7 - Training Script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 3 'main' block depending on what we want in the tfidf matrix to compute cosine similarity. <br/>\n",
    "We just have too run the block to compute the training of a classifier. Take care to select a correct percentage of the dataset to limit the computation time. <br/>\n",
    "You have to comment/uncomment the line of 'extract_features' if you want to use cite or citer/cited in feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### DATA = ABSTRACT FOR TFIDF_MATRIX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------ \n",
      "#########  DOWNLOAD FILES  ######### \n",
      "-----------------------------------\n",
      "#########  CREATE ID_POS, data, cite and dates ###### \n",
      "Nombre de nodes dans ID_pos :  27770\n",
      "-----------------------------------\n",
      "#########  CHECK PAPERS CITED AND CITER ###### \n",
      "-----------------------------------\n",
      "#########  CREATE DICTIONNARY CITE ###### \n",
      "Number of '1' labels in training_set :  335130  of  615512\n",
      "---------------------------------------\n",
      "#####  REDUCTION OF TRAINING_SET ###### \n",
      "We keep  30776  elements of training_set\n",
      "Valid_ids' length :  22865\n",
      "-----------------------------------\n",
      "#########  SCALING FEATURES ###### \n",
      "-----------------------------------\n",
      "#########  CONVERT LABELS ###### \n",
      "-----------------------------------\n",
      "#########  CLASSIFICATION ###### \n",
      "Resultat final : \n",
      "0.683393942064\n"
     ]
    }
   ],
   "source": [
    "# DATA = ABSTRACT FOR TFIDF_MATRIX\n",
    "# LOAD FILES / TREATMENT\n",
    "testing_set,training_set,node_info = download_files()\n",
    "ID_pos, data,dates,cite0 = create_ID_pos(node_info) # init dictionaire des citations\n",
    "cited,citer = cite_cited2(training_set,cite0)\n",
    "cite = cite_cited1(training_set,cite0)\n",
    "\n",
    "training_set, valid_ids = actualisation_set(training_set,0.05) #5%\n",
    "#node_info = actualisation_node_info(node_info) # peu de difference entre set entier (27,000) et subset 22,000\n",
    "\n",
    "# CREATE FEATURES\n",
    "tfidf_matrix = create_tfidf(data,my_preprocessor)\n",
    "\n",
    "overlap_title,comm_auth,citer_source,cited_target,similarity_abstract = extract_autor_title_abstract_similarity2(training_set,citer,cited,ID_pos,node_info,tfidf_matrix)\n",
    "#overlap_title,comm_auth,citer_source,cited_target,similarity_abstract = extract_autor_title_abstract_similarity3(training_set,cite,ID_pos,node_info,tfidf_matrix)\n",
    "\n",
    "date_source, date_target,diff = extract_date(training_set,dates,ID_pos)\n",
    "\n",
    "# FEATURE SELECTION :\n",
    "features = [similarity_abstract,overlap_title,comm_auth,citer_source,cited_target,date_source, date_target]\n",
    "training_features = scaling(features)\n",
    "labels_array = convert_labels(training_set)\n",
    "\n",
    "# CLASSIFICATION : \n",
    "classifier = classification(training_set,labels_array,training_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30776 30776 30776 615512 615512 30776 30776\n"
     ]
    }
   ],
   "source": [
    "print len(similarity_abstract),len(overlap_title),len(comm_auth),len(cite_source),len(cite_target),len(date_source),len(date_target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### DATA = ABSTRACT + AUTHOR FOR TFIDF_MATRIX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------\n",
      "#########  DOWNLOAD FILES ###### \n",
      "-----------------------------------\n",
      "#########  CREATE ID_POS ###### \n",
      "Nombre de nodes dans ID_pos :  27770\n",
      "-----------------------------------\n",
      "#########  CHECK PAPERS CITED AND CITER ###### \n",
      "-----------------------------------\n",
      "#########  REDUCTION OF TRAINING_SET ###### \n",
      "On garde 30776 elements de training_set\n",
      "Taille de valid_ids :  22857\n",
      "-----------------------------------\n",
      "#########  CREATE ID_POS ###### \n",
      "Nombre de nodes dans ID_pos :  27770\n",
      "-----------------------------------\n",
      "#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \n",
      "nombre d'echantillons à traiter :  30776\n",
      "0  examples processsed\n",
      "1000  examples processsed\n",
      "2000  examples processsed\n",
      "3000  examples processsed\n",
      "4000  examples processsed\n",
      "5000  examples processsed\n",
      "6000  examples processsed\n",
      "7000  examples processsed\n",
      "8000  examples processsed\n",
      "9000  examples processsed\n",
      "10000  examples processsed\n",
      "11000  examples processsed\n",
      "12000  examples processsed\n",
      "13000  examples processsed\n",
      "14000  examples processsed\n",
      "15000  examples processsed\n",
      "16000  examples processsed\n",
      "17000  examples processsed\n",
      "18000  examples processsed\n",
      "19000  examples processsed\n",
      "20000  examples processsed\n",
      "21000  examples processsed\n",
      "22000  examples processsed\n",
      "23000  examples processsed\n",
      "24000  examples processsed\n",
      "25000  examples processsed\n",
      "26000  examples processsed\n",
      "27000  examples processsed\n",
      "28000  examples processsed\n",
      "29000  examples processsed\n",
      "30000  examples processsed\n",
      "-----------------------------------\n",
      "nombre de similitudes sur titres :  15734\n",
      "nombre de similitudes sur auteurs :  2561\n",
      "nombre de similitudes sur abstracts :  190740\n",
      "-----------------------------------\n",
      "#########  EXTRACTION OF DATE FEATURE ###### \n",
      "0  examples processsed\n",
      "1000  examples processsed\n",
      "2000  examples processsed\n",
      "3000  examples processsed\n",
      "4000  examples processsed\n",
      "5000  examples processsed\n",
      "6000  examples processsed\n",
      "7000  examples processsed\n",
      "8000  examples processsed\n",
      "9000  examples processsed\n",
      "10000  examples processsed\n",
      "11000  examples processsed\n",
      "12000  examples processsed\n",
      "13000  examples processsed\n",
      "14000  examples processsed\n",
      "15000  examples processsed\n",
      "16000  examples processsed\n",
      "17000  examples processsed\n",
      "18000  examples processsed\n",
      "19000  examples processsed\n",
      "20000  examples processsed\n",
      "21000  examples processsed\n",
      "22000  examples processsed\n",
      "23000  examples processsed\n",
      "24000  examples processsed\n",
      "25000  examples processsed\n",
      "26000  examples processsed\n",
      "27000  examples processsed\n",
      "28000  examples processsed\n",
      "29000  examples processsed\n",
      "30000  examples processsed\n",
      "-----------------------------------\n",
      "#########  SCALING FEATURES ###### \n",
      "-----------------------------------\n",
      "#########  CONVERT LABELS ###### \n",
      "-----------------------------------\n",
      "#########  CLASSIFICATION ###### \n",
      "Resultat final : \n",
      "0.92516093588\n"
     ]
    }
   ],
   "source": [
    "# DATA_ABS_TIT = ABSTRACT + AUTHOR FOR TFIDF_MATRIX\n",
    "# LOAD FILES / TREATMENT\n",
    "testing_set,training_set,node_info = download_files()\n",
    "ID_pos, data,dates,cite0 = create_ID_pos(node_info) # init dictionaire des citations\n",
    "cited,citer = cite_cited2(training_set,cite0) # Sur 80,000 echant de training_set\n",
    "cite = cite_cited1(training_set,cite0)\n",
    "\n",
    "training_set, valid_ids = actualisation_set(training_set,0.05) #5%\n",
    "#node_info = actualisation_node_info(node_info) # peu de difference entre set entier (27,000) et subset 22,000\n",
    "ID_pos, data_abs_tit,dates,cite0 = create_ID_pos2(node_info) # init dictionaire des citations\n",
    "\n",
    "\n",
    "# CREATE FEATURES\n",
    "#tfidf_matrix_abs_tit = create_tfidf(data_abs_tit,my_preprocessor)\n",
    "\n",
    "comm_auth,cite_source,cite_target,similarity_ = extract_autor_title_abstract_similarity4(training_set,citer,cited,ID_pos,node_info,tfidf_matrix_abs_tit)\n",
    "#comm_auth,cite_source,cite_target,similarity_abstract = extract_autor_title_abstract_similarity5(training_set,cite,ID_pos,node_info,tfidf_matrix_abs_tit)\n",
    "\n",
    "date_source, date_target,diff = extract_date(training_set,dates,ID_pos)\n",
    "\n",
    "# FEATURE SELECTION :\n",
    "features = [similarity_,comm_auth,cite_source,cite_target,diff]\n",
    "features = [similarity_,comm_auth,cite_source,cite_target,date_source, date_target]\n",
    "training_features = scaling(features)\n",
    "labels_array = convert_labels(training_set)\n",
    "\n",
    "# CLASSIFICATION : \n",
    "classifier = classification(training_set,labels_array,training_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### DATA = ABSTRACT + TITLE + AUTHOR (+journal) FOR TFIDF_MATRIX\n",
    "<p style='color:red'>Best combinaison of features : </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------\n",
      "#########  DOWNLOAD FILES ###### \n",
      "-----------------------------------\n",
      "#########  CREATE ID_POS ###### \n",
      "Nombre de nodes dans ID_pos :  27770\n",
      "-----------------------------------\n",
      "#########  CHECK PAPERS CITED AND CITER ###### \n",
      "-----------------------------------\n",
      "#########  REDUCTION OF TRAINING_SET ###### \n",
      "On garde 615512 elements de training_set\n",
      "Taille de valid_ids :  27770\n",
      "-----------------------------------\n",
      "#########  CREATE ID_POS ###### \n",
      "Nombre de nodes dans ID_pos :  27770\n",
      "-----------------------------------\n",
      "#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \n",
      "nombre d'echantillons à traiter :  615512\n",
      "0  examples processsed\n",
      "1000  examples processsed\n",
      "2000  examples processsed\n",
      "3000  examples processsed\n",
      "4000  examples processsed\n",
      "5000  examples processsed\n",
      "6000  examples processsed\n",
      "7000  examples processsed\n",
      "8000  examples processsed\n",
      "9000  examples processsed\n",
      "10000  examples processsed\n",
      "11000  examples processsed\n",
      "12000  examples processsed\n",
      "13000  examples processsed\n",
      "14000  examples processsed\n",
      "15000  examples processsed\n",
      "16000  examples processsed\n",
      "17000  examples processsed\n",
      "18000  examples processsed\n",
      "19000  examples processsed\n",
      "20000  examples processsed\n",
      "21000  examples processsed\n",
      "22000  examples processsed\n",
      "23000  examples processsed\n",
      "24000  examples processsed\n",
      "25000  examples processsed\n",
      "26000  examples processsed\n",
      "27000  examples processsed\n",
      "28000  examples processsed\n",
      "29000  examples processsed\n",
      "30000  examples processsed\n",
      "31000  examples processsed\n",
      "32000  examples processsed\n",
      "33000  examples processsed\n",
      "34000  examples processsed\n",
      "35000  examples processsed\n",
      "36000  examples processsed\n",
      "37000  examples processsed\n",
      "38000  examples processsed\n",
      "39000  examples processsed\n",
      "40000  examples processsed\n",
      "41000  examples processsed\n",
      "42000  examples processsed\n",
      "43000  examples processsed\n",
      "44000  examples processsed\n",
      "45000  examples processsed\n",
      "46000  examples processsed\n",
      "47000  examples processsed\n",
      "48000  examples processsed\n",
      "49000  examples processsed\n",
      "50000  examples processsed\n",
      "51000  examples processsed\n",
      "52000  examples processsed\n",
      "53000  examples processsed\n",
      "54000  examples processsed\n",
      "55000  examples processsed\n",
      "56000  examples processsed\n",
      "57000  examples processsed\n",
      "58000  examples processsed\n",
      "59000  examples processsed\n",
      "60000  examples processsed\n",
      "61000  examples processsed\n",
      "62000  examples processsed\n",
      "63000  examples processsed\n",
      "64000  examples processsed\n",
      "65000  examples processsed\n",
      "66000  examples processsed\n",
      "67000  examples processsed\n",
      "68000  examples processsed\n",
      "69000  examples processsed\n",
      "70000  examples processsed\n",
      "71000  examples processsed\n",
      "72000  examples processsed\n",
      "73000  examples processsed\n",
      "74000  examples processsed\n",
      "75000  examples processsed\n",
      "76000  examples processsed\n",
      "77000  examples processsed\n",
      "78000  examples processsed\n",
      "79000  examples processsed\n",
      "80000  examples processsed\n",
      "81000  examples processsed\n",
      "82000  examples processsed\n",
      "83000  examples processsed\n",
      "84000  examples processsed\n",
      "85000  examples processsed\n",
      "86000  examples processsed\n",
      "87000  examples processsed\n",
      "88000  examples processsed\n",
      "89000  examples processsed\n",
      "90000  examples processsed\n",
      "91000  examples processsed\n",
      "92000  examples processsed\n",
      "93000  examples processsed\n",
      "94000  examples processsed\n",
      "95000  examples processsed\n",
      "96000  examples processsed\n",
      "97000  examples processsed\n",
      "98000  examples processsed\n",
      "99000  examples processsed\n",
      "100000  examples processsed\n",
      "101000  examples processsed\n",
      "102000  examples processsed\n",
      "103000  examples processsed\n",
      "104000  examples processsed\n",
      "105000  examples processsed\n",
      "106000  examples processsed\n",
      "107000  examples processsed\n",
      "108000  examples processsed\n",
      "109000  examples processsed\n",
      "110000  examples processsed\n",
      "111000  examples processsed\n",
      "112000  examples processsed\n",
      "113000  examples processsed\n",
      "114000  examples processsed\n",
      "115000  examples processsed\n",
      "116000  examples processsed\n",
      "117000  examples processsed\n",
      "118000  examples processsed\n",
      "119000  examples processsed\n",
      "120000  examples processsed\n",
      "121000  examples processsed\n",
      "122000  examples processsed\n",
      "123000  examples processsed\n",
      "124000  examples processsed\n",
      "125000  examples processsed\n",
      "126000  examples processsed\n",
      "127000  examples processsed\n",
      "128000  examples processsed\n",
      "129000  examples processsed\n",
      "130000  examples processsed\n",
      "131000  examples processsed\n",
      "132000  examples processsed\n",
      "133000  examples processsed\n",
      "134000  examples processsed\n",
      "135000  examples processsed\n",
      "136000  examples processsed\n",
      "137000  examples processsed\n",
      "138000  examples processsed\n",
      "139000  examples processsed\n",
      "140000  examples processsed\n",
      "141000  examples processsed\n",
      "142000  examples processsed\n",
      "143000  examples processsed\n",
      "144000  examples processsed\n",
      "145000  examples processsed\n",
      "146000  examples processsed\n",
      "147000  examples processsed\n",
      "148000  examples processsed\n",
      "149000  examples processsed\n",
      "150000  examples processsed\n",
      "151000  examples processsed\n",
      "152000  examples processsed\n",
      "153000  examples processsed\n",
      "154000  examples processsed\n",
      "155000  examples processsed\n",
      "156000  examples processsed\n",
      "157000  examples processsed\n",
      "158000  examples processsed\n",
      "159000  examples processsed\n",
      "160000  examples processsed\n",
      "161000  examples processsed\n",
      "162000  examples processsed\n",
      "163000  examples processsed\n",
      "164000  examples processsed\n",
      "165000  examples processsed\n",
      "166000  examples processsed\n",
      "167000  examples processsed\n",
      "168000  examples processsed\n",
      "169000  examples processsed\n",
      "170000  examples processsed\n",
      "171000  examples processsed\n",
      "172000  examples processsed\n",
      "173000  examples processsed\n",
      "174000  examples processsed\n",
      "175000  examples processsed\n",
      "176000  examples processsed\n",
      "177000  examples processsed\n",
      "178000  examples processsed\n",
      "179000  examples processsed\n",
      "180000  examples processsed\n",
      "181000  examples processsed\n",
      "182000  examples processsed\n",
      "183000  examples processsed\n",
      "184000  examples processsed\n",
      "185000  examples processsed\n",
      "186000  examples processsed\n",
      "187000  examples processsed\n",
      "188000  examples processsed\n",
      "189000  examples processsed\n",
      "190000  examples processsed\n",
      "191000  examples processsed\n",
      "192000  examples processsed\n",
      "193000  examples processsed\n",
      "194000  examples processsed\n",
      "195000  examples processsed\n",
      "196000  examples processsed\n",
      "197000  examples processsed\n",
      "198000  examples processsed\n",
      "199000  examples processsed\n",
      "200000  examples processsed\n",
      "201000  examples processsed\n",
      "202000  examples processsed\n",
      "203000  examples processsed\n",
      "204000  examples processsed\n",
      "205000  examples processsed\n",
      "206000  examples processsed\n",
      "207000  examples processsed\n",
      "208000  examples processsed\n",
      "209000  examples processsed\n",
      "210000  examples processsed\n",
      "211000  examples processsed\n",
      "212000  examples processsed\n",
      "213000  examples processsed\n",
      "214000  examples processsed\n",
      "215000  examples processsed\n",
      "216000  examples processsed\n",
      "217000  examples processsed\n",
      "218000  examples processsed\n",
      "219000  examples processsed\n",
      "220000  examples processsed\n",
      "221000  examples processsed\n",
      "222000  examples processsed\n",
      "223000  examples processsed\n",
      "224000  examples processsed\n",
      "225000  examples processsed\n",
      "226000  examples processsed\n",
      "227000  examples processsed\n",
      "228000  examples processsed\n",
      "229000  examples processsed\n",
      "230000  examples processsed\n",
      "231000  examples processsed\n",
      "232000  examples processsed\n",
      "233000  examples processsed\n",
      "234000  examples processsed\n",
      "235000  examples processsed\n",
      "236000  examples processsed\n",
      "237000  examples processsed\n",
      "238000  examples processsed\n",
      "239000  examples processsed\n",
      "240000  examples processsed\n",
      "241000  examples processsed\n",
      "242000  examples processsed\n",
      "243000  examples processsed\n",
      "244000  examples processsed\n",
      "245000  examples processsed\n",
      "246000  examples processsed\n",
      "247000  examples processsed\n",
      "248000  examples processsed\n",
      "249000  examples processsed\n",
      "250000  examples processsed\n",
      "251000  examples processsed\n",
      "252000  examples processsed\n",
      "253000  examples processsed\n",
      "254000  examples processsed\n",
      "255000  examples processsed\n",
      "256000  examples processsed\n",
      "257000  examples processsed\n",
      "258000  examples processsed\n",
      "259000  examples processsed\n",
      "260000  examples processsed\n",
      "261000  examples processsed\n",
      "262000  examples processsed\n",
      "263000  examples processsed\n",
      "264000  examples processsed\n",
      "265000  examples processsed\n",
      "266000  examples processsed\n",
      "267000  examples processsed\n",
      "268000  examples processsed\n",
      "269000  examples processsed\n",
      "270000  examples processsed\n",
      "271000  examples processsed\n",
      "272000  examples processsed\n",
      "273000  examples processsed\n",
      "274000  examples processsed\n",
      "275000  examples processsed\n",
      "276000  examples processsed\n",
      "277000  examples processsed\n",
      "278000  examples processsed\n",
      "279000  examples processsed\n",
      "280000  examples processsed\n",
      "281000  examples processsed\n",
      "282000  examples processsed\n",
      "283000  examples processsed\n",
      "284000  examples processsed\n",
      "285000  examples processsed\n",
      "286000  examples processsed\n",
      "287000  examples processsed\n",
      "288000  examples processsed\n",
      "289000  examples processsed\n",
      "290000  examples processsed\n",
      "291000  examples processsed\n",
      "292000  examples processsed\n",
      "293000  examples processsed\n",
      "294000  examples processsed\n",
      "295000  examples processsed\n",
      "296000  examples processsed\n",
      "297000  examples processsed\n",
      "298000  examples processsed\n",
      "299000  examples processsed\n",
      "300000  examples processsed\n",
      "301000  examples processsed\n",
      "302000  examples processsed\n",
      "303000  examples processsed\n",
      "304000  examples processsed\n",
      "305000  examples processsed\n",
      "306000  examples processsed\n",
      "307000  examples processsed\n",
      "308000  examples processsed\n",
      "309000  examples processsed\n",
      "310000  examples processsed\n",
      "311000  examples processsed\n",
      "312000  examples processsed\n",
      "313000  examples processsed\n",
      "314000  examples processsed\n",
      "315000  examples processsed\n",
      "316000  examples processsed\n",
      "317000  examples processsed\n",
      "318000  examples processsed\n",
      "319000  examples processsed\n",
      "320000  examples processsed\n",
      "321000  examples processsed\n",
      "322000  examples processsed\n",
      "323000  examples processsed\n",
      "324000  examples processsed\n",
      "325000  examples processsed\n",
      "326000  examples processsed\n",
      "327000  examples processsed\n",
      "328000  examples processsed\n",
      "329000  examples processsed\n",
      "330000  examples processsed\n",
      "331000  examples processsed\n",
      "332000  examples processsed\n",
      "333000  examples processsed\n",
      "334000  examples processsed\n",
      "335000  examples processsed\n",
      "336000  examples processsed\n",
      "337000  examples processsed\n",
      "338000  examples processsed\n",
      "339000  examples processsed\n",
      "340000  examples processsed\n",
      "341000  examples processsed\n",
      "342000  examples processsed\n",
      "343000  examples processsed\n",
      "344000  examples processsed\n",
      "345000  examples processsed\n",
      "346000  examples processsed\n",
      "347000  examples processsed\n",
      "348000  examples processsed\n",
      "349000  examples processsed\n",
      "350000  examples processsed\n",
      "351000  examples processsed\n",
      "352000  examples processsed\n",
      "353000  examples processsed\n",
      "354000  examples processsed\n",
      "355000  examples processsed\n",
      "356000  examples processsed\n",
      "357000  examples processsed\n",
      "358000  examples processsed\n",
      "359000  examples processsed\n",
      "360000  examples processsed\n",
      "361000  examples processsed\n",
      "362000  examples processsed\n",
      "363000  examples processsed\n",
      "364000  examples processsed\n",
      "365000  examples processsed\n",
      "366000  examples processsed\n",
      "367000  examples processsed\n",
      "368000  examples processsed\n",
      "369000  examples processsed\n",
      "370000  examples processsed\n",
      "371000  examples processsed\n",
      "372000  examples processsed\n",
      "373000  examples processsed\n",
      "374000  examples processsed\n",
      "375000  examples processsed\n",
      "376000  examples processsed\n",
      "377000  examples processsed\n",
      "378000  examples processsed\n",
      "379000  examples processsed\n",
      "380000  examples processsed\n",
      "381000  examples processsed\n",
      "382000  examples processsed\n",
      "383000  examples processsed\n",
      "384000  examples processsed\n",
      "385000  examples processsed\n",
      "386000  examples processsed\n",
      "387000  examples processsed\n",
      "388000  examples processsed\n",
      "389000  examples processsed\n",
      "390000  examples processsed\n",
      "391000  examples processsed\n",
      "392000  examples processsed\n",
      "393000  examples processsed\n",
      "394000  examples processsed\n",
      "395000  examples processsed\n",
      "396000  examples processsed\n",
      "397000  examples processsed\n",
      "398000  examples processsed\n",
      "399000  examples processsed\n",
      "400000  examples processsed\n",
      "401000  examples processsed\n",
      "402000  examples processsed\n",
      "403000  examples processsed\n",
      "404000  examples processsed\n",
      "405000  examples processsed\n",
      "406000  examples processsed\n",
      "407000  examples processsed\n",
      "408000  examples processsed\n",
      "409000  examples processsed\n",
      "410000  examples processsed\n",
      "411000  examples processsed\n",
      "412000  examples processsed\n",
      "413000  examples processsed\n",
      "414000  examples processsed\n",
      "415000  examples processsed\n",
      "416000  examples processsed\n",
      "417000  examples processsed\n",
      "418000  examples processsed\n",
      "419000  examples processsed\n",
      "420000  examples processsed\n",
      "421000  examples processsed\n",
      "422000  examples processsed\n",
      "423000  examples processsed\n",
      "424000  examples processsed\n",
      "425000  examples processsed\n",
      "426000  examples processsed\n",
      "427000  examples processsed\n",
      "428000  examples processsed\n",
      "429000  examples processsed\n",
      "430000  examples processsed\n",
      "431000  examples processsed\n",
      "432000  examples processsed\n",
      "433000  examples processsed\n",
      "434000  examples processsed\n",
      "435000  examples processsed\n",
      "436000  examples processsed\n",
      "437000  examples processsed\n",
      "438000  examples processsed\n",
      "439000  examples processsed\n",
      "440000  examples processsed\n",
      "441000  examples processsed\n",
      "442000  examples processsed\n",
      "443000  examples processsed\n",
      "444000  examples processsed\n",
      "445000  examples processsed\n",
      "446000  examples processsed\n",
      "447000  examples processsed\n",
      "448000  examples processsed\n",
      "449000  examples processsed\n",
      "450000  examples processsed\n",
      "451000  examples processsed\n",
      "452000  examples processsed\n",
      "453000  examples processsed\n",
      "454000  examples processsed\n",
      "455000  examples processsed\n",
      "456000  examples processsed\n",
      "457000  examples processsed\n",
      "458000  examples processsed\n",
      "459000  examples processsed\n",
      "460000  examples processsed\n",
      "461000  examples processsed\n",
      "462000  examples processsed\n",
      "463000  examples processsed\n",
      "464000  examples processsed\n",
      "465000  examples processsed\n",
      "466000  examples processsed\n",
      "467000  examples processsed\n",
      "468000  examples processsed\n",
      "469000  examples processsed\n",
      "470000  examples processsed\n",
      "471000  examples processsed\n",
      "472000  examples processsed\n",
      "473000  examples processsed\n",
      "474000  examples processsed\n",
      "475000  examples processsed\n",
      "476000  examples processsed\n",
      "477000  examples processsed\n",
      "478000  examples processsed\n",
      "479000  examples processsed\n",
      "480000  examples processsed\n",
      "481000  examples processsed\n",
      "482000  examples processsed\n",
      "483000  examples processsed\n",
      "484000  examples processsed\n",
      "485000  examples processsed\n",
      "486000  examples processsed\n",
      "487000  examples processsed\n",
      "488000  examples processsed\n",
      "489000  examples processsed\n",
      "490000  examples processsed\n",
      "491000  examples processsed\n",
      "492000  examples processsed\n",
      "493000  examples processsed\n",
      "494000  examples processsed\n",
      "495000  examples processsed\n",
      "496000  examples processsed\n",
      "497000  examples processsed\n",
      "498000  examples processsed\n",
      "499000  examples processsed\n",
      "500000  examples processsed\n",
      "501000  examples processsed\n",
      "502000  examples processsed\n",
      "503000  examples processsed\n",
      "504000  examples processsed\n",
      "505000  examples processsed\n",
      "506000  examples processsed\n",
      "507000  examples processsed\n",
      "508000  examples processsed\n",
      "509000  examples processsed\n",
      "510000  examples processsed\n",
      "511000  examples processsed\n",
      "512000  examples processsed\n",
      "513000  examples processsed\n",
      "514000  examples processsed\n",
      "515000  examples processsed\n",
      "516000  examples processsed\n",
      "517000  examples processsed\n",
      "518000  examples processsed\n",
      "519000  examples processsed\n",
      "520000  examples processsed\n",
      "521000  examples processsed\n",
      "522000  examples processsed\n",
      "523000  examples processsed\n",
      "524000  examples processsed\n",
      "525000  examples processsed\n",
      "526000  examples processsed\n",
      "527000  examples processsed\n",
      "528000  examples processsed\n",
      "529000  examples processsed\n",
      "530000  examples processsed\n",
      "531000  examples processsed\n",
      "532000  examples processsed\n",
      "533000  examples processsed\n",
      "534000  examples processsed\n",
      "535000  examples processsed\n",
      "536000  examples processsed\n",
      "537000  examples processsed\n",
      "538000  examples processsed\n",
      "539000  examples processsed\n",
      "540000  examples processsed\n",
      "541000  examples processsed\n",
      "542000  examples processsed\n",
      "543000  examples processsed\n",
      "544000  examples processsed\n",
      "545000  examples processsed\n",
      "546000  examples processsed\n",
      "547000  examples processsed\n",
      "548000  examples processsed\n",
      "549000  examples processsed\n",
      "550000  examples processsed\n",
      "551000  examples processsed\n",
      "552000  examples processsed\n",
      "553000  examples processsed\n",
      "554000  examples processsed\n",
      "555000  examples processsed\n",
      "556000  examples processsed\n",
      "557000  examples processsed\n",
      "558000  examples processsed\n",
      "559000  examples processsed\n",
      "560000  examples processsed\n",
      "561000  examples processsed\n",
      "562000  examples processsed\n",
      "563000  examples processsed\n",
      "564000  examples processsed\n",
      "565000  examples processsed\n",
      "566000  examples processsed\n",
      "567000  examples processsed\n",
      "568000  examples processsed\n",
      "569000  examples processsed\n",
      "570000  examples processsed\n",
      "571000  examples processsed\n",
      "572000  examples processsed\n",
      "573000  examples processsed\n",
      "574000  examples processsed\n",
      "575000  examples processsed\n",
      "576000  examples processsed\n",
      "577000  examples processsed\n",
      "578000  examples processsed\n",
      "579000  examples processsed\n",
      "580000  examples processsed\n",
      "581000  examples processsed\n",
      "582000  examples processsed\n",
      "583000  examples processsed\n",
      "584000  examples processsed\n",
      "585000  examples processsed\n",
      "586000  examples processsed\n",
      "587000  examples processsed\n",
      "588000  examples processsed\n",
      "589000  examples processsed\n",
      "590000  examples processsed\n",
      "591000  examples processsed\n",
      "592000  examples processsed\n",
      "593000  examples processsed\n",
      "594000  examples processsed\n",
      "595000  examples processsed\n",
      "596000  examples processsed\n",
      "597000  examples processsed\n",
      "598000  examples processsed\n",
      "599000  examples processsed\n",
      "600000  examples processsed\n",
      "601000  examples processsed\n",
      "602000  examples processsed\n",
      "603000  examples processsed\n",
      "604000  examples processsed\n",
      "605000  examples processsed\n",
      "606000  examples processsed\n",
      "607000  examples processsed\n",
      "608000  examples processsed\n",
      "609000  examples processsed\n",
      "610000  examples processsed\n",
      "611000  examples processsed\n",
      "612000  examples processsed\n",
      "613000  examples processsed\n",
      "614000  examples processsed\n",
      "615000  examples processsed\n",
      "-----------------------------------\n",
      "#########  EXTRACTION OF DATE FEATURE ###### \n",
      "0  examples processsed\n",
      "1000  examples processsed\n",
      "2000  examples processsed\n",
      "3000  examples processsed\n",
      "4000  examples processsed\n",
      "5000  examples processsed\n",
      "6000  examples processsed\n",
      "7000  examples processsed\n",
      "8000  examples processsed\n",
      "9000  examples processsed\n",
      "10000  examples processsed\n",
      "11000  examples processsed\n",
      "12000  examples processsed\n",
      "13000  examples processsed\n",
      "14000  examples processsed\n",
      "15000  examples processsed\n",
      "16000  examples processsed\n",
      "17000  examples processsed\n",
      "18000  examples processsed\n",
      "19000  examples processsed\n",
      "20000  examples processsed\n",
      "21000  examples processsed\n",
      "22000  examples processsed\n",
      "23000  examples processsed\n",
      "24000  examples processsed\n",
      "25000  examples processsed\n",
      "26000  examples processsed\n",
      "27000  examples processsed\n",
      "28000  examples processsed\n",
      "29000  examples processsed\n",
      "30000  examples processsed\n",
      "31000  examples processsed\n",
      "32000  examples processsed\n",
      "33000  examples processsed\n",
      "34000  examples processsed\n",
      "35000  examples processsed\n",
      "36000  examples processsed\n",
      "37000  examples processsed\n",
      "38000  examples processsed\n",
      "39000  examples processsed\n",
      "40000  examples processsed\n",
      "41000  examples processsed\n",
      "42000  examples processsed\n",
      "43000  examples processsed\n",
      "44000  examples processsed\n",
      "45000  examples processsed\n",
      "46000  examples processsed\n",
      "47000  examples processsed\n",
      "48000  examples processsed\n",
      "49000  examples processsed\n",
      "50000  examples processsed\n",
      "51000  examples processsed\n",
      "52000  examples processsed\n",
      "53000  examples processsed\n",
      "54000  examples processsed\n",
      "55000  examples processsed\n",
      "56000  examples processsed\n",
      "57000  examples processsed\n",
      "58000  examples processsed\n",
      "59000  examples processsed\n",
      "60000  examples processsed\n",
      "61000  examples processsed\n",
      "62000  examples processsed\n",
      "63000  examples processsed\n",
      "64000  examples processsed\n",
      "65000  examples processsed\n",
      "66000  examples processsed\n",
      "67000  examples processsed\n",
      "68000  examples processsed\n",
      "69000  examples processsed\n",
      "70000  examples processsed\n",
      "71000  examples processsed\n",
      "72000  examples processsed\n",
      "73000  examples processsed\n",
      "74000  examples processsed\n",
      "75000  examples processsed\n",
      "76000  examples processsed\n",
      "77000  examples processsed\n",
      "78000  examples processsed\n",
      "79000  examples processsed\n",
      "80000  examples processsed\n",
      "81000  examples processsed\n",
      "82000  examples processsed\n",
      "83000  examples processsed\n",
      "84000  examples processsed\n",
      "85000  examples processsed\n",
      "86000  examples processsed\n",
      "87000  examples processsed\n",
      "88000  examples processsed\n",
      "89000  examples processsed\n",
      "90000  examples processsed\n",
      "91000  examples processsed\n",
      "92000  examples processsed\n",
      "93000  examples processsed\n",
      "94000  examples processsed\n",
      "95000  examples processsed\n",
      "96000  examples processsed\n",
      "97000  examples processsed\n",
      "98000  examples processsed\n",
      "99000  examples processsed\n",
      "100000  examples processsed\n",
      "101000  examples processsed\n",
      "102000  examples processsed\n",
      "103000  examples processsed\n",
      "104000  examples processsed\n",
      "105000  examples processsed\n",
      "106000  examples processsed\n",
      "107000  examples processsed\n",
      "108000  examples processsed\n",
      "109000  examples processsed\n",
      "110000  examples processsed\n",
      "111000  examples processsed\n",
      "112000  examples processsed\n",
      "113000  examples processsed\n",
      "114000  examples processsed\n",
      "115000  examples processsed\n",
      "116000  examples processsed\n",
      "117000  examples processsed\n",
      "118000  examples processsed\n",
      "119000  examples processsed\n",
      "120000  examples processsed\n",
      "121000  examples processsed\n",
      "122000  examples processsed\n",
      "123000  examples processsed\n",
      "124000  examples processsed\n",
      "125000  examples processsed\n",
      "126000  examples processsed\n",
      "127000  examples processsed\n",
      "128000  examples processsed\n",
      "129000  examples processsed\n",
      "130000  examples processsed\n",
      "131000  examples processsed\n",
      "132000  examples processsed\n",
      "133000  examples processsed\n",
      "134000  examples processsed\n",
      "135000  examples processsed\n",
      "136000  examples processsed\n",
      "137000  examples processsed\n",
      "138000  examples processsed\n",
      "139000  examples processsed\n",
      "140000  examples processsed\n",
      "141000  examples processsed\n",
      "142000  examples processsed\n",
      "143000  examples processsed\n",
      "144000  examples processsed\n",
      "145000  examples processsed\n",
      "146000  examples processsed\n",
      "147000  examples processsed\n",
      "148000  examples processsed\n",
      "149000  examples processsed\n",
      "150000  examples processsed\n",
      "151000  examples processsed\n",
      "152000  examples processsed\n",
      "153000  examples processsed\n",
      "154000  examples processsed\n",
      "155000  examples processsed\n",
      "156000  examples processsed\n",
      "157000  examples processsed\n",
      "158000  examples processsed\n",
      "159000  examples processsed\n",
      "160000  examples processsed\n",
      "161000  examples processsed\n",
      "162000  examples processsed\n",
      "163000  examples processsed\n",
      "164000  examples processsed\n",
      "165000  examples processsed\n",
      "166000  examples processsed\n",
      "167000  examples processsed\n",
      "168000  examples processsed\n",
      "169000  examples processsed\n",
      "170000  examples processsed\n",
      "171000  examples processsed\n",
      "172000  examples processsed\n",
      "173000  examples processsed\n",
      "174000  examples processsed\n",
      "175000  examples processsed\n",
      "176000  examples processsed\n",
      "177000  examples processsed\n",
      "178000  examples processsed\n",
      "179000  examples processsed\n",
      "180000  examples processsed\n",
      "181000  examples processsed\n",
      "182000  examples processsed\n",
      "183000  examples processsed\n",
      "184000  examples processsed\n",
      "185000  examples processsed\n",
      "186000  examples processsed\n",
      "187000  examples processsed\n",
      "188000  examples processsed\n",
      "189000  examples processsed\n",
      "190000  examples processsed\n",
      "191000  examples processsed\n",
      "192000  examples processsed\n",
      "193000  examples processsed\n",
      "194000  examples processsed\n",
      "195000  examples processsed\n",
      "196000  examples processsed\n",
      "197000  examples processsed\n",
      "198000  examples processsed\n",
      "199000  examples processsed\n",
      "200000  examples processsed\n",
      "201000  examples processsed\n",
      "202000  examples processsed\n",
      "203000  examples processsed\n",
      "204000  examples processsed\n",
      "205000  examples processsed\n",
      "206000  examples processsed\n",
      "207000  examples processsed\n",
      "208000  examples processsed\n",
      "209000  examples processsed\n",
      "210000  examples processsed\n",
      "211000  examples processsed\n",
      "212000  examples processsed\n",
      "213000  examples processsed\n",
      "214000  examples processsed\n",
      "215000  examples processsed\n",
      "216000  examples processsed\n",
      "217000  examples processsed\n",
      "218000  examples processsed\n",
      "219000  examples processsed\n",
      "220000  examples processsed\n",
      "221000  examples processsed\n",
      "222000  examples processsed\n",
      "223000  examples processsed\n",
      "224000  examples processsed\n",
      "225000  examples processsed\n",
      "226000  examples processsed\n",
      "227000  examples processsed\n",
      "228000  examples processsed\n",
      "229000  examples processsed\n",
      "230000  examples processsed\n",
      "231000  examples processsed\n",
      "232000  examples processsed\n",
      "233000  examples processsed\n",
      "234000  examples processsed\n",
      "235000  examples processsed\n",
      "236000  examples processsed\n",
      "237000  examples processsed\n",
      "238000  examples processsed\n",
      "239000  examples processsed\n",
      "240000  examples processsed\n",
      "241000  examples processsed\n",
      "242000  examples processsed\n",
      "243000  examples processsed\n",
      "244000  examples processsed\n",
      "245000  examples processsed\n",
      "246000  examples processsed\n",
      "247000  examples processsed\n",
      "248000  examples processsed\n",
      "249000  examples processsed\n",
      "250000  examples processsed\n",
      "251000  examples processsed\n",
      "252000  examples processsed\n",
      "253000  examples processsed\n",
      "254000  examples processsed\n",
      "255000  examples processsed\n",
      "256000  examples processsed\n",
      "257000  examples processsed\n",
      "258000  examples processsed\n",
      "259000  examples processsed\n",
      "260000  examples processsed\n",
      "261000  examples processsed\n",
      "262000  examples processsed\n",
      "263000  examples processsed\n",
      "264000  examples processsed\n",
      "265000  examples processsed\n",
      "266000  examples processsed\n",
      "267000  examples processsed\n",
      "268000  examples processsed\n",
      "269000  examples processsed\n",
      "270000  examples processsed\n",
      "271000  examples processsed\n",
      "272000  examples processsed\n",
      "273000  examples processsed\n",
      "274000  examples processsed\n",
      "275000  examples processsed\n",
      "276000  examples processsed\n",
      "277000  examples processsed\n",
      "278000  examples processsed\n",
      "279000  examples processsed\n",
      "280000  examples processsed\n",
      "281000  examples processsed\n",
      "282000  examples processsed\n",
      "283000  examples processsed\n",
      "284000  examples processsed\n",
      "285000  examples processsed\n",
      "286000  examples processsed\n",
      "287000  examples processsed\n",
      "288000  examples processsed\n",
      "289000  examples processsed\n",
      "290000  examples processsed\n",
      "291000  examples processsed\n",
      "292000  examples processsed\n",
      "293000  examples processsed\n",
      "294000  examples processsed\n",
      "295000  examples processsed\n",
      "296000  examples processsed\n",
      "297000  examples processsed\n",
      "298000  examples processsed\n",
      "299000  examples processsed\n",
      "300000  examples processsed\n",
      "301000  examples processsed\n",
      "302000  examples processsed\n",
      "303000  examples processsed\n",
      "304000  examples processsed\n",
      "305000  examples processsed\n",
      "306000  examples processsed\n",
      "307000  examples processsed\n",
      "308000  examples processsed\n",
      "309000  examples processsed\n",
      "310000  examples processsed\n",
      "311000  examples processsed\n",
      "312000  examples processsed\n",
      "313000  examples processsed\n",
      "314000  examples processsed\n",
      "315000  examples processsed\n",
      "316000  examples processsed\n",
      "317000  examples processsed\n",
      "318000  examples processsed\n",
      "319000  examples processsed\n",
      "320000  examples processsed\n",
      "321000  examples processsed\n",
      "322000  examples processsed\n",
      "323000  examples processsed\n",
      "324000  examples processsed\n",
      "325000  examples processsed\n",
      "326000  examples processsed\n",
      "327000  examples processsed\n",
      "328000  examples processsed\n",
      "329000  examples processsed\n",
      "330000  examples processsed\n",
      "331000  examples processsed\n",
      "332000  examples processsed\n",
      "333000  examples processsed\n",
      "334000  examples processsed\n",
      "335000  examples processsed\n",
      "336000  examples processsed\n",
      "337000  examples processsed\n",
      "338000  examples processsed\n",
      "339000  examples processsed\n",
      "340000  examples processsed\n",
      "341000  examples processsed\n",
      "342000  examples processsed\n",
      "343000  examples processsed\n",
      "344000  examples processsed\n",
      "345000  examples processsed\n",
      "346000  examples processsed\n",
      "347000  examples processsed\n",
      "348000  examples processsed\n",
      "349000  examples processsed\n",
      "350000  examples processsed\n",
      "351000  examples processsed\n",
      "352000  examples processsed\n",
      "353000  examples processsed\n",
      "354000  examples processsed\n",
      "355000  examples processsed\n",
      "356000  examples processsed\n",
      "357000  examples processsed\n",
      "358000  examples processsed\n",
      "359000  examples processsed\n",
      "360000  examples processsed\n",
      "361000  examples processsed\n",
      "362000  examples processsed\n",
      "363000  examples processsed\n",
      "364000  examples processsed\n",
      "365000  examples processsed\n",
      "366000  examples processsed\n",
      "367000  examples processsed\n",
      "368000  examples processsed\n",
      "369000  examples processsed\n",
      "370000  examples processsed\n",
      "371000  examples processsed\n",
      "372000  examples processsed\n",
      "373000  examples processsed\n",
      "374000  examples processsed\n",
      "375000  examples processsed\n",
      "376000  examples processsed\n",
      "377000  examples processsed\n",
      "378000  examples processsed\n",
      "379000  examples processsed\n",
      "380000  examples processsed\n",
      "381000  examples processsed\n",
      "382000  examples processsed\n",
      "383000  examples processsed\n",
      "384000  examples processsed\n",
      "385000  examples processsed\n",
      "386000  examples processsed\n",
      "387000  examples processsed\n",
      "388000  examples processsed\n",
      "389000  examples processsed\n",
      "390000  examples processsed\n",
      "391000  examples processsed\n",
      "392000  examples processsed\n",
      "393000  examples processsed\n",
      "394000  examples processsed\n",
      "395000  examples processsed\n",
      "396000  examples processsed\n",
      "397000  examples processsed\n",
      "398000  examples processsed\n",
      "399000  examples processsed\n",
      "400000  examples processsed\n",
      "401000  examples processsed\n",
      "402000  examples processsed\n",
      "403000  examples processsed\n",
      "404000  examples processsed\n",
      "405000  examples processsed\n",
      "406000  examples processsed\n",
      "407000  examples processsed\n",
      "408000  examples processsed\n",
      "409000  examples processsed\n",
      "410000  examples processsed\n",
      "411000  examples processsed\n",
      "412000  examples processsed\n",
      "413000  examples processsed\n",
      "414000  examples processsed\n",
      "415000  examples processsed\n",
      "416000  examples processsed\n",
      "417000  examples processsed\n",
      "418000  examples processsed\n",
      "419000  examples processsed\n",
      "420000  examples processsed\n",
      "421000  examples processsed\n",
      "422000  examples processsed\n",
      "423000  examples processsed\n",
      "424000  examples processsed\n",
      "425000  examples processsed\n",
      "426000  examples processsed\n",
      "427000  examples processsed\n",
      "428000  examples processsed\n",
      "429000  examples processsed\n",
      "430000  examples processsed\n",
      "431000  examples processsed\n",
      "432000  examples processsed\n",
      "433000  examples processsed\n",
      "434000  examples processsed\n",
      "435000  examples processsed\n",
      "436000  examples processsed\n",
      "437000  examples processsed\n",
      "438000  examples processsed\n",
      "439000  examples processsed\n",
      "440000  examples processsed\n",
      "441000  examples processsed\n",
      "442000  examples processsed\n",
      "443000  examples processsed\n",
      "444000  examples processsed\n",
      "445000  examples processsed\n",
      "446000  examples processsed\n",
      "447000  examples processsed\n",
      "448000  examples processsed\n",
      "449000  examples processsed\n",
      "450000  examples processsed\n",
      "451000  examples processsed\n",
      "452000  examples processsed\n",
      "453000  examples processsed\n",
      "454000  examples processsed\n",
      "455000  examples processsed\n",
      "456000  examples processsed\n",
      "457000  examples processsed\n",
      "458000  examples processsed\n",
      "459000  examples processsed\n",
      "460000  examples processsed\n",
      "461000  examples processsed\n",
      "462000  examples processsed\n",
      "463000  examples processsed\n",
      "464000  examples processsed\n",
      "465000  examples processsed\n",
      "466000  examples processsed\n",
      "467000  examples processsed\n",
      "468000  examples processsed\n",
      "469000  examples processsed\n",
      "470000  examples processsed\n",
      "471000  examples processsed\n",
      "472000  examples processsed\n",
      "473000  examples processsed\n",
      "474000  examples processsed\n",
      "475000  examples processsed\n",
      "476000  examples processsed\n",
      "477000  examples processsed\n",
      "478000  examples processsed\n",
      "479000  examples processsed\n",
      "480000  examples processsed\n",
      "481000  examples processsed\n",
      "482000  examples processsed\n",
      "483000  examples processsed\n",
      "484000  examples processsed\n",
      "485000  examples processsed\n",
      "486000  examples processsed\n",
      "487000  examples processsed\n",
      "488000  examples processsed\n",
      "489000  examples processsed\n",
      "490000  examples processsed\n",
      "491000  examples processsed\n",
      "492000  examples processsed\n",
      "493000  examples processsed\n",
      "494000  examples processsed\n",
      "495000  examples processsed\n",
      "496000  examples processsed\n",
      "497000  examples processsed\n",
      "498000  examples processsed\n",
      "499000  examples processsed\n",
      "500000  examples processsed\n",
      "501000  examples processsed\n",
      "502000  examples processsed\n",
      "503000  examples processsed\n",
      "504000  examples processsed\n",
      "505000  examples processsed\n",
      "506000  examples processsed\n",
      "507000  examples processsed\n",
      "508000  examples processsed\n",
      "509000  examples processsed\n",
      "510000  examples processsed\n",
      "511000  examples processsed\n",
      "512000  examples processsed\n",
      "513000  examples processsed\n",
      "514000  examples processsed\n",
      "515000  examples processsed\n",
      "516000  examples processsed\n",
      "517000  examples processsed\n",
      "518000  examples processsed\n",
      "519000  examples processsed\n",
      "520000  examples processsed\n",
      "521000  examples processsed\n",
      "522000  examples processsed\n",
      "523000  examples processsed\n",
      "524000  examples processsed\n",
      "525000  examples processsed\n",
      "526000  examples processsed\n",
      "527000  examples processsed\n",
      "528000  examples processsed\n",
      "529000  examples processsed\n",
      "530000  examples processsed\n",
      "531000  examples processsed\n",
      "532000  examples processsed\n",
      "533000  examples processsed\n",
      "534000  examples processsed\n",
      "535000  examples processsed\n",
      "536000  examples processsed\n",
      "537000  examples processsed\n",
      "538000  examples processsed\n",
      "539000  examples processsed\n",
      "540000  examples processsed\n",
      "541000  examples processsed\n",
      "542000  examples processsed\n",
      "543000  examples processsed\n",
      "544000  examples processsed\n",
      "545000  examples processsed\n",
      "546000  examples processsed\n",
      "547000  examples processsed\n",
      "548000  examples processsed\n",
      "549000  examples processsed\n",
      "550000  examples processsed\n",
      "551000  examples processsed\n",
      "552000  examples processsed\n",
      "553000  examples processsed\n",
      "554000  examples processsed\n",
      "555000  examples processsed\n",
      "556000  examples processsed\n",
      "557000  examples processsed\n",
      "558000  examples processsed\n",
      "559000  examples processsed\n",
      "560000  examples processsed\n",
      "561000  examples processsed\n",
      "562000  examples processsed\n",
      "563000  examples processsed\n",
      "564000  examples processsed\n",
      "565000  examples processsed\n",
      "566000  examples processsed\n",
      "567000  examples processsed\n",
      "568000  examples processsed\n",
      "569000  examples processsed\n",
      "570000  examples processsed\n",
      "571000  examples processsed\n",
      "572000  examples processsed\n",
      "573000  examples processsed\n",
      "574000  examples processsed\n",
      "575000  examples processsed\n",
      "576000  examples processsed\n",
      "577000  examples processsed\n",
      "578000  examples processsed\n",
      "579000  examples processsed\n",
      "580000  examples processsed\n",
      "581000  examples processsed\n",
      "582000  examples processsed\n",
      "583000  examples processsed\n",
      "584000  examples processsed\n",
      "585000  examples processsed\n",
      "586000  examples processsed\n",
      "587000  examples processsed\n",
      "588000  examples processsed\n",
      "589000  examples processsed\n",
      "590000  examples processsed\n",
      "591000  examples processsed\n",
      "592000  examples processsed\n",
      "593000  examples processsed\n",
      "594000  examples processsed\n",
      "595000  examples processsed\n",
      "596000  examples processsed\n",
      "597000  examples processsed\n",
      "598000  examples processsed\n",
      "599000  examples processsed\n",
      "600000  examples processsed\n",
      "601000  examples processsed\n",
      "602000  examples processsed\n",
      "603000  examples processsed\n",
      "604000  examples processsed\n",
      "605000  examples processsed\n",
      "606000  examples processsed\n",
      "607000  examples processsed\n",
      "608000  examples processsed\n",
      "609000  examples processsed\n",
      "610000  examples processsed\n",
      "611000  examples processsed\n",
      "612000  examples processsed\n",
      "613000  examples processsed\n",
      "614000  examples processsed\n",
      "615000  examples processsed\n",
      "-----------------------------------\n",
      "#########  SCALING FEATURES ###### \n",
      "-----------------------------------\n",
      "#########  CONVERT LABELS ###### \n",
      "-----------------------------------\n",
      "#########  CLASSIFICATION ###### \n",
      "Resultat final : \n",
      "0.933783258665\n"
     ]
    }
   ],
   "source": [
    "# DATA_ABS_TIT_AU = ABSTRACT + TITLE + AUTHOR (+journal) FOR TFIDF_MATRIX\n",
    "# LOAD FILES / TREATMENT\n",
    "testing_set,training_set,node_info = download_files()\n",
    "ID_pos, data,dates,cite0 = create_ID_pos(node_info) # init dictionaire des citations\n",
    "cited,citer = cite_cited2(training_set,cite0) # Sur 80,000 echant de training_set\n",
    "cite = cite_cited1(training_set,cite0)\n",
    "\n",
    "training_set, valid_ids = actualisation_set(training_set,1) #100%\n",
    "ID_pos, data_abs_tit_au,dates,cite0 = create_ID_pos4(node_info) # init dictionaire des citations\n",
    "\n",
    "# CREATE FEATURES\n",
    "tfidf_matrix_abs_tit_au = create_tfidf(data_abs_tit_au,my_preprocessor)\n",
    "\n",
    "cite_source,cite_target,similarity_= extract_autor_title_abstract_similarity6(training_set,citer,cited,ID_pos,node_info,tfidf_matrix_abs_tit_au)\n",
    "#cite_source,cite_target,similarity_abstract = extract_autor_title_abstract_similarity7(training_set,cite,ID_pos,node_info,tfidf_matrix_abs_tit_au)\n",
    "\n",
    "date_source, date_target,diff = extract_date(training_set,dates,ID_pos)\n",
    "\n",
    "# FEATURE SELECTION :\n",
    "#features = [similarity_,cite_source,cite_target,diff]\n",
    "features = [similarity_,cite_source,cite_target,date_source, date_target]\n",
    "\n",
    "training_features = scaling(features)\n",
    "labels_array = convert_labels(training_set)\n",
    "\n",
    "# CLASSIFICATION : \n",
    "classifier = classification(training_set,labels_array,training_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30776\n",
      "30776\n",
      "30776\n",
      "30776\n",
      "30776\n",
      "30776\n"
     ]
    }
   ],
   "source": [
    "# correct 'setting an array with a sequence' error which appears often when sizes are different\n",
    "print len(similarity_abstract)\n",
    "print len(overlap_title)\n",
    "print len(comm_auth)\n",
    "print len(citer_source)\n",
    "print len(cited_target)\n",
    "print len(diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evolution of results depending on classifier and features :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLP : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 88,15% on 5% of training_set ; features citer/cited/date_source/date_target : 83,31% on leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts and authors : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 94,81% on 5% of training_set ; features citer/cited/date_source/date_target : 86,23% on leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 94,89% sur 5% citer/cited/date_source/date_target : 86,52% sur leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p>92,51% sur 5% citer/cited/date_source/date_target : 92,38% sur leaderboard</p> with CITER/CITED on 600,000 samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p >89,03% sur 5% citer/cited/diff : 88,94% sur leaderboard</p> AVEC CITER/CITED SUR Training_set réduit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p>89,93% sur 5% citer/cited/diff : 89,76% sur leaderboard</p> AVEC CITER/CITED SUR 80,000 echant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p >90,83% sur 5% citer/cited/diff : 90,49% sur leaderboard</p> AVEC CITER/CITED SUR 600,000 echant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 88,4% sur 5% cite/date_source/date_target : 82,32% sur leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts, authors and journal : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p>92,32% sur 5% citer/cited/date_source/date_target : 92,51% sur leaderboard</p> AVEC CITER/CITED SUR 600,000 echant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p>90,51% sur 5% citer/cited/diff : - de 92,51% sur leaderboard</p> AVEC CITER/CITED SUR 600,000 echant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts, authors and title : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p> 92,58% sur 5% citer/cited/date_source/date_target : 92,52 Marche plus sur leaderboard</p> AVEC CITER/CITED SUR 600,000 echant predict_test2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts, authors, title and journal : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p> 93,13% sur 5% citer/cited/date_source/date_target : 92,97 sur leaderboard</p> AVEC CITER/CITED SUR 600,000 echant predict_test3.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p> 93,33% sur 50% citer/cited/date_source/date_target : 93,26 sur leaderboard </p> AVEC CITER/CITED SUR 600,000 echant ; temps calcul training : 8min env ; classif prend le plus de temps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <p style=' color:red;'> 93,37% on 100% of training_set ; fetaures: citer/cited/date_source/date_target : 93,37% on leaderboard </p> with CITER/CITED on 600,000 samples ; time of training : 13min ; classification take long time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 86% sur 5% citer/cited/date_source/date_target : 79% sur leaderboard\n",
    "- 88% sur 5% cite/date_source/date_target : 81% sur leaderboard\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts and authors : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 93% sur 5% citer/cited/date_source/date_target : 79% sur leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 87,8% sur 5% citer/cited/date_source/date_target : 80,74% sur leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVM : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts and authors : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 88,80% sur 5% citer/cited/date_source/date_target : 88,84% leaderboard AVEC CITER/CITED SUR 600,00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 90% avec citer/cited/date_source/date_target sur 5%\n",
    "- 88% avec cite/date_source/date_target sur 5%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With TFIDF_matrix composed of abstracts : "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 84,09% sur 5% citer/cited/date_source/date_target : 72,17% leaderboard AVEC CITER/CITED SUR 600,00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 83,87% sur 5% citer/cited/diff : 72,35% leaderboard AVEC CITER/CITED SUR 600,00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- citer/cited/date_source/date_target : 88.59% leaderboard \n",
    "- citer/cited/diff : 88.29% leaderboard \n",
    "- 92.59% sur 5% ; 94.2% sur 2% cite/date_source/date_target/diff : (?)% sur leaderboard \n",
    "- 90,23% sur 5% ; 92.2% sur 2% cite/date_source/date_target : 87% sur leaderboard\n",
    "- (?)90,23% sur 5% cite/date_source/date_target :  83.09% sur leaderboard<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary : <br/><br/>\n",
    "I started with a SVM classifier, I tested several combinaisons of features beetween cite or citer/cited and date_source/date_target or diff. I also tested differences between abstract+author or abstract+author+journal or abstract+author+title+journal in tfidf_matrix to compute similarity. I concluded that the more information I put in the better prediction is. And I concluded that the better features were citer/cited/date_source/date_target. <br/><br/>\n",
    "Then I tried RandomForest classifier but, even with different features, were worst than with the SVM one.<br/> <br/>\n",
    "So I started using MLP classifier. Results were at the moment much better.<br/>\n",
    "\n",
    "I also noticed that I limited the computation of citer/cited on the reduced training_set. In that case citer/cited contained not much information so I expended it to the whole dataset. The computation didn't take much time. Results were much better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8 - Testing Script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each node pair in the testing set, your model should predict whether there is an edge between the two nodes (1) or not (0). <br/>\n",
    "The testing set contains 50% of true edges (the ones that have been removed from the original network) and 50% of synthetic, wrong edges (pairs of randomly selected nodes between which there was no edge)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This script is used to save prediction of labels from a trained classifier. It save labels in a prediction_test.csv file. Itake into account that variables such as cite/Cited/citer/dates/ID_pos/tfidf_matrix were computed before in the training script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------\n",
      "#########  EXTRACTION OF AUTOR/TITLE/ABSTRACT/CITATION FEATURES ###### \n",
      "nombre d'echantillons à traiter :  32648\n",
      "0  examples processsed\n",
      "1000  examples processsed\n",
      "2000  examples processsed\n",
      "3000  examples processsed\n",
      "4000  examples processsed\n",
      "5000  examples processsed\n",
      "6000  examples processsed\n",
      "7000  examples processsed\n",
      "8000  examples processsed\n",
      "9000  examples processsed\n",
      "10000  examples processsed\n",
      "11000  examples processsed\n",
      "12000  examples processsed\n",
      "13000  examples processsed\n",
      "14000  examples processsed\n",
      "15000  examples processsed\n",
      "16000  examples processsed\n",
      "17000  examples processsed\n",
      "18000  examples processsed\n",
      "19000  examples processsed\n",
      "20000  examples processsed\n",
      "21000  examples processsed\n",
      "22000  examples processsed\n",
      "23000  examples processsed\n",
      "24000  examples processsed\n",
      "25000  examples processsed\n",
      "26000  examples processsed\n",
      "27000  examples processsed\n",
      "28000  examples processsed\n",
      "29000  examples processsed\n",
      "30000  examples processsed\n",
      "31000  examples processsed\n",
      "32000  examples processsed\n",
      "-----------------------------------\n",
      "#########  EXTRACTION OF DATE FEATURE ###### \n",
      "0  examples processsed\n",
      "1000  examples processsed\n",
      "2000  examples processsed\n",
      "3000  examples processsed\n",
      "4000  examples processsed\n",
      "5000  examples processsed\n",
      "6000  examples processsed\n",
      "7000  examples processsed\n",
      "8000  examples processsed\n",
      "9000  examples processsed\n",
      "10000  examples processsed\n",
      "11000  examples processsed\n",
      "12000  examples processsed\n",
      "13000  examples processsed\n",
      "14000  examples processsed\n",
      "15000  examples processsed\n",
      "16000  examples processsed\n",
      "17000  examples processsed\n",
      "18000  examples processsed\n",
      "19000  examples processsed\n",
      "20000  examples processsed\n",
      "21000  examples processsed\n",
      "22000  examples processsed\n",
      "23000  examples processsed\n",
      "24000  examples processsed\n",
      "25000  examples processsed\n",
      "26000  examples processsed\n",
      "27000  examples processsed\n",
      "28000  examples processsed\n",
      "29000  examples processsed\n",
      "30000  examples processsed\n",
      "31000  examples processsed\n",
      "32000  examples processsed\n",
      "-----------------------------------\n",
      "#########  SCALING FEATURES ###### \n",
      "##### PREDICTION TERMINATED ########\n",
      "taille de testing_set :  32648  taille de labels_predicted :  32648\n",
      "##### PREDICTION SAVED ########\n"
     ]
    }
   ],
   "source": [
    "# LOAD FILES, CREATE TFIDF_MATRIX, CITE, CITED, CITER, DATES MADE DURING TRAINING BLOCK\n",
    "\n",
    "#overlap_title_test,comm_auth_test,citer_source_test,cited_target_test,similarity_abstract_test = extract_autor_title_abstract_similarity2(testing_set,citer,cited,ID_pos,node_info,tfidf_matrix)\n",
    "#overlap_title_test,comm_auth_test,citer_source_test,cited_target_test,similarity_abstract_test = extract_autor_title_abstract_similarity3(testing_set,cite,ID_pos,node_info,tfidf_matrix)\n",
    "\n",
    "#comm_auth_test,citer_source_test,cited_target_test,similarity_abstract_test = extract_autor_title_abstract_similarity5(testing_set,cite,ID_pos,node_info,tfidf_matrix_abs_tit)\n",
    "#comm_auth_test,cite_source_test,cite_target_test,similarity_abstract_test = extract_autor_title_abstract_similarity4(testing_set,citer,cited,ID_pos,node_info,tfidf_matrix_abs_tit)\n",
    "\n",
    "citer_source_test,cited_target_test,similarity_abstract_test= extract_autor_title_abstract_similarity6(testing_set,citer,cited,ID_pos,node_info,tfidf_matrix_abs_tit_au)\n",
    "#citer_source_test,cited_target_test,similarity_abstract_test = extract_autor_title_abstract_similarity7(testing_set,cite,ID_pos,node_info,tfidf_matrix_abs_tit_au)\n",
    "\n",
    "\n",
    "date_source_test, date_target_test,diff_test = extract_date(testing_set,dates,ID_pos)\n",
    "\n",
    "# FEATURE SELECTION :\n",
    "#features_test = [overlap_title_test,similarity_abstract_test,comm_auth_test,cite_source_test,cite_target_test,date_source_test, date_target_test]\n",
    "#features_test = [similarity_abstract_test,comm_auth_test,cite_source_test,cite_target_test,date_source_test, date_target_test]\n",
    "features_test = [similarity_abstract_test,citer_source_test,cited_target_test,date_source_test, date_target_test]\n",
    "\n",
    "testing_features = scaling(features_test)\n",
    "\n",
    "# CLASSIFICATION\n",
    "labels_predicted = classifier.predict(testing_features)\n",
    "print \"##### PREDICTION TERMINATED ########\"\n",
    "print \"taille de testing_set : \",len(testing_set),\" taille de labels_predicted : \",len(labels_predicted)\n",
    "# SAUVEGARDE DE LABELS_PREDICTED\n",
    "with open('predictions_test.csv', 'wb') as csvfile:\n",
    "    spamwriter = csv.writer(csvfile)\n",
    "    spamwriter.writerow(['Id','prediction'])\n",
    "    for i in range(len(labels_predicted)):\n",
    "        spamwriter.writerow([i,int(labels_predicted[i])])\n",
    "print \"##### PREDICTION SAVED ########\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32648\n",
      "32648\n",
      "32648\n",
      "32648\n",
      "30776\n"
     ]
    }
   ],
   "source": [
    "# correct 'setting an array with a sequence' error\n",
    "print len(similarity_abstract_test)\n",
    "#print len(overlap_title)\n",
    "print len(comm_auth_test)\n",
    "print len(citer_source_test)\n",
    "print len(cited_target_test)\n",
    "print len(diff)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}