asgn01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "ff790970842b144adf6d9cba4fc24da2",
     "grade": false,
     "grade_id": "cell-29af7c6b95714435",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Document Analysis Assignment 1: Information Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "abade6bdab3a02908c43d1f0a84151a5",
     "grade": false,
     "grade_id": "cell-47305e931ef5437d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "In this assignment, your task is to index a new document collection into *Elasticsearch* and measure search performance based on predefined queries. \n",
    "A set of new documents collection containing more than 10000 goverment sites description and a set of predefined queries will be provided for this assignment.\n",
    "Throughout this assginment, \n",
    "1. you will get better understanding of indexer including tokeniser, parser, and normalisers to improve the search performance given a predefined evaluation metric, \n",
    "2. you will get better understanding of search algorithm to obtain better search results, and \n",
    "3. you will find the best way to combine indexer and search algorithm to maximise the performance.\n",
    "\n",
    "Below, you will solve five programming assignments (Q1-Q5), and three written assignments after that. We will check the correctness of your code, but the score of the programming assignments will be graded based on your performance on Kaggle competition.\n",
    "- Write your code after `### Your code here`, and remove `raise NotImplementedError` after implementation.\n",
    "- Written assignments should be written in the given notebook cells. Please write them direcly in to the designated cells, and upload the notebook file to Wattle page.\n",
    "- Write answers in this notebook file, and upload the file to Wattle submission site. **Please rename and submit jupyter notebook file (`Assignment1.ipynb`) to `your_uid.ipynb` (e.g. `u6000001.ipynb`) with your written answers therein**. Do not upload any other files to Wattle except this notebook file.\n",
    "\n",
    "For the Kaggle competition:\n",
    "- Note that you are only allowed to upload **3 copies** of your results to Kaggle per day. Make every upload count, and don't waste your opportunities!\n",
    "\n",
    "Score distribution (total 10 points)\n",
    "- Kaggle competition (Q1-Q5): 4 points\n",
    "- Written assignment 1: 2 points\n",
    "- Written assignment 2: 2 points\n",
    "- Written assignment 3: 2 points\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "a365e56c45d35686c209865f6d5db2bd",
     "grade": false,
     "grade_id": "cell-4eb6f686aa493e2f",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Coding assignment (Q1 - Q5), 4 points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "4674684480b8ea208284f1610f9b9b50",
     "grade": false,
     "grade_id": "cell-37e4a49ffb49455e",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Q1. Index Gov dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "30cf82081447ba18b53aeca01a460789",
     "grade": false,
     "grade_id": "cell-1f5bd984facb5788",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Download a collection of government documents `gov.zip` from Wattle, and unzip the file. The unzipped file contains two sub-folders; `documents`, and `topics`. `documents` folder consists of sub-folders, each of which contains multiple documents. Your first job is to index the documents as we have done in the lab exercises.\n",
    "\n",
    "- Note that depending on your machine, indexing may take several minutes to a few hours. You may implement multi-threaded version of indexing to mitigate the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "578ed1cbb1af0ba2105f00224058934e",
     "grade": false,
     "grade_id": "cell-5e6d4a6f55f250ef",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Below we provide the basic configuration for indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "c1a0606aa7784018abc6977bb4fa4ff0",
     "grade": false,
     "grade_id": "cell-cef73deb3dfdc0dc",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# basic configuration for indexing\n",
    "basic_settings = {\n",
    "  \"mappings\": {\n",
    "    \"doc\": {\n",
    "      \"properties\": {\n",
    "        \"filename\": {\n",
    "          \"type\": \"keyword\",\n",
    "          \"index\": False,\n",
    "        },\n",
    "        \"path\": {\n",
    "          \"type\": \"keyword\",\n",
    "          \"index\": False,\n",
    "        },\n",
    "        \"text\": {\n",
    "          \"type\": \"text\",\n",
    "          \"similarity\": \"boolean\",\n",
    "          \"analyzer\": \"my_analyzer\",\n",
    "          \"search_analyzer\": \"my_analyzer\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"settings\": {\n",
    "    \"analysis\": {\n",
    "      \"analyzer\": {\n",
    "        \"my_analyzer\": {\n",
    "          \"filter\": [\n",
    "            \"stop\"\n",
    "          ],\n",
    "          \"char_filter\": [\n",
    "            \"html_strip\"\n",
    "          ],\n",
    "          \"type\": \"custom\",\n",
    "          \"tokenizer\": \"whitespace\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }   \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "4344c89893e1b1c6fbcc6fe74629beb7",
     "grade": false,
     "grade_id": "cell-0dc69ad1d421eb01",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "You need to implement function `build_gov_index` below. Don't forget to remove `raise NotImplementedError` after implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "5cb5236e66797287871e49e28e7c8c8a",
     "grade": true,
     "grade_id": "cell-2ad53a8c14acab88",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "No module named elasticsearch",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-f5a422a1fcc2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0melasticsearch\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mElasticsearch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mES_HOSTS\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'http://localhost:9200'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mDOCS_PATH\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'gov/documents'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mINDEX_NAME\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'gov'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mImportError\u001b[0m: No module named elasticsearch"
     ]
    }
   ],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "\n",
    "ES_HOSTS = ['http://localhost:9200']\n",
    "DOCS_PATH = 'gov/documents'\n",
    "INDEX_NAME = 'gov'\n",
    "DOC_TYPE = 'doc'\n",
    "\n",
    "def build_gov_index(es_conn, index_name, doc_path, settings):\n",
    "    # TODO implement function that \n",
    "    # 1. create index with `index_name`, if `index_name` already exists, remove the index first.\n",
    "    # 2. index the documents under doc_path including subfolders into elasticsearch (hint: read demo carefully)\n",
    "    # Note that this function will be used throughout this assignment    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "\n",
    "es_conn = Elasticsearch(ES_HOSTS)\n",
    "build_gov_index(es_conn, INDEX_NAME, DOCS_PATH, basic_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "fc2430fad40e9648f288c10549f1a9ed",
     "grade": false,
     "grade_id": "cell-fc12976b47c62f6a",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Q2. Search and measure performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "aac9d7d67b22a29b28a10e12f41db348",
     "grade": false,
     "grade_id": "cell-f24c11e452feb323",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "For the second task, you first need to read `topics/gov.topics` file. As we have done in lab demo session, each file of this file is formatted as\n",
    "\n",
    "`query_id query_terms`\n",
    "\n",
    "`query_id` is a numerical number, and `query_terms` consists of multiple keywords as search terms. Your job is to read the query file and search using the provided `search` function. You need to write the outputs of search results to `output.csv` file. The first line of `output.csv` file should start with the following header:\n",
    "\n",
    "`QueryId,DocumentNumber,Similarity,Iteration,RunId,Rank`, \n",
    "\n",
    "but you only need to fill QueryId and DocumentNumber for this assignment. Specifically, for each query, you will\n",
    "\n",
    "1. rank the output of search results based on their scores,\n",
    "2. write top-10 documents to `output.csv` based on the highest score. Except `QueryID` and `DocumentNumber` the other fields should be filled with 0s. For example, each line of the output file will be formatted as:\n",
    "    \n",
    "    `1,G00-27-1804490,0,0,0,0`\n",
    "    \n",
    "    where `1` is `query_id`, and `G00-27-1804490` is the file name of retrived document (=`DocumentNumber`).\n",
    "    \n",
    "    White space is now allowe between fields, i.e. `1, G00-27-1804490, 0, 0, 0, 0` will not be evaluated properly. \n",
    "    - **Note that you are only allowed to write 10-documents at most for each query**. If your output file contains more than 10 documents per query, **you will get 0 score for the programming assignment**.\n",
    "    \n",
    "3. upload `output.csv` file to [kaggle competition site](#Upload-output-file-to-Kaggle-competition-site) (see the details below). Check the performance of the basic search algorithm in terms of Precision@10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "d103a5c1fc0c2538b41031c8d9ebc569",
     "grade": true,
     "grade_id": "cell-76dc8b9da4ebc7c2",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def search(query_string, es_conn, index_name):\n",
    "    '''\n",
    "        searches for query_string with default search algorithm\n",
    "        input:\n",
    "            - query_string: a query\n",
    "            - es_conn: elasticsearch connection\n",
    "            - index_name: name of index\n",
    "        output:\n",
    "            - a generator of tuple (filename, score)\n",
    "\n",
    "    '''\n",
    "    res = es_conn.search(index = index_name,\n",
    "        body = {\n",
    "            \"_source\": [ \"filename\"],\n",
    "            \"query\": {\n",
    "                \"query_string\": {\n",
    "                    \"query\": query_string,\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    )\n",
    "    return res['hits']['hits']\n",
    "\n",
    "# TODO: read query file from `query_path`, search using `search_fn`,\n",
    "#       and write top 10 outputs per query to `output_file`\n",
    "#       Note that the function takes a search function as an argument, you can directly call the search function\n",
    "#          as `result = search_fn(query_string, es_conn, index_name)` within the function.\n",
    "#       This function will be used throughout this assignment\n",
    "def read_search_write_output(search_fn, query_path, output_file):\n",
    "    with open(output_file, 'w') as output:\n",
    "        output.write('QueryId,DocumentNumber,Similarity,Iteration,RunId,Rank\\n')  #for your convenience\n",
    "\n",
    "        # YOUR CODE HERE\n",
    "        raise NotImplementedError()\n",
    "\n",
    "query_path = 'topics/gov.topics'\n",
    "output_file = 'output.csv'\n",
    "read_search_write_output(search, query_path, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "d3bd9b6e33dd6e1585698f05256b8e23",
     "grade": false,
     "grade_id": "cell-c66326adeb8c9a53",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Q3. Improve indexer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "1df59eeb9fc0a012f54925c4797a3699",
     "grade": false,
     "grade_id": "cell-9b36a3646fd7bd72",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "You will be asked to change the configuration of indexer (`basic_settings`) to improve the search performance.\n",
    "\n",
    "Please look at the elastic search official document [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis.html) for better understanding of configuration and other options.\n",
    "\n",
    "Note that you can check how your tokeniser tokenises your input string via `analyze_query` function provided in the demo code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "fdcfc7c7fce31f8623f29aaa8fd239c0",
     "grade": true,
     "grade_id": "cell-884cf4f296f09710",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# TODO: configure settings to define your own analyzer for indexing\n",
    "q3_settings = {\n",
    "  \"mappings\": {\n",
    "    \"doc\": {\n",
    "      \"properties\": {\n",
    "        \"filename\": {\n",
    "          \"type\": \"keyword\",\n",
    "          \"index\": False,\n",
    "        },\n",
    "        \"path\": {\n",
    "          \"type\": \"keyword\",\n",
    "          \"index\": False,\n",
    "        },\n",
    "        \"text\": {\n",
    "          \"type\": \"text\",\n",
    "          \"similarity\": \"boolean\",\n",
    "          \"analyzer\": \"my_analyzer\",\n",
    "          \"search_analyzer\": \"my_analyzer\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"settings\": {\n",
    "    \"analysis\": {\n",
    "      \"analyzer\": {\n",
    "          \"my_analyzer\": {\n",
    "            # YOUR CODE HERE\n",
    "            raise NotImplementedError()\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "ff79ec9422b088b1e2d8b96844e79fd4",
     "grade": false,
     "grade_id": "cell-942792d0eb35960a",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TODO: run this block to generate an output based on q3_settings defined above.\n",
    "build_gov_index(es_conn, INDEX_NAME, DOCS_PATH, q3_settings)\n",
    "read_search_write_output(search, query_path, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "ef47fa609296ea03356a244b7ba0691a",
     "grade": false,
     "grade_id": "cell-df229524d91669a3",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Upload the final output to Kaggle to check the difference in Precision@10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "8dd91df1f4b66856a1707b4561b439cf",
     "grade": false,
     "grade_id": "cell-0661220c4435ba52",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Q4. Improve search algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "86877e836d76b8e5b24c1eb5cbf54ae0",
     "grade": false,
     "grade_id": "cell-80c53442f59e8cea",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "*Elasticsearch* also provides multiple configurable scoring algorithms. For this task, you will be asked to find a better similarity module to improve the search performance. Please refer [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/similarity.html) for better understanding of configurable elasticsearch similarity modules.\n",
    "\n",
    "You can also change the `search` function to improve performance. Please refer [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html) for better understanding of Query DSL used in *elasticsearch*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "de2a7de410381a20a9658b107cd33e99",
     "grade": true,
     "grade_id": "cell-5533d04f371dadea",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# TODO: define your own analyzer for indexing and searching\n",
    "q4_settings = {\n",
    "  \"mappings\": {\n",
    "    \"doc\": {\n",
    "      \"properties\": {\n",
    "        \"filename\": {\n",
    "          \"type\": \"keyword\",\n",
    "          \"index\": False,\n",
    "        },\n",
    "        \"path\": {\n",
    "          \"type\": \"keyword\",\n",
    "          \"index\": False,\n",
    "        },\n",
    "        \"text\": {\n",
    "            # YOUR CODE HERE\n",
    "            raise NotImplementedError()\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"settings\": {\n",
    "    \"analysis\": {\n",
    "      \"analyzer\": {\n",
    "            # YOUR CODE HERE\n",
    "            raise NotImplementedError()\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "# TODO: change search algorithm to improve the search results, the return type should be the same as that of `search` function\n",
    "def my_search(query_string, es_conn, index_name):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "16db429fd1208e1bf50ca9a00a400b17",
     "grade": false,
     "grade_id": "cell-0a5e21039c59945f",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TODO: run this block to generate an output based on q4_settings and my_search defined above.\n",
    "build_gov_index(es_conn, INDEX_NAME, DOCS_PATH, q4_settings)\n",
    "read_search_write_output(my_search, query_path, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "a8ed8fc938a4880ff5bad6abad2839e0",
     "grade": false,
     "grade_id": "cell-d03c7164f0db2d79",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Q5. Find the best combination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "c9ac0f956006573f1729419cb5824414",
     "grade": false,
     "grade_id": "cell-9da5cf70072f7baf",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Now it's time to explorer the best configuration of indexer and search algorithms. Each combination will yield a different search outcome. Try different combinations and report best results below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "2b042473ea09d65fdff546dc279ecf1d",
     "grade": true,
     "grade_id": "cell-4dc16abdf8d3131b",
     "locked": false,
     "points": 0,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# TODO: find the best combination of indexer configuration and search algorithm to maximise the performance of search result.\n",
    "best_settings = {\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "}\n",
    "\n",
    "def best_search(query_string, es_conn, index_name):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "cfda5ab63489f562294ff8fba1907dcb",
     "grade": false,
     "grade_id": "cell-1f9fa6f5bf378d0a",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TODO: run this block to generate the output\n",
    "build_gov_index(es_conn, INDEX_NAME, DOCS_PATH, best_settings)\n",
    "read_search_write_output(my_search, query_path, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "a127c8b9b0d15cb21a615f889290bdd1",
     "grade": false,
     "grade_id": "cell-4abac591c7ef7d4a",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Answer following questions based on your implementation from Q1 to Q5.\n",
    "\n",
    "# Written Assignment 1 (2pt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "584c2dae13eaf4a48c903b6aa31a68d5",
     "grade": false,
     "grade_id": "cell-7e9bfe1b4b37ced9",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "What changes did you make for the **indexer** to improve the performance of the system? Why do you think it improves the performance?\n",
    "\n",
    "(provide answer using bullet list with 2~3 items (Check [this](https://sourceforge.net/p/jupiter/wiki/markdown_syntax/#md_ex_lists) if you are not familiar with markdown syntax))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "8a5f441f9d6f0226f883d6dcee3095fe",
     "grade": true,
     "grade_id": "cell-0ff68bc511be791d",
     "locked": false,
     "points": 2,
     "schema_version": 1,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "9b054534102f5bd8c924bd69ad631c62",
     "grade": false,
     "grade_id": "cell-c83048719fa97fc7",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Written Assignment 2  (2pt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "843d24bad4cd528299f6039268f568e3",
     "grade": false,
     "grade_id": "cell-61f3446826c211ef",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "What changes did you make for the **search algorithm** to improve the performance of the system? Why do you think it improves the performance?\n",
    "\n",
    "(provide answer using bullet list with 2~3 items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "840de108e8c95bebf0ae029b1121c3bc",
     "grade": true,
     "grade_id": "cell-1f6c7740fd8a8aca",
     "locked": false,
     "points": 2,
     "schema_version": 1,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "be9dfdacdc8b3c3fb7ba7540f2710d9d",
     "grade": false,
     "grade_id": "cell-3b168c5fa7737d37",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Written Assignment 3 (2pt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "7374fa7271265a61538acc6f1f6abca6",
     "grade": false,
     "grade_id": "cell-a3d94aebec568d16",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "In the Kaggle competition, we only use Precision@10 as an evaluation metric. What other metrics can be used to measure the performance of IR system for the government document collection? Provide two metrics and explain why.\n",
    "\n",
    "(provide answer using bullet list with 2~3 items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "0cbf0931584afb2dcd4c9f9e95ecacea",
     "grade": true,
     "grade_id": "cell-af0ec75627f339fe",
     "locked": false,
     "points": 2,
     "schema_version": 1,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "5e446199b1e1bacba30aa6a2451bce72",
     "grade": false,
     "grade_id": "cell-9ca7c23c29ce4944",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Upload output file to Kaggle competition site"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "3a76cb2385760b6b925c0754fe2c8c17",
     "grade": false,
     "grade_id": "cell-fed539f3cce80673",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Once you generate `output.csv` file, you can upload your result on Kaggle competition site. To upload and evaluate your result\n",
    "\n",
    "1. Go to Kaggle competition site: [Click here](https://www.kaggle.com/t/26d21af1d3ea4447b86737fe889160ff).\n",
    "1. Sign up for Kaggle if you do not have an account. Go back to the [original kaggle page](https://www.kaggle.com/t/26d21af1d3ea4447b86737fe889160ff).\n",
    "1. Before submitting the result, first go to `team` manu and change **your team name as your university id**.\n",
    "![ChangeUID](images/changeuid.png)\n",
    "1. Time to submit your own result. Click `submit predictions` in the manu, you may need to agree the competetion rules before submitting your result.\n",
    "1. Upload your output csv file, you can write additional description of your submission in the description box.\n",
    "    Note that you are only allowed to submit **3 results per day**. Do not upload an arbitrary result and think which algorithm or parser will perform the best.\n",
    "1. If your output format is correct, the system will generate your score automatically.\n",
    "1. Go to `Leaderboard` menu. The leaderboard will show the current score of the other students.\n",
    "![Leaderboard](images/leaderboard.png)\n",
    "\n",
    "\n",
    "Note that you can check all of your submission from `my submission` menu. Please select one best performing submission before the assignment due. The selected submission will be used to measure the performance of *hidden* test case (see below for details).\n",
    "![Check](images/check.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "89e6c1b923777e9df37d3e5e3c538f8a",
     "grade": false,
     "grade_id": "cell-7614e05f0b4dac58",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "checksum": "2852643251d989cace90bb5a1509837b",
     "grade": false,
     "grade_id": "cell-48142ad1c5e9cb85",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "The system uses *precision@10* to measure the performance of each submission, which basically measure the accuracy of your top 10 returned documents.\n",
    "\n",
    "It is important to understand that the leaderboard score will be only computed based on the half of the test cases, and the remaining half will be computed after the deadline based on your selected submission. This process will ensure that your performance is not only applicable for the known test cases, but also generalised to the unknown test cases. We will combine these two performances to score the first assignment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}