examples/quickstart/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Qucik Start: Use Cases and Examples\n",
    "---\n",
    "In this notebook, we show an example of conducting an offline evaluation of the performance of BernoulliTS using *Inverse Probability Weighting* as an OPE estimator and the Random policy as a behavior policy.\n",
    "\n",
    "Our example contains the follwoing three major steps:\n",
    "- (1) Data loading and preprocessing\n",
    "- (2) Offline Bandit Simulation\n",
    "- (3) Off-Policy Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# import open bandit pipeline (obp)\n",
    "from obp.dataset import OpenBanditDataset\n",
    "from obp.policy import BernoulliTS, Random\n",
    "from obp.simulator import OfflineBanditSimulator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (1) Data loading and preprocessing\n",
    "\n",
    "We prepare easy-to-use data loader for Open Bandit Dataset, **OpenBanditDataset** class in `dataset` module. <br>\n",
    "It takes `behavior policy ('bts' or 'random')` and `campaign ('all', 'men', or 'women')` as inputs and provides dataset preprocessing as well as standardized train/test splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "{'n_data': 7000,\n 'n_actions': 34,\n 'action': array([22, 10,  7, ..., 12, 14, 12]),\n 'position': array([0, 1, 2, ..., 0, 1, 0]),\n 'reward': array([0, 0, 0, ..., 0, 0, 0]),\n 'pscore': array([0.02941176, 0.02941176, 0.02941176, ..., 0.02941176, 0.02941176,\n        0.02941176]),\n 'X_policy': array([[0, 1, 0, ..., 0, 1, 0],\n        [1, 0, 1, ..., 0, 0, 0],\n        [0, 1, 0, ..., 0, 0, 0],\n        ...,\n        [0, 1, 0, ..., 1, 0, 0],\n        [0, 1, 0, ..., 1, 0, 0],\n        [0, 1, 0, ..., 0, 0, 0]], dtype=uint8),\n 'X_reg': array([[ 0.        ,  2.        ,  0.        , ..., 11.        ,\n          3.        , -0.69874138],\n        [ 1.        ,  1.        ,  1.        , ...,  3.        ,\n          0.        ,  1.5864346 ],\n        [ 2.        ,  2.        ,  0.        , ...,  3.        ,\n          0.        ,  2.85837217],\n        ...,\n        [ 0.        ,  2.        ,  4.        , ...,  1.        ,\n          0.        ,  1.19838585],\n        [ 1.        ,  2.        ,  4.        , ..., 12.        ,\n          3.        , -1.00055707],\n        [ 0.        ,  2.        ,  0.        , ...,  1.        ,\n          0.        ,  1.19838585]]),\n 'X_user': array([[2, 0, 2, 6],\n        [1, 1, 2, 0],\n        [2, 0, 2, 0],\n        ...,\n        [2, 4, 4, 5],\n        [2, 4, 7, 5],\n        [2, 0, 1, 4]])}"
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "source": [
    "# (1) Data loading and preprocessing\n",
    "# specify the path of sample dataset\n",
    "data_path = Path('.').resolve().parents[1] / 'obd'\n",
    "# Load and preprocess raw data in \"All\" campaign collected by the Random policy\n",
    "dataset = OpenBanditDataset(behavior_policy='random', campaign='men', data_path=data_path)\n",
    "# Split the data into 70% training and 30% test sets\n",
    "train, test = dataset.split_data(test_size=0.3, random_state=42)\n",
    "\n",
    "# `train` is a dictionary storing action, position, reward, pscore, and some feature vectors\n",
    "train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2) Offline Bandit Simulation\n",
    "\n",
    "After preparing a dataset, we now run an **offline bandit simulation** on the logged bandit feedback. <br>\n",
    "Below, we use **OfflineBanditSimulator** class in `simulator` module. <br>\n",
    "We also use **Bernoulli Thompsom Sampling (Bernoulli TS)** as a counterfacutal policy to be evaluated, which is impelemted in `policy` module. <br>\n",
    "The `simulate` method of the Simulator class takes the counterfactual policy and the training dataset as inputs and run the policy on the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": "WARNING:root:regression model is not given, and thus model dependent estimators\n                such as DM and DR cannot be used.\n100%|██████████| 7000/7000 [00:00<00:00, 26096.62it/s]\n"
    }
   ],
   "source": [
    "# (2) Offline Bandit Simulation\n",
    "# Define a simulator object\n",
    "simulator = OfflineBanditSimulator(train=train)\n",
    "# Define a counterfacutal policy, which is the Bernoulli TS policy here\n",
    "counterfactual_policy = BernoulliTS(\n",
    "  n_actions=dataset.n_actions,\n",
    "  len_list=dataset.len_list,\n",
    "  random_state=42)\n",
    "# Run an offline bandit simulation on the training set\n",
    "simulator.simulate(policy=counterfactual_policy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (3) Off-Policy Evaluation (OPE)\n",
    "\n",
    "Our final step is **off-policy evaluation** (OPE), which attempts to estimate the performance of bandit algorithms using log data generated by offline bandit simulations. <br>\n",
    "Here we use the *InverseProbabilityWeighting* estimator as an OPE estimator and estiamte the performance of Bernoulli TS using the simulation log data. <br>\n",
    "Finally, we compare the estimated performance of Bernoulli TS with the ground-truth performance of the behavior policy, which is the Random policy here. <br>\n",
    "The ground-truth performance can easily be estimated with the empirical mean of factual rewads in the `test` set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "1.2142857142857142\n"
    }
   ],
   "source": [
    "# Estimate the policy value of BernoulliTS based on actions selected by that policy\n",
    "estimated_policy_value = simulator.inverse_probability_weighting()\n",
    "\n",
    "# Comapre the estimated performance of BernoulliTS (counterfactual policy)\n",
    "# with the ground-truth performance of Random (behavior policy)\n",
    "relative_policy_value_of_bernoulli_ts = estimated_policy_value / test['reward'].mean()\n",
    "# Our OPE procedure estimates that BernoulliTS improves Random by 21.4%\n",
    "print(relative_policy_value_of_bernoulli_ts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python37364bitpyenvpyenv99df861ad12e4e8792812ed413ad34d2",
   "display_name": "Python 3.7.3 64-bit ('.pyenv': pyenv)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}