notebooks/Defining-a-POMDP-with-the-Explicit-Interface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining a POMDP with the Explicit Interface\n",
    "\n",
    "In this tutorial we will define a version of the Tiger POMDP example [1] with the explicit interface of POMDPs.jl. To find out more about the explicit interface, please see this section of the POMDPs.jl documentation: [Explicit POMDP Interface](https://juliapomdp.github.io/POMDPs.jl/latest/explicit).\n",
    "\n",
    "**Note:** This tutorial assumes familiarity with object-oriented programming in Julia. If you have a small to medium sized problem, you may find the [QuickPOMDPs interface](https://github.com/JuliaPOMDP/QuickPOMDPs.jl) easier to get started with. It requires no object oriented code.\n",
    "\n",
    "[1] L. Pack Kaelbling, M. L. Littman, A. R. Cassandra, \"Planning and Action in Partially Observable Domain\", *Artificial Intelligence*, 1998."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "using POMDPs\n",
    "using POMDPModelTools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model type\n",
    "\n",
    "In the tiger POMDP, the agent is tasked with escaping from a room. There are two doors leading out of the room. Behind one of the doors is a tiger, and behind the other is sweet, sweet freedom. If the agent opens the door and finds the tiger, it gets eaten (and receives a reward of -100). If the agent opens the other door, it escapes and receives a reward of 10. The agent can also listen. Listening gives a noisy measurement of which door the tiger is hiding behind. Listening gives the agent the correct location of the tiger 85% of the time. The agent receives a reward of -1 for listening.\n",
    "\n",
    "The POMDP model of the problem is as follows:\n",
    "- The *state* is a Boolean value representing whether the tiger is on the left door (true) or on the right door (false)\n",
    "- The *action* is a Symbol indicating listening, opening the left door or opening the right door\n",
    "- The *observation* is a Boolean indicating whether the agent hears the tiger on the left (true) or on the right (false)\n",
    "- At each step there is a probability of hearing the tiger at the right location \n",
    "- Once the agent opens a door, it receives a reward and restarts at the initial state.\n",
    "\n",
    "The `TigerPOMDP` model type is a subtype of the `POMDP`. The `POMDP` abstract type is parameterized by the types used to represent the state, action, and observation respectively.\n",
    "The type also includes fields that will be used in the reward and transitions definitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TigerPOMDP"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "struct TigerPOMDP <: POMDP{Bool, Symbol, Bool} # POMDP{State, Action, Observation}\n",
    "    r_listen::Float64 # reward for listening (default -1)\n",
    "    r_findtiger::Float64 # reward for finding the tiger (default -100)\n",
    "    r_escapetiger::Float64 # reward for escaping (default 10)\n",
    "    p_listen_correctly::Float64 # prob of correctly listening (default 0.85)\n",
    "    discount_factor::Float64 # discount\n",
    "end\n",
    "\n",
    "TigerPOMDP() = TigerPOMDP(-1., -100., 10., 0.85, 0.95)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## States\n",
    "\n",
    "We define our state with a boolean that indicates weather or not the tiger is hiding behind the left door. If our state is true, the tiger is behind the left door. If its false, the tiger is behind the right door. \n",
    "\n",
    "We must implement the `states` function that returns the state space along with a `stateindex` function that returns the integer index of state `s`. For simple state spaces like this, a vector representation works well, but for more complex continuous or hybrid spaces, it may be best to implement a custom type. Such a type should support appropriate methods from the [POMDPs.jl spaces interface](https://juliapomdp.github.io/POMDPs.jl/latest/interfaces/#Spaces-1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "POMDPs.states(pomdp::TigerPOMDP) = [true, false]\n",
    "POMDPs.stateindex(pomdp::TigerPOMDP, s::Bool) = s ? 1 : 2 ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Actions\n",
    "\n",
    "There are three actions in our problem. Once again, we represent the action space as an array of the actions in our problem. The actions function serve a similar purpose to the `states` function above. Since the action space is discrete, we can define the `actionindex` function that associates an integer index to each action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "POMDPs.actions(pomdp::TigerPOMDP) = [:open_left, :open_right, :listen]\n",
    "function POMDPs.actionindex(pomdp::TigerPOMDP, a::Symbol)\n",
    "    if a==:open_left\n",
    "        return 1\n",
    "    elseif a==:open_right\n",
    "        return 2\n",
    "    elseif a==:listen\n",
    "        return 3\n",
    "    end\n",
    "    error(\"invalid TigerPOMDP action: $a\")\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** if the actions available depends on the state, one should additionally implement the function `actions(pomdp, s)`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## State transitions\n",
    "\n",
    "Now that the states and actions are defined, the transition distribution can be specified. We can do so by implementing the `transition` function. It takes as input the pomdp model, a state and an action and returns a distribution over the next states. \n",
    "\n",
    "We first need a data type to represent the transition distribution. We must be able to sample a state from this object using `rand` and also query the probability mass of a certain state using `pdf`.\n",
    "A new distribution type is sometimes appropriate, but POMDPModelTools provides useful [types](https://juliapomdp.github.io/POMDPModelTools.jl/latest/distributions) for simple cases like this one. \n",
    "\n",
    "In the tiger problem, since there are two state we can represent our distribution with one parameter corresponding to the probability of being in state `true` (tiger on the left). This can be done using the `BoolDistribution` type provided by POMDPModelTools. The transition model is described as follows: \n",
    "- The tiger always stays on the same side \n",
    "- Once we open the door, the problem resets, that is the tiger is spawned with equal probability behind one of the two doors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "function POMDPs.transition(pomdp::TigerPOMDP, s::Bool, a::Symbol)\n",
    "    if a == :open_left || a == :open_right\n",
    "        # problem resets\n",
    "        return BoolDistribution(0.5) \n",
    "    elseif s\n",
    "        # tiger on the left stays on the left \n",
    "        return BoolDistribution(1.0)\n",
    "    else\n",
    "        return BoolDistribution(0.0)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observations\n",
    "\n",
    "In the tiger problem there are two possible observations: hearing the tiger on the left or on the right. We represent them by a boolean. Similarly as for states and actions we must implement `observations` and `obsindex`.\n",
    "\n",
    "**Note:** `obsindex` for boolean observation is provided as a convenience function by POMDPModelTools, but we implement it again here to make modifying this code less error-prone. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "POMDPs.observations(pomdp::TigerPOMDP) = [true, false]\n",
    "POMDPs.obsindex(pomdp::TigerPOMDP, o::Bool) = o+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The observation model captures the uncertainty in the agent's listening ability. When we listen, we receive a noisy measurement of the tiger's location. \n",
    "To implement the observation model with the explicit interface, one must implement the function `observation` which returns a distribution. Remember that observations are also represented by booleans, we can again use the `BoolDistribution` type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "function POMDPs.observation(pomdp::TigerPOMDP, a::Symbol, s::Bool)\n",
    "    pc = pomdp.p_listen_correctly\n",
    "    if a == :listen \n",
    "        if s \n",
    "            return BoolDistribution(pc)\n",
    "        else\n",
    "            return BoolDistribution(1 - pc)\n",
    "        end\n",
    "    else\n",
    "        return BoolDistribution(0.5)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rewards\n",
    "\n",
    "The reward model caputres the immediate objectives of the agent. It recieves a large negative reward for opening the door with the tiger behind it (-100), gets a positive reward for opening the other door (+10), and a small penalty for listening (-1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reward model\n",
    "function POMDPs.reward(pomdp::TigerPOMDP, s::Bool, a::Symbol)\n",
    "    r = 0.0\n",
    "    if a == :listen\n",
    "        r+=pomdp.r_listen\n",
    "    elseif a == :open_left\n",
    "        s ? (r += pomdp.r_findtiger) : (r += pomdp.r_escapetiger)\n",
    "    elseif a == :open_right\n",
    "        s ? (r += pomdp.r_escapetiger) : (r += pomdp.r_findtiger)\n",
    "    end\n",
    "    return r\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Belief\n",
    "\n",
    "In POMDPs, we often represent our estimate of the current state with a belief, a distribution over states. Since we have two possible state, we can use the convenient `BoolDistribution` one more time. \n",
    "\n",
    "Implementing beliefs and their updaters can be tricky. Luckily, our solvers abstract away the belief updating. All you need to do is define a function that returns an initial distribution over states. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "POMDPs.initialstate_distribution(pomdp::TigerPOMDP) = BoolDistribution(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn more about beliefs and belief updating you may look at [BeliefUpdaters.jl](https://github.com/JuliaPOMDP/BeliefUpdaters.jl) where a collection of belief representations and belief updaters. \n",
    "**Note:** It is also possible to create your own belief representation and update schemes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Model Properties\n",
    "\n",
    "Other model properties such as the discount factor or terminal states are specified by implementing new methods of interface functions. A complete list of these functions can be found in the [Defining Basic (PO)MDP Properties section of the documentation](https://juliapomdp.github.io/POMDPs.jl/latest/basic_properties) (this link may be broken because it was set up in a transition phase; please submit an issue if it needs to be fixed).\n",
    "\n",
    "For the tiger problem, we need only the discount factor implemented via the `discount` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "POMDPs.discount(pomdp::TigerPOMDP) = pomdp.discount_factor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seeing the model in action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now implemented all the functions necessary to solve or simulate the tiger pomdp with the explicit interface!\n",
    "\n",
    "To learn how to solve POMDPs offline, use [this tutorial](Using-an-Offline-Solver.ipynb).\n",
    "\n",
    "To learn how to solve POMDPs online, use [this tutorial](Using-an-Online-Solver.ipynb).\n",
    "\n",
    "We can run a simulation of our model using the `stepthrough` function\n",
    "\n",
    "For more information on running simulations, see the [simulation example](Running-Simulations.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s = false\n",
      "a = :listen\n",
      "r = -1.0\n",
      "\n",
      "s = false\n",
      "a = :open_left\n",
      "r = 10.0\n",
      "\n",
      "s = true\n",
      "a = :open_left\n",
      "r = -100.0\n",
      "\n",
      "s = true\n",
      "a = :open_right\n",
      "r = 10.0\n",
      "\n",
      "s = false\n",
      "a = :open_left\n",
      "r = 10.0\n",
      "\n",
      "s = true\n",
      "a = :open_left\n",
      "r = -100.0\n",
      "\n",
      "s = false\n",
      "a = :open_left\n",
      "r = 10.0\n",
      "\n",
      "s = false\n",
      "a = :listen\n",
      "r = -1.0\n",
      "\n",
      "s = false\n",
      "a = :open_left\n",
      "r = 10.0\n",
      "\n",
      "s = false\n",
      "a = :open_left\n",
      "r = 10.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "using POMDPSimulators\n",
    "using POMDPPolicies\n",
    "\n",
    "m = TigerPOMDP()\n",
    "\n",
    "# policy that takes a random action\n",
    "policy = RandomPolicy(m)\n",
    "\n",
    "for (s, a, r) in stepthrough(m, policy, \"s,a,r\", max_steps=10)\n",
    "    @show s\n",
    "    @show a\n",
    "    @show r\n",
    "    println()\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}