Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0baf691e-b0d5-4acd-8057-a4c4e99ddb49",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb84e58-8b54-4d61-b5ff-08c845b13d4f",
   "metadata": {},
   "source": [
    "This notebook illustrates how to access the SBOX-COST problems using IOHexperimenter.\n",
    "To install IOHexperimenter, the following command can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56f53a36-e40c-4ff3-9c08-12b8745524a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install ioh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d6d79b-8aa4-419e-80c7-15b7fd0c2492",
   "metadata": {},
   "source": [
    "Once installed, simply import the package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3bfc62ad-24cd-4dc6-b3d1-cd9d2293631b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ioh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b509ddaa-446d-4629-b118-f4dbc08ab44f",
   "metadata": {},
   "source": [
    "# Accessing problems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87a4346-c909-4c6e-8de8-6f80821fce69",
   "metadata": {},
   "source": [
    "In ioh, everything revolves around the problem class, so we start by creating an individual SBOX-COST problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a8a575fc-2eed-4591-88a1-8b7df038dbc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = ioh.get_problem(1, instance=1, dimension=5, problem_type=ioh.ProblemClass.SBOX) #in ioh 0.3.9, ProblemType was renamed to ProblemClass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01d7975c-490e-4a19-8b33-2b0e8f1903fd",
   "metadata": {},
   "source": [
    "This problem contains all kinds of information about the way it is constructed. It also keeps track of all evaluations, which can be seen as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6af1db0b-2ac5-4b51-8ced-1a03bb81272f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MetaData: Sphere id: 1 iid: 1 dim: 5>\n",
      "<State evaluations: 1 optimum_found: false current_best: <Solution x: [0, 0, 0, 0, 0] y: 99.51746200000001>>\n"
     ]
    }
   ],
   "source": [
    "f([0,0,0,0,0])\n",
    "print(f.meta_data)\n",
    "print(f.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d775830e-1365-47e5-860b-7f35ffc703eb",
   "metadata": {},
   "source": [
    "For the SBOX-COST suite, the bounds are considered as hard constrains, so evaluating outside the domain will give an infinite value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2b4eaefb-df83-4f84-ad9a-ea2d72705f3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "inf"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f([6,0,0,0,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da96f9fe-87fe-4302-9f3e-b33bcc9917e5",
   "metadata": {},
   "source": [
    "This is not the case for the BBOB version of the functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "55240c2b-db03-41b6-b6f0-c4603b021af3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "125.27037568"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_bbob = ioh.get_problem(1, instance=1, dimension=5, problem_type=ioh.ProblemClass.BBOB)\n",
    "f_bbob([6,0,0,0,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89426fbd-bf71-4442-9c7d-b535fb30bbfc",
   "metadata": {},
   "source": [
    "Since this problem can be evaluated as a common python function, it should work directly with any optimizer. We make a basic random search to illustrate this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e4879f7e-1df2-4bf4-b5e8-7319b851c20d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class RandomSearch:\n",
    "    'Simple random search algorithm'\n",
    "    def __init__(self, n: int, length: float = 0.0):\n",
    "        self.n: int = n\n",
    "        self.length: float = length\n",
    "        \n",
    "    def __call__(self, problem: ioh.problem.RealSingleObjective) -> None:\n",
    "        'Evaluate the problem n times with a randomly generated solution'\n",
    "        \n",
    "        for _ in range(self.n):\n",
    "            # We can use the problems bounds accessor to get information about the problem bounds\n",
    "            x = np.random.uniform(problem.bounds.lb, problem.bounds.ub)\n",
    "            self.length = np.linalg.norm(x)\n",
    "            \n",
    "            problem(x)   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a541dde-9cf3-4af4-a660-5091e70b62e0",
   "metadata": {},
   "source": [
    "If we want to run this algorithm on our problem, we can do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a38de893-a238-4ab4-bfe9-d3590d4fed02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<State evaluations: 21 optimum_found: false current_best: <Solution x: [-0.9841754443299529, -0.546437645418079, 1.8182627272526375, 3.5814976760988753, -3.0087394534988876] y: 90.89006757496374>>\n"
     ]
    }
   ],
   "source": [
    "r = RandomSearch(10)\n",
    "r(f)\n",
    "print(f.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79320baa-fc59-450c-8ecd-ef00da3a0121",
   "metadata": {},
   "source": [
    "To run multiple independent runs on the same problem, we can reset the state as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "063cf170-024a-402c-b578-0865643da891",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34a5275f-c80d-4c3c-9f0a-670a5f32499c",
   "metadata": {},
   "source": [
    "# Logging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ffb6ea2-0ccd-4ae1-86de-eb872648b292",
   "metadata": {},
   "source": [
    "The default usage of IOHExperimenter is in generating logs of benchmarking experiments which can be analyzed in IOHAnalyzer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "84a76cbe-520a-44e4-9b1a-525eb4cfe4c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Analyzer /mnt/c/Users/Diedie/Documents/Projects/SBOX-COST/my-experiment>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "logger = ioh.logger.Analyzer(\n",
    "    root=os.getcwd(),                  # Store data in the current working directory\n",
    "    folder_name=\"my-experiment\",       # in a folder named: 'my-experiment'\n",
    "    algorithm_name=\"random-search\",    # meta-data for the algorithm used to generate these results\n",
    "    store_positions=False               # disable storing x-variables in the logged files\n",
    ")\n",
    "\n",
    "# this automatically creates a folder 'my-experiment' in the current working directory\n",
    "# if the folder already exists, it will given an additional number to make the name unique\n",
    "logger"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cb27e5f-4cca-4707-afbb-54b9987aba6e",
   "metadata": {},
   "source": [
    "We can add this logger to a problem so we can store the data when running our algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5a306a7c-773e-4251-a3e3-05ac752dac8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.attach_logger(logger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "416f61ff-1a16-4d29-b167-957eb657ccda",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = RandomSearch(100)\n",
    "r(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95a7cf21-0a70-4bf9-a325-82f0c79b6851",
   "metadata": {},
   "source": [
    "Once finished with the run, we can close the logger to force it to write the data (happens automatically when running as a python script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8a060dec-a11a-43ca-9f76-a695f8d04bac",
   "metadata": {},
   "outputs": [],
   "source": [
    "logger.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ee1d180-1b7f-4b28-b127-3e6eeae41ab4",
   "metadata": {},
   "source": [
    "# Experiment class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203e6d30-4de2-497c-b9b3-2ed202dec8bb",
   "metadata": {},
   "source": [
    "In Python, we provide the Experiment class which can be used to easily run a given algorithm over a larger number of problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e0886d3e-2d58-41a9-ab65-775d56aa19f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/diedie/.local/lib/python3.8/site-packages/ioh/__init__.py:344: UserWarning: There is already a folder named /mnt/c/Users/Diedie/Documents/Projects/SBOX-COST/ioh_data and merge_output is set to True. The results of this experiment will be merged with the contents of this folder\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "experiment = ioh.Experiment(\n",
    "    algorithm = RandomSearch(10), # An algorithm instance\n",
    "    fids = [1, 2, 3],               # the id's of the problems we want to test\n",
    "    iids = [1, 2],               # the instances \n",
    "    dims = [5, 20],               # the dimensions\n",
    "    reps = 3,                     # the number of runs,\n",
    "    problem_class=ioh.ProblemClass.SBOX, #the problem type\n",
    "    zip_output = True       \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ff45b6-9067-4b85-9c66-226cffcff507",
   "metadata": {},
   "source": [
    "Running this experiment creates a zip-file, which can directly be processed by IOHanalyzer (https://iohanalyzer.liacs.nl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "74265079-9b37-47f6-897a-2aaa6af3052d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<ioh.Experiment at 0x7fa87ee31be0>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c20e04-a75a-4646-ad39-49702976f072",
   "metadata": {},
   "source": [
    "For more background and other information on using IOHexperimenter, we refer to https://iohprofiler.github.io/IOHexp/\n",
    "\n",
    "If you have any questions about the usage of SBOX-COST, or about IOHprofiler in general, please make an issue on https://github.com/sbox-cost/Examples"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}