examples/kdd22/05_bias.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KDD 2022 Hands-on Tutorial on \"Gradual AutoML using Lale\"\n",
    "\n",
    "# 5. Bias Mitigation\n",
    "\n",
    "Datasets and predictive models can exhibit bias, which means they are\n",
    "unfair against certain groups or individuals.\n",
    "We would of course like to avoid that when possible.\n",
    "Mitigating bias cannot be solved by data scientists alone; it requires\n",
    "a discussion with more stake-holders, for instance, to determine what\n",
    "bias to target and how to measure it.\n",
    "That said, there are algorithmic bias mitigation techniques that can\n",
    "contribute to a solution.\n",
    "This notebook presents some fairness metrics and bias mitigators\n",
    "distributed with Lale, and shows how to use them either with manual\n",
    "machine learning or with AutoML.\n",
    "\n",
    "This notebook has the following sections:\n",
    "\n",
    "- [5.1 Datasets and Fairness Information](#5.1-Dataset-and-Fairness-Information)\n",
    "- [5.2 Metrics](#5.2-Metrics)\n",
    "- [5.3 Non-Mitigated Pipeline](#5.3-Non-Mitigated-Pipeline)\n",
    "- [5.4 Mitigators](#5.4-Mitigators)\n",
    "- [5.5 AutoML with Mitigators](#5.5-AutoML-with-Mitigators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Dataset and Fairness Information\n",
    "\n",
    "Like other notebooks in this tutorial, this notebook uses the\n",
    "[credit-g](https://www.openml.org/search?type=data&status=active&id=31)\n",
    "dataset from OpenML.\n",
    "First, we import the Lale wrappers for\n",
    "[AIF360](https://aif360.mybluemix.net/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "WARNING:root:No module named 'tensorflow': AdversarialDebiasing will be unavailable. To install, run:\n",
      "pip install 'aif360[AdversarialDebiasing]'\n",
      "WARNING:root:No module named 'tensorflow': AdversarialDebiasing will be unavailable. To install, run:\n",
      "pip install 'aif360[AdversarialDebiasing]'\n"
     ]
    }
   ],
   "source": [
    "import lale.lib.aif360 as lf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import lale.lib.aif360 as lf\n",
    "except UnicodeWarning:\n",
    "    warnings.warn('Oh Non', stacklevel=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on your Python environment, the above imports may yield some\n",
    "warnings about missing tensorflow or numba; you can ignore those,\n",
    "since this notebook does not need those packages.\n",
    "Next, we fetch the dataset along with its `fairness_info`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'favorable_labels': ['good'],\n",
       " 'protected_attributes': [{'feature': 'personal_status',\n",
       "   'reference_group': ['male div/sep', 'male mar/wid', 'male single']},\n",
       "  {'feature': 'age', 'reference_group': [[26, 1000]]}]}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y, fairness_info = lf.fetch_creditg_df()\n",
    "fairness_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fairness_info` is a Python dictionary of fairness-related\n",
    "meta-data about the dataset.\n",
    "It tells us which labels in `y` indicate a favorable outcome.\n",
    "Furthermore, it tells us which columns in `X` are protected\n",
    "attributes, i.e., features whose value indicates membership in\n",
    "a privileged group, and what those values are.\n",
    "To make this concrete, let's look at the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>checking_status</th>\n",
       "      <th>duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings_status</th>\n",
       "      <th>employment</th>\n",
       "      <th>installment_commitment</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>other_parties</th>\n",
       "      <th>residence_since</th>\n",
       "      <th>property_magnitude</th>\n",
       "      <th>age</th>\n",
       "      <th>other_payment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_credits</th>\n",
       "      <th>job</th>\n",
       "      <th>num_dependents</th>\n",
       "      <th>own_telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>good</td>\n",
       "      <td>&lt;0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>critical/other existing credit</td>\n",
       "      <td>radio/tv</td>\n",
       "      <td>1169.0</td>\n",
       "      <td>no known savings</td>\n",
       "      <td>&gt;=7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>real estate</td>\n",
       "      <td>67.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bad</td>\n",
       "      <td>0&lt;=X&lt;200</td>\n",
       "      <td>48.0</td>\n",
       "      <td>existing paid</td>\n",
       "      <td>radio/tv</td>\n",
       "      <td>5951.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>1&lt;=X&lt;4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>female div/dep/mar</td>\n",
       "      <td>none</td>\n",
       "      <td>2.0</td>\n",
       "      <td>real estate</td>\n",
       "      <td>22.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>good</td>\n",
       "      <td>no checking</td>\n",
       "      <td>12.0</td>\n",
       "      <td>critical/other existing credit</td>\n",
       "      <td>education</td>\n",
       "      <td>2096.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>4&lt;=X&lt;7</td>\n",
       "      <td>2.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>none</td>\n",
       "      <td>3.0</td>\n",
       "      <td>real estate</td>\n",
       "      <td>49.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1.0</td>\n",
       "      <td>unskilled resident</td>\n",
       "      <td>2.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>good</td>\n",
       "      <td>&lt;0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>existing paid</td>\n",
       "      <td>furniture/equipment</td>\n",
       "      <td>7882.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>4&lt;=X&lt;7</td>\n",
       "      <td>2.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>guarantor</td>\n",
       "      <td>4.0</td>\n",
       "      <td>life insurance</td>\n",
       "      <td>45.0</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>1.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>2.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bad</td>\n",
       "      <td>&lt;0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>delayed previously</td>\n",
       "      <td>new car</td>\n",
       "      <td>4870.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>1&lt;=X&lt;4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>no known property</td>\n",
       "      <td>53.0</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>2.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>2.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>good</td>\n",
       "      <td>no checking</td>\n",
       "      <td>12.0</td>\n",
       "      <td>existing paid</td>\n",
       "      <td>furniture/equipment</td>\n",
       "      <td>1736.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>4&lt;=X&lt;7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>female div/dep/mar</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>real estate</td>\n",
       "      <td>31.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1.0</td>\n",
       "      <td>unskilled resident</td>\n",
       "      <td>1.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>good</td>\n",
       "      <td>&lt;0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>existing paid</td>\n",
       "      <td>used car</td>\n",
       "      <td>3857.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>1&lt;=X&lt;4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>male div/sep</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>life insurance</td>\n",
       "      <td>40.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1.0</td>\n",
       "      <td>high qualif/self emp/mgmt</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>good</td>\n",
       "      <td>no checking</td>\n",
       "      <td>12.0</td>\n",
       "      <td>existing paid</td>\n",
       "      <td>radio/tv</td>\n",
       "      <td>804.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>&gt;=7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>car</td>\n",
       "      <td>38.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>bad</td>\n",
       "      <td>&lt;0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>existing paid</td>\n",
       "      <td>radio/tv</td>\n",
       "      <td>1845.0</td>\n",
       "      <td>&lt;100</td>\n",
       "      <td>1&lt;=X&lt;4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>no known property</td>\n",
       "      <td>23.0</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>1.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>good</td>\n",
       "      <td>0&lt;=X&lt;200</td>\n",
       "      <td>45.0</td>\n",
       "      <td>critical/other existing credit</td>\n",
       "      <td>used car</td>\n",
       "      <td>4576.0</td>\n",
       "      <td>100&lt;=X&lt;500</td>\n",
       "      <td>unemployed</td>\n",
       "      <td>3.0</td>\n",
       "      <td>male single</td>\n",
       "      <td>none</td>\n",
       "      <td>4.0</td>\n",
       "      <td>car</td>\n",
       "      <td>27.0</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1.0</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1.0</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    class checking_status  duration                  credit_history  \\\n",
       "0    good              <0       6.0  critical/other existing credit   \n",
       "1     bad        0<=X<200      48.0                   existing paid   \n",
       "2    good     no checking      12.0  critical/other existing credit   \n",
       "3    good              <0      42.0                   existing paid   \n",
       "4     bad              <0      24.0              delayed previously   \n",
       "..    ...             ...       ...                             ...   \n",
       "995  good     no checking      12.0                   existing paid   \n",
       "996  good              <0      30.0                   existing paid   \n",
       "997  good     no checking      12.0                   existing paid   \n",
       "998   bad              <0      45.0                   existing paid   \n",
       "999  good        0<=X<200      45.0  critical/other existing credit   \n",
       "\n",
       "                 purpose  credit_amount    savings_status  employment  \\\n",
       "0               radio/tv         1169.0  no known savings         >=7   \n",
       "1               radio/tv         5951.0              <100      1<=X<4   \n",
       "2              education         2096.0              <100      4<=X<7   \n",
       "3    furniture/equipment         7882.0              <100      4<=X<7   \n",
       "4                new car         4870.0              <100      1<=X<4   \n",
       "..                   ...            ...               ...         ...   \n",
       "995  furniture/equipment         1736.0              <100      4<=X<7   \n",
       "996             used car         3857.0              <100      1<=X<4   \n",
       "997             radio/tv          804.0              <100         >=7   \n",
       "998             radio/tv         1845.0              <100      1<=X<4   \n",
       "999             used car         4576.0        100<=X<500  unemployed   \n",
       "\n",
       "     installment_commitment     personal_status other_parties  \\\n",
       "0                       4.0         male single          none   \n",
       "1                       2.0  female div/dep/mar          none   \n",
       "2                       2.0         male single          none   \n",
       "3                       2.0         male single     guarantor   \n",
       "4                       3.0         male single          none   \n",
       "..                      ...                 ...           ...   \n",
       "995                     3.0  female div/dep/mar          none   \n",
       "996                     4.0        male div/sep          none   \n",
       "997                     4.0         male single          none   \n",
       "998                     4.0         male single          none   \n",
       "999                     3.0         male single          none   \n",
       "\n",
       "     residence_since property_magnitude   age other_payment_plans   housing  \\\n",
       "0                4.0        real estate  67.0                none       own   \n",
       "1                2.0        real estate  22.0                none       own   \n",
       "2                3.0        real estate  49.0                none       own   \n",
       "3                4.0     life insurance  45.0                none  for free   \n",
       "4                4.0  no known property  53.0                none  for free   \n",
       "..               ...                ...   ...                 ...       ...   \n",
       "995              4.0        real estate  31.0                none       own   \n",
       "996              4.0     life insurance  40.0                none       own   \n",
       "997              4.0                car  38.0                none       own   \n",
       "998              4.0  no known property  23.0                none  for free   \n",
       "999              4.0                car  27.0                none       own   \n",
       "\n",
       "     existing_credits                        job  num_dependents  \\\n",
       "0                 2.0                    skilled             1.0   \n",
       "1                 1.0                    skilled             1.0   \n",
       "2                 1.0         unskilled resident             2.0   \n",
       "3                 1.0                    skilled             2.0   \n",
       "4                 2.0                    skilled             2.0   \n",
       "..                ...                        ...             ...   \n",
       "995               1.0         unskilled resident             1.0   \n",
       "996               1.0  high qualif/self emp/mgmt             1.0   \n",
       "997               1.0                    skilled             1.0   \n",
       "998               1.0                    skilled             1.0   \n",
       "999               1.0                    skilled             1.0   \n",
       "\n",
       "    own_telephone foreign_worker  \n",
       "0             yes            yes  \n",
       "1            none            yes  \n",
       "2            none            yes  \n",
       "3            none            yes  \n",
       "4            none            yes  \n",
       "..            ...            ...  \n",
       "995          none            yes  \n",
       "996           yes            yes  \n",
       "997          none            yes  \n",
       "998           yes            yes  \n",
       "999          none            yes  \n",
       "\n",
       "[1000 rows x 21 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.max_columns = None\n",
    "pd.concat([y, X], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two labels, `class='good'` and `class='bad'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "favorable: ['good']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot: >"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAoYElEQVR4nO3df1RU54H/8Q/oMIA6EEwYtAF1t4lKY6oLjcyabFLDjxiOx6yctjbWkK5Ntiy6UbYmpUeNShqybqI2KcZt16rZxO3WdOOqaw0jabRVFCS1x19Bu0kWd3Vgq4Hxx3EY4H7/6JdbCcYwOLM84Pt1Dudk7n3uM8+d/HHf586ME2VZliUAAACDRPf1AgAAAD6JQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgnMF9vYDe6Ojo0JkzZzRs2DBFRUX19XIAAEAPWJalCxcuaOTIkYqOvv49kn4ZKGfOnFFqampfLwMAAPTC6dOndfvtt193TL8MlGHDhkn6wwm6XK6wzh0MBlVZWanc3Fw5HI6wzg0AQH8QqWuh3+9XamqqfR2/nn4ZKJ1v67hcrogESnx8vFwuF4ECALgpRfpa2JOPZ/AhWQAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABgnpEAZPXq0oqKiuv0VFxdLkq5cuaLi4mINHz5cQ4cOVUFBgRobG7vM0dDQoPz8fMXHxys5OVmLFi1SW1tb+M4IAAD0eyEFSm1trc6ePWv/eb1eSdJXvvIVSdLChQu1fft2bdmyRXv27NGZM2c0c+ZM+/j29nbl5+ertbVV+/fv16ZNm7Rx40YtXbo0jKcEAAD6u5AC5bbbblNKSor9t2PHDv3pn/6p7r//frW0tGj9+vVatWqVpk6dqoyMDG3YsEH79+/XgQMHJEmVlZU6fvy4Xn/9dU2cOFHTpk1TWVmZKioq1NraGpETBAAA/U+vf824tbVVr7/+ukpKShQVFaW6ujoFg0FlZ2fbY8aNG6e0tDRVV1crKytL1dXVmjBhgtxutz0mLy9PRUVFOnbsmCZNmnTN5woEAgoEAvZjv98v6Q+/thgMBnt7CtfUOV+45wUAoL+I1LUwlPl6HShbt25Vc3OzHn/8cUmSz+dTTEyMEhMTu4xzu93y+Xz2mKvjpHN/575PU15eruXLl3fbXllZqfj4+N6ewnV1vn0FAMDNKtzXwsuXL/d4bK8DZf369Zo2bZpGjhzZ2yl6rLS0VCUlJfZjv9+v1NRU5ebmyuVyhfW5gsGgvF6vlhyKVqAjKqxzR9LRZXl9vQQAwADReS3MycmRw+EI27yd74D0RK8C5b/+67+0e/du/du//Zu9LSUlRa2trWpubu5yF6WxsVEpKSn2mJqami5zdX7Lp3PMtTidTjmdzm7bHQ5HWF+4qwU6ohRo7z+BEqnXAQBw8wr3dTaUuXr176Bs2LBBycnJys/Pt7dlZGTI4XCoqqrK3lZfX6+GhgZ5PB5Jksfj0ZEjR9TU1GSP8Xq9crlcSk9P781SAADAABTyHZSOjg5t2LBBhYWFGjz4j4cnJCRo7ty5KikpUVJSklwul+bPny+Px6OsrCxJUm5urtLT0zVnzhytXLlSPp9PixcvVnFx8TXvkAAAgJtTyIGye/duNTQ06K/+6q+67Vu9erWio6NVUFCgQCCgvLw8rV271t4/aNAg7dixQ0VFRfJ4PBoyZIgKCwu1YsWKGzsLAAAwoIQcKLm5ubIs65r7YmNjVVFRoYqKik89ftSoUdq5c2eoTwsAAG4i/BYPAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjhBwo//M//6NvfOMbGj58uOLi4jRhwgQdOnTI3m9ZlpYuXaoRI0YoLi5O2dnZOnXqVJc5zp8/r9mzZ8vlcikxMVFz587VxYsXb/xsAADAgBBSoHz88ceaMmWKHA6HfvGLX+j48eN66aWXdMstt9hjVq5cqZdfflnr1q3TwYMHNWTIEOXl5enKlSv2mNmzZ+vYsWPyer3asWOH9u7dqyeffDJ8ZwUAAPq1waEM/vu//3ulpqZqw4YN9rYxY8bY/21ZltasWaPFixdrxowZkqTXXntNbrdbW7du1axZs3TixAnt2rVLtbW1yszMlCS98sorevjhh/Xiiy9q5MiR4TgvAADQj4V0B2Xbtm3KzMzUV77yFSUnJ2vSpEn68Y9/bO//8MMP5fP5lJ2dbW9LSEjQ5MmTVV1dLUmqrq5WYmKiHSeSlJ2drejoaB08ePBGzwcAAAwAId1B+eCDD/Tqq6+qpKRE3/ve91RbW6u//du/VUxMjAoLC+Xz+SRJbre7y3Fut9ve5/P5lJyc3HURgwcrKSnJHvNJgUBAgUDAfuz3+yVJwWBQwWAwlFP4TJ3zOaOtsM4baeF+HQAAN6/Oa0qkrrE9EVKgdHR0KDMzU88//7wkadKkSTp69KjWrVunwsLC0FYZgvLyci1fvrzb9srKSsXHx0fkOcsyOyIyb6Ts3Lmzr5cAABhgvF5vWOe7fPlyj8eGFCgjRoxQenp6l23jx4/Xz3/+c0lSSkqKJKmxsVEjRoywxzQ2NmrixIn2mKampi5ztLW16fz58/bxn1RaWqqSkhL7sd/vV2pqqnJzc+VyuUI5hc8UDAbl9Xq15FC0Ah1RYZ07ko4uy+vrJQAABojOa2FOTo4cDkfY5u18B6QnQgqUKVOmqL6+vsu2kydPatSoUZL+8IHZlJQUVVVV2UHi9/t18OBBFRUVSZI8Ho+am5tVV1enjIwMSdI777yjjo4OTZ48+ZrP63Q65XQ6u213OBxhfeGuFuiIUqC9/wRKpF4HAMDNK9zX2VDmCilQFi5cqD//8z/X888/r69+9auqqanRj370I/3oRz+SJEVFRWnBggV67rnndMcdd2jMmDFasmSJRo4cqUceeUTSH+64PPTQQ3riiSe0bt06BYNBzZs3T7NmzeIbPAAAQFKIgfKlL31Jb731lkpLS7VixQqNGTNGa9as0ezZs+0xTz/9tC5duqQnn3xSzc3Nuvfee7Vr1y7FxsbaY9544w3NmzdPDz74oKKjo1VQUKCXX345fGcFAAD6tSjLsvrX11X0h7eNEhIS1NLSEpHPoOzcuVNP1wzqV2/xfPRCfl8vAQAwQHReCx9++OGwfwalp9dvfosHAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGCckAJl2bJlioqK6vI3btw4e/+VK1dUXFys4cOHa+jQoSooKFBjY2OXORoaGpSfn6/4+HglJydr0aJFamtrC8/ZAACAAWFwqAd84Qtf0O7du/84weA/TrFw4UL9x3/8h7Zs2aKEhATNmzdPM2fO1L59+yRJ7e3tys/PV0pKivbv36+zZ8/qsccek8Ph0PPPPx+G0wEAAANByIEyePBgpaSkdNve0tKi9evXa/PmzZo6daokacOGDRo/frwOHDigrKwsVVZW6vjx49q9e7fcbrcmTpyosrIyPfPMM1q2bJliYmJu/IwAAEC/F3KgnDp1SiNHjlRsbKw8Ho/Ky8uVlpamuro6BYNBZWdn22PHjRuntLQ0VVdXKysrS9XV1ZowYYLcbrc9Ji8vT0VFRTp27JgmTZp0zecMBAIKBAL2Y7/fL0kKBoMKBoOhnsJ1dc7njLbCOm+khft1AADcvDqvKZG6xvZESIEyefJkbdy4UWPHjtXZs2e1fPly3XfffTp69Kh8Pp9iYmKUmJjY5Ri32y2fzydJ8vl8XeKkc3/nvk9TXl6u5cuXd9teWVmp+Pj4UE6hx8oyOyIyb6Ts3Lmzr5cAABhgvF5vWOe7fPlyj8eGFCjTpk2z//vuu+/W5MmTNWrUKP3sZz9TXFxcKFOFpLS0VCUlJfZjv9+v1NRU5ebmyuVyhfW5gsGgvF6vlhyKVqAjKqxzR9LRZXl9vQQAwADReS3MycmRw+EI27yd74D0RMhv8VwtMTFRd955p373u98pJydHra2tam5u7nIXpbGx0f7MSkpKimpqarrM0fktn2t9rqWT0+mU0+nstt3hcIT1hbtaoCNKgfb+EyiReh0AADevcF9nQ5nrhv4dlIsXL+o///M/NWLECGVkZMjhcKiqqsreX19fr4aGBnk8HkmSx+PRkSNH1NTUZI/xer1yuVxKT0+/kaUAAIABJKQ7KN/5znc0ffp0jRo1SmfOnNGzzz6rQYMG6etf/7oSEhI0d+5clZSUKCkpSS6XS/Pnz5fH41FWVpYkKTc3V+np6ZozZ45Wrlwpn8+nxYsXq7i4+Jp3SAAAwM0ppED57//+b33961/XuXPndNttt+nee+/VgQMHdNttt0mSVq9erejoaBUUFCgQCCgvL09r1661jx80aJB27NihoqIieTweDRkyRIWFhVqxYkV4zwoAAPRrIQXKT3/60+vuj42NVUVFhSoqKj51zKhRo/jGCQAAuC5+iwcAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHFuKFBeeOEFRUVFacGCBfa2K1euqLi4WMOHD9fQoUNVUFCgxsbGLsc1NDQoPz9f8fHxSk5O1qJFi9TW1nYjSwEAAANIrwOltrZW//iP/6i77767y/aFCxdq+/bt2rJli/bs2aMzZ85o5syZ9v729nbl5+ertbVV+/fv16ZNm7Rx40YtXbq092cBAAAGlF4FysWLFzV79mz9+Mc/1i233GJvb2lp0fr167Vq1SpNnTpVGRkZ2rBhg/bv368DBw5IkiorK3X8+HG9/vrrmjhxoqZNm6aysjJVVFSotbU1PGcFAAD6tcG9Oai4uFj5+fnKzs7Wc889Z2+vq6tTMBhUdna2vW3cuHFKS0tTdXW1srKyVF1drQkTJsjtdttj8vLyVFRUpGPHjmnSpEndni8QCCgQCNiP/X6/JCkYDCoYDPbmFD5V53zOaCus80ZauF8HAMDNq/OaEqlrbE+EHCg//elP9d5776m2trbbPp/Pp5iYGCUmJnbZ7na75fP57DFXx0nn/s5911JeXq7ly5d3215ZWan4+PhQT6FHyjI7IjJvpOzcubOvlwAAGGC8Xm9Y57t8+XKPx4YUKKdPn9ZTTz0lr9er2NjYkBfWW6WlpSopKbEf+/1+paamKjc3Vy6XK6zPFQwG5fV6teRQtAIdUWGdO5KOLsvr6yUAAAaIzmthTk6OHA5H2ObtfAekJ0IKlLq6OjU1NenP/uzP7G3t7e3au3evfvjDH+rtt99Wa2urmpubu9xFaWxsVEpKiiQpJSVFNTU1Xebt/JZP55hPcjqdcjqd3bY7HI6wvnBXC3REKdDefwIlUq8DAODmFe7rbChzhfQh2QcffFBHjhzR4cOH7b/MzEzNnj3b/m+Hw6Gqqir7mPr6ejU0NMjj8UiSPB6Pjhw5oqamJnuM1+uVy+VSenp6KMsBAAADVEh3UIYNG6a77rqry7YhQ4Zo+PDh9va5c+eqpKRESUlJcrlcmj9/vjwej7KysiRJubm5Sk9P15w5c7Ry5Ur5fD4tXrxYxcXF17xLAgAAbj69+hbP9axevVrR0dEqKChQIBBQXl6e1q5da+8fNGiQduzYoaKiInk8Hg0ZMkSFhYVasWJFuJcCAAD6qRsOlHfffbfL49jYWFVUVKiiouJTjxk1ahTfOgEAAJ+K3+IBAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGCckALl1Vdf1d133y2XyyWXyyWPx6Nf/OIX9v4rV66ouLhYw4cP19ChQ1VQUKDGxsYuczQ0NCg/P1/x8fFKTk7WokWL1NbWFp6zAQAAA0JIgXL77bfrhRdeUF1dnQ4dOqSpU6dqxowZOnbsmCRp4cKF2r59u7Zs2aI9e/bozJkzmjlzpn18e3u78vPz1draqv3792vTpk3auHGjli5dGt6zAgAA/drgUAZPnz69y+Pvf//7evXVV3XgwAHdfvvtWr9+vTZv3qypU6dKkjZs2KDx48frwIEDysrKUmVlpY4fP67du3fL7XZr4sSJKisr0zPPPKNly5YpJiYmfGcGAAD6rZAC5Wrt7e3asmWLLl26JI/Ho7q6OgWDQWVnZ9tjxo0bp7S0NFVXVysrK0vV1dWaMGGC3G63PSYvL09FRUU6duyYJk2adM3nCgQCCgQC9mO/3y9JCgaDCgaDvT2Fa+qczxlthXXeSAv36wAAuHl1XlMidY3tiZAD5ciRI/J4PLpy5YqGDh2qt956S+np6Tp8+LBiYmKUmJjYZbzb7ZbP55Mk+Xy+LnHSub9z36cpLy/X8uXLu22vrKxUfHx8qKfQI2WZHRGZN1J27tzZ10sAAAwwXq83rPNdvny5x2NDDpSxY8fq8OHDamlp0ZtvvqnCwkLt2bMn1GlCUlpaqpKSEvux3+9XamqqcnNz5XK5wvpcwWBQXq9XSw5FK9ARFda5I+nosry+XgIA4BruWvZ2Xy8hZM5oS2WZHcrJyZHD4QjbvJ3vgPREyIESExOjz3/+85KkjIwM1dbW6gc/+IG+9rWvqbW1Vc3NzV3uojQ2NiolJUWSlJKSopqami7zdX7Lp3PMtTidTjmdzm7bHQ5HWF+4qwU6ohRo7z+BEqnXAQBwY/rTteSTwn2dDWWuG/53UDo6OhQIBJSRkSGHw6Gqqip7X319vRoaGuTxeCRJHo9HR44cUVNTkz3G6/XK5XIpPT39RpcCAAAGiJDuoJSWlmratGlKS0vThQsXtHnzZr377rt6++23lZCQoLlz56qkpERJSUlyuVyaP3++PB6PsrKyJEm5ublKT0/XnDlztHLlSvl8Pi1evFjFxcXXvEMCAABuTiEFSlNTkx577DGdPXtWCQkJuvvuu/X2228rJydHkrR69WpFR0eroKBAgUBAeXl5Wrt2rX38oEGDtGPHDhUVFcnj8WjIkCEqLCzUihUrwntWAACgXwspUNavX3/d/bGxsaqoqFBFRcWnjhk1ahTfOAEAANfFb/EAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIwTUqCUl5frS1/6koYNG6bk5GQ98sgjqq+v7zLmypUrKi4u1vDhwzV06FAVFBSosbGxy5iGhgbl5+crPj5eycnJWrRokdra2m78bAAAwIAQUqDs2bNHxcXFOnDggLxer4LBoHJzc3Xp0iV7zMKFC7V9+3Zt2bJFe/bs0ZkzZzRz5kx7f3t7u/Lz89Xa2qr9+/dr06ZN2rhxo5YuXRq+swIAAP3a4FAG79q1q8vjjRs3Kjk5WXV1dfqLv/gLtbS0aP369dq8ebOmTp0qSdqwYYPGjx+vAwcOKCsrS5WVlTp+/Lh2794tt9utiRMnqqysTM8884yWLVummJiY8J0dAADol0IKlE9qaWmRJCUlJUmS6urqFAwGlZ2dbY8ZN26c0tLSVF1draysLFVXV2vChAlyu932mLy8PBUVFenYsWOaNGlSt+cJBAIKBAL2Y7/fL0kKBoMKBoM3cgrddM7njLbCOm+khft1AACEh3NQ/7qeSH+8BkbqGtsTvQ6Ujo4OLViwQFOmTNFdd90lSfL5fIqJiVFiYmKXsW63Wz6fzx5zdZx07u/cdy3l5eVavnx5t+2VlZWKj4/v7SlcV1lmR0TmjZSdO3f29RIAANew8p6+XkHveb3esM53+fLlHo/tdaAUFxfr6NGj+vWvf93bKXqstLRUJSUl9mO/36/U1FTl5ubK5XKF9bmCwaC8Xq+WHIpWoCMqrHNH0tFleX29BADANdy17O2+XkLInNGWyjI7lJOTI4fDEbZ5O98B6YleBcq8efO0Y8cO7d27V7fffru9PSUlRa2trWpubu5yF6WxsVEpKSn2mJqami7zdX7Lp3PMJzmdTjmdzm7bHQ5HWF+4qwU6ohRo7z+BEqnXAQBwY/rTteSTwn2dDWWukL7FY1mW5s2bp7feekvvvPOOxowZ02V/RkaGHA6Hqqqq7G319fVqaGiQx+ORJHk8Hh05ckRNTU32GK/XK5fLpfT09FCWAwAABqiQ7qAUFxdr8+bN+vd//3cNGzbM/sxIQkKC4uLilJCQoLlz56qkpERJSUlyuVyaP3++PB6PsrKyJEm5ublKT0/XnDlztHLlSvl8Pi1evFjFxcXXvEsCAABuPiEFyquvvipJeuCBB7ps37Bhgx5//HFJ0urVqxUdHa2CggIFAgHl5eVp7dq19thBgwZpx44dKioqksfj0ZAhQ1RYWKgVK1bc2JkAAIABI6RAsazP/qpUbGysKioqVFFR8aljRo0axbdOAADAp+K3eAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGCfkQNm7d6+mT5+ukSNHKioqSlu3bu2y37IsLV26VCNGjFBcXJyys7N16tSpLmPOnz+v2bNny+VyKTExUXPnztXFixdv6EQAAMDAEXKgXLp0SV/84hdVUVFxzf0rV67Uyy+/rHXr1ungwYMaMmSI8vLydOXKFXvM7NmzdezYMXm9Xu3YsUN79+7Vk08+2fuzAAAAA8rgUA+YNm2apk2bds19lmVpzZo1Wrx4sWbMmCFJeu211+R2u7V161bNmjVLJ06c0K5du1RbW6vMzExJ0iuvvKKHH35YL774okaOHHkDpwMAAAaCkAPlej788EP5fD5lZ2fb2xISEjR58mRVV1dr1qxZqq6uVmJioh0nkpSdna3o6GgdPHhQf/mXf9lt3kAgoEAgYD/2+/2SpGAwqGAwGM5TsOdzRlthnTfSwv06AADCwzmof11PpD9eAyN1je2JsAaKz+eTJLnd7i7b3W63vc/n8yk5ObnrIgYPVlJSkj3mk8rLy7V8+fJu2ysrKxUfHx+OpXdTltkRkXkjZefOnX29BADANay8p69X0Hterzes812+fLnHY8MaKJFSWlqqkpIS+7Hf71dqaqpyc3PlcrnC+lzBYFBer1dLDkUr0BEV1rkj6eiyvL5eAgDgGu5a9nZfLyFkzmhLZZkdysnJkcPhCNu8ne+A9ERYAyUlJUWS1NjYqBEjRtjbGxsbNXHiRHtMU1NTl+Pa2tp0/vx5+/hPcjqdcjqd3bY7HI6wvnBXC3REKdDefwIlUq8DAODG9KdrySeF+zobylxh/XdQxowZo5SUFFVVVdnb/H6/Dh48KI/HI0nyeDxqbm5WXV2dPeadd95RR0eHJk+eHM7lAACAfirkOygXL17U7373O/vxhx9+qMOHDyspKUlpaWlasGCBnnvuOd1xxx0aM2aMlixZopEjR+qRRx6RJI0fP14PPfSQnnjiCa1bt07BYFDz5s3TrFmz+AYPAACQ1ItAOXTokL785S/bjzs/G1JYWKiNGzfq6aef1qVLl/Tkk0+qublZ9957r3bt2qXY2Fj7mDfeeEPz5s3Tgw8+qOjoaBUUFOjll18Ow+kAAICBIORAeeCBB2RZn/6VqaioKK1YsUIrVqz41DFJSUnavHlzqE8NAABuEvwWDwAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4/RpoFRUVGj06NGKjY3V5MmTVVNT05fLAQAAhuizQPnXf/1XlZSU6Nlnn9V7772nL37xi8rLy1NTU1NfLQkAABiizwJl1apVeuKJJ/TNb35T6enpWrduneLj4/WTn/ykr5YEAAAMMbgvnrS1tVV1dXUqLS21t0VHRys7O1vV1dXdxgcCAQUCAftxS0uLJOn8+fMKBoNhXVswGNTly5c1OBit9o6osM4dSefOnevrJQAArmFw26W+XkLIBndYuny5Q+fOnZPD4QjbvBcuXJAkWZb12WsI27OG4Pe//73a29vldru7bHe73Xr//fe7jS8vL9fy5cu7bR8zZkzE1tjf3PpSX68AADCQPBrBuS9cuKCEhITrjumTQAlVaWmpSkpK7McdHR06f/68hg8frqio8N7l8Pv9Sk1N1enTp+VyucI6NwAA/UGkroWWZenChQsaOXLkZ47tk0C59dZbNWjQIDU2NnbZ3tjYqJSUlG7jnU6nnE5nl22JiYmRXKJcLheBAgC4qUXiWvhZd0469cmHZGNiYpSRkaGqqip7W0dHh6qqquTxePpiSQAAwCB99hZPSUmJCgsLlZmZqXvuuUdr1qzRpUuX9M1vfrOvlgQAAAzRZ4Hyta99Tf/7v/+rpUuXyufzaeLEidq1a1e3D87+X3M6nXr22We7vaUEAMDNwoRrYZTVk+/6AAAA/B/it3gAAIBxCBQAAGAcAgUAABiHQAmj0aNHa82aNX29DAAAunnggQe0YMGCsM757rvvKioqSs3NzWGdVyJQAACAgQgUAABgnAEZKBcuXNDs2bM1ZMgQjRgxQqtXr+5ya+vjjz/WY489pltuuUXx8fGaNm2aTp061WWOn//85/rCF74gp9Op0aNH66WXuv4aX1NTk6ZPn664uDiNGTNGb7zxxv/V6QEA0CttbW2aN2+eEhISdOutt2rJkiX2Lwv/8z//szIzMzVs2DClpKTo0UcfVVNTU5fjd+7cqTvvvFNxcXH68pe/rI8++ihiax2QgVJSUqJ9+/Zp27Zt8nq9+tWvfqX33nvP3v/444/r0KFD2rZtm6qrq2VZlh5++GEFg0FJUl1dnb761a9q1qxZOnLkiJYtW6YlS5Zo48aNXeY4ffq0fvnLX+rNN9/U2rVru/2PBADAJJs2bdLgwYNVU1OjH/zgB1q1apX+6Z/+SZIUDAZVVlam3/72t9q6das++ugjPf744/axp0+f1syZMzV9+nQdPnxY3/rWt/Td7343cou1Bhi/3285HA5ry5Yt9rbm5mYrPj7eeuqpp6yTJ09akqx9+/bZ+3//+99bcXFx1s9+9jPLsizr0UcftXJycrrMu2jRIis9Pd2yLMuqr6+3JFk1NTX2/hMnTliSrNWrV0fw7AAA6J3777/fGj9+vNXR0WFve+aZZ6zx48dfc3xtba0lybpw4YJlWZZVWlpqXwevPl6S9fHHH4d9vQPuDsoHH3ygYDCoe+65x96WkJCgsWPHSpJOnDihwYMHa/Lkyfb+4cOHa+zYsTpx4oQ9ZsqUKV3mnTJlik6dOqX29nZ7joyMDHv/uHHjIv4LywAA3IisrCxFRUXZjz0ej31tq6ur0/Tp05WWlqZhw4bp/vvvlyQ1NDRI+sO18eprZ+fxkTLgAgUAAITmypUrysvLk8vl0htvvKHa2lq99dZbkqTW1tY+WdOAC5Q/+ZM/kcPhUG1trb2tpaVFJ0+elCSNHz9ebW1tOnjwoL3/3Llzqq+vV3p6uj1m3759Xebdt2+f7rzzTg0aNEjjxo1TW1ub6urq7P319fUR+R44AADhcvW1T5IOHDigO+64Q++//77OnTunF154Qffdd5/GjRvX7XOV48ePV01NTbfjI2XABcqwYcNUWFioRYsW6Ze//KWOHTumuXPnKjo6WlFRUbrjjjs0Y8YMPfHEE/r1r3+t3/72t/rGN76hz33uc5oxY4Yk6e/+7u9UVVWlsrIynTx5Ups2bdIPf/hDfec735EkjR07Vg899JD++q//WgcPHlRdXZ2+9a1vKS4uri9PHQCA62poaFBJSYnq6+v1L//yL3rllVf01FNPKS0tTTExMXrllVf0wQcfaNu2bSorK+ty7Le//W2dOnVKixYtUn19vTZv3tzlyyNhF/ZPtRjA7/dbjz76qBUfH2+lpKRYq1atsu655x7ru9/9rmVZlnX+/Hlrzpw5VkJCghUXF2fl5eVZJ0+e7DLHm2++aaWnp1sOh8NKS0uz/uEf/qHL/rNnz1r5+fmW0+m00tLSrNdee80aNWoUH5IFABjp/vvvt/7mb/7G+va3v225XC7rlltusb73ve/ZH5rdvHmzNXr0aMvpdFoej8fatm2bJcn6zW9+Y8+xfft26/Of/7zldDqt++67z/rJT34SsQ/JRlnW//8C9AB26dIlfe5zn9NLL72kuXPn9vVyAADAZxjc1wuIhN/85jd6//33dc8996ilpUUrVqyQJPstHAAAYLYBGSiS9OKLL6q+vl4xMTHKyMjQr371K9166619vSwAANADN8VbPAAAoH8ZcN/iAQAA/R+BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADDO/wMaFFw0cENVrgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"favorable: {fairness_info['favorable_labels']}\")\n",
    "y.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fairness_info` tells us that `'good'` is in `favorable_labels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first protected attribute: {'feature': 'personal_status', 'reference_group': ['male div/sep', 'male mar/wid', 'male single']}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot: >"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAGdCAYAAAAVEKdkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0uElEQVR4nO3deXwV9b3/8XcSspDlkCaSBJQAli2UVRByKAhFSMSUgmCLiCyWipcbEEihlFt21Fi0xaW41oK9ShHrgiBCAhaoErYgspbtYsFCEgUhLOUkJN/fH/4y9Ri2wDkc4Pt6Ph4+Hs53Zr7znTOfM7zPnJmTIGOMEQAAgGWCAz0AAACAQCAEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsVC3QA7gc5eXlOnTokGJiYhQUFBTo4QAAgEtgjNGJEydUu3ZtBQcH/jrMdRmCDh06pDp16gR6GAAA4DIcPHhQt9xyS6CHcX2GoJiYGEnfvIgul8unfZeWlionJ0dpaWkKDQ31ad+wF3UFf6Cu4C/+qq3i4mLVqVPH+Xc80K7LEFTxFZjL5fJLCIqMjJTL5eKkAp+hruAP1BX8xd+1da3cyhL4L+QAAAACgBAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwErVAj2Aa1WzqcvkKQsK9DAu2edPZAR6CAAAXFe4EgQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFaqUgiaOnWqgoKCvP5r0qSJM//MmTPKzMxUfHy8oqOj1bdvXxUWFnr1ceDAAWVkZCgyMlIJCQkaN26czp4965u9AQAAuETVqrrCD37wAy1fvvw/HVT7TxdjxozRBx98oLfeeks1atTQiBEj1KdPH33yySeSpLKyMmVkZCgpKUlr1qzR4cOHNWjQIIWGhurxxx/3we4AAABcmiqHoGrVqikpKalS+/Hjx/Xqq69q3rx56tq1qyRpzpw5SklJ0dq1a5WamqqcnBzt2LFDy5cvV2Jiolq1aqUZM2Zo/Pjxmjp1qsLCwq58jwAAAC5BlUPQnj17VLt2bUVERMjtdis7O1vJycnKz89XaWmpunXr5izbpEkTJScnKy8vT6mpqcrLy1Pz5s2VmJjoLJOenq7hw4dr+/btat269Tm36fF45PF4nOni4mJJUmlpqUpLS6u6CxdU0V94sPFpv/7m69cBvlVxfDhO8CXqCv7ir9q61mq1SiGoffv2mjt3rho3bqzDhw9r2rRp6tSpk7Zt26aCggKFhYUpNjbWa53ExEQVFBRIkgoKCrwCUMX8innnk52drWnTplVqz8nJUWRkZFV24ZLNaFvul379ZcmSJYEeAi5Bbm5uoIeAGxB1BX/xdW2dPn3ap/1dqSqFoB49ejj/36JFC7Vv315169bVggULVL16dZ8PrsKECROUlZXlTBcXF6tOnTpKS0uTy+Xy6bZKS0uVm5urSRuD5SkP8mnf/rRtanqgh4ALqKir7t27KzQ0NNDDwQ2CuoK/+Ku2Kr7JuVZU+euwb4uNjVWjRo20d+9ede/eXSUlJTp27JjX1aDCwkLnHqKkpCStX7/eq4+Kp8fOdZ9RhfDwcIWHh1dqDw0N9dsb31MeJE/Z9ROCOAFeH/xZs7AXdQV/8XVtXWt1ekW/E3Ty5Ent27dPtWrVUps2bRQaGqoVK1Y483ft2qUDBw7I7XZLktxut7Zu3aqioiJnmdzcXLlcLjVt2vRKhgIAAFAlVboSNHbsWPXs2VN169bVoUOHNGXKFIWEhKh///6qUaOGhg4dqqysLMXFxcnlcmnkyJFyu91KTU2VJKWlpalp06YaOHCgZs6cqYKCAk2cOFGZmZnnvNIDAADgL1UKQV988YX69++vI0eOqGbNmurYsaPWrl2rmjVrSpJmzZql4OBg9e3bVx6PR+np6Xr++eed9UNCQrR48WINHz5cbrdbUVFRGjx4sKZPn+7bvQIAALiIKoWg+fPnX3B+RESEZs+erdmzZ593mbp16/IkEwAACDj+dhgAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACtdUQh64oknFBQUpNGjRzttZ86cUWZmpuLj4xUdHa2+ffuqsLDQa70DBw4oIyNDkZGRSkhI0Lhx43T27NkrGQoAAECVXHYI2rBhg1566SW1aNHCq33MmDFatGiR3nrrLa1atUqHDh1Snz59nPllZWXKyMhQSUmJ1qxZo9dee01z587V5MmTL38vAAAAquiyQtDJkyc1YMAAvfLKK/re977ntB8/flyvvvqqfv/736tr165q06aN5syZozVr1mjt2rWSpJycHO3YsUOvv/66WrVqpR49emjGjBmaPXu2SkpKfLNXAAAAF1HtclbKzMxURkaGunXrpkcffdRpz8/PV2lpqbp16+a0NWnSRMnJycrLy1Nqaqry8vLUvHlzJSYmOsukp6dr+PDh2r59u1q3bl1pex6PRx6Px5kuLi6WJJWWlqq0tPRyduG8KvoLDzY+7dfffP06wLcqjg/HCb5EXcFf/FVb11qtVjkEzZ8/X5s2bdKGDRsqzSsoKFBYWJhiY2O92hMTE1VQUOAs8+0AVDG/Yt65ZGdna9q0aZXac3JyFBkZWdVduCQz2pb7pV9/WbJkSaCHgEuQm5sb6CHgBkRdwV98XVunT5/2aX9Xqkoh6ODBgxo1apRyc3MVERHhrzFVMmHCBGVlZTnTxcXFqlOnjtLS0uRyuXy6rdLSUuXm5mrSxmB5yoN82rc/bZuaHugh4AIq6qp79+4KDQ0N9HBwg6Cu4C/+qq2Kb3KuFVUKQfn5+SoqKtJtt93mtJWVlWn16tX6wx/+oGXLlqmkpETHjh3zuhpUWFiopKQkSVJSUpLWr1/v1W/F02MVy3xXeHi4wsPDK7WHhob67Y3vKQ+Sp+z6CUGcAK8P/qxZ2Iu6gr/4urautTqt0o3Rd955p7Zu3arNmzc7/7Vt21YDBgxw/j80NFQrVqxw1tm1a5cOHDggt9stSXK73dq6dauKioqcZXJzc+VyudS0aVMf7RYAAMCFVelKUExMjJo1a+bVFhUVpfj4eKd96NChysrKUlxcnFwul0aOHCm3263U1FRJUlpampo2baqBAwdq5syZKigo0MSJE5WZmXnOqz0AAAD+cFlPh13IrFmzFBwcrL59+8rj8Sg9PV3PP/+8Mz8kJESLFy/W8OHD5Xa7FRUVpcGDB2v69Om+HgoAAMB5XXEIWrlypdd0RESEZs+erdmzZ593nbp16/I0EwAACCj+dhgAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxULdADAK4n9X79wWWtFx5iNLOd1GzqMnnKgnw8qgv7/ImMq7o9ALhecCUIAABYqUoh6IUXXlCLFi3kcrnkcrnkdrv14YcfOvPPnDmjzMxMxcfHKzo6Wn379lVhYaFXHwcOHFBGRoYiIyOVkJCgcePG6ezZs77ZGwAAgEtUpRB0yy236IknnlB+fr42btyorl27qlevXtq+fbskacyYMVq0aJHeeustrVq1SocOHVKfPn2c9cvKypSRkaGSkhKtWbNGr732mubOnavJkyf7dq8AAAAuokr3BPXs2dNr+rHHHtMLL7ygtWvX6pZbbtGrr76qefPmqWvXrpKkOXPmKCUlRWvXrlVqaqpycnK0Y8cOLV++XImJiWrVqpVmzJih8ePHa+rUqQoLC/PdngEAAFzAZd8TVFZWpvnz5+vUqVNyu93Kz89XaWmpunXr5izTpEkTJScnKy8vT5KUl5en5s2bKzEx0VkmPT1dxcXFztUkAACAq6HKT4dt3bpVbrdbZ86cUXR0tN599101bdpUmzdvVlhYmGJjY72WT0xMVEFBgSSpoKDAKwBVzK+Ydz4ej0cej8eZLi4uliSVlpaqtLS0qrtwQRX9hQcbn/brb75+HXBu4SGXVxcV9RSIuqI2blwVx5ZjDF/zV21da7Va5RDUuHFjbd68WcePH9df//pXDR48WKtWrfLH2BzZ2dmaNm1apfacnBxFRkb6ZZsz2pb7pV9/WbJkSaCHYIWZ7a5s/UDUFbVx48vNzQ30EHCD8nVtnT592qf9Xakqh6CwsDA1aNBAktSmTRtt2LBBzzzzjPr166eSkhIdO3bM62pQYWGhkpKSJElJSUlav369V38VT49VLHMuEyZMUFZWljNdXFysOnXqKC0tTS6Xq6q7cEGlpaXKzc3VpI3B8pRf3d9zuRLbpqYHeghWaDZ12WWtFx5sNKNteUDqitq4cVWcr7p3767Q0NBADwc3EH/VVsU3OdeKK/6xxPLycnk8HrVp00ahoaFasWKF+vbtK0natWuXDhw4ILfbLUlyu9167LHHVFRUpISEBEnfpEyXy6WmTZuedxvh4eEKDw+v1B4aGuq3N76nPOiq/6jdleAEeHVcaU0Eoq6ojRufP8+FsJuva+taq9MqhaAJEyaoR48eSk5O1okTJzRv3jytXLlSy5YtU40aNTR06FBlZWUpLi5OLpdLI0eOlNvtVmpqqiQpLS1NTZs21cCBAzVz5kwVFBRo4sSJyszMPGfIAQAA8JcqhaCioiINGjRIhw8fVo0aNdSiRQstW7ZM3bt3lyTNmjVLwcHB6tu3rzwej9LT0/X8888764eEhGjx4sUaPny43G63oqKiNHjwYE2fPt23ewUAAHARVQpBr7766gXnR0REaPbs2Zo9e/Z5l6lbty43agIAgIDjb4cBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFipSiEoOztbt99+u2JiYpSQkKDevXtr165dXsucOXNGmZmZio+PV3R0tPr27avCwkKvZQ4cOKCMjAxFRkYqISFB48aN09mzZ698bwAAAC5RlULQqlWrlJmZqbVr1yo3N1elpaVKS0vTqVOnnGXGjBmjRYsW6a233tKqVat06NAh9enTx5lfVlamjIwMlZSUaM2aNXrttdc0d+5cTZ482Xd7BQAAcBHVqrLw0qVLvabnzp2rhIQE5efn64477tDx48f16quvat68eerataskac6cOUpJSdHatWuVmpqqnJwc7dixQ8uXL1diYqJatWqlGTNmaPz48Zo6darCwsJ8t3cAAADnUaUQ9F3Hjx+XJMXFxUmS8vPzVVpaqm7dujnLNGnSRMnJycrLy1Nqaqry8vLUvHlzJSYmOsukp6dr+PDh2r59u1q3bl1pOx6PRx6Px5kuLi6WJJWWlqq0tPRKdqGSiv7Cg41P+/U3X78OOLfwkMuri4p6CkRdURs3ropjyzGGr/mrtq61Wr3sEFReXq7Ro0frhz/8oZo1ayZJKigoUFhYmGJjY72WTUxMVEFBgbPMtwNQxfyKeeeSnZ2tadOmVWrPyclRZGTk5e7CBc1oW+6Xfv1lyZIlgR6CFWa2u7L1A1FX1MaNLzc3N9BDwA3K17V1+vRpn/Z3pS47BGVmZmrbtm36+OOPfTmec5owYYKysrKc6eLiYtWpU0dpaWlyuVw+3VZpaalyc3M1aWOwPOVBPu3bn7ZNTQ/0EKzQbOqyy1ovPNhoRtvygNQVtXHjqjhfde/eXaGhoYEeDm4g/qqtim9yrhWXFYJGjBihxYsXa/Xq1brllluc9qSkJJWUlOjYsWNeV4MKCwuVlJTkLLN+/Xqv/iqeHqtY5rvCw8MVHh5eqT00NNRvb3xPeZA8ZddPCOIEeHVcaU0Eoq6ojRufP8+FsJuva+taq9MqPR1mjNGIESP07rvv6qOPPlL9+vW95rdp00ahoaFasWKF07Zr1y4dOHBAbrdbkuR2u7V161YVFRU5y+Tm5srlcqlp06ZXsi8AAACXrEpXgjIzMzVv3jwtXLhQMTExzj08NWrUUPXq1VWjRg0NHTpUWVlZiouLk8vl0siRI+V2u5WamipJSktLU9OmTTVw4EDNnDlTBQUFmjhxojIzM895tQcAAMAfqhSCXnjhBUlSly5dvNrnzJmjIUOGSJJmzZql4OBg9e3bVx6PR+np6Xr++eedZUNCQrR48WINHz5cbrdbUVFRGjx4sKZPn35lewIAAFAFVQpBxlz88d6IiAjNnj1bs2fPPu8ydevW5YkVAAAQUPztMAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAVqpyCFq9erV69uyp2rVrKygoSO+9957XfGOMJk+erFq1aql69erq1q2b9uzZ47XM0aNHNWDAALlcLsXGxmro0KE6efLkFe0IAABAVVQ5BJ06dUotW7bU7Nmzzzl/5syZevbZZ/Xiiy9q3bp1ioqKUnp6us6cOeMsM2DAAG3fvl25ublavHixVq9erWHDhl3+XgAAAFRRtaqu0KNHD/Xo0eOc84wxevrppzVx4kT16tVLkvTnP/9ZiYmJeu+993Tfffdp586dWrp0qTZs2KC2bdtKkp577jndfffdeuqpp1S7du0r2B0AAIBLU+UQdCH79+9XQUGBunXr5rTVqFFD7du3V15enu677z7l5eUpNjbWCUCS1K1bNwUHB2vdunW65557KvXr8Xjk8Xic6eLiYklSaWmpSktLfbkLTn/hwcan/fqbr18HnFt4yOXVRUU9BaKuqI0bV8Wx5RjD1/xVW9darfo0BBUUFEiSEhMTvdoTExOdeQUFBUpISPAeRLVqiouLc5b5ruzsbE2bNq1Se05OjiIjI30x9EpmtC33S7/+smTJkkAPwQoz213Z+oGoK2rjxpebmxvoIeAG5evaOn36tE/7u1I+DUH+MmHCBGVlZTnTxcXFqlOnjtLS0uRyuXy6rdLSUuXm5mrSxmB5yoN82rc/bZuaHughWKHZ1GWXtV54sNGMtuUBqStq48ZVcb7q3r27QkNDAz0c3ED8VVsV3+RcK3wagpKSkiRJhYWFqlWrltNeWFioVq1aOcsUFRV5rXf27FkdPXrUWf+7wsPDFR4eXqk9NDTUb298T3mQPGXXTwjiBHh1XGlNBKKuqI0bnz/PhbCbr2vrWqtTn/5OUP369ZWUlKQVK1Y4bcXFxVq3bp3cbrckye1269ixY8rPz3eW+eijj1ReXq727dv7cjgAAADnVeUrQSdPntTevXud6f3792vz5s2Ki4tTcnKyRo8erUcffVQNGzZU/fr1NWnSJNWuXVu9e/eWJKWkpOiuu+7SQw89pBdffFGlpaUaMWKE7rvvPp4MAwAAV02VQ9DGjRv1ox/9yJmuuFdn8ODBmjt3rn71q1/p1KlTGjZsmI4dO6aOHTtq6dKlioiIcNZ54403NGLECN15550KDg5W37599eyzz/pgdwAAuLbU+/UHgR5ClYWHmCt+EOR6UOUQ1KVLFxlz/sd8g4KCNH36dE2fPv28y8TFxWnevHlV3TQAAIDP8LfDAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlaoFegAAcCOo9+sPrvo2w0OMZraTmk1dJk9ZUJXX//yJDD+MCrh+cCUIAABYiRAEAACsRAgCAABWIgQBAAArEYIAAICVCEEAAMBKhCAAAGAlQhAAALASIQgAAFiJEAQAAKxECAIAAFYiBAEAACsRggAAgJUIQQAAwEqEIAAAYCVCEAAAsBIhCAAAWIkQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABYiRAEAACsRAgCAABWCmgImj17turVq6eIiAi1b99e69evD+RwAACARQIWgt58801lZWVpypQp2rRpk1q2bKn09HQVFRUFakgAAMAiAQtBv//97/XQQw/pwQcfVNOmTfXiiy8qMjJSf/rTnwI1JAAAYJFqgdhoSUmJ8vPzNWHCBKctODhY3bp1U15eXqXlPR6PPB6PM338+HFJ0tGjR1VaWurTsZWWlur06dOqVhqssvIgn/btT0eOHAn0EKxQ7eypy1uv3Oj06fKA1BW1cXVcbm1c0TavsK6ojasjELVxpSpq68iRIwoNDfVZvydOnJAkGWN81ueVCEgI+uqrr1RWVqbExESv9sTERP3jH/+otHx2dramTZtWqb1+/fp+G+P15qbfBXoEuJj7A7RdauPGdiV1RW3gQvx5zjpx4oRq1Kjhxy1cmoCEoKqaMGGCsrKynOny8nIdPXpU8fHxCgry7afq4uJi1alTRwcPHpTL5fJp37AXdQV/oK7gL/6qLWOMTpw4odq1a/uszysRkBB00003KSQkRIWFhV7thYWFSkpKqrR8eHi4wsPDvdpiY2P9OUS5XC5OKvA56gr+QF3BX/xRW9fCFaAKAbkxOiwsTG3atNGKFSuctvLycq1YsUJutzsQQwIAAJYJ2NdhWVlZGjx4sNq2bat27drp6aef1qlTp/Tggw8GakgAAMAiAQtB/fr105dffqnJkyeroKBArVq10tKlSyvdLH21hYeHa8qUKZW+fgOuBHUFf6Cu4C+21FaQuVaeUwMAALiK+NthAADASoQgAABgJUIQAACw0g0XgoYMGaLevXv7rf8uXbpo9OjRPu1z5cqVCgoK0rFjx3za77XAGKNhw4YpLi5OQUFB2rx5c8DG8vnnn/tlDHPnzvX63aqpU6eqVatWPt2G5J/aw9Xlr/NTvXr19PTTTzvTQUFBeu+993y+HeBGc8OFIH975513NGPGjEAP47qxdOlSzZ07V4sXL9bhw4fVrFmzQA/J78aOHev1G1iXatWqVapTp44fRnRxr732mjp27BiQbcP3Dh8+rB49elR5vR/96Ef64x//6IcRwdf8/YH/WnSxcH85H3QJQVUUFxenmJiYQA/jurFv3z7VqlVLHTp0UFJSkqpVuy7+UssViY6OVnx8fJXXW7hwoXr27OmHEV3atn/yk59c1W2WlJRc1e3ZJCkpqcqPNh89elSffPJJwGoQ9qhfv76WL19e5fUuN9xfSMBCUJcuXTRy5EiNHj1a3/ve95SYmKhXXnnF+cHEmJgYNWjQQB9++KGzTllZmYYOHar69eurevXqaty4sZ555pkLbqe8vFzZ2dnOOi1bttRf//rXC67z/PPPq2HDhoqIiFBiYqLuvfder3F/+yuJevXq6fHHH9fPf/5zxcTEKDk5WS+//LJXf2vWrFGrVq0UERGhtm3b6r333rtoWv3444/VqVMnVa9eXXXq1NEjjzyiU6eur79EPGTIEI0cOVIHDhxQUFCQ6tWrJ+nix6Ti68Fly5apdevWql69urp27aqioiJ9+OGHSklJkcvl0v3336/Tp0876y1dulQdO3ZUbGys4uPj9eMf/1j79u274Bi3bdumHj16KDo6WomJiRo4cKC++uqrC64zd+5cJScnKzIyUvfcc0+lv8T97a/DcnJyFBERUemrzlGjRqlr165ebe+//74TRE6dOqVBgwYpOjpatWrV0u9+V/kvXXo8Ho0dO1Y333yzoqKi1L59e61cudJrnLGxsXrvvfecek5PT9fBgwe9+jlz5oxycnKcbderV0+PPvqos/26devq/fff15dffqlevXopOjpaLVq00MaNG50+jhw5ov79++vmm29WZGSkmjdvrr/85S9e2+nSpYtGjBih0aNH66abblJ6evoFX+dAupbPT0VFRerZs6eqV6+u+vXr64033qi0zLc/MXfo0EHjx4/3mv/ll18qNDRUq1evdto++OAD3XbbbUpMTNTXX3+tAQMGqGbNmqpevboaNmyoOXPmOMsePHhQP/vZzxQbG6u4uDj16tVLn3/+uTO/4irFtGnTVLNmTblcLv3Xf/0XwddS3z7uW7Zs0ddff63OnTtXuZ/LCfcXZQKkc+fOJiYmxsyYMcPs3r3bzJgxw4SEhJgePXqYl19+2ezevdsMHz7cxMfHm1OnThljjCkpKTGTJ082GzZsMP/3f/9nXn/9dRMZGWnefPNNp9/BgwebXr16OdOPPvqoadKkiVm6dKnZt2+fmTNnjgkPDzcrV64857g2bNhgQkJCzLx588znn39uNm3aZJ555hmvcY8aNcqZrlu3romLizOzZ882e/bsMdnZ2SY4ONj84x//MMYYc/z4cRMXF2ceeOABs337drNkyRLTqFEjI8l8+umnxhhj/va3vxlJ5uuvvzbGGLN3714TFRVlZs2aZXbv3m0++eQT07p1azNkyBAfvPJXz7Fjx8z06dPNLbfcYg4fPmyKioqMMRc/JhWvR2pqqvn444/Npk2bTIMGDUznzp1NWlqa2bRpk1m9erWJj483TzzxhLO9v/71r+btt982e/bsMZ9++qnp2bOnad68uSkrKzPGGLN//36v1/3rr782NWvWNBMmTDA7d+40mzZtMt27dzc/+tGPzrtPa9euNcHBwea3v/2t2bVrl3nmmWdMbGysqVGjhrPMlClTTMuWLY0xxpw9e9YkJiaaP/7xj878c7Vt27bNxMTEGI/HY4wxZvjw4SY5OdksX77cbNmyxfz4xz82MTExXrX3i1/8wnTo0MGsXr3a7N271zz55JMmPDzc7N692xhjzJw5c0xoaKhp27atWbNmjdm4caNp166d6dChg9c+LV682DRq1MiZrqjpF1980Xkfulwuc9ddd5kFCxaYXbt2md69e5uUlBRTXl5ujDHmiy++ME8++aT59NNPzb59+8yzzz5rQkJCzLp165x+O3fubKKjo824cePMP/7xD+c9ci26Vs9PxhjTo0cP07JlS5OXl2c2btxoOnToYKpXr25mzZrlLCPJvPvuu8YYY/7whz+Y5ORk51gZY8xzzz1Xqe3ee+81jz/+uDHGmMzMTNOqVSuzYcMGs3//fpObm2vef/99Zz9TUlLMz3/+c7NlyxazY8cOc//995vGjRs79Tt48GATHR1t+vXrZ7Zt22YWL15satasaf7nf/7n8g7Idaxz585mxIgRZtSoUSY2NtYkJCSYl19+2Zw8edIMGTLEREdHm+9///tmyZIlzjpnz541P//5z029evVMRESEadSokXn66ae9+v1uLZWVlZnHH3/cWadFixbmrbfeuuDY6tata2bMmGEGDhxooqKiTHJyslm4cKEpKioyP/nJT0xUVJRp3ry52bBhg7POV199Ze677z5Tu3ZtU716ddOsWTMzb968SvucmZlpRo0aZeLj402XLl2cedOnTzf9+vUz5eXl5qabbvIaY8uWLU1SUpIz/fe//92EhYU577Fv17Uxxqxbt860atXKhIeHmzZt2ph33nnH6xx/KQIagjp27OhMnz171kRFRZmBAwc6bYcPHzaSTF5e3nn7yczMNH379nWmv10YZ86cMZGRkWbNmjVe6wwdOtT079//nP29/fbbxuVymeLi4vOO+7sh6IEHHnCmy8vLTUJCgnnhhReMMca88MILJj4+3vz73/92lnnllVcuGIKGDh1qhg0b5rXdv//97yY4ONirn+vBrFmzTN26dZ3pSzkmFa/H8uXLnfnZ2dlGktm3b5/T9vDDD5v09PTzbvvLL780kszWrVuNMZVD0IwZM0xaWprXOgcPHjSSzK5du87ZZ//+/c3dd9/t1davX7/zhiBjjBk1apTp2rWrM71s2TITHh7uHG9jjHnsscfMvffea4wx5sSJEyYsLMwsWLDAmX/kyBFTvXp1p/b++c9/mpCQEPOvf/3Layx33nmnmTBhgjHmmxAkyaxdu9aZv3PnTiPJK5w89NBDZuzYsc70d2u64n04adIkpy0vL89IMocPHz7n62SMMRkZGeaXv/ylM925c2fTunXr8y5/LblWz0+7du0yksz69eudtopjer4QVFRUZKpVq2ZWr17tzHe73Wb8+PHO9JkzZ0x0dLTZtm2bMcaYnj17mgcffPCcY/jf//1f07hxY68A5fF4TPXq1c2yZcuc/YyLi3P+8TLmm3NhdHS086HEFtdyoA7EB562bds6oalPnz4mMzPTGGPM0aNHTVhYmKlRo4bZuXOns08//OEPnXW/XdcnTpwwNWvWNPfff7/Ztm2bWbRokbn11lurHIICeoNGixYtnP8PCQlRfHy8mjdv7rRV/AmNoqIip2327Nn605/+pAMHDujf//63SkpKzvskzt69e3X69Gl1797dq72kpEStW7c+5zrdu3dX3bp1deutt+quu+7SXXfdpXvuuUeRkZGXtB9BQUFKSkpyxrxr1y61aNFCERERzjLt2rU7b1+S9Nlnn2nLli1el7mNMSovL9f+/fuVkpJywfWvZVU5Jt9+XRMTExUZGalbb73Vq239+vXO9J49ezR58mStW7dOX331lcrLyyVJBw4cOOcN2Z999pn+9re/KTo6utK8ffv2qVGjRpXad+7cqXvuucerze12a+nSpefd5wEDBig1NVWHDh1S7dq19cYbbygjI8PribKFCxdqxIgRzrZLSkrUvn17Z35cXJwaN27sTG/dulVlZWWVxujxeLzuR6pWrZpuv/12Z7pJkyaKjY3Vzp071a5dOxljtGjRIi1YsMCrn+++9pLO+95MSkpSWVmZHn/8cS1YsED/+te/VFJSIo/HU+l906ZNm/O+Tteaa/H8tHPnTlWrVs3rdaw4pudTs2ZNpaWl6Y033lCnTp20f/9+5eXl6aWXXnKW+eijj5SQkKAf/OAHkqThw4erb9++2rRpk9LS0tS7d2916NBB0jfvm71791a6N/LMmTNeXz+3bNnS6/i73W6dPHlSBw8eVN26dc873htRy5YtNXHiREnShAkT9MQTT+imm27SQw89JEmaPHmyXnjhBW3ZskWpqakKDQ3VtGnTnPXr16+vvLw8LViwQD/72c8q9e/xePT4449r+fLlzh8hv/XWW/Xxxx/rpZdeuuBXT3fffbcefvhhr3Hcfvvt+ulPfypJGj9+vNxutwoLC5WUlKSbb75ZY8eOddYfOXKkli1bpgULFnj929awYUPNnDnTa1v/+te/tGXLFue+ni5dujh1uHr1arVu3VpJSUlauXKlmjRpopUrV5537PPmzVN5ebleffVVRURE6Ac/+IG++OILDR8+/Lz7ei4BDUGhoaFe00FBQV5tQUFBkuT8YzZ//nyNHTtWv/vd7+R2uxUTE6Mnn3xS69atO2f/J0+elPTNd90333yz17zzfa8YExOjTZs2aeXKlcrJydHkyZM1depUbdiw4bwnmnPtR8WYL8fJkyf18MMP65FHHqk0Lzk5+bL7vRZU5Zh8txYu9jr37NlTdevW1SuvvKLatWurvLxczZo1O+99CCdPnlTPnj3129/+ttK8WrVqVW3HLuD222/X97//fc2fP1/Dhw/Xu+++q7lz5zrzDx8+rE8//VQZGRmX3OfJkycVEhKi/Px8hYSEeM07V6g7n/Xr1+vs2bPOP3AVzvU+vNB788knn9Qzzzyjp59+Ws2bN1dUVJRGjx5d6bWPioq65LEF2rV4frpcAwYM0COPPKLnnntO8+bNU/Pmzb0C3bfvR5OkHj166J///KeWLFmi3Nxc3XnnncrMzNRTTz2lkydPqk2bNue8F6lmzZo+HfeN4loM1Ocam78/8Lz//vvOfZuS1LlzZ40aNUpffvmlVq1apS5dujghaOjQoVqzZo1+9atfnXPcO3furHSBoSIAVsV19ajOJ598og4dOui///u/nbYL3fjatGlThYeH68CBA1W6CatatWrq1q2bunXrpilTpig2NlYfffSR+vTpU+UxN27cWK+//ro8Ho9zYtuwYcMF17ntttu0Y8cONWjQoMrbu9Zd7jG5mCNHjmjXrl165ZVX1KlTJ0nf3Fx+Ibfddpvefvtt1atX75KfWktJSan0j9ratWsvut6AAQP0xhtv6JZbblFwcLBX4Fm0aJE6dOiguLg4SdL3v/99hYaGat26dU7o/frrr7V7927nNWvdurXKyspUVFTk7O+5nD17Vhs3bnQ+oe3atUvHjh1zriYuXLhQGRkZlYJUVX3yySfq1auXHnjgAUnfBIPdu3eradOmV9Tv9eRqnJ+aNGmis2fPKj8/37nCV3FML6RXr14aNmyYli5dqnnz5mnQoEHOvIqrga+//rrXOjVr1tTgwYM1ePBgderUSePGjdNTTz2l2267TW+++aYSEhLkcrnOu83PPvtM//73v1W9enVJ37xPoqOjA/YzEIF0LQfqq/mB57thu3nz5oqLi9OqVau0atUqPfbYY0pKStJvf/tbbdiwQaWlpZU+oPnadfWIfMOGDbVx40YtW7ZMu3fv1qRJky4YKGJiYjR27FiNGTNGr732mvbt26dNmzbpueee02uvvXbOdRYvXqxnn31Wmzdv1j//+U/9+c9/Vnl5uddXEVVx//33q7y8XMOGDdPOnTu1bNkyPfXUU5L+U1zfNX78eK1Zs0YjRozQ5s2btWfPHq+vS65nl3NMLsX3vvc9xcfH6+WXX9bevXv10UcfKSsr64LrZGZm6ujRo+rfv782bNigffv2admyZXrwwQdVVlZ2znUeeeQRLV26VE899ZT27NmjP/zhDxf8KqzCgAEDtGnTJj322GO69957vU5M3z0xREdHa+jQoRo3bpw++ugjbdu2TUOGDFFw8H/ero0aNdKAAQM0aNAgvfPOO9q/f7/Wr1+v7OxsffDBB85yoaGhGjlypNatW6f8/HwNGTJEqampTij67rYvV8OGDZWbm6s1a9Zo586devjhh1VYWHjF/V5Prsb5qXHjxrrrrrv08MMPO8f0F7/4hRM0zicqKkq9e/fWpEmTtHPnTvXv39+Zl5+fr9OnT3v9TtTkyZO1cOFC7d27V9u3b9fixYud4DxgwADddNNN6tWrl/7+979r//79WrlypR555BF98cUXTh8lJSUaOnSoduzYoSVLlmjKlCkaMWKEVx3j3L4dqFu3bq0GDRpccqBu0KCB13++Dp3f/sDTsmVL3Xrrrdq9e/dF1zt58qT+9re/qVevXk5bUFCQOnXqpIULF2r79u3q2LGjWrRoIY/Ho5deeklt27Y979XjlJQUbdmyRWfOnHHaLuUD6XddV9X48MMPq0+fPurXr5/at2+vI0eOeH3qOpcZM2Zo0qRJys7OVkpKiu666y598MEHql+//jmXj42N1TvvvKOuXbsqJSVFL774ov7yl78435VXlcvl0qJFi7R582a1atVKv/nNbzR58mRJ8rqM920tWrTQqlWrtHv3bnXq1EmtW7fW5MmTVbt27csaw7WmqsfkUgQHB2v+/PnKz89Xs2bNNGbMGD355JMXXKd27dr65JNPVFZWprS0NDVv3lyjR49WbGzseU/UqampeuWVV/TMM8+oZcuWysnJcb7rv5AGDRqoXbt22rJliwYMGOC0nzp1SitWrKgURJ588kl16tRJPXv2VLdu3dSxY8dKl5fnzJmjQYMG6Ze//KUaN26s3r17a8OGDV5fmUZGRmr8+PG6//779cMf/lDR0dF68803JX1zlWLv3r0+eVR94sSJuu2225Senu5c0rbth9yuxvlJ+ua4165dW507d1afPn00bNgwJSQkXHR8AwYM0GeffaZOnTp51cjChQt19913e10NDQsL04QJE9SiRQvdcccdCgkJ0fz58yV9U1OrV69WcnKy+vTpo5SUFA0dOlRnzpzxujJ05513qmHDhrrjjjvUr18//eQnP9HUqVMvOk5cnUB9JWO7nA88S5cuVaNGjZyfSqnQpUsX/eUvf1GrVq0UHR2t4OBg3XHHHXrjjTcueIX0/vvvV1BQkB566CEnaFdcYKiSS76FGj7z+uuvm9DQUHP69OlADwUB9vbbb5uUlBS/9D1nzhyvp9a+63e/+53p0aOHX7aN60fz5s29njryhe8+uWSz7z5RbMw3T2V9+2k+Y7yffDpz5owZMmSIqVGjhomNjTXDhw83v/71r72eOv3ua1xeXm6efvpp07hxYxMaGmpq1qxp0tPTzapVq847touNw5jKT9UeOXLE9OrVy0RHR5uEhAQzceJEM2jQIK+xnGufH3jgAfOb3/ym0hg+/fRTI8nracVZs2YZSWbp0qUXHFteXp5p2bKlCQsLM61atTJvv/12lZ8OC/r/HcOP/vznP+vWW2/VzTffrM8++0wjRoxQly5dKn0HD/vk5OTI4/H45Vd6586dq9GjR5/3fpEFCxaoVq1aF7ynCDe2kpISZWdnKysry6e/hD9kyBAdO3aMv18GSd/cm5iYmKgPP/zwok9HX23X1Y3R16uCggJNnjxZBQUFqlWrln7605/qscceC/SwcA1IS0sL2LbP9agt7BIWFqYpU6YEehi4wR09elRjxozx+rmOawVXggAAgJWuqxujAQAAfIUQBAAArEQIAgAAViIEAQAAKxGCAACAlQhBAADASoQgAABgJUIQAACwEiEIAABY6f8BLUl3aIF1HFAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"first protected attribute: {fairness_info['protected_attributes'][0]}\")\n",
    "X[\"personal_status\"].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fairness_info` tells us that one of the\n",
    "`protected_attributes` is `personal_status`, and that values\n",
    "in `['male div/sep', 'male mar/wid', 'male single']`\n",
    "indicate members of the privileged group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "second protected attribute: {'feature': 'age', 'reference_group': [[26, 1000]]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot: >"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAHNCAYAAAC+QxloAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAn7UlEQVR4nO3df3RU9Z3/8deEDBNimEQiJGFJAH9goBSsQWAWtwoEspRDoeS4ttpTRW2PNlgg1RVOqwRbS7RVV89GcC0buttmcWkXlSKENJawKkESFxXXzYINwppfrTYJP8wwzXy+f+w3sw0JSSaTfCaTeT7OmSNz75173/edm+TlJ/OZ6zDGGAEAAFgSE+4CAABAdCF8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMCq2HAXcDG/36+6ujqNHj1aDocj3OUAAIA+MMbozJkzGj9+vGJieh7bGHLho66uTunp6eEuAwAA9MPp06c1YcKEHrcZcuFj9OjRkv63eLfbbfXYPp9P+/fv1+LFi+V0Oq0eezigf6Gjh6Gjh6Ghf6GL1h62trYqPT098Hu8J0MufHT8qcXtdoclfMTHx8vtdkfVBTNQ6F/o6GHo6GFo6F/oor2HfXnLBG84BQAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgVWy4C4Bdk9bv6XH9ycKllioBAEQrRj4AAIBVhA8AAGBVUOGjoKBADoej0yMzMzOwvq2tTXl5eUpOTlZCQoJyc3PV2Ng44EUDAIDIFfTIx+c+9znV19cHHq+//npg3bp167R7927t3LlTFRUVqqur08qVKwe0YAAAENmCfsNpbGysUlNTuyxvaWnRtm3bVFJSogULFkiSiouLNXXqVFVWVmru3LmhVwsAACJe0OHj+PHjGj9+vOLi4uTxeLR582ZlZGSourpaPp9P2dnZgW0zMzOVkZGhQ4cOXTJ8eL1eeb3ewPPW1lZJks/nk8/nC7a8kHQcz/ZxbXKNMD2uD+Xco6F/g40eho4ehob+hS5aexjM+TqMMT3/Nvoze/fu1dmzZ3Xttdeqvr5emzZt0scff6xjx45p9+7dWrVqVacgIUmzZ8/W/Pnz9fjjj3e7z4KCAm3atKnL8pKSEsXHx/f5RAAAQPicP39et912m1paWuR2u3vcNqjwcbHm5mZNnDhRTz31lEaNGtWv8NHdyEd6err+8Ic/9Fr8QPP5fCorK9OiRYvkdDqtHnugTC8oDen1xwpy+v3a4dC/cKOHoaOHoaF/oYvWHra2tuqKK67oU/gI6UPGkpKSNGXKFJ04cUKLFi3ShQsX1NzcrKSkpMA2jY2N3b5HpIPL5ZLL5eqy3Ol0hu2LFs5jh8rb7gjp9QNx3pHcv6GCHoaOHoaG/oUu2noYzLmG9DkfZ8+e1Ycffqi0tDRlZWXJ6XSqvLw8sL6mpkanTp2Sx+MJ5TAAAGAYCWrk44EHHtCyZcs0ceJE1dXVaePGjRoxYoS+9rWvKTExUXfffbfy8/M1ZswYud1u3X///fJ4PMx0AQAAAUGFj//5n//R1772NX3yyScaO3asbrzxRlVWVmrs2LGSpKeffloxMTHKzc2V1+tVTk6OnnvuuUEpHAAARKagwseOHTt6XB8XF6eioiIVFRWFVBQAABi+uLcLAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKtCurcLgjdp/Z4e158sXGqpEgAAwoORDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWBUb7gLQ2aT1e3pcf7JwqaVKAAAYHIx8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqptqiE6b6AgAGGyMfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKu5qG2F6u+ssAABDHSMfAADAKsIHAACwKqTwUVhYKIfDobVr1waWtbW1KS8vT8nJyUpISFBubq4aGxtDrRMAAAwT/Q4fR44c0fPPP68ZM2Z0Wr5u3Trt3r1bO3fuVEVFherq6rRy5cqQCwUAAMNDv8LH2bNndfvtt+uFF17Q5ZdfHlje0tKibdu26amnntKCBQuUlZWl4uJivfnmm6qsrBywogEAQOTq12yXvLw8LV26VNnZ2frhD38YWF5dXS2fz6fs7OzAsszMTGVkZOjQoUOaO3dul315vV55vd7A89bWVkmSz+eTz+frT3n91nG8wTyua4QZtH3b0FNvbPRvuKOHoaOHoaF/oYvWHgZzvkGHjx07dujtt9/WkSNHuqxraGjQyJEjlZSU1Gl5SkqKGhoaut3f5s2btWnTpi7L9+/fr/j4+GDLGxBlZWWDtu8nZg/arq149dVXe91mMPsXLehh6OhhaOhf6KKth+fPn+/ztkGFj9OnT2vNmjUqKytTXFxc0IV1Z8OGDcrPzw88b21tVXp6uhYvXiy32z0gx+grn8+nsrIyLVq0SE6nc1COMb2gdFD2a8uxgpxLruvo38NVMfL6Hf3ahw29fQ3CWZ+Na3C4o4ehoX+hi9Yedvzloi+CCh/V1dVqamrS9ddfH1jW3t6ugwcP6u///u9VWlqqCxcuqLm5udPoR2Njo1JTU7vdp8vlksvl6rLc6XSG7Ys2mMf2tl/6l3Ik6EtfvH5Hj+cZ7m/G3r4G4a6vo4ahUEcko4ehoX+hi7YeBnOuQYWPhQsX6r333uu0bNWqVcrMzNRDDz2k9PR0OZ1OlZeXKzc3V5JUU1OjU6dOyePxBHMoAAAwTAUVPkaPHq3p06d3WnbZZZcpOTk5sPzuu+9Wfn6+xowZI7fbrfvvv18ej6fbN5sCAIDoM+D3dnn66acVExOj3Nxceb1e5eTk6LnnnhvowwAAgAgVcvg4cOBAp+dxcXEqKipSUVFRqLsGAADDEHe1BS7S252DTxYutVQJAAxP3FgOAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYx1RbWMZUVAKIbIx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIqptghKT9NkXSOMnphtsRgAQERi5AMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVbHhLgC42KT1e3pcf7JwqaVKAACDgZEPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFjFVFtEHKbiAkBkY+QDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYx1RbDTm9TcQEA4cXIBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALAqqPCxZcsWzZgxQ263W263Wx6PR3v37g2sb2trU15enpKTk5WQkKDc3Fw1NjYOeNEAACByBRU+JkyYoMLCQlVXV6uqqkoLFizQ8uXL9f7770uS1q1bp927d2vnzp2qqKhQXV2dVq5cOSiFAwCAyBTUvV2WLVvW6fljjz2mLVu2qLKyUhMmTNC2bdtUUlKiBQsWSJKKi4s1depUVVZWau7cuQNXNQAAiFj9vrFce3u7du7cqXPnzsnj8ai6ulo+n0/Z2dmBbTIzM5WRkaFDhw5dMnx4vV55vd7A89bWVkmSz+eTz+frb3n90nG8wTyua4QZtH2HmyvGdPrvcDWY14eNa3C4o4ehoX+hi9YeBnO+DmNMUL8p3nvvPXk8HrW1tSkhIUElJSX60pe+pJKSEq1atapTkJCk2bNna/78+Xr88ce73V9BQYE2bdrUZXlJSYni4+ODKQ0AAITJ+fPnddttt6mlpUVut7vHbYMe+bj22mt19OhRtbS06Je//KXuuOMOVVRU9LvYDRs2KD8/P/C8tbVV6enpWrx4ca/FDzSfz6eysjItWrRITqdzUI4xvaB0UPY7FLhijH4wy6+Hq2Lk9TvCXc6gOVaQM2j7tnENDnf0MDT0L3TR2sOOv1z0RdDhY+TIkbr66qslSVlZWTpy5IieeeYZ3Xrrrbpw4YKam5uVlJQU2L6xsVGpqamX3J/L5ZLL5eqy3Ol0hu2LNpjH9rYP31/KHbx+x7A+TxvXZTiv/+GCHoaG/oUu2noYzLmG/Dkffr9fXq9XWVlZcjqdKi8vD6yrqanRqVOn5PF4Qj0MAAAYJoIa+diwYYOWLFmijIwMnTlzRiUlJTpw4IBKS0uVmJiou+++W/n5+RozZozcbrfuv/9+eTweZroAAICAoMJHU1OTvvGNb6i+vl6JiYmaMWOGSktLtWjRIknS008/rZiYGOXm5srr9SonJ0fPPffcoBQOAAAiU1DhY9u2bT2uj4uLU1FRkYqKikIqCgAADF/9/pwPIFpNWr+nx/UnC5daqgQAIhM3lgMAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFZxV9sB1tsdTwEAiHaMfAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKzixnLAEDS9oFTedke3604WLrVcDQAMLEY+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAVU22BATZp/Z4e1zNVFkC0Y+QDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYxV1tgWGGu+oCGOoY+QAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVUy1BSzraSqsa4TRE7MtFgMAYcDIBwAAsIrwAQAArAoqfGzevFk33HCDRo8erXHjxmnFihWqqanptE1bW5vy8vKUnJyshIQE5ebmqrGxcUCLBgAAkSuo8FFRUaG8vDxVVlaqrKxMPp9Pixcv1rlz5wLbrFu3Trt379bOnTtVUVGhuro6rVy5csALBwAAkSmoN5zu27ev0/Pt27dr3Lhxqq6u1he/+EW1tLRo27ZtKikp0YIFCyRJxcXFmjp1qiorKzV37tyBqxwAAESkkGa7tLS0SJLGjBkjSaqurpbP51N2dnZgm8zMTGVkZOjQoUPdhg+v1yuv1xt43traKkny+Xzy+XyhlBe0juOFclzXCDNQ5UQcV4zp9F8Ery897O367O0atP19ZdtAfB9HM/oXumjtYTDn6zDG9Os3hd/v15e//GU1Nzfr9ddflySVlJRo1apVncKEJM2ePVvz58/X448/3mU/BQUF2rRpU5flJSUlio+P709pAADAsvPnz+u2225TS0uL3G53j9v2e+QjLy9Px44dCwSP/tqwYYPy8/MDz1tbW5Wenq7Fixf3WvxA8/l8Kisr06JFi+R0Ovu1j+kFpQNcVeRwxRj9YJZfD1fFyOt3hLuciNSXHh4ryOlxH71dg729PtINxPdxNKN/oYvWHnb85aIv+hU+Vq9erV//+tc6ePCgJkyYEFiempqqCxcuqLm5WUlJSYHljY2NSk1N7XZfLpdLLpery3Kn0xm2L1oox/a280vX63fQhxD11MPers3eeh8tPwzD+TNkOKB/oYu2HgZzrkHNdjHGaPXq1dq1a5dee+01TZ48udP6rKwsOZ1OlZeXB5bV1NTo1KlT8ng8wRwKAAAMU0GNfOTl5amkpEQvv/yyRo8erYaGBklSYmKiRo0apcTERN19993Kz8/XmDFj5Ha7df/998vj8TDTBQAASAoyfGzZskWSdPPNN3daXlxcrDvvvFOS9PTTTysmJka5ubnyer3KycnRc889NyDFAgCAyBdU+OjLxJi4uDgVFRWpqKio30UBAIDhi3u7AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCq3zeWA4DuTFq/p8f1JwuXWqoEwFDFyAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArGKqLRBhepvKOtivD3WqLFNxATDyAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIobywEISqg3pgMARj4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBVTbbsxvaBU3nZHuMsA0I3epvq6Rhg9MdtSMQD6hZEPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFXe1BTCk9HbX2r7q6e7UJwuXDsgxAPQPIx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAqqDDx8GDB7Vs2TKNHz9eDodDL730Uqf1xhg98sgjSktL06hRo5Sdna3jx48PVL0AACDCBR0+zp07p5kzZ6qoqKjb9U888YSeffZZbd26VYcPH9Zll12mnJwctbW1hVwsAACIfEHf22XJkiVasmRJt+uMMfq7v/s7ff/739fy5cslSf/0T/+klJQUvfTSS/rqV78aWrUAACDiDeiN5Wpra9XQ0KDs7OzAssTERM2ZM0eHDh3qNnx4vV55vd7A89bWVkmSz+eTz+cbyPJ61XE8V4yxetzhoqNv9K//6GHo+tJD2z9bIklHb+hR/0VrD4M53wENHw0NDZKklJSUTstTUlIC6y62efNmbdq0qcvy/fv3Kz4+fiDL67MfzPKH5bjDBf0LHT0MXU89fPXVVy1WEpnKysrCXULEi7Yenj9/vs/bDmj46I8NGzYoPz8/8Ly1tVXp6elavHix3G631Vp8Pp/Kysr0cFWMvP7ub8WNS3PFGP1glp/+hYAehq4vPTxWkDOoNUwvKO1x/WAfPxQdPwcXLVokp9MZ7nIiUrT2sOMvF30xoOEjNTVVktTY2Ki0tLTA8sbGRl133XXdvsblcsnlcnVZ7nQ6w/ZF8/od8rbzg7+/6F/o6GHoeurhYP9s6e1rFwm/kML5M3i4iLYeBnOuA/o5H5MnT1ZqaqrKy8sDy1pbW3X48GF5PJ6BPBQAAIhQQY98nD17VidOnAg8r62t1dGjRzVmzBhlZGRo7dq1+uEPf6hrrrlGkydP1sMPP6zx48drxYoVA1k3AACIUEGHj6qqKs2fPz/wvOP9GnfccYe2b9+uv/3bv9W5c+f0rW99S83Nzbrxxhu1b98+xcXFDVzVAAAgYgUdPm6++WYZc+kpbA6HQ48++qgeffTRkAoDAADDU9hnuwDAUDNp/Z4e158sXGqpku4N9fqA3nBjOQAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYxVRbAFGnt6mq4T7+cJ8qG+3nD0Y+AACAZYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAVU20BIEjhnqobbkyVRagY+QAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVUy1BQB0Mb2gVN52R7jLQD8N9enQjHwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKuY7QIAw8xQn+nQm8G+cd9QP/9owMgHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwirvaAsAQM9h3de1p/64RRk/MHtTDh12o/eWuuKFj5AMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVkXdVNton2IGAINtsKcKI/Ix8gEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAq6Juqi0AADZMLyiVt93R7bpovzMuIx8AAMAqwgcAALBq0MJHUVGRJk2apLi4OM2ZM0dvvfXWYB0KAABEkEEJHy+++KLy8/O1ceNGvf3225o5c6ZycnLU1NQ0GIcDAAARZFDCx1NPPaVvfvObWrVqlaZNm6atW7cqPj5e//iP/zgYhwMAABFkwGe7XLhwQdXV1dqwYUNgWUxMjLKzs3Xo0KEu23u9Xnm93sDzlpYWSdKnn34qn8830OUp9k/nLr3Ob3T+vF+xvhi1+7t/hzIujf6Fjh6Gjh6Ghv717pNPPulxvc/n0/nz53vsYW/7CFVPv+sG6/hnzpyRJBljet/YDLCPP/7YSDJvvvlmp+UPPvigmT17dpftN27caCTx4MGDBw8ePIbB4/Tp071mhbB/zseGDRuUn58feO73+/Xpp58qOTlZDofd1N3a2qr09HSdPn1abrfb6rGHA/oXOnoYOnoYGvoXumjtoTFGZ86c0fjx43vddsDDxxVXXKERI0aosbGx0/LGxkalpqZ22d7lcsnlcnValpSUNNBlBcXtdkfEBVNeXq4XX3xRW7duVUzM0Jk1HSn9G8roYejoYWjoX+iisYeJiYl92m7Af2ONHDlSWVlZKi8vDyzz+/0qLy+Xx+MZ6MNFrfb2dn3rW9/SCy+8oFdeeSXc5QAA0GeD8r/L+fn5euGFF/Szn/1MH3zwge677z6dO3dOq1atGozDRaUdO3bod7/7nRwOhx5++OG+vcEHAIAhYFDe83Hrrbfq97//vR555BE1NDTouuuu0759+5SSkjIYhxswLpdLGzdu7PJnoKGmvb1djzzyiBwOh4wxOnbsmF5++WWtWLEirHVFSv+GMnoYOnoYGvoXOnrYO4fhf5kjzi9+8Qt9/etfDzyPiYnRtGnT9O6771p/ky4AAMEifESY9vZ2TZkyRbW1tV3+1LJr166wj34AANCboTNFAn3S8V6Pi4NHTEwM7/0AAEQERj4iSE+jHh0Y/QAADHWMfESQS416dIiJidH3v/99+f1+y5UBANB3URc+Nm/erBtuuEGjR4/WuHHjtGLFCtXU1HTapq2tTXl5eUpOTlZCQoJyc3O7fGiabX8+w+VS/H6/3n///UH/3I8tW7ZoxowZgQ/Q8Xg82rt3b2D9UOzfUFZYWCiHw6G1a9cGltHDnhUUFMjhcHR6ZGZmBtbTv959/PHH+vrXv67k5GSNGjVKn//851VVVRVYb4zRI488orS0NI0aNUrZ2dk6fvx4GCseWiZNmtTlGnQ4HMrLy5PENdibqAsfFRUVysvLU2VlpcrKyuTz+bR48WKdO/d/N+FZt26ddu/erZ07d6qiokJ1dXVauXJlGKvufdSjg433fkyYMEGFhYWqrq5WVVWVFixYoOXLl+v999+XNDT7N1QdOXJEzz//vGbMmNFpOT3s3ec+9znV19cHHq+//npgHf3r2R//+EfNmzdPTqdTe/fu1X/+53/qySef1OWXXx7Y5oknntCzzz6rrVu36vDhw7rsssuUk5Ojtra2MFY+dBw5cqTT9VdWViZJuuWWWyRxDfaqvzeQGy6ampqMJFNRUWGMMaa5udk4nU6zc+fOwDYffPCBkWQOHToUlhr/9Kc/mSuvvNI4HI4+39hn165dVmu8/PLLzU9/+tMh2b+h6syZM+aaa64xZWVl5qabbjJr1qwxxgzNa3Co2bhxo5k5c2a36+hf7x566CFz4403XnK93+83qamp5sc//nFgWXNzs3G5XOZf/uVfbJQYcdasWWOuuuoq4/f7uQb7IOpGPi7W0tIiSRozZowkqbq6Wj6fT9nZ2YFtMjMzlZGRoUOHDoWlxr6OenSwOfOlvb1dO3bs0Llz5+TxeIZk/4aqvLw8LV26tFOvpKF5DQ5Fx48f1/jx43XllVfq9ttv16lTpyTRv7545ZVXNGvWLN1yyy0aN26cvvCFL+iFF14IrK+trVVDQ0OnHiYmJmrOnDn0sBsXLlzQz3/+c911111yOBxcg30Q1eHD7/dr7dq1mjdvnqZPny5Jamho0MiRI7vc3C4lJUUNDQ3Wa+zLez0u5vf7A596Oljee+89JSQkyOVy6d5779WuXbs0bdq0Ide/oWrHjh16++23tXnz5i7r6GHv5syZo+3bt2vfvn3asmWLamtr9Vd/9Vc6c+YM/euD3/3ud9qyZYuuueYalZaW6r777tN3vvMd/exnP5OkQJ8u/lRqeti9l156Sc3Nzbrzzjsl8T3cF4Py8eqRIi8vT8eOHev0t+KhpmPUI1gdox/Lly8flE89vfbaa3X06FG1tLTol7/8pe644w5VVFQM+HGGo9OnT2vNmjUqKytTXFxcuMuJSEuWLAn8e8aMGZozZ44mTpyof/3Xf9WoUaPCWFlk8Pv9mjVrln70ox9Jkr7whS/o2LFj2rp1q+64444wVxd5tm3bpiVLlvTpVvL4X1E78rF69Wr9+te/1m9/+1tNmDAhsDw1NVUXLlxQc3Nzp+0bGxuVmppqtcb+jHp0GOzRj5EjR+rqq69WVlaWNm/erJkzZ+qZZ54ZUv0bqqqrq9XU1KTrr79esbGxio2NVUVFhZ599lnFxsYqJSWFHgYpKSlJU6ZM0YkTJ7gG+yAtLU3Tpk3rtGzq1KmBP1119Oni2Rn0sKuPPvpIv/nNb3TPPfcElnEN9i7qwocxRqtXr9auXbv02muvafLkyZ3WZ2Vlyel0qry8PLCspqZGp06dksfjsVprsO/1uJjNO976/X55vd4h1b+hauHChXrvvfd09OjRwGPWrFm6/fbbA/+mh8E5e/asPvzwQ6WlpXEN9sG8efO6fMTAf//3f2vixImSpMmTJys1NbVTD1tbW3X48GF6eJHi4mKNGzdOS5cuDSzjGuyDsL7dNQzuu+8+k5iYaA4cOGDq6+sDj/Pnzwe2uffee01GRoZ57bXXTFVVlfF4PMbj8VivddmyZX2e3dLT47/+678GtK7169ebiooKU1tba959912zfv1643A4zP79+40xQ6d/keTPZ7sYQw97893vftccOHDA1NbWmjfeeMNkZ2ebK664wjQ1NRlj6F9v3nrrLRMbG2see+wxc/z4cfOLX/zCxMfHm5///OeBbQoLC01SUpJ5+eWXzbvvvmuWL19uJk+ebD777LMwVj60tLe3m4yMDPPQQw91Wcc12LOoCx+X+gVdXFwc2Oazzz4z3/72t83ll19u4uPjzVe+8hVTX19vvdbf//73pqKiotPj2WefveQ5zJ07t8v2b7/99oDXddddd5mJEyeakSNHmrFjx5qFCxcGgocxQ6d/keTi8EEPe3brrbeatLQ0M3LkSPMXf/EX5tZbbzUnTpwIrKd/vdu9e7eZPn26cblcJjMz0/zDP/xDp/V+v988/PDDJiUlxbhcLrNw4UJTU1MTpmqHptLSUiOp275wDfaMe7tEmIMHD+qmm27qdt2XvvQl7dmzx3JFAAAEJ+re8wEAAMKL8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8RJjU1FS5XK5u11155ZWWqwEAIHgOY4wJdxEIjt/vV3dftpiYGDkcjjBUBABA3xE+AACAVfzZBQAAWEX4AAAAVhE+AACAVYQPAABgFeFjmDpw4IAcDoeam5utHnf79u1KSkoKaR8nT56Uw+HQ0aNHL7lNuM4PABA6wscwcfPNN2vt2rXhLgMAgF4RPhBw4cKFcJcAAIgChI9h4M4771RFRYWeeeYZORwOORwOnTx5UpJUXV2tWbNmKT4+Xn/5l3+pmpqawOsKCgp03XXX6ac//akmT56suLg4SVJzc7PuuecejR07Vm63WwsWLNA777wTeN0777yj+fPna/To0XK73crKylJVVVWnmkpLSzV16lQlJCTor//6r1VfXx9Y5/f79eijj2rChAlyuVy67rrrtG/fvh7P8dVXX9WUKVM0atQozZ8/P3B+AIDIQ/gYBp555hl5PB5985vfVH19verr65Weni5J+t73vqcnn3xSVVVVio2N1V133dXptSdOnNCvfvUr/du//VvgPRa33HKLmpqatHfvXlVXV+v666/XwoUL9emnn0qSbr/9dk2YMEFHjhxRdXW11q9fL6fTGdjn+fPn9ZOf/ET//M//rIMHD+rUqVN64IEHOtX75JNP6ic/+Yneffdd5eTk6Mtf/rKOHz/e7fmdPn1aK1eu1LJly3T06FHdc889Wr9+/UC2EABgk8GwcNNNN5k1a9YEnv/2t781ksxvfvObwLI9e/YYSeazzz4zxhizceNG43Q6TVNTU2Cbf//3fzdut9u0tbV12v9VV11lnn/+eWOMMaNHjzbbt2/vto7i4mIjyZw4cSKwrKioyKSkpASejx8/3jz22GOdXnfDDTeYb3/728YYY2pra40k8x//8R/GGGM2bNhgpk2b1mn7hx56yEgyf/zjH3tqCwBgCGLkY5ibMWNG4N9paWmSpKampsCyiRMnauzYsYHn77zzjs6ePavk5GQlJCQEHrW1tfrwww8lSfn5+brnnnuUnZ2twsLCwPIO8fHxuuqqqzodt+OYra2tqqur07x58zq9Zt68efrggw+6PYcPPvhAc+bM6bTM4/H0uQcAgKElNtwFYHD9+Z9DOm465/f7A8suu+yyTtufPXtWaWlpOnDgQJd9dUyhLSgo0G233aY9e/Zo79692rhxo3bs2KGvfOUrXY7ZcVzDLYQAAP8fIx/DxMiRI9Xe3h7yfq6//no1NDQoNjZWV199dafHFVdcEdhuypQpWrdunfbv36+VK1equLi4T/t3u90aP3683njjjU7L33jjDU2bNq3b10ydOlVvvfVWp2WVlZVBnhkAYKggfAwTkyZN0uHDh3Xy5En94Q9/6DS6EYzs7Gx5PB6tWLFC+/fv18mTJ/Xmm2/qe9/7nqqqqvTZZ59p9erVOnDggD766CO98cYbOnLkiKZOndrnYzz44IN6/PHH9eKLL6qmpkbr16/X0aNHtWbNmm63v/fee3X8+HE9+OCDqqmpUUlJibZv396v8wMAhB/hY5h44IEHNGLECE2bNk1jx47VqVOn+rUfh8OhV199VV/84he1atUqTZkyRV/96lf10UcfKSUlRSNGjNAnn3yib3zjG5oyZYr+5m/+RkuWLNGmTZv6fIzvfOc7ys/P13e/+119/vOf1759+/TKK6/ommuu6Xb7jIwM/epXv9JLL72kmTNnauvWrfrRj37Ur/MDAISfw/DHeAAAYBEjHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKz6f34ES+eRqfdhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"second protected attribute: {fairness_info['protected_attributes'][1]}\")\n",
    "ax = X.age.hist(bins=range(int(X.age.min()), int(X.age.max()) + 1))\n",
    "ax.annotate(\"threshold\", xy=(25.5, 0), xytext=(25.5, -10),\n",
    "            arrowprops=dict(facecolor='black', shrink=0.05))\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fairness_info` tells us that another one of the\n",
    "`protected_attributes` is `age`, and that values in the range\n",
    "`[26, 1000]` indicate members of the privileged group.\n",
    "To understand the overall distribution of data rows by positive\n",
    "outcomes and membership in the privileged group for each attribute, we\n",
    "can use the helper function `count_fairness_groups`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>y_true</th>\n",
       "      <th>personal_status</th>\n",
       "      <th>age</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">0</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>47</td>\n",
       "      <td>0.447619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>62</td>\n",
       "      <td>0.302439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>0</th>\n",
       "      <td>33</td>\n",
       "      <td>0.388235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>158</td>\n",
       "      <td>0.261157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">1</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>58</td>\n",
       "      <td>0.552381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>143</td>\n",
       "      <td>0.697561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>0</th>\n",
       "      <td>52</td>\n",
       "      <td>0.611765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>447</td>\n",
       "      <td>0.738843</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            count     ratio\n",
       "y_true personal_status age                 \n",
       "0      0               0       47  0.447619\n",
       "                       1       62  0.302439\n",
       "       1               0       33  0.388235\n",
       "                       1      158  0.261157\n",
       "1      0               0       58  0.552381\n",
       "                       1      143  0.697561\n",
       "       1               0       52  0.611765\n",
       "                       1      447  0.738843"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lf.count_fairness_groups(X, y, **fairness_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, for `y_true 0` (unfavorable outcome), `personal_status 0`\n",
    "(unprivileged gender), and `age 0` (unprivileged age), the dataset\n",
    "contains 47 samples.\n",
    "In contrast, for `y_true 1` (favorable outcome), `personal_status 0`\n",
    "(unprivileged gender), and `age 0` (unprivileged age), the dataset\n",
    "contains 58 samples.\n",
    "That means that among young females, the ratio of favorable outcomes is\n",
    "58 / (47 + 58) = 0.552, which is a worse ratio than for other groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  5.2 Metrics\n",
    "\n",
    "Lale provides several [fairness metrics](https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.html#metrics) from AIF360.\n",
    "Which metric to choose depends on the values and policies of you and\n",
    "your organization.\n",
    "For illustration purposes, this notebook demonstrates\n",
    "[disparate impact](https://doi.org/10.1145/2783258.2783311).\n",
    "We first configure it with the `fairness_info`, then use it to score\n",
    "the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "disparate impact 0.75\n"
     ]
    }
   ],
   "source": [
    "di_scorer = lf.disparate_impact(**fairness_info)\n",
    "print(f\"disparate impact {di_scorer.score_data(X=X, y_pred=y):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A score of 0.75 means that members of the unprivileged group\n",
    "experience a positive outcome only 75% as often as members of the\n",
    "privileged group.\n",
    "This is unfair; for instance, it falls short of the 80% rule in US Law.\n",
    "We can train a model that has less bias than the dataset.\n",
    "For instance, a [DummyClassifier](https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.dummy_classifier.html#lale.lib.sklearn.dummy_classifier.DummyClassifier)\n",
    "ignores its input and always returns the majority label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dummy DI 1.00\n",
      "['good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good', 'good']\n"
     ]
    }
   ],
   "source": [
    "from lale.lib.sklearn import DummyClassifier\n",
    "trained_dummy = DummyClassifier().fit(X, y)\n",
    "print(f\"dummy DI {di_scorer(trained_dummy, X, y):.2f}\")\n",
    "print(list(trained_dummy.predict(X))[:24])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `DummyClassifier` is trivially perfectly fair: irrespective of\n",
    "whether or not an individual is a member of the privileged group, the\n",
    "predicted label is always `'good'`.\n",
    "On the other hand, it is poor at predicting the ground truth labels.\n",
    "To better evaluate `DummyClassifier` as well as other models, we\n",
    "create a few more metrics directly from scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.metrics\n",
    "acc_scorer = sklearn.metrics.make_scorer(sklearn.metrics.accuracy_score)\n",
    "rc0_scorer = sklearn.metrics.make_scorer(sklearn.metrics.recall_score, pos_label=\"bad\")\n",
    "rc1_scorer = sklearn.metrics.make_scorer(sklearn.metrics.recall_score, pos_label=\"good\")\n",
    "bal_scorer = sklearn.metrics.make_scorer(sklearn.metrics.balanced_accuracy_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define a utility function `get_metrics` that uses\n",
    "cross-validation to evaluate a given estimator against a variety of\n",
    "metrics.\n",
    "For better stability, we are using a cross-validation split that\n",
    "stratifies the data by both fairness outcomes and groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate\n",
    "fair_cv = lf.FairStratifiedKFold(**fairness_info, n_splits=3)\n",
    "\n",
    "def get_metrics(estimator):\n",
    "    scores = cross_validate(estimator, X, y, cv=fair_cv, scoring={\n",
    "        \"disparate impact\": di_scorer,\n",
    "        \"balanced accuracy\": bal_scorer,\n",
    "        \"recall (unfavorable)\": rc0_scorer,\n",
    "        \"recall (favorable)\": rc1_scorer,\n",
    "        \"accuracy\": acc_scorer,\n",
    "    })\n",
    "    result = pd.DataFrame.from_dict(scores)\n",
    "    mean, std = result.mean(), result.std()\n",
    "    result.loc[\"mean\"] = mean\n",
    "    result.loc[\"std\"] = std\n",
    "    return result\n",
    "\n",
    "pd.options.display.float_format = \"{:.2f}\".format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at all these metrics for `DummyClassifier` as a baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_disparate impact</th>\n",
       "      <th>test_balanced accuracy</th>\n",
       "      <th>test_recall (unfavorable)</th>\n",
       "      <th>test_recall (favorable)</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.05</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.04</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_disparate impact  test_balanced accuracy  \\\n",
       "0         0.00        0.03                   1.00                    0.50   \n",
       "1         0.00        0.03                   1.00                    0.50   \n",
       "2         0.00        0.05                   1.00                    0.50   \n",
       "mean      0.00        0.04                   1.00                    0.50   \n",
       "std       0.00        0.01                   0.00                    0.00   \n",
       "\n",
       "      test_recall (unfavorable)  test_recall (favorable)  test_accuracy  \n",
       "0                          0.00                     1.00           0.70  \n",
       "1                          0.00                     1.00           0.70  \n",
       "2                          0.00                     1.00           0.70  \n",
       "mean                       0.00                     1.00           0.70  \n",
       "std                        0.00                     0.00           0.00  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_metrics(trained_dummy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rows `0`, `1`, and `2` represent the three cross-validation folds, and\n",
    "the rows `mean` and `std` show their average and standard deviation.\n",
    "Columns `fit_time` and `score_time` show that `DummyClassifier` is\n",
    "fast, which is no surprise.\n",
    "We already knew that its disparate impact is 1.00, so it is\n",
    "perfectly fair.\n",
    "Its balanced accuracy (across positive and negative outcomes) is\n",
    "0.50, because it has a recall of 0.00 for unfavorable labels and a\n",
    "recall of 1.00 for favorable labels.\n",
    "Finally, its accuracy is 0.70, because the dataset is imbalanced,\n",
    "with 70% of instances having positive outcomes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Non-Mitigated Pipeline\n",
    "\n",
    "How well does a simple pipeline perform on this dataset if it does not\n",
    "use any algorithmic bias mitigators?\n",
    "First, let's create a preprocessing sub-pipeline to ensure all columns are\n",
    "numeric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: cluster:(root) Pages: 1 -->\n",
       "<svg width=\"295pt\" height=\"107pt\"\n",
       " viewBox=\"0.00 0.00 294.51 107.28\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 103.28)\">\n",
       "<title>cluster:(root)</title>\n",
       "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-103.28 290.51,-103.28 290.51,4 -4,4\"/>\n",
       "</a>\n",
       "</g>\n",
       "<!-- project_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>project_0</title>\n",
       "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.project.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"project_0 = Project(columns={&#39;type&#39;: &#39;number&#39;})\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"129.59\" cy=\"-81.28\" rx=\"27.2\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"129.59\" y=\"-78.48\" font-family=\"Times,serif\" font-size=\"11.00\">Project</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- concat_features -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>concat_features</title>\n",
       "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.concat_features.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"concat_features = ConcatFeatures\">\n",
       "<ellipse fill=\"white\" stroke=\"black\" cx=\"245.49\" cy=\"-54.28\" rx=\"41.02\" ry=\"19.6\"/>\n",
       "<text text-anchor=\"middle\" x=\"245.49\" y=\"-57.48\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"245.49\" y=\"-45.48\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- project_0&#45;&gt;concat_features -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>project_0&#45;&gt;concat_features</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M155.59,-75.36C168.14,-72.38 183.77,-68.68 198.39,-65.21\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.39,-68.57 208.31,-62.86 197.78,-61.76 199.39,-68.57\"/>\n",
       "</g>\n",
       "<!-- project_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>project_1</title>\n",
       "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.project.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"project_1 = Project(drop_columns={&#39;type&#39;: &#39;number&#39;})\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"27.35\" cy=\"-28.28\" rx=\"27.2\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27.35\" y=\"-25.48\" font-family=\"Times,serif\" font-size=\"11.00\">Project</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- one_hot_encoder -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>one_hot_encoder</title>\n",
       "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.one_hot_encoder.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"one_hot_encoder = OneHotEncoder(handle_unknown=&#39;ignore&#39;)\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"129.59\" cy=\"-28.28\" rx=\"38.78\" ry=\"28.07\"/>\n",
       "<text text-anchor=\"middle\" x=\"129.59\" y=\"-37.48\" font-family=\"Times,serif\" font-size=\"11.00\">One&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"129.59\" y=\"-25.48\" font-family=\"Times,serif\" font-size=\"11.00\">Hot&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"129.59\" y=\"-13.48\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- project_1&#45;&gt;one_hot_encoder -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>project_1&#45;&gt;one_hot_encoder</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M54.95,-28.28C62.79,-28.28 71.62,-28.28 80.37,-28.28\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.54,-31.78 90.54,-28.28 80.54,-24.78 80.54,-31.78\"/>\n",
       "</g>\n",
       "<!-- one_hot_encoder&#45;&gt;concat_features -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>one_hot_encoder&#45;&gt;concat_features</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M166.94,-36.58C176.91,-38.86 187.85,-41.35 198.29,-43.74\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"197.57,-47.16 208.1,-45.98 199.12,-40.34 197.57,-47.16\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fe9f01eafd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from lale.lib.lale import Project, ConcatFeatures\n",
    "from lale.lib.sklearn import OneHotEncoder, MinMaxScaler\n",
    "\n",
    "prefix = (\n",
    "    Project(columns={\"type\": \"number\"})\n",
    "    & (\n",
    "        Project(drop_columns={\"type\": \"number\"})\n",
    "        >> OneHotEncoder(handle_unknown=\"ignore\")\n",
    "    )\n",
    ") >> ConcatFeatures\n",
    "prefix.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's add a random forest classifier as the final estimator to\n",
    "this pipeline and evaluate its performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_disparate impact</th>\n",
       "      <th>test_balanced accuracy</th>\n",
       "      <th>test_recall (unfavorable)</th>\n",
       "      <th>test_recall (favorable)</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.34</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.66</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.51</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.64</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.96</td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.38</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.44</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.41</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.41</td>\n",
       "      <td>0.92</td>\n",
       "      <td>0.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.09</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_disparate impact  test_balanced accuracy  \\\n",
       "0         0.34        0.12                   0.73                    0.66   \n",
       "1         0.51        0.09                   0.64                    0.67   \n",
       "2         0.38        0.10                   0.79                    0.68   \n",
       "mean      0.41        0.10                   0.72                    0.67   \n",
       "std       0.09        0.02                   0.07                    0.01   \n",
       "\n",
       "      test_recall (unfavorable)  test_recall (favorable)  test_accuracy  \n",
       "0                          0.42                     0.89           0.75  \n",
       "1                          0.38                     0.96           0.78  \n",
       "2                          0.44                     0.91           0.77  \n",
       "mean                       0.41                     0.92           0.77  \n",
       "std                        0.03                     0.03           0.02  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lale.lib.sklearn import RandomForestClassifier\n",
    "trainable_nomit = prefix >> RandomForestClassifier()\n",
    "get_metrics(trainable_nomit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared to the `DummyClassifier`, the `RandomForestClassifier` has\n",
    "worse fairness but better predictive performance, as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4 Mitigators\n",
    "\n",
    "Lale provides several [bias mitigators](https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.html#pre-estimator-mitigation-operators) from AIF360.\n",
    "This notebook demonstrates three representative ones.\n",
    "The first one, [GerryFairClassifier](https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.gerry_fair_classifier.html#lale.lib.aif360.gerry_fair_classifier.GerryFairClassifier),\n",
    "is an in-estimator mitigator, meaning that it has its own built-in estimator.\n",
    "We configure it with the previously defined `fairness_info` and\n",
    "`prefix`, which is a data preparation sub-pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_disparate impact</th>\n",
       "      <th>test_balanced accuracy</th>\n",
       "      <th>test_recall (unfavorable)</th>\n",
       "      <th>test_recall (favorable)</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.92</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.74</td>\n",
       "      <td>0.71</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.74</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.63</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.40</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.22</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.64</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.29</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.54</td>\n",
       "      <td>0.85</td>\n",
       "      <td>0.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.59</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.16</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_disparate impact  test_balanced accuracy  \\\n",
       "0         1.92        0.45                   0.76                    0.74   \n",
       "1         0.74        0.30                   0.63                    0.65   \n",
       "2         1.22        0.29                   0.64                    0.70   \n",
       "mean      1.29        0.35                   0.68                    0.70   \n",
       "std       0.59        0.09                   0.07                    0.04   \n",
       "\n",
       "      test_recall (unfavorable)  test_recall (favorable)  test_accuracy  \n",
       "0                          0.71                     0.76           0.75  \n",
       "1                          0.40                     0.90           0.75  \n",
       "2                          0.52                     0.88           0.77  \n",
       "mean                       0.54                     0.85           0.76  \n",
       "std                        0.16                     0.07           0.01  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lale.lib.aif360 import GerryFairClassifier as InMit\n",
    "trainable_inmit = InMit(**fairness_info, preparation=prefix)\n",
    "get_metrics(trainable_inmit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second example, [DisparateImpactRemover](https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.disparate_impact_remover.html#lale.lib.aif360.disparate_impact_remover.DisparateImpactRemover),\n",
    "is a pre-estimator mitigator, meaning that it transforms the data to\n",
    "make it more fair before it is piped to an estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_disparate impact</th>\n",
       "      <th>test_balanced accuracy</th>\n",
       "      <th>test_recall (unfavorable)</th>\n",
       "      <th>test_recall (favorable)</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.59</td>\n",
       "      <td>0.22</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.44</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.62</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.64</td>\n",
       "      <td>0.34</td>\n",
       "      <td>0.94</td>\n",
       "      <td>0.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.54</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.94</td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.58</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.66</td>\n",
       "      <td>0.40</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.04</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_disparate impact  test_balanced accuracy  \\\n",
       "0         0.59        0.22                   1.00                    0.68   \n",
       "1         0.62        0.24                   0.87                    0.64   \n",
       "2         0.54        0.26                   0.93                    0.68   \n",
       "mean      0.58        0.24                   0.93                    0.66   \n",
       "std       0.04        0.02                   0.06                    0.02   \n",
       "\n",
       "      test_recall (unfavorable)  test_recall (favorable)  test_accuracy  \n",
       "0                          0.44                     0.91           0.77  \n",
       "1                          0.34                     0.94           0.76  \n",
       "2                          0.42                     0.94           0.78  \n",
       "mean                       0.40                     0.93           0.77  \n",
       "std                        0.05                     0.01           0.01  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lale.lib.aif360 import DisparateImpactRemover as PreMit\n",
    "trainable_premit = PreMit(**fairness_info, preparation=prefix) >> RandomForestClassifier()\n",
    "get_metrics(trainable_premit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third example, [CalibratedEqOddsPostprocessing](https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.calibrated_eq_odds_postprocessing.html#lale.lib.aif360.calibrated_eq_odds_postprocessing.CalibratedEqOddsPostprocessing),\n",
    "is a post-estimator mitigator, meaning that it transforms the predictions\n",
    "after they are returned from an estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_disparate impact</th>\n",
       "      <th>test_balanced accuracy</th>\n",
       "      <th>test_recall (unfavorable)</th>\n",
       "      <th>test_recall (favorable)</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.47</td>\n",
       "      <td>0.27</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.62</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.63</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.82</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.47</td>\n",
       "      <td>0.21</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.52</td>\n",
       "      <td>0.22</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.09</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_disparate impact  test_balanced accuracy  \\\n",
       "0         0.47        0.27                   0.86                    0.62   \n",
       "1         0.63        0.19                   0.82                    0.61   \n",
       "2         0.47        0.21                   0.83                    0.60   \n",
       "mean      0.52        0.22                   0.84                    0.61   \n",
       "std       0.09        0.04                   0.02                    0.01   \n",
       "\n",
       "      test_recall (unfavorable)  test_recall (favorable)  test_accuracy  \n",
       "0                          0.57                     0.67           0.64  \n",
       "1                          0.55                     0.67           0.63  \n",
       "2                          0.52                     0.68           0.63  \n",
       "mean                       0.55                     0.67           0.63  \n",
       "std                        0.03                     0.00           0.01  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lale.lib.aif360 import CalibratedEqOddsPostprocessing as PostMit\n",
    "trainable_postmit = PostMit(**fairness_info, estimator=prefix >> RandomForestClassifier())\n",
    "get_metrics(trainable_postmit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, the three demonstrated mitigators vary in their effectiveness\n",
    "at mitigating bias for this particular dataset and in the predictive\n",
    "performance of the mitigated pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5 AutoML with Mitigators\n",
    "\n",
    "We can create a search space with a choice of mitigated pipelines by\n",
    "simply using Lale's or-combinator (`|`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "planned_mit3 = trainable_inmit | trainable_premit | trainable_postmit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can use `auto_configure` to run AutoML on this search space.\n",
    "This will not just tweak the algorithm choices but also tune\n",
    "hyperparameters, including those of mitigators.\n",
    "As the optimization objective, we use a combined scorer that lets\n",
    "AutoML optimize for both balanced accuracy (for predictive\n",
    "performance) and disparate impact (for fairness)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:12<00:00,  2.46s/trial, best loss: -0.7614278773442624]\n",
      "CPU times: user 15.6 s, sys: 10.4 s, total: 26 s\n",
      "Wall time: 13.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from lale.lib.lale import Hyperopt\n",
    "comb_scorer = lf.balanced_accuracy_and_disparate_impact(**fairness_info)\n",
    "trained_mit3 = planned_mit3.auto_configure(\n",
    "    X, y, optimizer=Hyperopt, max_evals=5, cv=fair_cv, scoring=comb_scorer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the best (with respect to the combined scorer)\n",
    "pipeline AutoML encountered in its trials, and we can look at its\n",
    "metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: cluster:(root) Pages: 1 -->\n",
       "<svg width=\"468pt\" height=\"178pt\"\n",
       " viewBox=\"0.00 0.00 467.60 178.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 174)\">\n",
       "<title>cluster:(root)</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-174 463.6,-174 463.6,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster:(root)</title>\n",
       "<g id=\"a_clust1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.calibrated_eq_odds_postprocessing.html#lale.lib.aif360.calibrated_eq_odds_postprocessing.CalibratedEqOddsPostprocessing\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"(root) = PostMit(favorable_labels=[&#39;good&#39;], protected_attributes=[{&#39;name&#39;: &#39;lale.lib.aif360.calibrated_eq_odds_postprocessing.CalibratedEqOddsPostprocessing&#39;, &#39;reference_group&#39;: [&#39;male div/sep&#39;, &#39;male mar/wid&#39;, &#39;male single&#39;], &#39;feature&#39;: &#39;personal_status&#39;}, {&#39;name&#39;: &#39;l...)\">\n",
       "<polygon fill=\"white\" stroke=\"black\" points=\"8,-8 8,-162 451.6,-162 451.6,-8 8,-8\"/>\n",
       "<text text-anchor=\"middle\" x=\"229.8\" y=\"-146.8\" font-family=\"Times,serif\" font-size=\"14.00\">PostMit</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<g id=\"clust2\" class=\"cluster\">\n",
       "<title>cluster:pipeline</title>\n",
       "<g id=\"a_clust2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.aif360.calibrated_eq_odds_postprocessing.html#lale.lib.aif360.calibrated_eq_odds_postprocessing.CalibratedEqOddsPostprocessing\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"pipeline = ...\">\n",
       "<path fill=\"#b0e2ff\" stroke=\"black\" d=\"M28,-16C28,-16 431.6,-16 431.6,-16 437.6,-16 443.6,-22 443.6,-28 443.6,-28 443.6,-119 443.6,-119 443.6,-125 437.6,-131 431.6,-131 431.6,-131 28,-131 28,-131 22,-131 16,-125 16,-119 16,-119 16,-28 16,-28 16,-22 22,-16 28,-16\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- project -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>project</title>\n",
       "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.project.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"project = Project(columns={&#39;type&#39;: &#39;number&#39;})\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"153.59\" cy=\"-105\" rx=\"27.2\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.59\" y=\"-102.2\" font-family=\"Times,serif\" font-size=\"11.00\">Project</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- concat_features -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>concat_features</title>\n",
       "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.concat_features.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
       "<ellipse fill=\"white\" stroke=\"black\" cx=\"269.49\" cy=\"-73\" rx=\"41.02\" ry=\"19.6\"/>\n",
       "<text text-anchor=\"middle\" x=\"269.49\" y=\"-76.2\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"269.49\" y=\"-64.2\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- project&#45;&gt;concat_features -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>project&#45;&gt;concat_features</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M179.03,-98.14C192.11,-94.46 208.64,-89.82 223.92,-85.52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"225.05,-88.84 233.73,-82.77 223.15,-82.1 225.05,-88.84\"/>\n",
       "</g>\n",
       "<!-- project_0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>project_0</title>\n",
       "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.project.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"project_0 = Project(drop_columns={&#39;type&#39;: &#39;number&#39;})\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"51.35\" cy=\"-52\" rx=\"27.2\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"51.35\" y=\"-49.2\" font-family=\"Times,serif\" font-size=\"11.00\">Project</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- one_hot_encoder -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>one_hot_encoder</title>\n",
       "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.one_hot_encoder.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"one_hot_encoder = OneHotEncoder(handle_unknown=&#39;ignore&#39;)\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"153.59\" cy=\"-52\" rx=\"38.78\" ry=\"28.07\"/>\n",
       "<text text-anchor=\"middle\" x=\"153.59\" y=\"-61.2\" font-family=\"Times,serif\" font-size=\"11.00\">One&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"153.59\" y=\"-49.2\" font-family=\"Times,serif\" font-size=\"11.00\">Hot&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"153.59\" y=\"-37.2\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- project_0&#45;&gt;one_hot_encoder -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>project_0&#45;&gt;one_hot_encoder</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M78.95,-52C86.79,-52 95.62,-52 104.37,-52\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"104.54,-55.5 114.54,-52 104.54,-48.5 104.54,-55.5\"/>\n",
       "</g>\n",
       "<!-- one_hot_encoder&#45;&gt;concat_features -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>one_hot_encoder&#45;&gt;concat_features</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M191.57,-58.82C200.92,-60.54 211.1,-62.42 220.9,-64.22\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"220.45,-67.7 230.92,-66.07 221.72,-60.82 220.45,-67.7\"/>\n",
       "</g>\n",
       "<!-- random_forest_classifier -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>random_forest_classifier</title>\n",
       "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.random_forest_classifier.html\" target=\"_blank\" rel=\"noopener noreferrer\" xlink:title=\"random_forest_classifier = RandomForestClassifier(max_features=0.7421091918485162, min_samples_leaf=2, min_samples_split=0.2129181299179138, n_estimators=70)\">\n",
       "<ellipse fill=\"#b0e2ff\" stroke=\"black\" cx=\"391.05\" cy=\"-73\" rx=\"44.6\" ry=\"28.07\"/>\n",
       "<text text-anchor=\"middle\" x=\"391.05\" y=\"-82.2\" font-family=\"Times,serif\" font-size=\"11.00\">Random&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"391.05\" y=\"-70.2\" font-family=\"Times,serif\" font-size=\"11.00\">Forest&#45;</text>\n",
       "<text text-anchor=\"middle\" x=\"391.05\" y=\"-58.2\" font-family=\"Times,serif\" font-size=\"11.00\">Classifier</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- concat_features&#45;&gt;random_forest_classifier -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>concat_features&#45;&gt;random_forest_classifier</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M310.64,-73C318.74,-73 327.38,-73 335.88,-73\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"336.11,-76.5 346.11,-73 336.11,-69.5 336.11,-76.5\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7fea4f8e4280>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_disparate impact</th>\n",
       "      <th>test_balanced accuracy</th>\n",
       "      <th>test_recall (unfavorable)</th>\n",
       "      <th>test_recall (favorable)</th>\n",
       "      <th>test_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.52</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.46</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.37</td>\n",
       "      <td>0.23</td>\n",
       "      <td>1.05</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.40</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.47</td>\n",
       "      <td>0.16</td>\n",
       "      <td>1.08</td>\n",
       "      <td>0.58</td>\n",
       "      <td>0.46</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.45</td>\n",
       "      <td>0.20</td>\n",
       "      <td>1.04</td>\n",
       "      <td>0.56</td>\n",
       "      <td>0.44</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.07</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_disparate impact  test_balanced accuracy  \\\n",
       "0         0.52        0.20                   0.99                    0.57   \n",
       "1         0.37        0.23                   1.05                    0.55   \n",
       "2         0.47        0.16                   1.08                    0.58   \n",
       "mean      0.45        0.20                   1.04                    0.56   \n",
       "std       0.07        0.04                   0.05                    0.01   \n",
       "\n",
       "      test_recall (unfavorable)  test_recall (favorable)  test_accuracy  \n",
       "0                          0.46                     0.68           0.61  \n",
       "1                          0.40                     0.70           0.61  \n",
       "2                          0.46                     0.69           0.62  \n",
       "mean                       0.44                     0.69           0.61  \n",
       "std                        0.03                     0.01           0.01  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trained_mit3.visualize()\n",
    "get_metrics(trained_mit3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the result prioritized fairness over predictive\n",
    "performance, performing only slightly better than `DummyClassifier`.\n",
    "You can experiment with a higher `max_evals` and/or different search\n",
    "objectives.\n",
    "\n",
    "The following papers provide more information on fairness support in\n",
    "Lale:\n",
    "\n",
    "```bibtex\n",
    "@InProceedings{feffer_et_al_2022,\n",
    "  title = \"An Empirical Study of Modular Bias Mitigators and Ensembles\",\n",
    "  author = \"Feffer, Michael and Hirzel, Martin and Hoffman, Samuel C. and Kate, Kiran and Ram, Parikshit and Shinnar, Avraham\",\n",
    "  booktitle = \"Workshop on Benchmarking Data for Data-Centric AI (DataPerf@ICML)\",\n",
    "  year = 2022,\n",
    "  month = jul,\n",
    "  url = \"http://hirzels.com/martin/papers/dataperf22-fair-ensembles.pdf\" }\n",
    "```\n",
    "\n",
    "```bibtex\n",
    "@InProceedings{hirzel_kate_ram_2021,\n",
    "  title = \"Engineering Fair Machine Learning Pipelines\",\n",
    "  author = \"Hirzel, Martin and Kate, Kiran and Ram, Parikshit\",\n",
    "  booktitle = \"ICLR Workshop on Responsible AI (RAI@ICLR)\",\n",
    "  year = 2021,\n",
    "  month = may,\n",
    "  url = \"http://hirzels.com/martin/papers/rai21-fairness.pdf\" }\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}