overviews/AndreMachado/main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3W Dataset's General Presentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a general presentation of the 3W Dataset, to the best of its authors' knowledge, the first realistic and public dataset with rare undesirable real events in oil wells that can be readily used as a benchmark dataset for development of machine learning techniques related to inherent difficulties of actual data.\n",
    "\n",
    "For more information about the theory behind this dataset, refer to the paper **A Realistic and Public Dataset with Rare Undesirable Real Events in Oil Wells** published in the **Journal of Petroleum Science and Engineering** (link [here](https://doi.org/10.1016/j.petrol.2019.106223))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Jupyter Notebook presents a 3W Dataset overview. For this, one **interactive plot graph** from a specific instance from an event class is presented. By default, the instance is downsampling (n=100) and applied Z-score Scaler. To help the visualization, transient labels were changed to '0.5'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Imports and Configurations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.simplefilter(\"ignore\", FutureWarning)\n",
    "\n",
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.append(os.path.join(\"..\", \"..\"))\n",
    "import toolkit as tk\n",
    "\n",
    "import plotly.offline as py\n",
    "import plotly.graph_objs as go\n",
    "import glob\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from ydata_profiling import ProfileReport\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'svg'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each instance is stored in a [Parquet file](https://parquet.apache.org/docs/) and loaded into a pandas DataFrame as follows:\n",
    "\n",
    "* All Parquet files are created and read with pandas functions, `pyarrow` engine and `brotli` compression;\n",
    "* For each instance, timestamps corresponding to observations are stored in Parquet file as its index and loaded into pandas DataFrame as its index;\n",
    "* Each observation is stored in a line of a Parquet file and loaded as a line of a pandas DataFrame; \n",
    "* All variables are stored as float in columns of Parquet files and loaded as float in columns of pandas DataFrame;\n",
    "* All labels are stored as `Int64` (not `int64`) in columns of Parquet files and loaded as `Int64` (not `int64`) in columns of pandas DataFrame.\n",
    "\n",
    "The variables and labels are as follows:\n",
    "\n",
    "* **ABER-CKGL**: Opening of the GLCK (gas lift choke) [%];\n",
    "* **ABER-CKP**: Opening of the PCK (production choke) [%];\n",
    "* **ESTADO-DHSV**: State of the DHSV (downhole safety valve) [0, 0.5, or 1];\n",
    "* **ESTADO-M1**: State of the PMV (production master valve) [0, 0.5, or 1];\n",
    "* **ESTADO-M2**: State of the AMV (annulus master valve) [0, 0.5, or 1];\n",
    "* **ESTADO-PXO**: State of the PXO (pig-crossover) valve [0, 0.5, or 1];\n",
    "* **ESTADO-SDV-GL**: State of the gas lift SDV (shutdown valve) [0, 0.5, or 1];\n",
    "* **ESTADO-SDV-P**: State of the production SDV (shutdown valve) [0, 0.5, or 1];\n",
    "* **ESTADO-W1**: State of the PWV (production wing valve) [0, 0.5, or 1];\n",
    "* **ESTADO-W2**: State of the AWV (annulus wing valve) [0, 0.5, or 1];\n",
    "* **ESTADO-XO**: State of the XO (crossover) valve [0, 0.5, or 1];\n",
    "* **P-ANULAR**: Pressure in the well annulus [Pa];\n",
    "* **P-JUS-BS**: Downstream pressure of the SP (service pump) [Pa];\n",
    "* **P-JUS-CKGL**: Downstream pressure of the GLCK (gas lift choke) [Pa];\n",
    "* **P-JUS-CKP**: Downstream pressure of the PCK (production choke) [Pa];\n",
    "* **P-MON-CKGL**: Upstream pressure of the GLCK (gas lift choke) [Pa];\n",
    "* **P-MON-CKP**: Upstream pressure of the PCK (production choke) [Pa];\n",
    "* **P-MON-SDV-P**: Upstream pressure of the production SDV (shutdown valve) [Pa];\n",
    "* **P-PDG**: Pressure at the PDG (permanent downhole gauge) [Pa];\n",
    "* **PT-P**: Downstream pressure of the PWV (production wing valve) in the production tube [Pa];\n",
    "* **P-TPT**: Pressure at the TPT (temperature and pressure transducer) [Pa];\n",
    "* **QBS**: Flow rate at the SP (service pump) [m3/s];\n",
    "* **QGL**: Gas lift flow rate [m3/s];\n",
    "* **T-JUS-CKP**: Downstream temperature of the PCK (production choke) [oC];\n",
    "* **T-MON-CKP**: Upstream temperature of the PCK (production choke) [oC];\n",
    "* **T-PDG**: Temperature at the PDG (permanent downhole gauge) [oC];\n",
    "* **T-TPT**: Temperature at the TPT (temperature and pressure transducer) [oC];\n",
    "* **class**: Label of the observation;\n",
    "* **state**: Well operational status.\n",
    "\n",
    "Other informations are also loaded into each pandas Dataframe:\n",
    "\n",
    "* **label**: instance label (event type);\n",
    "* **well**: well name. Hand-drawn and simulated instances have fixed names. Real instances have names masked with incremental id;\n",
    "* **id**: instance identifier. Hand-drawn and simulated instances have incremental id. Each real instance has an id generated from its first timestamp.\n",
    "\n",
    "More information about these variables can be obtained from the following publicly available documents:\n",
    "\n",
    "* ***Option in Portuguese***: R.E.V. Vargas. Base de dados e benchmarks para prognóstico de anomalias em sistemas de elevação de petróleo. Universidade Federal do Espírito Santo. Doctoral thesis. 2019. https://github.com/petrobras/3W/raw/main/docs/doctoral_thesis_ricardo_vargas.pdf.\n",
    "* ***Option in English***: B.G. Carvalho. Evaluating machine learning techniques for detection of flow instability events in offshore oil wells. Universidade Federal do Espírito Santo. Master's degree dissertation. 2021. https://github.com/petrobras/3W/raw/main/docs/master_degree_dissertation_bruno_carvalho.pdf."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Plot Instances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot one interactive graph from an especific event class and instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_number = 7\n",
    "instance_index = 5\n",
    "resample_factor = 100\n",
    "tk.plot_instance(class_number, instance_index, resample_factor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Profiling Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, we generate a complete interactive HTML report from the data set. It is possible to have a complete view of the 3W Dataset of one event class, such as the number of lines, number of columns (variables), number of missing values (null cells, NaNs), duplicate lines, size, and the types of variables that we have in the database. In addition, the tool also brings statistics, histograms, interactions, and correlations.\n",
    "\n",
    "In the Warnings field, the report already brings some things that we will have to be careful about when analyzing the dataset. With this, it is possible to assess the need or not to perform some initial treatment on the data, before starting the exploration.\n",
    "\n",
    "The original frequency rate is 1Hz. In some 3W classes, due to a large number of samples, the maximum allowed size is exceeded. Thus we reduce the frequency rate. The parameter, that determines the new frequency is \"resample_factor\". In this case, we downsampling 100 times. To visualize the original data use \"resize=1\", but it's no warranty that the report will be generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_number = 2\n",
    "resample_factor = 100\n",
    "df_all_instances_class = pd.concat(\n",
    "    [\n",
    "        tk.resample(pd.read_parquet(f, engine=\"pyarrow\"), resample_factor, class_number)\n",
    "        for f in glob.glob(\n",
    "            os.path.join(tk.PATH_DATASET, str(class_number), \"*.parquet\")\n",
    "        )\n",
    "    ],\n",
    "    ignore_index=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Genarate the Profile Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile = df_all_instances_class.profile_report(\n",
    "    title=tk.EVENT_NAMES[class_number] + \" Profiling Report\"\n",
    ")\n",
    "profile.to_file(tk.EVENT_NAMES[class_number].replace(\" \", \"\") + \"DataReport.html\")\n",
    "print(\n",
    "    \"Generated Profiling Report: \"\n",
    "    + tk.EVENT_NAMES[class_number].replace(\" \", \"\")\n",
    "    + \"DataReport.html\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the Interactive Report on new tab browser "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import webbrowser\n",
    "\n",
    "webbrowser.open_new_tab(\n",
    "    tk.EVENT_NAMES[class_number].replace(\" \", \"\") + \"DataReport.html\"\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Sumário",
   "title_sidebar": "Sumário",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "0fbbee2471b2eebdaf23673a6aac306d841dd0fe8256377f17a4a496f5c71b8e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}