nbs/12_text-data-language-modeling.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "skip_exec: true\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |default_exp text.data.language_modeling\n",
    "# |default_cls_lvl 3\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data\n",
    "\n",
    "> The `text.data.language_modeling` module contains the bits required to use the fastai DataBlock API and/or mid-level data processing pipelines to organize your data for causal and masked language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "import os, random, warnings\n",
    "from abc import ABC, abstractmethod\n",
    "from enum import Enum\n",
    "\n",
    "from datasets import Dataset\n",
    "from fastcore.all import *\n",
    "from fastai.imports import *\n",
    "from fastai.losses import CrossEntropyLossFlat\n",
    "from fastai.torch_core import *\n",
    "from fastai.torch_imports import *\n",
    "from transformers import (\n",
    "    AutoModelForCausalLM,\n",
    "    AutoModelForMaskedLM,\n",
    "    PretrainedConfig,\n",
    "    PreTrainedTokenizerBase,\n",
    "    PreTrainedModel,\n",
    "    BatchEncoding,\n",
    ")\n",
    "from transformers.utils import logging as hf_logging\n",
    "\n",
    "from blurr.text.data.core import TextInput, BatchTokenizeTransform, Preprocessor, first_blurr_tfm\n",
    "from blurr.text.utils import get_hf_objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What we're running with at the time this documentation was generated:\n",
      "torch: 1.9.0+cu102\n",
      "fastai: 2.7.9\n",
      "transformers: 4.21.2\n"
     ]
    }
   ],
   "source": [
    "# | hide\n",
    "import pdb\n",
    "\n",
    "from fastai.data.block import DataBlock, ColReader, ColSplitter\n",
    "from fastai.data.core import DataLoader, DataLoaders, TfmdDL\n",
    "from fastai.data.external import untar_data, URLs\n",
    "from fastai.data.transforms import *\n",
    "from fastcore.test import *\n",
    "from nbdev import nbdev_export\n",
    "from nbdev.showdoc import show_doc\n",
    "\n",
    "from blurr.utils import print_versions\n",
    "from blurr.text.data.core import TextBlock\n",
    "from blurr.text.utils import BlurrText\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "# silence all the HF warnings\n",
    "warnings.simplefilter(\"ignore\")\n",
    "hf_logging.set_verbosity_error()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# | echo: false\n",
    "NLP = BlurrText()\n",
    "\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
    "print(\"What we're running with at the time this documentation was generated:\")\n",
    "print_versions(\"torch fastai transformers\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GPU #1: GeForce GTX 1080 Ti\n"
     ]
    }
   ],
   "source": [
    "# |hide\n",
    "# |cuda\n",
    "torch.cuda.set_device(1)\n",
    "print(f\"Using GPU #{torch.cuda.current_device()}: {torch.cuda.get_device_name()}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "For this example, we'll use the `WIKITEXT_TINY` dataset available from fastai.  In addition to using the `Datasets` library from Hugging Face, fastai provides a lot of smaller datasets that are really useful when experimenting and/or in the early development of your training/validation/inference coding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('/home/wgilliam/.fastai/data/wikitext-2/train.csv'),Path('/home/wgilliam/.fastai/data/wikitext-2/test.csv')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_path = untar_data(URLs.WIKITEXT_TINY)\n",
    "wiki_path.ls()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "615 47\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the &lt;unk&gt; season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\n = Big Boy ( song ) = \\n \\n \" Big Boy \" &lt;unk&gt; \" I 'm A Big Boy Now \" was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including \" Big Boy \" . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \\n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n = The Remix ( Lady Gaga album ) = \\n \\n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and &lt;unk&gt; composit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n = New Year 's Eve ( Up All Night ) = \\n \\n \" New Year 's Eve \" is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica &lt;unk&gt; and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \\n During Reagan ( Christina Applegate ) and Chris 's ( Will &lt;unk&gt; ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\n = Geopyxis carbonaria = \\n \\n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family &lt;unk&gt; . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf &lt;unk&gt; cup , &lt;unk&gt; &lt;unk&gt; cup , or pixie cup . The small , &lt;unk&gt; @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         0\n",
       "0   \\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...\n",
       "1   \\n = Big Boy ( song ) = \\n \\n \" Big Boy \" <unk> \" I 'm A Big Boy Now \" was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including \" Big Boy \" . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \\n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...\n",
       "2   \\n = The Remix ( Lady Gaga album ) = \\n \\n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...\n",
       "3   \\n = New Year 's Eve ( Up All Night ) = \\n \\n \" New Year 's Eve \" is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \\n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...\n",
       "4   \\n = Geopyxis carbonaria = \\n \\n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ...."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = pd.read_csv(wiki_path / \"train.csv\", header=None)\n",
    "valid_df = pd.read_csv(wiki_path / \"test.csv\", header=None)\n",
    "\n",
    "print(len(train_df), len(valid_df))\n",
    "train_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the &lt;unk&gt; season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\n = Big Boy ( song ) = \\n \\n \" Big Boy \" &lt;unk&gt; \" I 'm A Big Boy Now \" was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including \" Big Boy \" . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \\n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n = The Remix ( Lady Gaga album ) = \\n \\n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and &lt;unk&gt; composit...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n = New Year 's Eve ( Up All Night ) = \\n \\n \" New Year 's Eve \" is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica &lt;unk&gt; and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \\n During Reagan ( Christina Applegate ) and Chris 's ( Will &lt;unk&gt; ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\n = Geopyxis carbonaria = \\n \\n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family &lt;unk&gt; . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf &lt;unk&gt; cup , &lt;unk&gt; &lt;unk&gt; cup , or pixie cup . The small , &lt;unk&gt; @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         0  \\\n",
       "0   \\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...   \n",
       "1   \\n = Big Boy ( song ) = \\n \\n \" Big Boy \" <unk> \" I 'm A Big Boy Now \" was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including \" Big Boy \" . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \\n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...   \n",
       "2   \\n = The Remix ( Lady Gaga album ) = \\n \\n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...   \n",
       "3   \\n = New Year 's Eve ( Up All Night ) = \\n \\n \" New Year 's Eve \" is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \\n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...   \n",
       "4   \\n = Geopyxis carbonaria = \\n \\n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....   \n",
       "\n",
       "   is_valid  \n",
       "0     False  \n",
       "1     False  \n",
       "2     False  \n",
       "3     False  \n",
       "4     False  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[\"is_valid\"] = False\n",
    "valid_df[\"is_valid\"] = True\n",
    "\n",
    "df = pd.concat([train_df, valid_df])\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using pad_token, but it is not set yet.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('[PAD]', 50256)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_cls = AutoModelForCausalLM\n",
    "hf_logging.set_verbosity_error()\n",
    "\n",
    "pretrained_model_name = \"gpt2\"\n",
    "hf_arch, hf_config, hf_tokenizer, hf_model = get_hf_objects(pretrained_model_name, model_cls=model_cls)\n",
    "\n",
    "# some tokenizers like gpt and gpt2 do not have a pad token, so we add it here mainly for the purpose\n",
    "# of setting the \"labels\" key appropriately (see below)\n",
    "if hf_tokenizer.pad_token is None:\n",
    "    hf_tokenizer.pad_token = \"[PAD]\"\n",
    "\n",
    "hf_tokenizer.pad_token, hf_tokenizer.pad_token_id\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# special_tokens_dict = {'additional_special_tokens': ['[C1]']}\n",
    "# num_added_toks = hf_tokenizer.add_special_tokens(special_tokens_dict)\n",
    "# hf_model.resize_token_embeddings(len(hf_tokenizer))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "Starting with version 2.0, `BLURR` provides a language preprocessing class that can be used to preprocess DataFrames or Hugging Face Datasets for both causal and masked language modeling tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `LMPreprocessor` -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class LMPreprocessor(Preprocessor):\n",
    "    def __init__(\n",
    "        self,\n",
    "        # A Hugging Face tokenizer\n",
    "        hf_tokenizer: PreTrainedTokenizerBase,\n",
    "        # The number of examples to process at a time\n",
    "        batch_size: int = 1000,\n",
    "        # How big each chunk of text should be (default: hf_tokenizer.model_max_length)\n",
    "        chunk_size: Optional[int] = None,\n",
    "        # How to indicate the beginning on a new text example (default is hf_tokenizer.eos_token|sep_token\n",
    "        sep_token: Optional[str] = None,\n",
    "        # The attribute holding the text\n",
    "        text_attr: str = \"text\",\n",
    "        # The attribute that should be created if your are processing individual training and validation\n",
    "        # datasets into a single dataset, and will indicate to which each example is associated\n",
    "        is_valid_attr: Optional[str] = \"is_valid\",\n",
    "        # Tokenization kwargs that will be applied with calling the tokenizer\n",
    "        tok_kwargs: dict = {},\n",
    "    ):\n",
    "        tok_kwargs = {**tok_kwargs, \"truncation\": False, \"return_offsets_mapping\": True}\n",
    "        super().__init__(hf_tokenizer, batch_size, text_attr, None, is_valid_attr, tok_kwargs)\n",
    "\n",
    "        self.chunk_size = chunk_size or hf_tokenizer.model_max_length\n",
    "        self.sep_token = sep_token or hf_tokenizer.eos_token or hf_tokenizer.sep_token\n",
    "\n",
    "    def process_df(self, training_df: pd.DataFrame, validation_df: Optional[pd.DataFrame] = None):\n",
    "        # process df in mini-batches\n",
    "        final_train_df = pd.DataFrame()\n",
    "        for g, batch_df in training_df.groupby(np.arange(len(training_df)) // self.batch_size):\n",
    "            final_train_df = final_train_df.append(self._process_df_batch(batch_df))\n",
    "            final_train_df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "        final_val_df = pd.DataFrame() if validation_df is not None else None\n",
    "        if final_val_df is not None:\n",
    "            for g, batch_df in validation_df.groupby(np.arange(len(validation_df)) // self.batch_size):\n",
    "                final_val_df = final_val_df.append(self._process_df_batch(batch_df))\n",
    "                final_val_df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "        final_df = super().process_df(final_train_df, final_val_df)\n",
    "        return final_df\n",
    "\n",
    "    def process_hf_dataset(self, training_ds: Dataset, validation_ds: Optional[Dataset] = None):\n",
    "        ds = super().process_hf_dataset(training_ds, validation_ds)\n",
    "        return Dataset.from_pandas(self.process_df(pd.DataFrame(ds)))\n",
    "\n",
    "    # ----- utility methods -----\n",
    "    def _process_df_batch(self, batch_df):\n",
    "        batch_df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "        # concatenate our texts\n",
    "        concat_txts = {self.text_attr: f\" {self.sep_token} \".join(batch_df[self.text_attr].values.tolist())}\n",
    "        inputs = self._tokenize_function(concat_txts)\n",
    "\n",
    "        # compute the length of our concatenated texts\n",
    "        n_total_toks = len(inputs[\"input_ids\"])\n",
    "\n",
    "        # need to modify chunk_size to included the # of special tokens added\n",
    "        max_chunk_size = self.chunk_size - self.hf_tokenizer.num_special_tokens_to_add() - 1\n",
    "\n",
    "        # drop the last chunk of text if it is smaller than chunk size (see the HF course, section 7 on training MLMs)\n",
    "        total_length = (n_total_toks // max_chunk_size) * max_chunk_size\n",
    "\n",
    "        # break our concatenated into chunks of text of size max_chunk_size\n",
    "        examples = []\n",
    "        for i in range(0, total_length, max_chunk_size):\n",
    "            chunked_offsets = inputs[\"offset_mapping\"][i : i + max_chunk_size]\n",
    "            chunked_text = concat_txts[self.text_attr][min(chunked_offsets)[0] : max(chunked_offsets)[1]]\n",
    "            examples.append(chunked_text)\n",
    "\n",
    "        return pd.DataFrame(examples, columns=[f\"proc_{self.text_attr}\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using a `DataFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21330\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>proc_0</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the &lt;unk&gt; season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>above the relegation zone on goal difference , before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two . This meant York qualified for the play @-@ offs , and they were eliminated in the semi @-@ final by Fleetwood Town . York were knocked out of the 2013 – 14 FA Cup , Football League Cup and Football League Trophy in their opening round matches . \\n 35 players made at least one appearance in nationally organised first @-@ team competition , and there were 12 different &lt;unk&gt; . Defender Ben Davies missed</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               proc_0  \\\n",
       "0        \\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only   \n",
       "1   above the relegation zone on goal difference , before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two . This meant York qualified for the play @-@ offs , and they were eliminated in the semi @-@ final by Fleetwood Town . York were knocked out of the 2013 – 14 FA Cup , Football League Cup and Football League Trophy in their opening round matches . \\n 35 players made at least one appearance in nationally organised first @-@ team competition , and there were 12 different <unk> . Defender Ben Davies missed   \n",
       "\n",
       "   is_valid  \n",
       "0     False  \n",
       "1     False  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor = LMPreprocessor(hf_tokenizer, chunk_size=128, text_attr=0)\n",
    "proc_df = preprocessor.process_df(train_df, valid_df)\n",
    "\n",
    "print(len(proc_df))\n",
    "proc_df.head(2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using a Hugging Face `Dataset`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LM Strategies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `LMType` -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class LMType(Enum):\n",
    "    \"\"\"Use this enum to indicate what kind of language model you are training\"\"\"\n",
    "\n",
    "    CAUSAL = 1\n",
    "    MASKED = 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `BaseLMStrategy` and implementations -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class BaseLMStrategy(ABC):\n",
    "    \"\"\"ABC for various language modeling strategies (e.g., causal, BertMLM, WholeWordMLM, etc...)\"\"\"\n",
    "\n",
    "    def __init__(self, hf_tokenizer, ignore_token_id=CrossEntropyLossFlat().ignore_index):\n",
    "        store_attr([\"hf_tokenizer\", \"ignore_token_id\"])\n",
    "\n",
    "    @abstractmethod\n",
    "    def build_inputs_targets(self, samples, include_labels: bool = True, inputs: Optional[BatchEncoding] = None):\n",
    "        pass\n",
    "\n",
    "    # utility methods\n",
    "    def _get_random_token_id(self, n):\n",
    "        return random.sample(list(self.hf_tokenizer.get_vocab().values()), n)\n",
    "\n",
    "    @classmethod\n",
    "    @abstractmethod\n",
    "    def get_lm_type(cls):\n",
    "        pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we include a `BaseLMStrategy` abstract class and several different strategies for building your inputs and targets for causal and masked language modeling tasks.  With CLMs, the objective is to simply predict the next token, but with MLMs, a variety of masking strategies may be used (e.g., mask random tokens, mask random words, mask spans, etc...).  A `BertMLMStrategy` is introduced below that follows the \"mask random tokens\" strategy used in the BERT paper, but users can create their own `BaseLMStrategy` subclass to support any masking strategy they desire."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `CausalLMStrategy` -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class CausalLMStrategy(BaseLMStrategy):\n",
    "    \"\"\"For next token prediction language modeling tasks, we want to use the `CausalLMStrategy` which makes the\n",
    "    necessary changes in your inputs/targets for causal LMs\n",
    "    \"\"\"\n",
    "\n",
    "    def build_inputs_targets(self, samples, include_labels: bool = True, inputs: Optional[BatchEncoding] = None):\n",
    "        updated_samples = []\n",
    "        for s in samples:\n",
    "            if include_labels:\n",
    "                s[0][\"labels\"] = s[0][\"input_ids\"].clone()\n",
    "                s[0][\"labels\"][s[0][\"labels\"] == self.hf_tokenizer.pad_token_id] = self.ignore_token_id\n",
    "\n",
    "            targ_ids = torch.cat([s[0][\"input_ids\"][1:], tensor([self.hf_tokenizer.eos_token_id])])\n",
    "\n",
    "            updated_samples.append((s[0], targ_ids))\n",
    "\n",
    "        return updated_samples\n",
    "\n",
    "    @classmethod\n",
    "    def get_lm_type(cls: LMType):\n",
    "        return LMType.CAUSAL\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `BertMLMStrategy` - "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class BertMLMStrategy(BaseLMStrategy):\n",
    "    \"\"\"A masked language modeling strategy using the default BERT masking definition.\"\"\"\n",
    "\n",
    "    def __init__(self, hf_tokenizer, ignore_token_id=CrossEntropyLossFlat().ignore_index):\n",
    "        super().__init__(hf_tokenizer, ignore_token_id)\n",
    "\n",
    "        vocab = hf_tokenizer.get_vocab()\n",
    "        self.dnm_tok_ids = [\n",
    "            vocab[tok] for tok in list(hf_tokenizer.special_tokens_map.values()) if vocab[tok] != hf_tokenizer.mask_token_id\n",
    "        ]\n",
    "\n",
    "    def build_inputs_targets(self, samples, include_labels: bool = True, inputs: Optional[BatchEncoding] = None):\n",
    "        updated_samples = []\n",
    "        for s in samples:\n",
    "            # mask the input_ids\n",
    "            masked_input_ids = s[0][\"input_ids\"].clone()\n",
    "\n",
    "            # we want to mask 15% of the non-special tokens(e.g., special tokens inclue [CLS], [SEP], etc...)\n",
    "            idxs = torch.randperm(len(masked_input_ids))\n",
    "            total_masked_idxs = int(len(idxs) * 0.15)\n",
    "\n",
    "            # of the 15% for masking, replace 80% with [MASK] token, 10% with random token, and 10% with correct token\n",
    "            n_mask_idxs = int(total_masked_idxs * 0.8)\n",
    "            n_rnd_idxs = int(total_masked_idxs * 0.1)\n",
    "\n",
    "            # we only want non-special tokens\n",
    "            mask_idxs = [idx for idx in idxs if masked_input_ids[idx] not in self.dnm_tok_ids][:total_masked_idxs]\n",
    "\n",
    "            # replace 80% with [MASK]\n",
    "            if n_mask_idxs > 0 and len(mask_idxs) >= n_mask_idxs:\n",
    "                masked_input_ids[[mask_idxs[:n_mask_idxs]]] = self.hf_tokenizer.mask_token_id\n",
    "\n",
    "            # replace 10% with a random token\n",
    "            if n_rnd_idxs > 0 and len(mask_idxs) >= (n_mask_idxs + n_rnd_idxs):\n",
    "                rnd_tok_ids = self._get_random_token_id(n_rnd_idxs)\n",
    "                masked_input_ids[[mask_idxs[n_mask_idxs : (n_mask_idxs + n_rnd_idxs)]]] = tensor(rnd_tok_ids)\n",
    "\n",
    "            # ignore padding when calculating the loss\n",
    "            lbls = s[0][\"input_ids\"].clone()\n",
    "            lbls[[[idx for idx in idxs if idx not in mask_idxs]]] = self.ignore_token_id\n",
    "\n",
    "            # update the inputs to use our masked input_ids and labels; set targ_ids = labels (will use when\n",
    "            # we calculate the loss ourselves)\n",
    "            s[0][\"input_ids\"] = masked_input_ids\n",
    "            targ_ids = lbls\n",
    "\n",
    "            if include_labels:\n",
    "                s[0][\"labels\"] = targ_ids.clone()\n",
    "\n",
    "            updated_samples.append((s[0], targ_ids))\n",
    "\n",
    "        return updated_samples\n",
    "\n",
    "    @classmethod\n",
    "    def get_lm_type(cls: LMType):\n",
    "        return LMType.MASKED\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follows the masking strategy used in the [BERT paper](https://arxiv.org/abs/1810.04805) for random token masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mid-level API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `CausalLMTextInput` and `MLMTextInput` -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class CausalLMTextInput(TextInput):\n",
    "    pass\n",
    "\n",
    "\n",
    "# export\n",
    "class MLMTextInput(TextInput):\n",
    "    pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we define a custom classes for the `@typedispatch`ed methods to use so that we can override how both causal and masked language modeling inputs/targets are assembled, as well as, how the data is shown via methods like `show_batch` and `show_results`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `LMBatchTokenizeTransform` -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "class LMBatchTokenizeTransform(BatchTokenizeTransform):\n",
    "    def __init__(\n",
    "        self,\n",
    "        # The abbreviation/name of your Hugging Face transformer architecture (e.b., bert, bart, etc..)\n",
    "        hf_arch: str,\n",
    "        # A specific configuration instance you want to use\n",
    "        hf_config: PretrainedConfig,\n",
    "        # A Hugging Face tokenizer\n",
    "        hf_tokenizer: PreTrainedTokenizerBase,\n",
    "        # A Hugging Face model\n",
    "        hf_model: PreTrainedModel,\n",
    "        # To control whether the \"labels\" are included in your inputs. If they are, the loss will be calculated in\n",
    "        # the model's forward function and you can simply use `PreCalculatedLoss` as your `Learner`'s loss function to use it\n",
    "        include_labels: bool = True,\n",
    "        # The token ID that should be ignored when calculating the loss\n",
    "        ignore_token_id: int = CrossEntropyLossFlat().ignore_index,\n",
    "        # The language modeling strategy (or objective)\n",
    "        lm_strategy_cls: BaseLMStrategy = CausalLMStrategy,\n",
    "        # To control the length of the padding/truncation. It can be an integer or None,\n",
    "        # in which case it will default to the maximum length the model can accept. If the model has no\n",
    "        # specific maximum input length, truncation/padding to max_length is deactivated.\n",
    "        # See [Everything you always wanted to know about padding and truncation](https://huggingface.co/transformers/preprocessing.html#everything-you-always-wanted-to-know-about-padding-and-truncation)\n",
    "        max_length: int = None,\n",
    "        # To control the `padding` applied to your `hf_tokenizer` during tokenization. If None, will default to\n",
    "        # `False` or `'do_not_pad'.\n",
    "        # See [Everything you always wanted to know about padding and truncation](https://huggingface.co/transformers/preprocessing.html#everything-you-always-wanted-to-know-about-padding-and-truncation)\n",
    "        padding: Union[bool, str] = True,\n",
    "        # To control `truncation` applied to your `hf_tokenizer` during tokenization. If None, will default to\n",
    "        # `False` or `do_not_truncate`.\n",
    "        # See [Everything you always wanted to know about padding and truncation](https://huggingface.co/transformers/preprocessing.html#everything-you-always-wanted-to-know-about-padding-and-truncation)\n",
    "        truncation: Union[bool, str] = True,\n",
    "        # The `is_split_into_words` argument applied to your `hf_tokenizer` during tokenization. Set this to `True`\n",
    "        # if your inputs are pre-tokenized (not numericalized)\n",
    "        is_split_into_words: bool = False,\n",
    "        # Any other keyword arguments you want included when using your `hf_tokenizer` to tokenize your inputs\n",
    "        tok_kwargs={},\n",
    "        # Any keyword arguments you want included when generated text\n",
    "        # See [How to generate text](https://huggingface.co/blog/how-to-generate)\n",
    "        text_gen_kwargs={},\n",
    "        # Keyword arguments to apply to `BatchTokenizeTransform`\n",
    "        **kwargs\n",
    "    ):\n",
    "        super().__init__(\n",
    "            hf_arch,\n",
    "            hf_config,\n",
    "            hf_tokenizer,\n",
    "            hf_model,\n",
    "            include_labels=include_labels,\n",
    "            ignore_token_id=ignore_token_id,\n",
    "            max_length=max_length,\n",
    "            padding=padding,\n",
    "            truncation=truncation,\n",
    "            is_split_into_words=is_split_into_words,\n",
    "            tok_kwargs=tok_kwargs.copy(),\n",
    "            **kwargs\n",
    "        )\n",
    "\n",
    "        self.lm_strategy = lm_strategy_cls(hf_tokenizer=hf_tokenizer, ignore_token_id=ignore_token_id)\n",
    "        self.text_gen_kwargs, self.ignore_token_id = text_gen_kwargs, ignore_token_id\n",
    "\n",
    "    def encodes(self, samples, return_batch_encoding=False):\n",
    "        # because no target is specific in CLM, fastai will duplicate the inputs (which is just the raw text)\n",
    "        samples, inputs = super().encodes(samples, return_batch_encoding=True)\n",
    "        if len(samples[0]) == 1:\n",
    "            return samples\n",
    "\n",
    "        updated_samples = self.lm_strategy.build_inputs_targets(samples, self.include_labels, inputs)\n",
    "\n",
    "        if return_batch_encoding:\n",
    "            return updated_samples, inputs\n",
    "\n",
    "        return updated_samples\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our `LMBatchTokenizeTransform` allows us to update the input's `labels` and our targets appropriately given any language modeling task. \n",
    "\n",
    "The `labels` argument allows you to forgo calculating the loss yourself by letting Hugging Face return it for you should you choose to do that. Padding tokens are set to -100 by default (e.g., `CrossEntropyLossFlat().ignore_index`) and prevent cross entropy loss from considering token prediction for tokens it should ... i.e., the padding tokens. For more information on the meaning of this argument, see the [Hugging Face glossary entry for \"Labels\"](https://huggingface.co/transformers/glossary.html#labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the mid-level API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Causal LM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 1: Get your Hugging Face objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using pad_token, but it is not set yet.\n"
     ]
    }
   ],
   "source": [
    "model_cls = AutoModelForCausalLM\n",
    "hf_logging.set_verbosity_error()\n",
    "\n",
    "pretrained_model_name = \"gpt2\"\n",
    "hf_arch, hf_config, hf_tokenizer, hf_model = get_hf_objects(pretrained_model_name, model_cls=model_cls)\n",
    "\n",
    "# some tokenizers like gpt and gpt2 do not have a pad token, so we add it here mainly for the purpose\n",
    "# of setting the \"labels\" key appropriately (see below)\n",
    "if hf_tokenizer.pad_token is None:\n",
    "    hf_tokenizer.pad_token = \"[PAD]\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Step 2: Preprocess data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21330\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>proc_0</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the &lt;unk&gt; season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>above the relegation zone on goal difference , before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two . This meant York qualified for the play @-@ offs , and they were eliminated in the semi @-@ final by Fleetwood Town . York were knocked out of the 2013 – 14 FA Cup , Football League Cup and Football League Trophy in their opening round matches . \\n 35 players made at least one appearance in nationally organised first @-@ team competition , and there were 12 different &lt;unk&gt; . Defender Ben Davies missed</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               proc_0  \\\n",
       "0        \\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only   \n",
       "1   above the relegation zone on goal difference , before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two . This meant York qualified for the play @-@ offs , and they were eliminated in the semi @-@ final by Fleetwood Town . York were knocked out of the 2013 – 14 FA Cup , Football League Cup and Football League Trophy in their opening round matches . \\n 35 players made at least one appearance in nationally organised first @-@ team competition , and there were 12 different <unk> . Defender Ben Davies missed   \n",
       "\n",
       "   is_valid  \n",
       "0     False  \n",
       "1     False  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor = LMPreprocessor(hf_tokenizer, chunk_size=128, text_attr=0)\n",
    "proc_df = preprocessor.process_df(train_df, valid_df)\n",
    "\n",
    "print(len(proc_df))\n",
    "proc_df.head(2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Step 3: Create your `DataBlock`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_tok_tfm = LMBatchTokenizeTransform(hf_arch, hf_config, hf_tokenizer, hf_model, lm_strategy_cls=CausalLMStrategy)\n",
    "\n",
    "blocks = (TextBlock(batch_tokenize_tfm=batch_tok_tfm, input_return_type=CausalLMTextInput), noop)\n",
    "\n",
    "dblock = DataBlock(blocks=blocks, get_x=ColReader(\"proc_0\"), splitter=ColSplitter(col=\"is_valid\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 4: Build your `DataLoaders`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = dblock.dataloaders(proc_df, bs=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = dls.one_batch()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 129]), torch.Size([4, 129]), torch.Size([4, 129]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[0][\"input_ids\"].shape, b[0][\"labels\"].shape, b[1].shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{tuple: [dict, torch.Tensor]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explode_types(b)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "@typedispatch\n",
    "def show_batch(\n",
    "    # This typedispatched `show_batch` will be called for `CausalLMTextInput` typed inputs\n",
    "    x: CausalLMTextInput,\n",
    "    # Your targets\n",
    "    y,\n",
    "    # Your raw inputs/targets\n",
    "    samples,\n",
    "    # Your `DataLoaders`. This is required so as to get at the Hugging Face objects for\n",
    "    # decoding them into something understandable\n",
    "    dataloaders,\n",
    "    # Your `show_batch` context\n",
    "    ctxs=None,\n",
    "    # The maximum number of items to show\n",
    "    max_n=6,\n",
    "    # Any truncation your want applied to your decoded inputs\n",
    "    trunc_at=None,\n",
    "    # Any other keyword arguments you want applied to `show_batch`\n",
    "    **kwargs\n",
    "):\n",
    "    # grab our tokenizer and ignore token to decode\n",
    "    tfm = first_blurr_tfm(dataloaders)\n",
    "    hf_tokenizer = tfm.hf_tokenizer\n",
    "    ignore_token_id = tfm.ignore_token_id\n",
    "\n",
    "    res = L(\n",
    "        [\n",
    "            (\n",
    "                hf_tokenizer.decode(s[0], skip_special_tokens=False)[:trunc_at],\n",
    "                hf_tokenizer.decode(s[1][s[1] != ignore_token_id], skip_special_tokens=True)[:trunc_at],\n",
    "            )\n",
    "            for s in samples\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    display_df(pd.DataFrame(res, columns=[\"text\", \"target\"])[:max_n])\n",
    "    return ctxs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>₹ 40 million ( US $ 590 @,@ 000 ) was spent solely on VFX for Magadheera. \\n \\n = = = &lt;unk&gt; = = = \\n \\n During the film's shoot at Ramoji Film City in late November 2008, a 500 square feet ( 46 m2 ) film can, containing two or three scenes, was discovered missing from Rainbow lab. The filmmakers filed a case at &lt;unk&gt; police station. Security personnel and film unit members searched, but failed to recover the reels. Rajamouli's unit said it was not important if the scenes from</td>\n",
       "      <td>�� 40 million ( US $ 590 @,@ 000 ) was spent solely on VFX for Magadheera. \\n \\n = = = &lt;unk&gt; = = = \\n \\n During the film's shoot at Ramoji Film City in late November 2008, a 500 square feet ( 46 m2 ) film can, containing two or three scenes, was discovered missing from Rainbow lab. The filmmakers filed a case at &lt;unk&gt; police station. Security personnel and film unit members searched, but failed to recover the reels. Rajamouli's unit said it was not important if the scenes from</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ederation. Described as \" the most organized of the Northern Arabian tribes \", at the peak of its power in the 6th century BCE it controlled a large region between the Persian Gulf and the Sinai Peninsula. \\n Biblical tradition holds that the Qedarites are named for Qedar, the second son of Ishmael, mentioned in the Bible's books of Genesis ( 25 : 13 ) and 1 Chronicles ( 1 : 29 ), where there are also frequent references to Qedar as a tribe. The earliest &lt;unk&gt; inscriptions discovered by archaeol</td>\n",
       "      <td>eration. Described as \" the most organized of the Northern Arabian tribes \", at the peak of its power in the 6th century BCE it controlled a large region between the Persian Gulf and the Sinai Peninsula. \\n Biblical tradition holds that the Qedarites are named for Qedar, the second son of Ishmael, mentioned in the Bible's books of Genesis ( 25 : 13 ) and 1 Chronicles ( 1 : 29 ), where there are also frequent references to Qedar as a tribe. The earliest &lt;unk&gt; inscriptions discovered by archaeologi</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(dataloaders=dls, max_n=2, trunc_at=500)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Masked LM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 1: Get your Hugging Face objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_cls = AutoModelForMaskedLM\n",
    "hf_logging.set_verbosity_error()\n",
    "\n",
    "pretrained_model_name = \"bert-base-uncased\"\n",
    "hf_arch, hf_config, hf_tokenizer, hf_model = get_hf_objects(pretrained_model_name, model_cls=model_cls)\n",
    "\n",
    "# some tokenizers like gpt and gpt2 do not have a pad token, so we add it here mainly for the purpose\n",
    "# of setting the \"labels\" key appropriately (see below)\n",
    "if hf_tokenizer.pad_token is None:\n",
    "    hf_tokenizer.pad_token = \"[PAD]\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Step 2: Preprocess data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using eos_token, but it is not set yet.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21227\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>proc_0</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the &lt;unk&gt; season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>goal difference , before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two . This meant York qualified for the play @-@ offs , and they were eliminated in the semi @-@ final by Fleetwood Town . York were knocked out of the 2013 – 14 FA Cup , Football League Cup and Football League Trophy in their opening round matches . \\n 35 players made at least one appearance in nationally organised first @-@ team competition , and there were 12 different &lt;unk&gt; . Defender Ben Davies missed only five of the fifty @</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    proc_0  \\\n",
       "0   \\n = 2013 – 14 York City F.C. season = \\n \\n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \\n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...   \n",
       "1                            goal difference , before a 17 @-@ match unbeaten run saw the team finish in seventh @-@ place in the 24 @-@ team 2013 – 14 Football League Two . This meant York qualified for the play @-@ offs , and they were eliminated in the semi @-@ final by Fleetwood Town . York were knocked out of the 2013 – 14 FA Cup , Football League Cup and Football League Trophy in their opening round matches . \\n 35 players made at least one appearance in nationally organised first @-@ team competition , and there were 12 different <unk> . Defender Ben Davies missed only five of the fifty @   \n",
       "\n",
       "   is_valid  \n",
       "0     False  \n",
       "1     False  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor = LMPreprocessor(hf_tokenizer, chunk_size=128, text_attr=0)\n",
    "proc_df = preprocessor.process_df(train_df, valid_df)\n",
    "\n",
    "print(len(proc_df))\n",
    "proc_df.head(2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Step 3: Create your `DataBlock`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_tok_tfm = LMBatchTokenizeTransform(hf_arch, hf_config, hf_tokenizer, hf_model, lm_strategy_cls=BertMLMStrategy)\n",
    "\n",
    "blocks = (TextBlock(batch_tokenize_tfm=batch_tok_tfm, input_return_type=MLMTextInput), noop)\n",
    "\n",
    "dblock = DataBlock(blocks=blocks, get_x=ColReader(\"proc_0\"), splitter=ColSplitter(col=\"is_valid\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Step 4: Build your `DataLoaders`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = dblock.dataloaders(proc_df, bs=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 128]), torch.Size([4, 128]), torch.Size([4, 128]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = dls.one_batch()\n",
    "b[0][\"input_ids\"].shape, b[0][\"labels\"].shape, b[1].shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 101, 2003, 2098, 2340,  103, 2101,  103, 1026, 4895,  103, 1028, 1026,\n",
       "          103, 2243, 1028, 1998, 1996, 2674, 2736, 1037], device='cuda:1'),\n",
       " tensor([-100, -100, -100, -100, 2781, -100, 2083, -100, -100, 2243, -100, -100,\n",
       "         4895, -100, 1028, -100, -100, -100, -100, -100], device='cuda:1'),\n",
       " tensor([-100, -100, -100, -100, 2781, -100, 2083, -100, -100, 2243, -100, -100,\n",
       "         4895, -100, 1028, -100, -100, -100, -100, -100], device='cuda:1'))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[0][\"input_ids\"][0][:20], b[0][\"labels\"][0][:20], b[1][0][:20]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{tuple: [dict, torch.Tensor]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explode_types(b)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |export\n",
    "@typedispatch\n",
    "def show_batch(\n",
    "    # This typedispatched `show_batch` will be called for `MLMTextInput` typed inputs\n",
    "    x: MLMTextInput,\n",
    "    # Your targets\n",
    "    y,\n",
    "    # Your raw inputs/targets\n",
    "    samples,\n",
    "    # Your `DataLoaders`. This is required so as to get at the Hugging Face objects for\n",
    "    # decoding them into something understandable\n",
    "    dataloaders,\n",
    "    # Your `show_batch` context\n",
    "    ctxs=None,\n",
    "    # The maximum number of items to show\n",
    "    max_n=6,\n",
    "    # Any truncation your want applied to your decoded inputs\n",
    "    trunc_at=None,\n",
    "    # Any other keyword arguments you want applied to `show_batch`\n",
    "    **kwargs,\n",
    "):\n",
    "    # grab our tokenizer and ignore token to decode\n",
    "    tfm = first_blurr_tfm(dataloaders)\n",
    "    hf_tokenizer = tfm.hf_tokenizer\n",
    "    ignore_token_id = tfm.ignore_token_id\n",
    "\n",
    "    # grab our mask token id and do-not-mask token ids\n",
    "    mask_token_id = hf_tokenizer.mask_token_id\n",
    "\n",
    "    vocab = hf_tokenizer.get_vocab()\n",
    "    dnm_tok_ids = [vocab[tok] for tok in list(hf_tokenizer.special_tokens_map.values()) if vocab[tok] != mask_token_id]\n",
    "\n",
    "    res = L()\n",
    "    for s in samples:\n",
    "        # exclue dnm tokens from input\n",
    "        inps = [\n",
    "            hf_tokenizer.decode(tok_id) if (tok_id == mask_token_id or s[1][idx] == ignore_token_id) else f\"[{hf_tokenizer.decode(tok_id)}]\"\n",
    "            for idx, tok_id in enumerate(s[0])\n",
    "            if (tok_id not in dnm_tok_ids)\n",
    "        ]\n",
    "\n",
    "        # replaced masked tokens with \"[{actual_token}]\"\n",
    "        trgs = [\n",
    "            hf_tokenizer.decode(s[0][idx]) if (tok_id == ignore_token_id) else f\"[{hf_tokenizer.decode(tok_id)}]\"\n",
    "            for idx, tok_id in enumerate(s[1])\n",
    "            if (s[0][idx] not in dnm_tok_ids)\n",
    "        ]\n",
    "\n",
    "        res.append((\" \".join(inps[:trunc_at]).strip(), \" \".join(trgs[:trunc_at]).strip()))\n",
    "\n",
    "    display_df(pd.DataFrame(res, columns=[\"text\", \"target\"])[:max_n])\n",
    "    return ctxs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[##las] ##ed 11 minutes [MASK] through &lt; [un] ##k &gt; [MASK] un ##k &gt; and the match finished a 1 [MASK] 1 [MASK] . york [were] knocked out of the fa cup after losing 3 – 2 at home to bristol rovers in a first round [MASK] ; the [MASK] were 3 [–] 0 up by 50 @ - @ minutes before fletcher pulled two back [MASK] york with a penalty [MASK] a long @ - @ range strike . defender keith [MASK] , of cheltenham , and [MASK] nick pope [MASK] of charlton athletic , were signed on loan until january 2014 . they [MASK] played in york ' s first league [MASK] [MASK] four weeks , 2 – 1 [MASK] , to southend united</td>\n",
       "      <td>[is] ##ed 11 minutes [later] through &lt; [un] ##k &gt; [&lt;] un ##k &gt; and the match finished a 1 [–] 1 [draw] . york [were] knocked out of the fa cup after losing 3 – 2 at home to bristol rovers in a first round [replay] ; the [visitors] were 3 [–] 0 up by 50 @ - @ minutes before fletcher pulled two back [for] york with a penalty [and] a long @ - @ range strike . defender keith [lowe] , of cheltenham , and [goalkeeper] nick pope [,] of charlton athletic , were signed on loan until january 2014 . they [both] played in york ' s first league [defeat] [in] four weeks , 2 – 1 [away] , to southend united</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[MASK] ##on . [MASK] 134 ##5 [MASK] iii was planning a [MASK] assault on france . a three [MASK] [MASK] [MASK] &lt; un ##k &gt; attack would have the earl of northampton attacking from brittany [,] the [MASK] himself from flanders , while gr [bubble] ##mont was dispatched [MASK] &lt; un ##k &gt; to prepare a campaign in the south [MASK] moving rapidly through the country , he confronted the [comte] d ’ [MASK] at &lt; un ##k &gt; [on] [MASK] october and there achieved a victory described as \" [MASK] greatest single achievement of lancaster ' s entire military career \" . the ransom from the prisoners has been [MASK] at £ 50 @ , @ 000 . [MASK] next year , while edward was</td>\n",
       "      <td>[ign] ##on . [in] 134 ##5 [edward] iii was planning a [major] assault on france . a three [@] [-] [@] &lt; un ##k &gt; attack would have the earl of northampton attacking from brittany [,] the [king] himself from flanders , while gr [##os] ##mont was dispatched [to] &lt; un ##k &gt; to prepare a campaign in the south [.] moving rapidly through the country , he confronted the [comte] d ’ [isle] at &lt; un ##k &gt; [on] [21] october and there achieved a victory described as \" [the] greatest single achievement of lancaster ' s entire military career \" . the ransom from the prisoners has been [estimated] at £ 50 @ , @ 000 . [the] next year , while edward was</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(dataloaders=dls, max_n=2, trunc_at=250)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# |hide\n",
    "nbdev_export()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 ('blurr')",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}