12-HuggingFace/03-huggingface-bert-kor-text-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "06d7e9c4",
   "metadata": {},
   "source": [
    "가장 간단한 방법으로 허깅페이스 사전학습 모델을 활용한 텍스트 분류를 진행합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3f202caa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "사용 디바이스: cuda:1\n",
      "\n",
      "    Korpora 는 다른 분들이 연구 목적으로 공유해주신 말뭉치들을\n",
      "    손쉽게 다운로드, 사용할 수 있는 기능만을 제공합니다.\n",
      "\n",
      "    말뭉치들을 공유해 주신 분들에게 감사드리며, 각 말뭉치 별 설명과 라이센스를 공유 드립니다.\n",
      "    해당 말뭉치에 대해 자세히 알고 싶으신 분은 아래의 description 을 참고,\n",
      "    해당 말뭉치를 연구/상용의 목적으로 이용하실 때에는 아래의 라이센스를 참고해 주시기 바랍니다.\n",
      "\n",
      "    # Description\n",
      "    Author : e9t@github\n",
      "    Repository : https://github.com/e9t/nsmc\n",
      "    References : www.lucypark.kr/docs/2015-pyconkr/#39\n",
      "\n",
      "    Naver sentiment movie corpus v1.0\n",
      "    This is a movie review dataset in the Korean language.\n",
      "    Reviews were scraped from Naver Movies.\n",
      "\n",
      "    The dataset construction is based on the method noted in\n",
      "    [Large movie review dataset][^1] from Maas et al., 2011.\n",
      "\n",
      "    [^1]: http://ai.stanford.edu/~amaas/data/sentiment/\n",
      "\n",
      "    # License\n",
      "    CC0 1.0 Universal (CC0 1.0) Public Domain Dedication\n",
      "    Details in https://creativecommons.org/publicdomain/zero/1.0/\n",
      "\n",
      "[Korpora] Corpus `nsmc` is already installed at /root/Korpora/nsmc/ratings_train.txt\n",
      "[Korpora] Corpus `nsmc` is already installed at /root/Korpora/nsmc/ratings_test.txt\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "# 한글 자연어 처리 데이터셋\n",
    "from Korpora import Korpora\n",
    "\n",
    "# 토크나이저 관련 경고 무시하기 위하여 설정\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = 'true'\n",
    "\n",
    "# device 지정\n",
    "device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')\n",
    "print(f'사용 디바이스: {device}')\n",
    "\n",
    "corpus = Korpora.load(\"nsmc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe1375e",
   "metadata": {},
   "source": [
    "본 튜토리얼은 `Korpora`의 **네이버 영화 댓글 데이터셋**으로 진행합니다.\n",
    "\n",
    "- github 주소: https://github.com/e9t/nsmc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "43d8d9ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "train = pd.read_csv('~/Korpora/nsmc/ratings_train.txt', sep='\\t')\n",
    "test = pd.read_csv('~/Korpora/nsmc/ratings_test.txt', sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb57a10",
   "metadata": {},
   "source": [
    "문장의 길이를 계산하여 `length` 컬럼에 담습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eb1c9ff3",
   "metadata": {},
   "outputs": [],
   "source": [
    "train['length'] = train['document'].apply(lambda x: len(str(x)))\n",
    "test['length'] = test['document'].apply(lambda x: len(str(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb814f6f",
   "metadata": {},
   "source": [
    "문장의 길이가 단어 기준으로 **5 이상**인 문장만 가져옵니다. 너무 짧은 문장은 식별하기 어려워 제외토록 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9bda8639",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>document</th>\n",
       "      <th>label</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>73093</th>\n",
       "      <td>9626744</td>\n",
       "      <td>킬링타임용도 아까움. .. 외계인 불쌍함. .. 연기도 전부 개판. . 내용개판. ..</td>\n",
       "      <td>0</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39458</th>\n",
       "      <td>8632842</td>\n",
       "      <td>관람하면서 몸을 움직이고 싶어지는 영화</td>\n",
       "      <td>1</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47733</th>\n",
       "      <td>8030303</td>\n",
       "      <td>절대 10점을 쥬지않는다</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84687</th>\n",
       "      <td>10075627</td>\n",
       "      <td>이 영화는 대체 몇 번 째인지...봐도봐도 또 보고싶은 영화!</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75734</th>\n",
       "      <td>7916982</td>\n",
       "      <td>짝 퉁 쓰 레 기</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114578</th>\n",
       "      <td>8618070</td>\n",
       "      <td>기대는 안했는데 가족이란? 생각나게 하는 매력적인 영화입니다.</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34309</th>\n",
       "      <td>8832611</td>\n",
       "      <td>그나이에 전혀 현실감각 없는 이야기 소설책 부터 허접하더니 영화는 쩝...</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41251</th>\n",
       "      <td>4377580</td>\n",
       "      <td>이연걸의 액션이 멋진 영화</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24006</th>\n",
       "      <td>3895859</td>\n",
       "      <td>정치에 있어서 소신의 중요함을 다시 보게 되었다</td>\n",
       "      <td>1</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103967</th>\n",
       "      <td>4966212</td>\n",
       "      <td>미드는 시즌이 지나면 개판된다는 진리를 보여줌. 시즌1최고, 2아침드라마급, 3쓰레기</td>\n",
       "      <td>0</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id                                          document  label  \\\n",
       "73093    9626744  킬링타임용도 아까움. .. 외계인 불쌍함. .. 연기도 전부 개판. . 내용개판. ..      0   \n",
       "39458    8632842                             관람하면서 몸을 움직이고 싶어지는 영화      1   \n",
       "47733    8030303                                     절대 10점을 쥬지않는다      0   \n",
       "84687   10075627                이 영화는 대체 몇 번 째인지...봐도봐도 또 보고싶은 영화!      1   \n",
       "75734    7916982                                         짝 퉁 쓰 레 기      0   \n",
       "...          ...                                               ...    ...   \n",
       "114578   8618070                기대는 안했는데 가족이란? 생각나게 하는 매력적인 영화입니다.      1   \n",
       "34309    8832611         그나이에 전혀 현실감각 없는 이야기 소설책 부터 허접하더니 영화는 쩝...      0   \n",
       "41251    4377580                                    이연걸의 액션이 멋진 영화      1   \n",
       "24006    3895859                        정치에 있어서 소신의 중요함을 다시 보게 되었다      1   \n",
       "103967   4966212   미드는 시즌이 지나면 개판된다는 진리를 보여줌. 시즌1최고, 2아침드라마급, 3쓰레기      0   \n",
       "\n",
       "        length  \n",
       "73093       48  \n",
       "39458       21  \n",
       "47733       13  \n",
       "84687       34  \n",
       "75734        9  \n",
       "...        ...  \n",
       "114578      34  \n",
       "34309       41  \n",
       "41251       14  \n",
       "24006       26  \n",
       "103967      47  \n",
       "\n",
       "[1000 rows x 4 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = train.loc[train['length'] > 5]\n",
    "# 전체 데이터셋 크기가 커서 1000개의 문장을 샘플링 합니다.\n",
    "train = train.sample(1000)\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "562ee14c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>document</th>\n",
       "      <th>label</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14218</th>\n",
       "      <td>4437078</td>\n",
       "      <td>연기가 자연스럽지못하고, 지루했다</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36410</th>\n",
       "      <td>9990755</td>\n",
       "      <td>굿굿재밌게봤어요~^^</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17343</th>\n",
       "      <td>9964083</td>\n",
       "      <td>몰입도 안되고 이도 저도 아닌 유치하기 짝이없네</td>\n",
       "      <td>0</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40714</th>\n",
       "      <td>1599945</td>\n",
       "      <td>내 인생 최고의 영화... 너무도 잔잔하고 예쁜 영화...</td>\n",
       "      <td>1</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24556</th>\n",
       "      <td>10516</td>\n",
       "      <td>정말 닮고 싶은 캐릭터네요.. 잘된 애니 같아여..^^</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38679</th>\n",
       "      <td>8680926</td>\n",
       "      <td>야근 시트콤임 지루함 밥먹는거 밖에 안나옴</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23279</th>\n",
       "      <td>8154982</td>\n",
       "      <td>두배우의 명성에 먹칠을하는 영화임</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30633</th>\n",
       "      <td>5780174</td>\n",
       "      <td>스토리가 만화같다 연기또한.,.,</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42485</th>\n",
       "      <td>8763674</td>\n",
       "      <td>진심 1시간 개똥만 보다가 날렸네</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21779</th>\n",
       "      <td>10164582</td>\n",
       "      <td>후속작은 만들지 말아야 했다. 충분히 전편에서 주제를 심도깊게 던졌다. 그래서 영화...</td>\n",
       "      <td>0</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>500 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             id                                           document  label  \\\n",
       "14218   4437078                                 연기가 자연스럽지못하고, 지루했다      0   \n",
       "36410   9990755                                        굿굿재밌게봤어요~^^      1   \n",
       "17343   9964083                         몰입도 안되고 이도 저도 아닌 유치하기 짝이없네      0   \n",
       "40714   1599945                   내 인생 최고의 영화... 너무도 잔잔하고 예쁜 영화...      1   \n",
       "24556     10516                     정말 닮고 싶은 캐릭터네요.. 잘된 애니 같아여..^^      1   \n",
       "...         ...                                                ...    ...   \n",
       "38679   8680926                            야근 시트콤임 지루함 밥먹는거 밖에 안나옴      0   \n",
       "23279   8154982                                 두배우의 명성에 먹칠을하는 영화임      0   \n",
       "30633   5780174                                 스토리가 만화같다 연기또한.,.,      0   \n",
       "42485   8763674                                 진심 1시간 개똥만 보다가 날렸네      0   \n",
       "21779  10164582  후속작은 만들지 말아야 했다. 충분히 전편에서 주제를 심도깊게 던졌다. 그래서 영화...      0   \n",
       "\n",
       "       length  \n",
       "14218      18  \n",
       "36410      11  \n",
       "17343      26  \n",
       "40714      32  \n",
       "24556      30  \n",
       "...       ...  \n",
       "38679      23  \n",
       "23279      18  \n",
       "30633      18  \n",
       "42485      18  \n",
       "21779      58  \n",
       "\n",
       "[500 rows x 4 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test 데이터셋에도 동일하게 적용합니다.\n",
    "test = test.loc[test['length'] > 5]\n",
    "test = test.sample(500)\n",
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a42a4817",
   "metadata": {},
   "source": [
    "## 토큰화가 적용된 데이터셋\n",
    "\n",
    "먼저 `huggingface` 에서 한글 데이터셋으로 사전학습된 모델을 지정합니다.\n",
    "\n",
    "이번에는 `'kykim/bert-kor-base'`을 사용할 예정이며, 관련 내용은 아래의 링크에서 확인할 수 있습니다.\n",
    "\n",
    "- 링크: https://huggingface.co/kykim/bert-kor-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "15fb4ae9",
   "metadata": {},
   "outputs": [],
   "source": [
    "CHECKPOINT_NAME = 'kykim/bert-kor-base'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a85b56b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import BertTokenizerFast\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "\n",
    "class TokenDataset(Dataset):\n",
    "  \n",
    "    def __init__(self, dataframe, tokenizer_pretrained):\n",
    "        # sentence, label 컬럼으로 구성된 데이터프레임 전달\n",
    "        self.data = dataframe        \n",
    "        # Huggingface 토크나이저 생성\n",
    "        self.tokenizer = BertTokenizerFast.from_pretrained(tokenizer_pretrained)\n",
    "  \n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "  \n",
    "    def __getitem__(self, idx):\n",
    "        sentence = self.data.iloc[idx]['document']\n",
    "        label = self.data.iloc[idx]['label']\n",
    "\n",
    "        # 토큰화 처리\n",
    "        tokens = self.tokenizer(\n",
    "            sentence,                # 1개 문장 \n",
    "            return_tensors='pt',     # 텐서로 반환\n",
    "            truncation=True,         # 잘라내기 적용\n",
    "            padding='max_length',    # 패딩 적용\n",
    "            add_special_tokens=True  # 스페셜 토큰 적용\n",
    "        )\n",
    "\n",
    "        input_ids = tokens['input_ids'].squeeze(0)           # 2D -> 1D\n",
    "        attention_mask = tokens['attention_mask'].squeeze(0) # 2D -> 1D\n",
    "        token_type_ids = torch.zeros_like(attention_mask)\n",
    "\n",
    "        # input_ids, attention_mask, token_type_ids 이렇게 3가지 요소를 반환하도록 합니다.\n",
    "        # input_ids: 토큰\n",
    "        # attention_mask: 실제 단어가 존재하면 1, 패딩이면 0 (패딩은 0이 아닐 수 있습니다)\n",
    "        # token_type_ids: 문장을 구분하는 id. 단일 문장인 경우에는 전부 0\n",
    "        return {\n",
    "            'input_ids': input_ids,\n",
    "            'attention_mask': attention_mask, \n",
    "            'token_type_ids': token_type_ids,\n",
    "        }, torch.tensor(label)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b658f4ae",
   "metadata": {},
   "source": [
    "`nn.Dataset`을 확장한 클래스인 `TokenDataset` 클래스 인스턴스를 생성합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "98cbadb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 토크나이저 지정\n",
    "tokenizer_pretrained = CHECKPOINT_NAME\n",
    "\n",
    "# train, test 데이터셋 생성\n",
    "train_data = TokenDataset(train, tokenizer_pretrained)\n",
    "test_data = TokenDataset(test, tokenizer_pretrained)\n",
    "\n",
    "# DataLoader로 이전에 생성한 Dataset를 지정하여, batch 구성, shuffle, num_workers 등을 설정합니다.\n",
    "train_loader = DataLoader(train_data, batch_size=8, shuffle=True, num_workers=8)\n",
    "test_loader = DataLoader(test_data, batch_size=8, shuffle=True, num_workers=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ce8de0f",
   "metadata": {},
   "source": [
    "생성된 `train_loader`로부터 1개의 batch 만 꺼내서 값을 출력해 보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c6d5613c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1, 1, 0, 1, 0, 0, 1, 1], device='cuda:1')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1개의 batch 꺼내기\n",
    "inputs, labels = next(iter(train_loader))\n",
    "\n",
    "# 데이터셋을 device 설정\n",
    "inputs = {k: v.to(device) for k, v in inputs.items()}\n",
    "labels.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15183c55",
   "metadata": {},
   "source": [
    "inputs 는 입력으로 들어가는 `x` 데이터 입니다.\n",
    "\n",
    "각각의 key 에 대한 설명은 다음과 같습니다.\n",
    "\n",
    "- `input_ids`: 토큰\n",
    "- `attention_mask`: 실제 단어가 존재하면 1, 패딩이면 0 (패딩은 0이 아닐 수 있습니다)\n",
    "- `token_type_ids`: 문장을 구분하는 id. 단일 문장인 경우에는 전부 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e4db8113",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['input_ids', 'attention_mask', 'token_type_ids'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 생성된 inputs의 key 값 출력\n",
    "inputs.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2874357",
   "metadata": {},
   "source": [
    "첫 번째 **8**은 `batch_size` 입니다.\n",
    "\n",
    "그렇다면 두 번째 `512`는 의미하는 바가 뭘까요?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bee2dc16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([8, 512]), torch.Size([8, 512]), torch.Size([8, 512]))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# key 별 shape 확인\n",
    "inputs['input_ids'].shape, inputs['attention_mask'].shape, inputs['token_type_ids'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a67f72ef",
   "metadata": {},
   "source": [
    "**512**에 대한 답변은 아래 `BertConfig`를 통해 확인할 수 있습니다.\n",
    "\n",
    "아래 `\"max_position_embeddings\"` 값에 **512**가 할당되어 있는 것을 볼 수 있는데, 1개 문장의 길이가 최대 512개의 단어로 이루어지도록 embedding 되었다는 의미입니다. 즉, 1개 문장의 길이가 **512** 단어로 토크나이저된 결과이며, `batch_size`가 **8** 이기 때문에 결과 shape가 `[8, 512]`로 나오게 되었습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "87b69313",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertConfig {\n",
       "  \"architectures\": [\n",
       "    \"BertForMaskedLM\"\n",
       "  ],\n",
       "  \"attention_probs_dropout_prob\": 0.1,\n",
       "  \"classifier_dropout\": null,\n",
       "  \"directionality\": \"bidi\",\n",
       "  \"embedding_size\": 768,\n",
       "  \"gradient_checkpointing\": false,\n",
       "  \"hidden_act\": \"gelu\",\n",
       "  \"hidden_dropout_prob\": 0.1,\n",
       "  \"hidden_size\": 768,\n",
       "  \"initializer_range\": 0.02,\n",
       "  \"intermediate_size\": 3072,\n",
       "  \"layer_norm_eps\": 1e-12,\n",
       "  \"max_position_embeddings\": 512,\n",
       "  \"model_type\": \"bert\",\n",
       "  \"num_attention_heads\": 12,\n",
       "  \"num_hidden_layers\": 12,\n",
       "  \"pad_token_id\": 0,\n",
       "  \"pooler_fc_size\": 768,\n",
       "  \"pooler_num_attention_heads\": 12,\n",
       "  \"pooler_num_fc_layers\": 3,\n",
       "  \"pooler_size_per_head\": 128,\n",
       "  \"pooler_type\": \"first_token_transform\",\n",
       "  \"position_embedding_type\": \"absolute\",\n",
       "  \"transformers_version\": \"4.22.0\",\n",
       "  \"type_vocab_size\": 2,\n",
       "  \"use_cache\": true,\n",
       "  \"vocab_size\": 42000\n",
       "}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import BertConfig\n",
    "\n",
    "config = BertConfig.from_pretrained(CHECKPOINT_NAME)\n",
    "config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6be897a",
   "metadata": {},
   "source": [
    "`labels`는 one-hot encoding이 되어 있지 않은 상태로 출력됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "83db2ded",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1, 1, 0, 1, 0, 0, 1, 1])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# labels 출력\n",
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c1378d",
   "metadata": {},
   "source": [
    "## 사전학습(pre-trained) Model 생성\n",
    "\n",
    "사전학습 BERT 모델은 `BertModel`의 `from_pretrained` 함수로 간단히 가져올 수 있습니다.\n",
    "\n",
    "GPU 학습을 위하여 `.to(device)` 도 설정하였습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "483f3202",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at kykim/bert-kor-base were not used when initializing BertModel: ['cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BertModel(\n",
       "  (embeddings): BertEmbeddings(\n",
       "    (word_embeddings): Embedding(42000, 768, padding_idx=0)\n",
       "    (position_embeddings): Embedding(512, 768)\n",
       "    (token_type_embeddings): Embedding(2, 768)\n",
       "    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       "  (encoder): BertEncoder(\n",
       "    (layer): ModuleList(\n",
       "      (0): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (1): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (2): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (3): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (4): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (5): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (6): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (7): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (8): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (9): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (10): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (11): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          (intermediate_act_fn): GELUActivation()\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (pooler): BertPooler(\n",
       "    (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "    (activation): Tanh()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import BertModel\n",
    "\n",
    "# 모델 생성\n",
    "model_bert = BertModel.from_pretrained(CHECKPOINT_NAME).to(device)\n",
    "model_bert"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5613c9",
   "metadata": {},
   "source": [
    "이전에 생성한 `inputs`를 한 번 대입해본 후 결과를 확인해 보도록 하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e986faf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['last_hidden_state', 'pooler_output'])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = model_bert(**inputs)\n",
    "output.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7920156",
   "metadata": {},
   "source": [
    "결과를 `output` 에 담은 후 key를 출력해 보면 `'last_hidden_state'` 와 `'pooler_output'`이렇게 2개가 출력됩니다.\n",
    "\n",
    "- `last_hidden_state`는 배치의 각 시퀀스에서 각 토큰에 대한 숨겨진 표현을 포함합니다. 따라서 크기는 `(batch_size, seq_len, hidden_size)`입니다.\n",
    "- `pooler_output`은 배치의 각 시퀀스의 \"표현\"을 포함하며 크기`(batch_size, hidden_size)`입니다. 기본적으로 하는 일은 배치(hidden_size 크기의 벡터)에서 각 시퀀스의 `[CLS]` 토큰의 숨겨진 표현을 가져온 다음 BertPooler를 통해 실행하는 것입니다. 이것은 선형 레이어와 Tanh 활성화 함수로 구성됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "234ff7f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([8, 512, 768]), torch.Size([8, 768]))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output['last_hidden_state'].shape, output['pooler_output'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1a5866aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 512, 768])\n",
      "tensor([[-0.2648, -0.7522, -0.0201,  ...,  0.1347, -1.0979,  0.5044],\n",
      "        [-0.8496,  0.3822, -0.1804,  ...,  0.0605, -1.2255,  0.1880],\n",
      "        [ 0.1074,  0.3704, -0.4008,  ..., -0.0877, -0.9114,  0.2701],\n",
      "        ...,\n",
      "        [ 0.5518,  0.0208,  0.2635,  ...,  0.5427, -2.0081,  0.9168],\n",
      "        [ 0.8714, -0.0267,  0.2738,  ...,  0.1526, -1.0668,  0.5521],\n",
      "        [ 0.1562,  0.0058, -0.0917,  ..., -1.7647,  0.1226, -0.9639]],\n",
      "       device='cuda:1', grad_fn=<SliceBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# last_hidden_state 출력\n",
    "last_hidden_state = output['last_hidden_state']\n",
    "print(last_hidden_state.shape)\n",
    "print(last_hidden_state[:, 0, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7d4473c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 768])\n",
      "tensor([[-0.1134,  0.3340, -0.9995,  ...,  0.1017,  0.2376,  0.7940],\n",
      "        [ 0.5536,  0.3353, -0.9998,  ...,  0.7139,  0.7452, -0.9895],\n",
      "        [-0.9024, -0.0207, -0.7576,  ..., -0.9268,  0.2992,  0.9743],\n",
      "        ...,\n",
      "        [-0.5301,  0.6815, -0.8834,  ...,  0.1480,  0.3735,  0.5666],\n",
      "        [-0.9325, -0.0130, -0.7675,  ..., -0.6921,  0.0119,  0.8636],\n",
      "        [ 0.6517, -0.2224, -0.9989,  ...,  0.9627,  0.3042, -0.4665]],\n",
      "       device='cuda:1', grad_fn=<TanhBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# pooler_output 출력\n",
    "pooler_output = output['pooler_output']\n",
    "print(pooler_output.shape)\n",
    "print(pooler_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd29981b",
   "metadata": {},
   "source": [
    "output으로 나온 `last_hidden_state[:, 0, :]`의 `[CLS]` 토큰을 가져온 뒤 FC에 입력으로 대입합니다.\n",
    "\n",
    "Binary-Classificition을 위하여 최종 출력 값은 **2** 로 설정합니다.\n",
    "\n",
    "결과 shape은 `[8, 2]` 입니다. **8개의 batch** 에 대해 2개의 출력이 나왔습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5aff89bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 2])\n",
      "tensor([1, 0, 1, 1, 1, 1, 1, 1], device='cuda:1')\n"
     ]
    }
   ],
   "source": [
    "fc = nn.Linear(768, 2)\n",
    "fc.to(device)\n",
    "fc_output = fc(last_hidden_state[:, 0, :])\n",
    "print(fc_output.shape)\n",
    "print(fc_output.argmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15015a98",
   "metadata": {},
   "source": [
    "## 사전학습(pre-trained) BERT 모델을 활용한 텍스트 분류 모델 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "053f646d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomBertModel(nn.Module):\n",
    "    def __init__(self, bert_pretrained, dropout_rate=0.5):\n",
    "        # 부모클래스 초기화\n",
    "        super(CustomBertModel, self).__init__()\n",
    "        # 사전학습 모델 지정\n",
    "        self.bert = BertModel.from_pretrained(bert_pretrained)\n",
    "        # dropout 설정\n",
    "        self.dr = nn.Dropout(p=dropout_rate)\n",
    "        # 최종 출력층 정의\n",
    "        self.fc = nn.Linear(768, 2)\n",
    "    \n",
    "    def forward(self, input_ids, attention_mask, token_type_ids):\n",
    "        # 입력을 pre-trained bert model 로 대입\n",
    "        output = self.bert(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)\n",
    "        # 결과의 last_hidden_state 가져옴\n",
    "        last_hidden_state = output['last_hidden_state']\n",
    "        # last_hidden_state[:, 0, :]는 [CLS] 토큰을 가져옴\n",
    "        x = self.dr(last_hidden_state[:, 0, :])\n",
    "        # FC 을 거쳐 최종 출력\n",
    "        x = self.fc(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41e8c34e",
   "metadata": {},
   "source": [
    "위의 정의한 `CustomBertModel` 클래스 인스턴스를 생성합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f05a2387",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at kykim/bert-kor-base were not used when initializing BertModel: ['cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CustomBertModel(\n",
       "  (bert): BertModel(\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(42000, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (token_type_embeddings): Embedding(2, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (1): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (2): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (3): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (4): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (5): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (6): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (7): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (8): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (9): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (10): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (11): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (pooler): BertPooler(\n",
       "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (activation): Tanh()\n",
       "    )\n",
       "  )\n",
       "  (dr): Dropout(p=0.5, inplace=False)\n",
       "  (fc): Linear(in_features=768, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# CustomBertModel 생성\n",
    "bert = CustomBertModel(CHECKPOINT_NAME)\n",
    "bert.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7338ceb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# loss 정의: CrossEntropyLoss\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "# 옵티마이저 정의: bert.paramters()와 learning_rate 설정\n",
    "optimizer = optim.Adam(bert.parameters(), lr=1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b265dca2",
   "metadata": {},
   "source": [
    "학습(train) 함수를 정의합니다. 기존의 PyTorch 모델 학습과 별반 다르지 않습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0eae61d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm  # Progress Bar 출력\n",
    "\n",
    "def model_train(model, data_loader, loss_fn, optimizer, device):\n",
    "    # 모델을 훈련모드로 설정합니다. training mode 일 때 Gradient 가 업데이트 됩니다. 반드시 train()으로 모드 변경을 해야 합니다.\n",
    "    model.train()\n",
    "    \n",
    "    # loss와 accuracy 계산을 위한 임시 변수 입니다. 0으로 초기화합니다.\n",
    "    running_loss = 0\n",
    "    corr = 0\n",
    "    counts = 0\n",
    "    \n",
    "    # 예쁘게 Progress Bar를 출력하면서 훈련 상태를 모니터링 하기 위하여 tqdm으로 래핑합니다.\n",
    "    prograss_bar = tqdm(data_loader, unit='batch', total=len(data_loader), mininterval=1)\n",
    "    \n",
    "    # mini-batch 학습을 시작합니다.\n",
    "    for idx, (inputs, labels) in enumerate(prograss_bar):\n",
    "        # inputs, label 데이터를 device 에 올립니다. (cuda:0 혹은 cpu)\n",
    "        inputs = {k:v.to(device) for k, v in inputs.items()}\n",
    "        labels = labels.to(device)\n",
    "        \n",
    "        # 누적 Gradient를 초기화 합니다.\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # Forward Propagation을 진행하여 결과를 얻습니다.\n",
    "        output = model(**inputs)\n",
    "        \n",
    "        # 손실함수에 output, label 값을 대입하여 손실을 계산합니다.\n",
    "        loss = loss_fn(output, labels)\n",
    "        \n",
    "        # 오차역전파(Back Propagation)을 진행하여 미분 값을 계산합니다.\n",
    "        loss.backward()\n",
    "        \n",
    "        # 계산된 Gradient를 업데이트 합니다.\n",
    "        optimizer.step()\n",
    "        \n",
    "        # output의 max(dim=1)은 max probability와 max index를 반환합니다.\n",
    "        # max probability는 무시하고, max index는 pred에 저장하여 label 값과 대조하여 정확도를 도출합니다.\n",
    "        _, pred = output.max(dim=1)\n",
    "        \n",
    "        # pred.eq(lbl).sum() 은 정확히 맞춘 label의 합계를 계산합니다. item()은 tensor에서 값을 추출합니다.\n",
    "        # 합계는 corr 변수에 누적합니다.\n",
    "        corr += pred.eq(labels).sum().item()\n",
    "        counts += len(labels)\n",
    "        \n",
    "        # loss 값은 1개 배치의 평균 손실(loss) 입니다. img.size(0)은 배치사이즈(batch size) 입니다.\n",
    "        # loss 와 img.size(0)를 곱하면 1개 배치의 전체 loss가 계산됩니다.\n",
    "        # 이를 누적한 뒤 Epoch 종료시 전체 데이터셋의 개수로 나누어 평균 loss를 산출합니다.\n",
    "        running_loss += loss.item() * labels.size(0)\n",
    "        \n",
    "        # 프로그레스바에 학습 상황 업데이트\n",
    "        prograss_bar.set_description(f\"training loss: {running_loss/(idx+1):.5f}, training accuracy: {corr / counts:.5f}\")\n",
    "        \n",
    "    # 누적된 정답수를 전체 개수로 나누어 주면 정확도가 산출됩니다.\n",
    "    acc = corr / len(data_loader.dataset)\n",
    "    \n",
    "    # 평균 손실(loss)과 정확도를 반환합니다.\n",
    "    # train_loss, train_acc\n",
    "    return running_loss / len(data_loader.dataset), acc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de546ae5",
   "metadata": {},
   "source": [
    "평가(evaluation)를 위한 함수를 정의합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5d4ba450",
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_evaluate(model, data_loader, loss_fn, device):\n",
    "    # model.eval()은 모델을 평가모드로 설정을 바꾸어 줍니다. \n",
    "    # dropout과 같은 layer의 역할 변경을 위하여 evaluation 진행시 꼭 필요한 절차 입니다.\n",
    "    model.eval()\n",
    "    \n",
    "    # Gradient가 업데이트 되는 것을 방지 하기 위하여 반드시 필요합니다.\n",
    "    with torch.no_grad():\n",
    "        # loss와 accuracy 계산을 위한 임시 변수 입니다. 0으로 초기화합니다.\n",
    "        corr = 0\n",
    "        running_loss = 0\n",
    "        \n",
    "        # 배치별 evaluation을 진행합니다.\n",
    "        for inputs, labels in data_loader:\n",
    "            # inputs, label 데이터를 device 에 올립니다. (cuda:0 혹은 cpu)\n",
    "            inputs = {k:v.to(device) for k, v in inputs.items()}\n",
    "            labels = labels.to(device)\n",
    "            \n",
    "            # 모델에 Forward Propagation을 하여 결과를 도출합니다.\n",
    "            output = model(**inputs)\n",
    "            \n",
    "            # output의 max(dim=1)은 max probability와 max index를 반환합니다.\n",
    "            # max probability는 무시하고, max index는 pred에 저장하여 label 값과 대조하여 정확도를 도출합니다.\n",
    "            _, pred = output.max(dim=1)\n",
    "            \n",
    "            # pred.eq(lbl).sum() 은 정확히 맞춘 label의 합계를 계산합니다. item()은 tensor에서 값을 추출합니다.\n",
    "            # 합계는 corr 변수에 누적합니다.\n",
    "            corr += torch.sum(pred.eq(labels)).item()\n",
    "            \n",
    "            # loss 값은 1개 배치의 평균 손실(loss) 입니다. img.size(0)은 배치사이즈(batch size) 입니다.\n",
    "            # loss 와 img.size(0)를 곱하면 1개 배치의 전체 loss가 계산됩니다.\n",
    "            # 이를 누적한 뒤 Epoch 종료시 전체 데이터셋의 개수로 나누어 평균 loss를 산출합니다.\n",
    "            running_loss += loss_fn(output, labels).item() * labels.size(0)\n",
    "        \n",
    "        # validation 정확도를 계산합니다.\n",
    "        # 누적한 정답숫자를 전체 데이터셋의 숫자로 나누어 최종 accuracy를 산출합니다.\n",
    "        acc = corr / len(data_loader.dataset)\n",
    "        \n",
    "        # 결과를 반환합니다.\n",
    "        # val_loss, val_acc\n",
    "        return running_loss / len(data_loader.dataset), acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cda17eb9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 5.03556, training accuracy: 0.67800: 100% 125/125 [00:20<00:00,  6.23batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] val_loss has been improved from inf to 0.37132. Saving Model!\n",
      "epoch 01, loss: 0.62945, acc: 0.67800, val_loss: 0.37132, val_accuracy: 0.84800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 3.53300, training accuracy: 0.81700: 100% 125/125 [00:20<00:00,  6.19batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 02, loss: 0.44163, acc: 0.81700, val_loss: 0.40988, val_accuracy: 0.82600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 2.23064, training accuracy: 0.90100: 100% 125/125 [00:20<00:00,  6.19batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] val_loss has been improved from 0.37132 to 0.36758. Saving Model!\n",
      "epoch 03, loss: 0.27883, acc: 0.90100, val_loss: 0.36758, val_accuracy: 0.86600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 1.40625, training accuracy: 0.93100: 100% 125/125 [00:20<00:00,  6.17batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 04, loss: 0.17578, acc: 0.93100, val_loss: 0.41405, val_accuracy: 0.86600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 1.00858, training accuracy: 0.95900: 100% 125/125 [00:20<00:00,  6.16batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 05, loss: 0.12607, acc: 0.95900, val_loss: 0.40552, val_accuracy: 0.86600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 0.46512, training accuracy: 0.98100: 100% 125/125 [00:20<00:00,  6.15batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 06, loss: 0.05814, acc: 0.98100, val_loss: 0.54145, val_accuracy: 0.86400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 0.21964, training accuracy: 0.99200: 100% 125/125 [00:20<00:00,  6.17batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 07, loss: 0.02746, acc: 0.99200, val_loss: 0.68765, val_accuracy: 0.85600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 0.34378, training accuracy: 0.98700: 100% 125/125 [00:20<00:00,  6.16batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 08, loss: 0.04297, acc: 0.98700, val_loss: 0.51284, val_accuracy: 0.87400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 0.32060, training accuracy: 0.98900: 100% 125/125 [00:20<00:00,  6.16batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 09, loss: 0.04008, acc: 0.98900, val_loss: 0.68871, val_accuracy: 0.84600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loss: 0.25765, training accuracy: 0.99100: 100% 125/125 [00:20<00:00,  6.14batch/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, loss: 0.03221, acc: 0.99100, val_loss: 0.65872, val_accuracy: 0.86000\n"
     ]
    }
   ],
   "source": [
    "# 최대 Epoch을 지정합니다.\n",
    "num_epochs = 10\n",
    "\n",
    "# checkpoint로 저장할 모델의 이름을 정의 합니다.\n",
    "model_name = 'bert-kor-base'\n",
    "\n",
    "min_loss = np.inf\n",
    "\n",
    "# Epoch 별 훈련 및 검증을 수행합니다.\n",
    "for epoch in range(num_epochs):\n",
    "    # Model Training\n",
    "    # 훈련 손실과 정확도를 반환 받습니다.\n",
    "    train_loss, train_acc = model_train(bert, train_loader, loss_fn, optimizer, device)\n",
    "\n",
    "    # 검증 손실과 검증 정확도를 반환 받습니다.\n",
    "    val_loss, val_acc = model_evaluate(bert, test_loader, loss_fn, device)   \n",
    "    \n",
    "    # val_loss 가 개선되었다면 min_loss를 갱신하고 model의 가중치(weights)를 저장합니다.\n",
    "    if val_loss < min_loss:\n",
    "        print(f'[INFO] val_loss has been improved from {min_loss:.5f} to {val_loss:.5f}. Saving Model!')\n",
    "        min_loss = val_loss\n",
    "        torch.save(bert.state_dict(), f'{model_name}.pth')\n",
    "    \n",
    "    # Epoch 별 결과를 출력합니다.\n",
    "    print(f'epoch {epoch+1:02d}, loss: {train_loss:.5f}, acc: {train_acc:.5f}, val_loss: {val_loss:.5f}, val_accuracy: {val_acc:.5f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c2538f5",
   "metadata": {},
   "source": [
    "학습시 저장한 state_dict를 로드합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "20bf353c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 저장한 state_dict를 로드 합니다.\n",
    "bert.load_state_dict(torch.load(f'{model_name}.pth'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "d103972c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomPredictor():\n",
    "    def __init__(self, model, tokenizer, labels: dict):\n",
    "        self.model = model\n",
    "        self.tokenizer = tokenizer\n",
    "        self.labels = labels\n",
    "        \n",
    "    def predict(self, sentence):\n",
    "        # 토큰화 처리\n",
    "        tokens = self.tokenizer(\n",
    "            sentence,                # 1개 문장 \n",
    "            return_tensors='pt',     # 텐서로 반환\n",
    "            truncation=True,         # 잘라내기 적용\n",
    "            padding='max_length',    # 패딩 적용\n",
    "            add_special_tokens=True  # 스페셜 토큰 적용\n",
    "        )\n",
    "        tokens.to(device)\n",
    "        prediction = self.model(**tokens)\n",
    "        prediction = F.softmax(prediction, dim=1)\n",
    "        output = prediction.argmax(dim=1).item()\n",
    "        prob, result = prediction.max(dim=1)[0].item(), self.labels[output]\n",
    "        print(f'[{result}]\\n확률은: {prob*100:.3f}% 입니다.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "365982b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Huggingface 토크나이저 생성\n",
    "tokenizer = BertTokenizerFast.from_pretrained(CHECKPOINT_NAME)\n",
    "\n",
    "labels = {\n",
    "    0: '부정 리뷰 입니다.', \n",
    "    1: '긍정 리뷰 입니다.'\n",
    "}\n",
    "\n",
    "# CustomPredictor 인스턴스를 생성합니다.\n",
    "predictor = CustomPredictor(bert, tokenizer, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "33e12c9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 사용자 입력에 대하여 예측 후 출력을 낼 수 있는 간단한 함수를 생성합니다.\n",
    "def predict_sentence(predictor):\n",
    "    input_sentence = input('문장을 입력해 주세요: ')\n",
    "    predictor.predict(input_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "2dbe4684",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "문장을 입력해 주세요: 이 영화는 정말 더럽게 재미없네... 다신 안보련다 비추!\n",
      "[부정 리뷰 입니다.]\n",
      "확률은: 93.257% 입니다.\n"
     ]
    }
   ],
   "source": [
    "# 부정 리뷰 입력 예시\n",
    "predict_sentence(predictor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "2f1cd581",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "문장을 입력해 주세요: 인생 최고의 영화였다. 진짜 배우들의 명연기가 돋보이는 영화\n",
      "[긍정 리뷰 입니다.]\n",
      "확률은: 99.676% 입니다.\n"
     ]
    }
   ],
   "source": [
    "# 긍정 리뷰 입력 예시\n",
    "predict_sentence(predictor)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}