02-PyTorch/23-seq2seq-챗봇.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "be745cb0",
   "metadata": {},
   "source": [
    "## 모듈 import & 데이터 로드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "id": "7aabd1bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12시 땡!</td>\n",
       "      <td>하루가 또 가네요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11818</th>\n",
       "      <td>훔쳐보는 것도 눈치 보임.</td>\n",
       "      <td>티가 나니까 눈치가 보이는 거죠!</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11819</th>\n",
       "      <td>훔쳐보는 것도 눈치 보임.</td>\n",
       "      <td>훔쳐보는 거 티나나봐요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11820</th>\n",
       "      <td>흑기사 해주는 짝남.</td>\n",
       "      <td>설렜겠어요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11821</th>\n",
       "      <td>힘든 연애 좋은 연애라는게 무슨 차이일까?</td>\n",
       "      <td>잘 헤어질 수 있는 사이 여부인 거 같아요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11822</th>\n",
       "      <td>힘들어서 결혼할까봐</td>\n",
       "      <td>도피성 결혼은 하지 않길 바라요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11823 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Q                         A  label\n",
       "0                       12시 땡!                하루가 또 가네요.      0\n",
       "1                  1지망 학교 떨어졌어                 위로해 드립니다.      0\n",
       "2                 3박4일 놀러가고 싶다               여행은 언제나 좋죠.      0\n",
       "3              3박4일 정도 놀러가고 싶다               여행은 언제나 좋죠.      0\n",
       "4                      PPL 심하네                눈살이 찌푸려지죠.      0\n",
       "...                        ...                       ...    ...\n",
       "11818           훔쳐보는 것도 눈치 보임.        티가 나니까 눈치가 보이는 거죠!      2\n",
       "11819           훔쳐보는 것도 눈치 보임.             훔쳐보는 거 티나나봐요.      2\n",
       "11820              흑기사 해주는 짝남.                    설렜겠어요.      2\n",
       "11821  힘든 연애 좋은 연애라는게 무슨 차이일까?  잘 헤어질 수 있는 사이 여부인 거 같아요.      2\n",
       "11822               힘들어서 결혼할까봐        도피성 결혼은 하지 않길 바라요.      2\n",
       "\n",
       "[11823 rows x 3 columns]"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "\n",
    "data_dir = 'data'\n",
    "\n",
    "df = pd.read_csv(os.path.join(data_dir, 'ChatbotData.csv'))\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "id": "a21e2a89",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = df['Q']\n",
    "answer = df['A']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "id": "6b84af8d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             12시 땡!\n",
       "1        1지망 학교 떨어졌어\n",
       "2       3박4일 놀러가고 싶다\n",
       "3    3박4일 정도 놀러가고 싶다\n",
       "4            PPL 심하네\n",
       "Name: Q, dtype: object"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "8e1e1f97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     하루가 또 가네요.\n",
       "1      위로해 드립니다.\n",
       "2    여행은 언제나 좋죠.\n",
       "3    여행은 언제나 좋죠.\n",
       "4     눈살이 찌푸려지죠.\n",
       "Name: A, dtype: object"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d719c14",
   "metadata": {},
   "source": [
    "## 1. 데이터 전처리"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd3ce925",
   "metadata": {},
   "source": [
    "### 1-1. 한글 정규화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "id": "0bdebcb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "re.compile(r'[^ ?,.!A-Za-z0-9가-힣+]', re.UNICODE)"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "# 한글, 영어, 숫자, 공백, ?!.,을 제외한 나머지 문자 제거\n",
    "korean_pattern = r'[^ ?,.!A-Za-z0-9가-힣+]'\n",
    "\n",
    "# 패턴 컴파일\n",
    "normalizer = re.compile(korean_pattern)\n",
    "normalizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "id": "7bd1ad37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "수정 전: SNS보면 나만 빼고 다 행복해보여\n",
      "수정 후: SNS보면 나만 빼고 다 행복해보여\n"
     ]
    }
   ],
   "source": [
    "print(f'수정 전: {question[10]}')\n",
    "print(f'수정 후: {normalizer.sub(\"\", question[10])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "id": "4c918464",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "수정 전: 자랑하는 자리니까요.\n",
      "수정 후: 자랑하는 자리니까요.\n"
     ]
    }
   ],
   "source": [
    "print(f'수정 전: {answer[10]}')\n",
    "print(f'수정 후: {normalizer.sub(\"\", answer[10])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "id": "2316a26e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'SNS보면 나만 빼고 다 행복해보여'"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def normalize(sentence):\n",
    "    return normalizer.sub(\"\", sentence)\n",
    "\n",
    "normalize(question[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "690d1d77",
   "metadata": {},
   "source": [
    "### 1-2. 한글 형태소 분석기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "003483bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from konlpy.tag import Mecab, Okt\n",
    "\n",
    "# 형태소 분석기\n",
    "mecab = Mecab()\n",
    "okt = Okt()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "aeca6d91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SNS', '보', '면', '나', '만', '빼', '고', '다', '행복', '해', '보여']"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mecab\n",
    "mecab.morphs(normalize(question[10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "b855766c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['자랑', '하는', '자리', '니까', '요', '.']"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# okt\n",
    "okt.morphs(normalize(answer[10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "a90d6f6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 한글 전처리를 함수화\n",
    "def clean_text(sentence, tagger):\n",
    "    sentence = normalize(sentence)\n",
    "    sentence = tagger.morphs(sentence)\n",
    "    sentence = ' '.join(sentence)\n",
    "    sentence = sentence.lower()\n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "id": "1f0d5ba3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sns 보면 나 만 빼고 다 행복 해보여'"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 한글\n",
    "clean_text(question[10], okt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "id": "cea788a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'자랑 하는 자리 니까 요 .'"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 영어\n",
    "clean_text(answer[10], okt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "730f1941",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11823, 11823)"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(question), len(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "4ce95632",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = [clean_text(sent, okt) for sent in question.values[:1000]]\n",
    "answers = [clean_text(sent, okt) for sent in answer.values[:1000]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "148bfa28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['12시 땡 !', '1 지망 학교 떨어졌어', '3 박 4일 놀러 가고 싶다', '3 박 4일 정도 놀러 가고 싶다', 'ppl 심하네']"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "73844c99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['하루 가 또 가네요 .',\n",
       " '위로 해 드립니다 .',\n",
       " '여행 은 언제나 좋죠 .',\n",
       " '여행 은 언제나 좋죠 .',\n",
       " '눈살 이 찌푸려지죠 .']"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answers[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fa240d8",
   "metadata": {},
   "source": [
    "### 1-3. 단어 사전 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "id": "6887a3ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda', index=0)"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data.dataset import Dataset\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "id": "e75b7101",
   "metadata": {},
   "outputs": [],
   "source": [
    "PAD_TOKEN = 0\n",
    "SOS_TOKEN = 1\n",
    "EOS_TOKEN = 2\n",
    "\n",
    "\n",
    "class WordVocab():\n",
    "    def __init__(self):\n",
    "        self.word2index = {\n",
    "            '<PAD>': PAD_TOKEN,\n",
    "            '<SOS>': SOS_TOKEN, \n",
    "            '<EOS>': EOS_TOKEN,\n",
    "        }\n",
    "        self.word2count = {}\n",
    "        self.index2word = {\n",
    "            PAD_TOKEN: '<PAD>', \n",
    "            SOS_TOKEN: '<SOS>', \n",
    "            EOS_TOKEN: '<EOS>'\n",
    "        }\n",
    "        \n",
    "        self.n_words = 3  # PAD, SOS, EOS 포함\n",
    "\n",
    "    def add_sentence(self, sentence):\n",
    "        for word in sentence.split(' '):\n",
    "            self.add_word(word)\n",
    "\n",
    "    def add_word(self, word):\n",
    "        if word not in self.word2index:\n",
    "            self.word2index[word] = self.n_words\n",
    "            self.word2count[word] = 1\n",
    "            self.index2word[self.n_words] = word\n",
    "            self.n_words += 1\n",
    "        else:\n",
    "            self.word2count[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "id": "60766793",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sns 보면 나 만 빼고 다 행복 해보여'"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "id": "5aa82bb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "원문: sns 보면 나 만 빼고 다 행복 해보여\n",
      "==============================\n",
      "단어사전\n",
      "{'<PAD>': 0, '<SOS>': 1, '<EOS>': 2, 'sns': 3, '보면': 4, '나': 5, '만': 6, '빼고': 7, '다': 8, '행복': 9, '해보여': 10}\n"
     ]
    }
   ],
   "source": [
    "print(f'원문: {questions[10]}')\n",
    "lang = WordVocab()\n",
    "lang.add_sentence(questions[10])\n",
    "print('==='*10)\n",
    "print('단어사전')\n",
    "print(lang.word2index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fb512c",
   "metadata": {},
   "source": [
    "### 1-4. padding to sequences\n",
    "\n",
    "- 하나의 배치 구성을 위해서는 문장의 길이가 맞아야 합니다.\n",
    "- 하지만, 문장 별로 길이가 다르기 때문에 길이를 맞춰 주는 작업을 수행해야 합니다.\n",
    "- 짧은 문장은 남은 공간에 PAD 토큰을 추가하여 길이를 맞춰 주도록 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "id": "158c431e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Sentence: [92, 29, 34, 32, 10, 26]\n",
      "Output: [92, 29, 34, 32, 10, 26, 2, 0, 0, 0]\n",
      "Total Length: 10\n"
     ]
    }
   ],
   "source": [
    "max_length = 10\n",
    "sentence_length = 6\n",
    "\n",
    "sentence_tokens = np.random.randint(low=3, high=100, size=(sentence_length,))\n",
    "sentence_tokens = sentence_tokens.tolist()\n",
    "print(f'Generated Sentence: {sentence_tokens}')\n",
    "\n",
    "sentence_tokens = sentence_tokens[:(max_length-1)]\n",
    "\n",
    "token_length = len(sentence_tokens)\n",
    "\n",
    "# 문장의 맨 끝부분에 <EOS> 토큰 추가\n",
    "sentence_tokens.append(2)\n",
    "\n",
    "for i in range(token_length, max_length-1):\n",
    "    # 나머지 빈 곳에 <PAD> 토큰 추가\n",
    "    sentence_tokens.append(0)\n",
    "\n",
    "print(f'Output: {sentence_tokens}')\n",
    "print(f'Total Length: {len(sentence_tokens)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1fd2384",
   "metadata": {},
   "source": [
    "### 1-5. 전처리 프로세스 클래스화\n",
    "\n",
    "- `torch.utils.data.Dataset`을 상속 받아 `TextDataset` 클래스를 구현합니다.\n",
    "- 데이터를 로드하고, 정규화 및 전처리, 토큰화를 진행합니다.\n",
    "- 단어 사전을 생성하고 이에 따라, 시퀀스로 변환합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "id": "303e404e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from konlpy.tag import Mecab, Okt\n",
    "\n",
    "\n",
    "class TextDataset(Dataset):\n",
    "    def __init__(self, csv_path, min_length=3, max_length=32):\n",
    "        super(TextDataset, self).__init__()\n",
    "        data_dir = 'data'\n",
    "        \n",
    "        # TOKEN 정의\n",
    "        self.PAD_TOKEN = 0 # Padding 토큰\n",
    "        self.SOS_TOKEN = 1 # SOS 토큰\n",
    "        self.EOS_TOKEN = 2 # EOS 토큰\n",
    "        \n",
    "        self.tagger = Mecab()   # 형태소 분석기\n",
    "        self.max_length = max_length # 한 문장의 최대 길이 지정\n",
    "        \n",
    "        # CSV 데이터 로드\n",
    "        df = pd.read_csv(os.path.join(data_dir, csv_path))\n",
    "        \n",
    "        # 한글 정규화\n",
    "        korean_pattern = r'[^ ?,.!A-Za-z0-9가-힣+]'\n",
    "        self.normalizer = re.compile(korean_pattern)\n",
    "        \n",
    "        # src: 질의, tgt: 답변\n",
    "        src_clean = []\n",
    "        tgt_clean = []\n",
    "        \n",
    "        # 단어 사전 생성\n",
    "        wordvocab = WordVocab()\n",
    "        \n",
    "        for _, row in df.iterrows():\n",
    "            src = row['Q']\n",
    "            tgt = row['A']\n",
    "            \n",
    "            # 한글 전처리\n",
    "            src = self.clean_text(src)\n",
    "            tgt = self.clean_text(tgt)\n",
    "            \n",
    "            if len(src.split()) > min_length and len(tgt.split()) > min_length:\n",
    "                # 최소 길이를 넘어가는 문장의 단어만 추가\n",
    "                wordvocab.add_sentence(src)\n",
    "                wordvocab.add_sentence(tgt)\n",
    "                src_clean.append(src)\n",
    "                tgt_clean.append(tgt)            \n",
    "        \n",
    "        self.srcs = src_clean\n",
    "        self.tgts = tgt_clean\n",
    "        self.wordvocab = wordvocab\n",
    "\n",
    "    \n",
    "    def normalize(self, sentence):\n",
    "        # 정규표현식에 따른 한글 정규화\n",
    "        return self.normalizer.sub(\"\", sentence)\n",
    "\n",
    "    def clean_text(self, sentence):\n",
    "        # 한글 정규화\n",
    "        sentence = self.normalize(sentence)\n",
    "        # 형태소 처리\n",
    "        sentence = self.tagger.morphs(sentence)\n",
    "        sentence = ' '.join(sentence)\n",
    "        sentence = sentence.lower()\n",
    "        return sentence\n",
    "    \n",
    "    def texts_to_sequences(self, sentence):\n",
    "        # 문장 -> 시퀀스로 변환\n",
    "        return [self.wordvocab.word2index[w] for w in sentence.split()]\n",
    "\n",
    "    def pad_sequence(self, sentence_tokens):\n",
    "        # 문장의 맨 끝 토큰은 제거\n",
    "        sentence_tokens = sentence_tokens[:(self.max_length-1)]\n",
    "        token_length = len(sentence_tokens)\n",
    "\n",
    "        # 문장의 맨 끝부분에 <EOS> 토큰 추가\n",
    "        sentence_tokens.append(self.EOS_TOKEN)\n",
    "\n",
    "        for i in range(token_length, (self.max_length-1)):\n",
    "            # 나머지 빈 곳에 <PAD> 토큰 추가\n",
    "            sentence_tokens.append(self.PAD_TOKEN)\n",
    "        return sentence_tokens\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        inputs = self.srcs[idx]\n",
    "        inputs_sequences = self.texts_to_sequences(inputs)\n",
    "        inputs_padded = self.pad_sequence(inputs_sequences)\n",
    "        \n",
    "        outputs = self.tgts[idx]\n",
    "        outputs_sequences = self.texts_to_sequences(outputs)\n",
    "        outputs_padded = self.pad_sequence(outputs_sequences)\n",
    "        \n",
    "        return torch.tensor(inputs_padded), torch.tensor(outputs_padded)\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.srcs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "id": "02edbeb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 한 문장의 최대 단어길이를 25로 설정\n",
    "MAX_LENGTH = 25\n",
    "\n",
    "dataset = TextDataset('ChatbotData.csv', min_length=3, max_length=MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "id": "b3369f04",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10번째 데이터 임의 추출\n",
    "x, y = dataset[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad58adee",
   "metadata": {},
   "source": [
    "- 문장의 맨 끝에는 2번 토큰(EOS 토큰)이 위치합니다.\n",
    "- EOS 토큰부터 max_length 까지는 PAD 토큰으로 채워집니다. 여기서 0번 토큰이 PAD 토큰 입니다.\n",
    "- x, y 데이터 모두 `max_length=25`의 크기를 가집니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "id": "5b910123",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x shape: torch.Size([25])\n",
      "tensor([83, 84, 51, 85, 86, 18,  2,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "         0,  0,  0,  0,  0,  0,  0])\n"
     ]
    }
   ],
   "source": [
    "print(f'x shape: {x.shape}')\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "id": "b8db4966",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y shape: torch.Size([25])\n",
      "tensor([87, 88, 58, 89, 63, 90, 11,  2,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
      "         0,  0,  0,  0,  0,  0,  0])\n"
     ]
    }
   ],
   "source": [
    "print(f'y shape: {y.shape}')\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b59ed00f",
   "metadata": {},
   "source": [
    "### 1-6. train / test 데이터셋 분할"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "id": "85f8825a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8168"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 80%의 데이터를 train에 할당합니다.\n",
    "train_size = int(len(dataset) * 0.8)\n",
    "train_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "id": "9b5d8817",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2042"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 나머지 20% 데이터를 test에 할당합니다.\n",
    "test_size = len(dataset) - train_size\n",
    "test_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "id": "ec552a83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import random_split\n",
    "\n",
    "# 랜덤 스플릿으로 분할을 완료합니다.\n",
    "train_dataset, test_dataset = random_split(dataset, [train_size, test_size])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6001bf5a",
   "metadata": {},
   "source": [
    "### 1-7. DataLoader 생성\n",
    "\n",
    "- 배치 구성을 쉽게 하기 위해서 `torch.utils.data.DataLoader`를 활용합니다.\n",
    "- train/test 데이터셋 모두 `batch_size=16` 으로 설정하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "id": "75ba4f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader, SubsetRandomSampler\n",
    "\n",
    "train_loader = DataLoader(train_dataset, \n",
    "                          batch_size=16, \n",
    "                          shuffle=True)\n",
    "\n",
    "test_loader = DataLoader(test_dataset, \n",
    "                         batch_size=16, \n",
    "                         shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "id": "e0a2d840",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1개의 배치 데이터를 추출합니다.\n",
    "x, y = next(iter(train_loader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "id": "5d8d2de0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 25]), torch.Size([16, 25]))"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shape: (batch_size, sequence_length)\n",
    "x.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44168ab8",
   "metadata": {},
   "source": [
    "## 2. 모델"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c15f6490",
   "metadata": {},
   "source": [
    "### 2-1. Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "id": "70706f67",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, num_vocabs, hidden_size, embedding_dim, num_layers):\n",
    "        super(Encoder, self).__init__()\n",
    "        \n",
    "        # 단어 사전의 개수 지정\n",
    "        self.num_vocabs = num_vocabs\n",
    "        # 임베딩 레이어 정의 (number of vocabs, embedding dimension)\n",
    "        self.embedding = nn.Embedding(num_vocabs, embedding_dim)\n",
    "        # GRU (embedding dimension)\n",
    "        self.gru = nn.GRU(embedding_dim, \n",
    "                          hidden_size, \n",
    "                          num_layers=num_layers, \n",
    "                          bidirectional=False)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.embedding(x).permute(1, 0, 2)\n",
    "        output, hidden = self.gru(x)\n",
    "        return output, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23f9b4a0",
   "metadata": {},
   "source": [
    "#### 2-1-1. Embedding Layer의 입/출력 shape에 대한 이해"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "id": "6e645c01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 25])\n",
      "torch.Size([16, 25, 64])\n"
     ]
    }
   ],
   "source": [
    "embedding_dim = 64 # 임베딩 차원\n",
    "embedding = nn.Embedding(dataset.wordvocab.n_words, embedding_dim)\n",
    "\n",
    "# x의 shape을 변경합니다.\n",
    "# (batch_size, sequence_length) => (sequence_length, batch_size)\n",
    "embedded = embedding(x)\n",
    "\n",
    "print(x.shape)\n",
    "print(embedded.shape)\n",
    "# input:  (sequence_length, batch_size)\n",
    "# output: (sequence_length, batch_size, embedding_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e4ff8e4",
   "metadata": {},
   "source": [
    "embedding 레이어를 통과한 출력을 `(batch_size, sequence_length, embedding_dim)` => `(sequence_length, batch_size, embedding_dim)` shape 변환을 위하여 `permute(1, 0, 2)`를 수행합니다.\n",
    "\n",
    "여기서 shape를 변환하는 이유는 GRU 레이어의 입력이 `(sequence_length, batch_size, embedding_dim)` 을 수용하기 때문입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "id": "3a9972d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([25, 16, 64])\n"
     ]
    }
   ],
   "source": [
    "embedded = embedded.permute(1, 0, 2)\n",
    "print(embedded.shape)\n",
    "# (sequence_length, batch_size, embedding_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15e0f2cf",
   "metadata": {},
   "source": [
    "#### 2-1-2.  GRU Layer의 입/출력 shape에 대한 이해"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "id": "581c15b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([25, 16, 32])\n",
      "torch.Size([1, 16, 32])\n"
     ]
    }
   ],
   "source": [
    "hidden_size = 32   \n",
    "\n",
    "gru = nn.GRU(embedding_dim,      # embedding 차원\n",
    "             hidden_size, \n",
    "             num_layers=1, \n",
    "             bidirectional=False)\n",
    "\n",
    "# input       : (sequence_length, batch_size, embedding_dim)\n",
    "# h0          : (Bidirectional(1) x number of layers(1), batch_size, hidden_size)\n",
    "o, h = gru(embedded, None)\n",
    "\n",
    "print(o.shape)\n",
    "print(h.shape)\n",
    "# output      : (sequence_length, batch_size, hidden_size x bidirectional(1))\n",
    "# hidden_state: (bidirectional(1) x number of layers(1), batch_size, hidden_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b84e3cee",
   "metadata": {},
   "source": [
    "#### 2-1-3.  Encoder의 입/출력 shape에 대한 이해"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "id": "bbb2ce08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of vocabs: 6417\n"
     ]
    }
   ],
   "source": [
    "NUM_VOCABS = dataset.wordvocab.n_words\n",
    "print(f'number of vocabs: {NUM_VOCABS}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "id": "78d30d4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encoder 정의\n",
    "encoder = Encoder(NUM_VOCABS, \n",
    "                  hidden_size=32, \n",
    "                  embedding_dim=64, \n",
    "                  num_layers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "id": "18357f63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([25, 16, 32])\n",
      "torch.Size([1, 16, 32])\n"
     ]
    }
   ],
   "source": [
    "# Encoder에 x 통과 후 output, hidden_size 의 shape 확인\n",
    "# input(x)    : (batch_size, sequence_length)\n",
    "o, h = encoder(x)\n",
    "\n",
    "print(o.shape)\n",
    "print(h.shape)\n",
    "# output      : (sequence_length, batch_size, hidden_size x bidirectional(1))\n",
    "# hidden_state: (bidirectional(1) x number of layers(1), batch_size, hidden_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4786ad2b",
   "metadata": {},
   "source": [
    "### 2-2. Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "id": "3e45f493",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, num_vocabs, hidden_size, embedding_dim, num_layers=1, dropout=0.2):\n",
    "        super(Decoder, self).__init__()\n",
    "        # 단어사전 개수\n",
    "        self.num_vocabs = num_vocabs\n",
    "        self.embedding = nn.Embedding(num_vocabs, embedding_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.gru = nn.GRU(embedding_dim, \n",
    "                          hidden_size, \n",
    "                          num_layers=num_layers, \n",
    "                          bidirectional=False)\n",
    "        \n",
    "        # 최종 출력은 단어사전의 개수\n",
    "        self.fc = nn.Linear(hidden_size, num_vocabs)\n",
    "        \n",
    "    def forward(self, x, hidden_state):\n",
    "        x = x.unsqueeze(0) # (1, batch_size) 로 변환\n",
    "        embedded = F.relu(self.embedding(x))\n",
    "        embedded = self.dropout(embedded)\n",
    "        output, hidden = self.gru(embedded, hidden_state)\n",
    "        output = self.fc(output.squeeze(0)) # (sequence_length, batch_size, hidden_size(32) x bidirectional(1))\n",
    "        return output, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2db142",
   "metadata": {},
   "source": [
    "#### 2-2-1. Embedding Layer의 입/출력 shape에 대한 이해"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "id": "576e0f66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 2, 1, 0]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 16])"
      ]
     },
     "execution_count": 265,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.abs(torch.randn(size=(1, 16)).long())\n",
    "print(x)\n",
    "x.shape\n",
    "# batch_size = 16 이라 가정했을 때,\n",
    "# (1, batch_size)\n",
    "# 여기서 batch_size => (1, batch_size) 로 shape 변환을 선행"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "id": "643c2b56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 16, 64])"
      ]
     },
     "execution_count": 266,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_dim = 64 # 임베딩 차원\n",
    "embedding = nn.Embedding(dataset.wordvocab.n_words, embedding_dim)\n",
    "\n",
    "embedded = embedding(x)\n",
    "embedded.shape\n",
    "# embedding 출력\n",
    "# (1, batch_size, embedding_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16c71b03",
   "metadata": {},
   "source": [
    "#### 2-2-2. GRU Layer의 입/출력 shape에 대한 이해"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "id": "69a0d8ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 16, 32])\n",
      "torch.Size([1, 16, 32])\n"
     ]
    }
   ],
   "source": [
    "hidden_size = 32\n",
    "\n",
    "gru = nn.GRU(embedding_dim, \n",
    "             hidden_size, \n",
    "             num_layers=1, \n",
    "             bidirectional=False, \n",
    "             batch_first=False, # batch_first=False로 지정\n",
    "            )\n",
    "\n",
    "o, h = gru(embedded)\n",
    "\n",
    "print(o.shape)\n",
    "# output shape: (sequence_length, batch_size, hidden_size(32) x bidirectional(1))\n",
    "print(h.shape)\n",
    "# hidden_state shape: (Bidirectional(1) x number of layers(1), batch_size, hidden_size(32))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab2f79c",
   "metadata": {},
   "source": [
    "#### 2-2-3. 최종 출력층(FC) shape에 대한 이해"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "id": "c6c6180e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 32])\n",
      "torch.Size([16, 6417])\n"
     ]
    }
   ],
   "source": [
    "fc = nn.Linear(32, NUM_VOCABS) # 출력은 단어사전의 개수로 가정\n",
    "\n",
    "output = fc(o[0])\n",
    "\n",
    "print(o[0].shape)\n",
    "print(output.shape)\n",
    "# input : (batch_size, output from GRU)\n",
    "# output: (batch_size, output dimension)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cdd9550",
   "metadata": {},
   "source": [
    "### 2-3. 인코더 -> 디코더 입출력 shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "id": "7937ff1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder = Decoder(num_vocabs=dataset.wordvocab.n_words, \n",
    "                  hidden_size=32, \n",
    "                  embedding_dim=64, \n",
    "                  num_layers=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f252cf92",
   "metadata": {},
   "source": [
    "디코더에 입력될 인코더의 `output`, `hidden_state`의 shape을 확인합니다.\n",
    "**\n",
    "- 여기서 **`hidden_state`만 디코더의 입력** 으로 활용합니다.\n",
    "- `x`는 SOS 토큰이 첫 번째 입력으로 들어갑니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "id": "9f81859c",
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper__index_select)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[270], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m x, y \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mnext\u001b[39m(\u001b[38;5;28miter\u001b[39m(train_loader))\n\u001b[0;32m----> 3\u001b[0m o, h \u001b[38;5;241m=\u001b[39m \u001b[43mencoder\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      5\u001b[0m \u001b[38;5;28mprint\u001b[39m(o\u001b[38;5;241m.\u001b[39mshape, h\u001b[38;5;241m.\u001b[39mshape)\n\u001b[1;32m      6\u001b[0m \u001b[38;5;66;03m# output      : (batch_size, sequence_length, hidden_size(32) x bidirectional(1))\u001b[39;00m\n\u001b[1;32m      7\u001b[0m \u001b[38;5;66;03m# hidden_state: (Bidirectional(1) x number of layers(1), batch_size, hidden_size(32))\u001b[39;00m\n",
      "File \u001b[0;32m/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py:1102\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1098\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1099\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1100\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1103\u001b[0m \u001b[38;5;66;03m# Do not call functions when jit is used\u001b[39;00m\n\u001b[1;32m   1104\u001b[0m full_backward_hooks, non_full_backward_hooks \u001b[38;5;241m=\u001b[39m [], []\n",
      "Cell \u001b[0;32mIn[236], line 16\u001b[0m, in \u001b[0;36mEncoder.forward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, x):\n\u001b[0;32m---> 16\u001b[0m     x \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43membedding\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mpermute(\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m2\u001b[39m)\n\u001b[1;32m     17\u001b[0m     output, hidden \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgru(x)\n\u001b[1;32m     18\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m output, hidden\n",
      "File \u001b[0;32m/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py:1102\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1098\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1099\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1100\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1103\u001b[0m \u001b[38;5;66;03m# Do not call functions when jit is used\u001b[39;00m\n\u001b[1;32m   1104\u001b[0m full_backward_hooks, non_full_backward_hooks \u001b[38;5;241m=\u001b[39m [], []\n",
      "File \u001b[0;32m/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/sparse.py:158\u001b[0m, in \u001b[0;36mEmbedding.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    157\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;28minput\u001b[39m: Tensor) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Tensor:\n\u001b[0;32m--> 158\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mF\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43membedding\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    159\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpadding_idx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmax_norm\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    160\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnorm_type\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscale_grad_by_freq\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msparse\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/functional.py:2044\u001b[0m, in \u001b[0;36membedding\u001b[0;34m(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)\u001b[0m\n\u001b[1;32m   2038\u001b[0m     \u001b[38;5;66;03m# Note [embedding_renorm set_grad_enabled]\u001b[39;00m\n\u001b[1;32m   2039\u001b[0m     \u001b[38;5;66;03m# XXX: equivalent to\u001b[39;00m\n\u001b[1;32m   2040\u001b[0m     \u001b[38;5;66;03m# with torch.no_grad():\u001b[39;00m\n\u001b[1;32m   2041\u001b[0m     \u001b[38;5;66;03m#   torch.embedding_renorm_\u001b[39;00m\n\u001b[1;32m   2042\u001b[0m     \u001b[38;5;66;03m# remove once script supports set_grad_enabled\u001b[39;00m\n\u001b[1;32m   2043\u001b[0m     _no_grad_embedding_renorm_(weight, \u001b[38;5;28minput\u001b[39m, max_norm, norm_type)\n\u001b[0;32m-> 2044\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43membedding\u001b[49m\u001b[43m(\u001b[49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpadding_idx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mscale_grad_by_freq\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msparse\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper__index_select)"
     ]
    }
   ],
   "source": [
    "x, y = next(iter(train_loader))\n",
    "\n",
    "o, h = encoder(x)\n",
    "\n",
    "print(o.shape, h.shape)\n",
    "# output      : (batch_size, sequence_length, hidden_size(32) x bidirectional(1))\n",
    "# hidden_state: (Bidirectional(1) x number of layers(1), batch_size, hidden_size(32))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee6789c1",
   "metadata": {},
   "source": [
    "인코더(Encoder)로부터 생성된 hidden_state(h)와 SOS 토큰을 디코더(Decoder)의 입력으로 넣어줍니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "id": "22ea6114",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([16])"
      ]
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.abs(torch.full(size=(16,), fill_value=SOS_TOKEN).long())\n",
    "print(x)\n",
    "x.shape\n",
    "# batch_size = 16 이라 가정(16개의 SOS 토큰)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "id": "723aa6e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 6417]), torch.Size([1, 16, 32]))"
      ]
     },
     "execution_count": 272,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_output, decoder_hidden = decoder(x, h)\n",
    "decoder_output.shape, decoder_hidden.shape\n",
    "# (batch_size, num_vocabs), (1, batch_size, hidden_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1a486dd",
   "metadata": {},
   "source": [
    "- `decoder_output`은 `(batch_size, num_vocabs)` shape로 출력\n",
    "- `decoder_hidden`의 shape는 입력으로 넣어준 shape와 동일함을 확인"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd459e0",
   "metadata": {},
   "source": [
    "### 2-4. Seq2Seq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "id": "6f8727da",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device):\n",
    "        super(Seq2Seq, self).__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.device = device\n",
    "        \n",
    "    def forward(self, inputs, outputs, teacher_forcing_ratio=0.5):\n",
    "        # inputs : (batch_size, sequence_length)\n",
    "        # outputs: (batch_size, sequence_length)\n",
    "        \n",
    "        batch_size, output_length = outputs.shape\n",
    "        output_num_vocabs = self.decoder.num_vocabs\n",
    "        \n",
    "        # 리턴할 예측된 outputs를 저장할 임시 변수\n",
    "        # (sequence_length, batch_size, num_vocabs)\n",
    "        predicted_outputs = torch.zeros(output_length, batch_size, output_num_vocabs).to(self.device)\n",
    "        \n",
    "        # 인코더에 입력 데이터 주입, encoder_output은 버리고 hidden_state 만 살립니다. \n",
    "        # 여기서 hidden_state가 디코더에 주입할 context vector 입니다.\n",
    "        # (Bidirectional(1) x number of layers(1), batch_size, hidden_size)\n",
    "        _, decoder_hidden = self.encoder(inputs)\n",
    "        \n",
    "        # (batch_size) shape의 SOS TOKEN으로 채워진 디코더 입력 생성\n",
    "        decoder_input = torch.full((batch_size,), SOS_TOKEN, device=self.device)\n",
    "        \n",
    "        # 순회하면서 출력 단어를 생성합니다.\n",
    "        # 0번째는 SOS TOKEN이 위치하므로, 1번째 인덱스부터 순회합니다.\n",
    "        for t in range(0, output_length):\n",
    "            # decoder_input : 디코더 입력 (batch_size) 형태의 SOS TOKEN로 채워진 입력\n",
    "            # decoder_output: (batch_size, num_vocabs)\n",
    "            # decoder_hidden: (Bidirectional(1) x number of layers(1), batch_size, hidden_size), context vector와 동일 shape\n",
    "            decoder_output, decoder_hidden = self.decoder(decoder_input, decoder_hidden)\n",
    "\n",
    "            # t번째 단어에 디코더의 output 저장\n",
    "            predicted_outputs[t] = decoder_output\n",
    "            \n",
    "            # teacher forcing 적용 여부 확률로 결정\n",
    "            # teacher forcing 이란: 정답치를 다음 RNN Cell의 입력으로 넣어주는 경우. 수렴속도가 빠를 수 있으나, 불안정할 수 있음\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "            \n",
    "            # top1 단어 토큰 예측\n",
    "            top1 = decoder_output.argmax(1) \n",
    "            \n",
    "            # teacher forcing 인 경우 ground truth 값을, 그렇지 않은 경우, 예측 값을 다음 input으로 지정\n",
    "            decoder_input = outputs[:, t] if teacher_force else top1\n",
    "        \n",
    "        return predicted_outputs.permute(1, 0, 2) # (batch_size, sequence_length, num_vocabs)로 변경"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2187c300",
   "metadata": {},
   "source": [
    "#### 2-4-1. Seq2Seq 입출력 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "id": "038673be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encoder 정의\n",
    "encoder = Encoder(num_vocabs=dataset.wordvocab.n_words, \n",
    "                       hidden_size=32, \n",
    "                       embedding_dim=64, \n",
    "                       num_layers=1)\n",
    "# Decoder 정의\n",
    "decoder = Decoder(num_vocabs=dataset.wordvocab.n_words, \n",
    "                       hidden_size=32, \n",
    "                       embedding_dim=64, \n",
    "                       num_layers=1)\n",
    "# Seq2Seq 정의\n",
    "seq2seq = Seq2Seq(encoder, decoder, 'cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "id": "cdbf3a7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 25]) torch.Size([16, 25])\n"
     ]
    }
   ],
   "source": [
    "x, y = next(iter(train_loader))\n",
    "print(x.shape, y.shape)\n",
    "# (batch_size, sequence_length), (batch_size, sequence_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "id": "b85bb34d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 25, 6417])\n"
     ]
    }
   ],
   "source": [
    "output = seq2seq(x, y)\n",
    "print(output.shape)\n",
    "# (batch_size, sequence_length, num_vocabs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0012cd56",
   "metadata": {},
   "source": [
    "## 3. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "id": "7be02adc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_vocabs: 6417\n",
      "======================\n",
      "Seq2Seq(\n",
      "  (encoder): Encoder(\n",
      "    (embedding): Embedding(6417, 256)\n",
      "    (gru): GRU(256, 512)\n",
      "  )\n",
      "  (decoder): Decoder(\n",
      "    (embedding): Embedding(6417, 256)\n",
      "    (dropout): Dropout(p=0.2, inplace=False)\n",
      "    (gru): GRU(256, 512)\n",
      "    (fc): Linear(in_features=512, out_features=6417, bias=True)\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "NUM_VOCABS = dataset.wordvocab.n_words\n",
    "HIDDEN_SIZE = 512\n",
    "EMBEDDIMG_DIM = 256\n",
    "\n",
    "print(f'num_vocabs: {NUM_VOCABS}\\n======================')\n",
    "\n",
    "# Encoder 정의\n",
    "encoder = Encoder(num_vocabs=NUM_VOCABS, \n",
    "                  hidden_size=HIDDEN_SIZE, \n",
    "                  embedding_dim=EMBEDDIMG_DIM, \n",
    "                  num_layers=1)\n",
    "# Decoder 정의\n",
    "decoder = Decoder(num_vocabs=NUM_VOCABS, \n",
    "                  hidden_size=HIDDEN_SIZE, \n",
    "                  embedding_dim=EMBEDDIMG_DIM, \n",
    "                  num_layers=1)\n",
    "\n",
    "# Seq2Seq 생성\n",
    "# encoder, decoder를 device 모두 지정\n",
    "model = Seq2Seq(encoder.to(device), decoder.to(device), device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c9c956e",
   "metadata": {},
   "source": [
    "### 3-1. Hyperparamter 정의"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "id": "e04b3d64",
   "metadata": {},
   "outputs": [],
   "source": [
    "class EarlyStopping:\n",
    "    def __init__(self, patience=3, delta=0.0, mode='min', verbose=True):\n",
    "        \"\"\"\n",
    "        patience (int): loss or score가 개선된 후 기다리는 기간. default: 3\n",
    "        delta  (float): 개선시 인정되는 최소 변화 수치. default: 0.0\n",
    "        mode     (str): 개선시 최소/최대값 기준 선정('min' or 'max'). default: 'min'.\n",
    "        verbose (bool): 메시지 출력. default: True\n",
    "        \"\"\"\n",
    "        self.early_stop = False\n",
    "        self.patience = patience\n",
    "        self.verbose = verbose\n",
    "        self.counter = 0\n",
    "        \n",
    "        self.best_score = np.Inf if mode == 'min' else 0\n",
    "        self.mode = mode\n",
    "        self.delta = delta\n",
    "        \n",
    "\n",
    "    def __call__(self, score):\n",
    "\n",
    "        if self.best_score is None:\n",
    "            self.best_score = score\n",
    "            self.counter = 0\n",
    "        elif self.mode == 'min':\n",
    "            if score < (self.best_score - self.delta):\n",
    "                self.counter = 0\n",
    "                self.best_score = score\n",
    "                if self.verbose:\n",
    "                    print(f'[EarlyStopping] (Update) Best Score: {self.best_score:.5f}')\n",
    "            else:\n",
    "                self.counter += 1\n",
    "                if self.verbose:\n",
    "                    print(f'[EarlyStopping] (Patience) {self.counter}/{self.patience}, ' \\\n",
    "                          f'Best: {self.best_score:.5f}' \\\n",
    "                          f', Current: {score:.5f}, Delta: {np.abs(self.best_score - score):.5f}')\n",
    "                \n",
    "        elif self.mode == 'max':\n",
    "            if score > (self.best_score + self.delta):\n",
    "                self.counter = 0\n",
    "                self.best_score = score\n",
    "                if self.verbose:\n",
    "                    print(f'[EarlyStopping] (Update) Best Score: {self.best_score:.5f}')\n",
    "            else:\n",
    "                self.counter += 1\n",
    "                if self.verbose:\n",
    "                    print(f'[EarlyStopping] (Patience) {self.counter}/{self.patience}, ' \\\n",
    "                          f'Best: {self.best_score:.5f}' \\\n",
    "                          f', Current: {score:.5f}, Delta: {np.abs(self.best_score - score):.5f}')\n",
    "                \n",
    "            \n",
    "        if self.counter >= self.patience:\n",
    "            if self.verbose:\n",
    "                print(f'[EarlyStop Triggered] Best Score: {self.best_score:.5f}')\n",
    "            # Early Stop\n",
    "            self.early_stop = True\n",
    "        else:\n",
    "            # Continue\n",
    "            self.early_stop = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54188410",
   "metadata": {},
   "source": [
    "훈련에 적용할 하이퍼파라미터 설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "id": "000fdc1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "LR = 1e-3\n",
    "optimizer = optim.Adam(model.parameters(), lr=LR)\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "es = EarlyStopping(patience=5, \n",
    "                   delta=0.001, \n",
    "                   mode='min', \n",
    "                   verbose=True\n",
    "                  )\n",
    "\n",
    "scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, \n",
    "                                                 mode='min', \n",
    "                                                 factor=0.5, \n",
    "                                                 patience=2,\n",
    "                                                 threshold_mode='abs',\n",
    "                                                 min_lr=1e-8, \n",
    "                                                 verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b20fe21b",
   "metadata": {},
   "source": [
    "### 3-2. train 함수 정의"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "id": "07b38a4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, data_loader, optimizer, loss_fn, device):\n",
    "    model.train()\n",
    "    running_loss = 0\n",
    "    \n",
    "    for x, y in data_loader:\n",
    "        x, y = x.to(device), y.to(device)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # output: (batch_size, sequence_length, num_vocabs)\n",
    "        output = model(x, y)\n",
    "        output_dim = output.size(2)\n",
    "        \n",
    "        # 1번 index 부터 슬라이싱한 이유는 0번 index가 SOS TOKEN 이기 때문\n",
    "        # (batch_size*sequence_length, num_vocabs) 로 변경\n",
    "        output = output.reshape(-1, output_dim)\n",
    "        \n",
    "        # (batch_size*sequence_length) 로 변경\n",
    "        y = y.view(-1)\n",
    "        \n",
    "        # Loss 계산\n",
    "        loss = loss_fn(output, y)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        running_loss += loss.item() * x.size(0)\n",
    "        \n",
    "    return running_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33b10132",
   "metadata": {},
   "source": [
    "### 3-3. evaluation 함수 정의"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "id": "81c765b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, data_loader, loss_fn, device):\n",
    "    model.eval()\n",
    "    \n",
    "    eval_loss = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for x, y in data_loader:\n",
    "            x, y = x.to(device), y.to(device)\n",
    "            output = model(x, y)\n",
    "            output_dim = output.size(2)\n",
    "            output = output.reshape(-1, output_dim)\n",
    "            y = y.view(-1)\n",
    "            \n",
    "            # Loss 계산\n",
    "            loss = loss_fn(output, y)\n",
    "            \n",
    "            eval_loss += loss.item() * x.size(0)\n",
    "            \n",
    "    return eval_loss / len(data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90690a6a",
   "metadata": {},
   "source": [
    "### 3-4. 랜덤 샘플링 후 결과 추론"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "id": "d2e58bc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sequence_to_sentence(sequences, index2word):\n",
    "    outputs = []\n",
    "    for p in sequences:\n",
    "\n",
    "        word = index2word[p]\n",
    "        if p not in [SOS_TOKEN, EOS_TOKEN, PAD_TOKEN]:\n",
    "            outputs.append(word)\n",
    "        if word == EOS_TOKEN:\n",
    "            break\n",
    "    return ' '.join(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc24d22b",
   "metadata": {},
   "source": [
    "sequence를 다시 문장으로 바꾸어 문장 형식으로 출력하기 위한 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "id": "e6c44255",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_evaluation(model, dataset, index2word, device, n=10):\n",
    "    \n",
    "    n_samples = len(dataset)\n",
    "    indices = list(range(n_samples))\n",
    "    np.random.shuffle(indices)      # Shuffle\n",
    "    sampled_indices = indices[:n]   # Sampling N indices\n",
    "    \n",
    "    # 샘플링한 데이터를 기반으로 DataLoader 생성\n",
    "    sampler = SubsetRandomSampler(sampled_indices)\n",
    "    sampled_dataloader = DataLoader(dataset, batch_size=10, sampler=sampler)\n",
    "    \n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for x, y in sampled_dataloader:\n",
    "            x, y = x.to(device), y.to(device)        \n",
    "            output = model(x, y, teacher_forcing_ratio=0)\n",
    "            # output: (number of samples, sequence_length, num_vocabs)\n",
    "            \n",
    "            preds = output.detach().cpu().numpy()\n",
    "            x = x.detach().cpu().numpy()\n",
    "            y = y.detach().cpu().numpy()\n",
    "            \n",
    "            for i in range(n):\n",
    "                print(f'질문   : {sequence_to_sentence(x[i], index2word)}')\n",
    "                print(f'답변   : {sequence_to_sentence(y[i], index2word)}')\n",
    "                print(f'예측답변: {sequence_to_sentence(preds[i].argmax(1), index2word)}')\n",
    "                print('==='*10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffacc0b4",
   "metadata": {},
   "source": [
    "### 3-5. 훈련 시작"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 284,
   "id": "db078cf1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1, loss: 32.2713, val_loss: 28.9118\n",
      "[EarlyStopping] (Update) Best Score: 32.27133\n",
      "[EarlyStopping] (Update) Best Score: 27.90689\n",
      "[EarlyStopping] (Update) Best Score: 25.78647\n",
      "[EarlyStopping] (Update) Best Score: 23.60776\n",
      "[EarlyStopping] (Update) Best Score: 20.68632\n",
      "epoch: 6, loss: 17.8025, val_loss: 27.6363\n",
      "[EarlyStopping] (Update) Best Score: 17.80247\n",
      "[EarlyStopping] (Update) Best Score: 14.78749\n",
      "[EarlyStopping] (Update) Best Score: 12.07858\n",
      "Epoch     8: reducing learning rate of group 0 to 5.0000e-04.\n",
      "[EarlyStopping] (Update) Best Score: 8.72596\n",
      "[EarlyStopping] (Update) Best Score: 6.87622\n",
      "epoch: 11, loss: 5.6122, val_loss: 31.1174\n",
      "[EarlyStopping] (Update) Best Score: 5.61224\n",
      "Epoch    11: reducing learning rate of group 0 to 2.5000e-04.\n",
      "[EarlyStopping] (Update) Best Score: 4.26539\n",
      "[EarlyStopping] (Update) Best Score: 3.53884\n",
      "[EarlyStopping] (Update) Best Score: 3.03033\n",
      "Epoch    14: reducing learning rate of group 0 to 1.2500e-04.\n",
      "[EarlyStopping] (Update) Best Score: 2.41656\n",
      "epoch: 16, loss: 2.1939, val_loss: 33.0993\n",
      "[EarlyStopping] (Update) Best Score: 2.19391\n",
      "[EarlyStopping] (Update) Best Score: 2.03166\n",
      "Epoch    17: reducing learning rate of group 0 to 6.2500e-05.\n",
      "[EarlyStopping] (Update) Best Score: 1.75492\n",
      "[EarlyStopping] (Update) Best Score: 1.62426\n",
      "[EarlyStopping] (Update) Best Score: 1.50042\n",
      "Epoch    20: reducing learning rate of group 0 to 3.1250e-05.\n"
     ]
    }
   ],
   "source": [
    "NUM_EPOCHS = 20\n",
    "STATEDICT_PATH = 'models/seq2seq-chatbot-kor.pt'\n",
    "\n",
    "best_loss = np.inf\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    loss = train(model, train_loader, optimizer, loss_fn, device)\n",
    "    \n",
    "    val_loss = evaluate(model, test_loader, loss_fn, device)\n",
    "    \n",
    "    if val_loss < best_loss:\n",
    "        best_loss = val_loss\n",
    "        torch.save(model.state_dict(), STATEDICT_PATH)\n",
    "    \n",
    "    if epoch % 5 == 0:\n",
    "        print(f'epoch: {epoch+1}, loss: {loss:.4f}, val_loss: {val_loss:.4f}')\n",
    "    \n",
    "    # Early Stop\n",
    "    es(loss)\n",
    "    if es.early_stop:\n",
    "        break\n",
    "    \n",
    "    # Scheduler\n",
    "    scheduler.step(val_loss)\n",
    "                   \n",
    "model.load_state_dict(torch.load(STATEDICT_PATH))\n",
    "torch.save(model.state_dict(), f'models/seq2seq-chatbot-kor-{best_loss:.4f}.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bafd07db",
   "metadata": {},
   "source": [
    "## 결과"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "id": "8d437e50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "질문   : 대 기업 아니 어도 될까 ?\n",
      "답변   : 어디 에서 일 하 든 상관 없 어요 .\n",
      "예측답변: 먼저 고백 해 보 세요 .\n",
      "==============================\n",
      "질문   : 술기운 에 연락 했 는데 .\n",
      "답변   : 후회 하 지 않 길 바라 요 .\n",
      "예측답변: 이렇게 고민 한 일 이 었 나 봐요 .\n",
      "==============================\n",
      "질문   : 별 이 안 보여\n",
      "답변   : 한적 한 시골 에서 하늘 을 올려 봐 보 세요 .\n",
      "예측답변: 눈 을 해 보 세요 .\n",
      "==============================\n",
      "질문   : 좋 아 하 는 이상형 이 계속 바뀌 어 .\n",
      "답변   : 성격 도 계속 바뀌 니 걱정 말 아요 .\n",
      "예측답변: 이상형 을 사랑 하 는 게 좋 겠 어요 .\n",
      "==============================\n",
      "질문   : 이 기회 잡 고 싶 다 .\n",
      "답변   : 행운 을 빌 게 요 !\n",
      "예측답변: 저 도 그럴 거 예요 .\n",
      "==============================\n",
      "질문   : 건강 이 최고 인 것 같 아\n",
      "답변   : 가장 중요 한 목표 네요 .\n",
      "예측답변: 눈 이 는 게 이 죠 .\n",
      "==============================\n",
      "질문   : 사무실 에 나왔 지만 손 에 안 잡히 네\n",
      "답변   : 복잡 한 심경 인가 봐요 .\n",
      "예측답변: as 에 에 가 고 나 봐요 .\n",
      "==============================\n",
      "질문   : 명치 쪽 이 답답 해\n",
      "답변   : 많이 답답 할 거 라 생각 해요 .\n",
      "예측답변: 기분 전환 해 보 세요 .\n",
      "==============================\n",
      "질문   : 6 년 그리고 남 은 것 들\n",
      "답변   : 좋 은 기억 들 만 남 았 길 바랄게요 .\n",
      "예측답변: 시간 이 시간 이 필요 하 겠 지만 잘 이겨낼 수 도 있 어요 .\n",
      "==============================\n",
      "질문   : 운동 좀 해야겠다 .\n",
      "답변   : 건강 생각 해서 꾸준히 하 세요 .\n",
      "예측답변: 네 말씀 해 보 세요 .\n",
      "==============================\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load(STATEDICT_PATH))\n",
    "random_evaluation(model, test_dataset, dataset.wordvocab.index2word, device)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}