13-ChatGPT/02-news-article.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f558c41f",
   "metadata": {},
   "source": [
    "## 진행 과정\n",
    "\n",
    "1. investing.com의 뉴스기사(본문) 크롤링.\n",
    "2. chatgpt 에 요약 요청.\n",
    "3. 요약을 한글로 변환."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7eec0f",
   "metadata": {},
   "source": [
    "## beautifulsoup4 + requests 로 크롤링 시도\n",
    "\n",
    "- 시도를 해봤으나 사이트에서 requests + beautifulsoup4 를 사용한 크롤링 시도가 막혀 있어서 **사용이 불가**했습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2cf8a15",
   "metadata": {},
   "source": [
    "실행하기 위한 설치코드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69a9cb15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install beautifulsoup4\n",
    "# !pip install requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccfe7c32",
   "metadata": {},
   "source": [
    "요청을 위한 코드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "33891407",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [403]>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup as bs\n",
    "from fake_useragent import UserAgent\n",
    "\n",
    "# user-agent 설정\n",
    "ua = UserAgent()\n",
    "headers = {'User-Agent':str(ua.chrome)}\n",
    "\n",
    "# 요청하고자 하는 샘플 뉴스기사 URL\n",
    "url = 'https://www.investing.com/analysis/us-stock-market-has-plenty-of-reasons-to-rally-after-feds-decision-200634857'\n",
    "\n",
    "# requests 요청\n",
    "page = requests.get(url, headers=headers)\n",
    "page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e4afcb",
   "metadata": {},
   "source": [
    "결과는 `403` response가 뜨는 것을 확인할 수 있습니다.\n",
    "\n",
    "따라서, `selenium` 으로 크롤링을 진행하는 것으로 우회하도록 하겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb20ff1b",
   "metadata": {},
   "source": [
    "## Selenium 으로 크롤링"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e25df33f",
   "metadata": {},
   "source": [
    "`Selenium` 설치 코드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "d0ca526f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "623eb1af",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from webdriver_manager.chrome import ChromeDriverManager\n",
    "from selenium.webdriver.chrome.service import Service\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "# 크롬드라이버 셋팅\n",
    "def set_chrome_driver(headless=True):\n",
    "    options = webdriver.ChromeOptions()\n",
    "    if headless:\n",
    "        options.add_argument('headless')\n",
    "    options.add_argument(\"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36\")\n",
    "    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)\n",
    "    return driver"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae56e979",
   "metadata": {},
   "source": [
    "`set_chrome_driver`는 chrome 드라이버를 셋업하는 함수입니다.\n",
    "\n",
    "만약 로컬에 chromedriver가 존재하지 않으면 알아서 **올바른 버전을 다운로드** 진행하게 됩니다(이전에는 chromedriver를 직접 다운로드 했었어야 합니다)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "cdee1fcd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<selenium.webdriver.remote.webelement.WebElement (session=\"d0e6dbbcd5f3fa889cfa228f445c9576\", element=\"0053d389-fa03-426c-b735-52c2dc17e736\")>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# driver 설정\n",
    "driver = set_chrome_driver(False)\n",
    "\n",
    "# URL 요청\n",
    "driver.get(url)\n",
    "\n",
    "# aritivlePage는 신문기사의 본문입니다\n",
    "article_page = driver.find_element(By.CLASS_NAME, 'articlePage')\n",
    "article_page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb043189",
   "metadata": {},
   "source": [
    "아래의 코드를 실행하여 신문기사의 본문이 잘 크롤링 되었는지 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "4cc7580c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Francesco Casarella/Investing.com\n",
      "Articles (31)\n",
      "Follow\n",
      "  US500\n",
      "-0.53%\n",
      "Markets are cautious ahead of the Fed\n",
      "After a tumultuous 2022, many investors are waiting on the sidelines, holding cash and waiting to enter\n",
      "With risk-off sentiment dominating and plenty of liquidity on the sidelines, markets could rally in the second half of the week\n",
      "Yesterday, the S&P 500 closed lower. This is nothing new, considering the same thing happened the last three times Powell spoke.\n",
      "S&P 500 Daily Chart\n",
      "I don't expect any surprises. A 25bp hike and Powell maintaining his stance on fighting inflation (\"we're improving, but it's not time to rest yet\") is likely. As always, the markets are pricing in such a scenario.\n",
      "In the meantime, while the focus is still on the recession and earnings (we will have a dedicated analysis as soon as the quarters are over), there are other situations worthy of consideration.\n",
      "Cash Levels at All-Time Highs\n",
      "After the sell-off in 2022, there is still a lot of liquidity on the sidelines that needs to be deployed. We can see above that several funds are at record highs not seen for years (curiously, they were also at very high levels in 2009 as the market recovered from the subprime bubble).\n",
      "The buyback announcements made by various companies in January could help support prices.\n",
      "US Share Buyback Authorizations\n",
      "Generally, we are not seeing the euphoria typical of bubble bursts, where the collapse comes after markets are taken by surprise.\n",
      "After a year like 2022, the markets are already negative as far as sentiment is concerned, and if we look at the chart below, we can see that traders are still in a risk-off mode.\n",
      "Usually, when traders are negative, there is a lot of caution, and as a result, it is difficult to be caught off guard if the market declines further.\n",
      "Risk Off/Risk On Sentiment\n",
      "However, the surprise could come from the opposite direction. A continuation of the rally could generate a buying frenzy in a self-reinforcing mechanism between closing shorts and new buying.\n",
      "In this sense, this week will not be so much about the FOMC's decision on the size of the hike, nor even about Powell's words (which I think will be confirmed as hawkish).\n",
      "Instead, it will be about how the markets react in the second half of the week and the week after.\n",
      "Disclaimer: This article is written for informational purposes only; it does not constitute a solicitation, offer, advice, counseling, or recommendation to invest as such it is not intended to incentivize the purchase of assets in any way. I want to remind you that any type of asset is evaluated from multiple points of view and is highly risky; therefore, any investment decision and the associated risk remain with the investor.\n"
     ]
    }
   ],
   "source": [
    "# 신문기사의 본문을 출력합니다.\n",
    "print(article_page.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f05a23d0",
   "metadata": {},
   "source": [
    "## OpenAI API 사용\n",
    "\n",
    "**API KEY 발급 방법**\n",
    "- API KEY 신청 주소\n",
    "- https://beta.openai.com/ 회원 가입 후\n",
    "- https://beta.openai.com/account/api-keys\n",
    "- create new secret key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "271dbf1c",
   "metadata": {},
   "source": [
    "`openai` 라이브러리 설치코드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a04a0011",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de1b419",
   "metadata": {},
   "source": [
    "먼저 `openai` 라이브러리 import 후 **API KEY**를 설정 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "0b0c8a5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "# API 키 설정\n",
    "openai.api_key = \"OpenAI API Key 입력\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8df3a3",
   "metadata": {},
   "source": [
    "요약 및 긍/부정 감정 분석을 진행하는 프롬프트를 생성하여 출력 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "249cd651",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summarize the paragraph below and interpret whether it is a positive or negative sentiment.\n",
      "\n",
      "Francesco Casarella/Investing.com\n",
      "Articles (31)\n",
      "Follow\n",
      "  US500\n",
      "-0.53%\n",
      "Markets are cautious ahead of the Fed\n",
      "After a tumultuous 2022, many investors are waiting on the sidelines, holding cash and waiting to enter\n",
      "With risk-off sentiment dominating and plenty of liquidity on the sidelines, markets could rally in the second half of the week\n",
      "Yesterday, the S&P 500 closed lower. This is nothing new, considering the same thing happened the last three times Powell spoke.\n",
      "S&P 500 Daily Chart\n",
      "I don't expect any surprises. A 25bp hike and Powell maintaining his stance on fighting inflation (\"we're improving, but it's not time to rest yet\") is likely. As always, the markets are pricing in such a scenario.\n",
      "In the meantime, while the focus is still on the recession and earnings (we will have a dedicated analysis as soon as the quarters are over), there are other situations worthy of consideration.\n",
      "Cash Levels at All-Time Highs\n",
      "After the sell-off in 2022, there is still a lot of liquidity on the sidelines that needs to be deployed. We can see above that several funds are at record highs not seen for years (curiously, they were also at very high levels in 2009 as the market recovered from the subprime bubble).\n",
      "The buyback announcements made by various companies in January could help support prices.\n",
      "US Share Buyback Authorizations\n",
      "Generally, we are not seeing the euphoria typical of bubble bursts, where the collapse comes after markets are taken by surprise.\n",
      "After a year like 2022, the markets are already negative as far as sentiment is concerned, and if we look at the chart below, we can see that traders are still in a risk-off mode.\n",
      "Usually, when traders are negative, there is a lot of caution, and as a result, it is difficult to be caught off guard if the market declines further.\n",
      "Risk Off/Risk On Sentiment\n",
      "However, the surprise could come from the opposite direction. A continuation of the rally could generate a buying frenzy in a self-reinforcing mechanism between closing shorts and new buying.\n",
      "In this sense, this week will not be so much about the FOMC's decision on the size of the hike, nor even about Powell's words (which I think will be confirmed as hawkish).\n",
      "Instead, it will be about how the markets react in the second half of the week and the week after.\n",
      "Disclaimer: This article is written for informational purposes only; it does not constitute a solicitation, offer, advice, counseling, or recommendation to invest as such it is not intended to incentivize the purchase of assets in any way. I want to remind you that any type of asset is evaluated from multiple points of view and is highly risky; therefore, any investment decision and the associated risk remain with the investor.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 프롬프트 (요약해줘 + 긍/부정 감정도 분석해줘)\n",
    "prompt = f'''\n",
    "Summarize the paragraph below and interpret whether it is a positive or negative sentiment.\n",
    "\n",
    "{article_page.text}\n",
    "'''\n",
    "print(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e888150c",
   "metadata": {},
   "source": [
    "`ChatGPT`에 요약을 요청합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "20cbed79",
   "metadata": {},
   "outputs": [],
   "source": [
    "def summarize(prompt):\n",
    "    # 모델 엔진 선택\n",
    "    model_engine = \"text-davinci-003\"\n",
    "\n",
    "    # 맥스 토큰\n",
    "    max_tokens = 3000\n",
    "\n",
    "    # 요약 요청\n",
    "    completion = openai.Completion.create(\n",
    "        engine=model_engine,\n",
    "        prompt=prompt,\n",
    "        max_tokens=max_tokens,\n",
    "        temperature=0.3,       # creativity\n",
    "        top_p=1,\n",
    "        frequency_penalty=0,\n",
    "        presence_penalty=0\n",
    "    )\n",
    "    return completion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "98647b63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "This paragraph is discussing the cautious sentiment of investors in the markets due to the tumultuous year of 2022 and the potential for a rally in the second half of the week. It also mentions the high levels of cash on the sidelines and the potential for a buying frenzy if the markets continue to rally. Overall, this is a positive sentiment.\n"
     ]
    }
   ],
   "source": [
    "# 요약 요청후 결과 return\n",
    "response = summarize(prompt)\n",
    "\n",
    "# 결과 출력\n",
    "print(response.choices[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5e89bb",
   "metadata": {},
   "source": [
    "## 파파고 번역\n",
    "\n",
    "요약본을 한글로 번역하기 위하여 PAPAGO 번역 사이트를 이용하였습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81eab9d2",
   "metadata": {},
   "source": [
    "**프로세스**\n",
    "\n",
    "1. 파파고 번역 요청을 위한 `selenium` 브라우저를 생성\n",
    "2. 영문 텍스트를 입력\n",
    "3. 번역된 한글 텍스트를 크롤링"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "5e9ae318",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 샘플 텍스트\n",
    "summarized = '''This paragraph is summarizing the current state of the stock market, with the Dow Jones Industrial Average, S&P 500, and NASDAQ Composite all down, while Gold Futures and Advanced Micro Devices Inc (AMD) rose. Investors are cautious ahead of the Federal Reserve's rate decision, and corporate earnings are mixed, with Electronic Arts Inc (EA) and Snap Inc (SNAP) falling and AMD rising. Oil prices are also down. Overall, the sentiment of the paragraph is negative.'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "6ba5a4c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "이 단락은 다우존스 산업평균지수, S&P 500, 나스닥 종합지수가 모두 하락한 반면 골드 퓨처스와 어드밴스드 마이크로 디바이스 주식회사(AMD)는 상승하는 등 주식시장의 현주소를 요약하고 있다. 미국 연방준비제도(Fed·연준)의 금리 결정을 앞두고 투자자들이 신중한 모습을 보이고 있고, 일렉트로닉아츠(EA)와 스냅인(SNAP)이 하락하고 AMD가 상승하는 등 기업 실적도 엇갈리고 있다. 유가도 하락했다. 전반적으로 문단의 정서는 부정적이다.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# 1. 파파고 번역 요청을 위한 selenium 생성\n",
    "papago = set_chrome_driver(False)\n",
    "papago.get(f'https://papago.naver.com/')\n",
    "\n",
    "# 2. 영문 문장 입력\n",
    "papago.find_element(By.ID, 'txtSource').send_keys(summarized)\n",
    "# 2. 번역 버튼 클릭\n",
    "papago.find_element(By.ID, 'btnTranslate').click()\n",
    "\n",
    "# 강제 지연 시간: 번역을 기다릴 수 있도록 2초 슬립\n",
    "time.sleep(2)\n",
    "\n",
    "# 3. 번역된 한글 텍스트 크롤링\n",
    "papago_translated = papago.find_element(By.ID, 'targetEditArea')\n",
    "print(papago_translated.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dc67d94",
   "metadata": {},
   "source": [
    "위의 내용을 아래에 함수화 합니다.\n",
    "\n",
    "번역된 텍스트 크롤링 시 오류가 날 수 있으므로, 이에 대한 **예외처리**도 적용합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "4fb51086",
   "metadata": {},
   "outputs": [],
   "source": [
    "def papago_translate(text):\n",
    "    try:\n",
    "        papago = set_chrome_driver(False)\n",
    "        papago.get('https://papago.naver.com/')\n",
    "        time.sleep(1)\n",
    "        papago.find_element(By.ID, 'txtSource').send_keys(text)\n",
    "        papago.find_element(By.ID, 'btnTranslate').click()\n",
    "        time.sleep(2)\n",
    "        papago_translated = papago.find_element(By.ID, 'targetEditArea')\n",
    "        result = papago_translated.text\n",
    "    except NoSuchElementException: # 예외처리 (요소를 찾지 못하는 경우)\n",
    "        result = '번역 오류ㅠㅠ'\n",
    "    finally:\n",
    "        papago.close()\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91a91690",
   "metadata": {},
   "source": [
    "## 최종 뉴스기사 크롤링+요약+번역 코드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "435ecd70",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from selenium import webdriver\n",
    "from webdriver_manager.chrome import ChromeDriverManager\n",
    "from selenium.webdriver.chrome.service import Service\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.common.exceptions import NoSuchElementException\n",
    "import openai\n",
    "\n",
    "\n",
    "# OPENAI API키 설정\n",
    "openai.api_key = \"OpenAI API Key 입력\"\n",
    "\n",
    "# 크롬드라이버 셋팅\n",
    "def set_chrome_driver(headless=True):\n",
    "    options = webdriver.ChromeOptions()\n",
    "    if headless:\n",
    "        options.add_argument('headless')\n",
    "    options.add_argument(\"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36\")\n",
    "    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)\n",
    "    return driver\n",
    "\n",
    "# 뉴스 페이지 크롤링\n",
    "def crawl_page(url):\n",
    "    try:\n",
    "        driver = set_chrome_driver(False)\n",
    "        driver.get(url)\n",
    "        # 요소 변경 가능\n",
    "        article_page = driver.find_element(By.CLASS_NAME, 'articlePage')\n",
    "        text = article_page.text\n",
    "        driver.close()\n",
    "    except NoSuchElementException:\n",
    "        text = \"\"\n",
    "    return text\n",
    "\n",
    "# ChatGPT 요약\n",
    "def summarize(text):\n",
    "    # 모델 엔진 선택\n",
    "    model_engine = \"text-davinci-003\"\n",
    "\n",
    "    # 맥스 토큰\n",
    "    max_tokens = 2500\n",
    "    \n",
    "    # 프롬프트 (요약해줘!)\n",
    "    prompt = f'''Summarize the paragraph below and interpret whether it is a positive or negative sentiment.\n",
    "\n",
    "    {text}\n",
    "    '''\n",
    "\n",
    "    # 요약 요청\n",
    "    completion = openai.Completion.create(\n",
    "        engine=model_engine,\n",
    "        prompt=prompt,\n",
    "        max_tokens=max_tokens,\n",
    "        temperature=0.3,      # creativity\n",
    "        top_p=1,\n",
    "        frequency_penalty=0,\n",
    "        presence_penalty=0\n",
    "    )\n",
    "    return completion.choices[0].text\n",
    "\n",
    "# 파파고 번역\n",
    "def papago_translate(text):\n",
    "    try:\n",
    "        papago = set_chrome_driver(False)\n",
    "        papago.get('https://papago.naver.com/')\n",
    "        time.sleep(1)\n",
    "        papago.find_element(By.ID, 'txtSource').send_keys(text)\n",
    "        papago.find_element(By.ID, 'btnTranslate').click()\n",
    "        time.sleep(2)\n",
    "        papago_translated = papago.find_element(By.ID, 'targetEditArea')\n",
    "        result = papago_translated.text\n",
    "    except NoSuchElementException: # 예외처리 (요소를 찾지 못하는 경우)\n",
    "        result = '번역 오류ㅠㅠ'\n",
    "    finally:\n",
    "        papago.close()\n",
    "    return result\n",
    "\n",
    "# 최종 wrapper\n",
    "def summarize_news(url):\n",
    "    page = crawl_page(url)\n",
    "    summarized = summarize(page)\n",
    "    print('<원문 요약>')\n",
    "    print(summarized)\n",
    "    korean_translated = papago_translate(summarized)\n",
    "    print('\\n<한글 요약본>')\n",
    "    print(korean_translated)\n",
    "    return korean_translated"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfd79e95",
   "metadata": {},
   "source": [
    "1개의 샘플 URL로 테스트를 진행합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "cfb7b06c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<원문 요약>\n",
      "\n",
      "This paragraph is discussing the current state of the wheat market, which has been affected by the Ukraine war and other factors. It is a neutral sentiment, as it is simply providing information about the current state of the market.\n",
      "\n",
      "<한글 요약본>\n",
      "이 단락은 우크라이나 전쟁과 다른 요인들의 영향을 받은 밀 시장의 현재 상태를 논의하고 있다. 그것은 단지 시장의 현재 상태에 대한 정보를 제공하는 것이기 때문에 중립적인 정서이다.\n"
     ]
    }
   ],
   "source": [
    "_ = summarize_news('https://www.investing.com/analysis/traders-send-wheat-prices-spiking-as-allied-tanks-aid-to-roll-into-ukraine-200634894')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dabaf798",
   "metadata": {},
   "source": [
    "## 더 나아가서\n",
    "\n",
    "- top5(Most Popular) 신문기사의 링크를 크롤링 해온 뒤 이를 모두 요약+번역할 수 있도록 발전시킨 코드입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59d8c7f",
   "metadata": {},
   "source": [
    "먼저, investing.com의 Most Popular에 게재된 Top5 신문기사를 크롤링합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "44a6790d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<selenium.webdriver.remote.webelement.WebElement (session=\"7af93f9bf399bacaaf635c50b3e0fd94\", element=\"ade4efab-ccba-4907-8527-12ddfd0f156c\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"7af93f9bf399bacaaf635c50b3e0fd94\", element=\"ad1ba5e8-443f-475c-874d-6f53b9dc9043\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"7af93f9bf399bacaaf635c50b3e0fd94\", element=\"8992e0f9-f6b5-4b82-a95e-741432b50039\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"7af93f9bf399bacaaf635c50b3e0fd94\", element=\"0eec2883-fa65-4b1f-9b5a-6b41b0487256\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"7af93f9bf399bacaaf635c50b3e0fd94\", element=\"3a1bbf5f-6fc0-44d7-92a8-9e73649c0c2e\")>]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# most popular news 의 신문기사 요소를 크롤링 합니다\n",
    "top5 = set_chrome_driver(False)\n",
    "# URL 요청\n",
    "top5.get('https://www.investing.com/news/most-popular-news')\n",
    "# 5개의 요소만 가져옵니다.\n",
    "top5.find_element(By.CLASS_NAME, 'largeTitle').find_elements(By.CLASS_NAME, 'js-article-item')[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca221429",
   "metadata": {},
   "source": [
    "크롤링 한 Top5 **신문기사의 URL을 추출**합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "6299bb0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.investing.com/news/economy/fed-hikes-by-025-in-further-downshift-on-tightening-but-sees-more-hikes-ahead-2993513',\n",
       " 'https://www.investing.com/news/economy/futures-dip-on-jitters-ahead-of-adp-report-fed-decision-2993229',\n",
       " 'https://www.investing.com/news/stock-market-news/us-stocks-are-falling-ahead-of-the-feds-rate-decision-2993346',\n",
       " 'https://www.investing.com/jp.php?v2=ZSU2aDZhYjhiM2xlZzZiZzNrZT83MDE6Z3BjMWBqNH1jJTQ9MWkydDc_OSdiPmE7M0BmOTY-MyUwZm89OntiIWUiNmg2ZmI3YjVsYWciYiMzb2U8NzgxNmdwYyBgag==',\n",
       " 'https://www.investing.com/news/economy/fed-decision-snaps-falling-sales-meta-earnings--whats-moving-markets-2993057']"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 5개의 신문기사 URL 만 추출 합니다.\n",
    "top5_links = []\n",
    "\n",
    "for link in top5.find_element(By.CLASS_NAME, 'largeTitle').find_elements(By.CLASS_NAME, 'js-article-item')[:5]:\n",
    "    top5_links.append(link.find_element(By.CSS_SELECTOR, 'a').get_attribute('href'))\n",
    "    \n",
    "top5_links"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa005d17",
   "metadata": {},
   "source": [
    "마지막으로, 이전에 생성한 `summarize_news` 함수를 활용하여 크롤링 + 요약 + 번역을 진행하고 결과를 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "42cfdd7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<원문 요약>\n",
      "\n",
      "This paragraph is summarizing the Federal Reserve's decision to raise interest rates by 0.25%, and the sentiment is generally positive. It notes that the move is in line with the Fed's goal of returning inflation to 2%, and that the labor market remains strong. It also mentions that the markets are expecting Powell to push back against the easing financial conditions, but that the response will likely be positive.\n",
      "\n",
      "<한글 요약본>\n",
      "이 문단은 연방준비제도이사회(FRB)가 금리를 0.25% 인상하기로 한 결정을 정리한 것으로 대체로 긍정적인 분위기다. 물가상승률을 2%로 되돌리겠다는 연준의 목표와 맞물려 노동시장이 강세를 유지하고 있다는 점에 주목한다. 그것은 또한 시장이 파월이 완화적인 금융 조건에 대해 반발할 것으로 기대하고 있지만, 반응은 긍정적일 가능성이 높다고 언급한다.\n",
      "\n",
      "<원문 요약>\n",
      "\n",
      "This paragraph is discussing the performance of the stock market in the US, with the Dow, S&P 500, and Nasdaq all falling ahead of the Federal Reserve's decision on interest rates. It also mentions that Advanced Micro Devices Inc had a positive outlook, while Snap Inc and Electronic Arts Inc had negative outlooks. Overall, the sentiment of this paragraph is negative as the stock market indexes are down and some companies had negative outlooks.\n",
      "\n",
      "<한글 요약본>\n",
      "이 단락은 연방준비제도의 금리 결정을 앞두고 다우지수, S&P 500지수, 나스닥지수가 모두 하락한 가운데 미국 증시의 실적을 논의하고 있다. 또한 Advanced Micro Devices Inc는 긍정적인 전망을, Snap Inc와 Electronic Arts Inc는 부정적인 전망을 나타냈습니다. 전반적으로 이 단락의 정서는 주가지수가 하락하고 일부 기업들이 부정적인 전망을 했기 때문에 부정적이다.\n",
      "\n",
      "<원문 요약>\n",
      "\n",
      "This paragraph is summarizing the current state of the stock market and the Federal Reserve's decision on interest rates. It also mentions the job market, corporate earnings, and oil prices. Overall, the sentiment is neutral as some stocks are rising while others are falling.\n",
      "\n",
      "<한글 요약본>\n",
      "이 단락은 주식 시장의 현황과 연방 준비 제도 이사회의 금리 결정을 요약하고 있다. 그것은 또한 고용 시장, 기업 수익, 유가를 언급한다. 일부 종목은 오르고 있는 반면 다른 종목은 하락하고 있어 전반적으로 심리가 중립적이다.\n",
      "\n",
      "<원문 요약>\n",
      "\n",
      "This product is not worth the money.\n",
      "\n",
      "This paragraph expresses a negative sentiment about a product not being worth the money.\n",
      "\n",
      "<한글 요약본>\n",
      "이 제품은 값어치가 없다.\n",
      "\n",
      "이 단락은 제품이 그 돈의 가치가 없다는 부정적인 감정을 표현한다.\n",
      "\n",
      "<원문 요약>\n",
      "\n",
      "This paragraph provides an overview of the financial markets on February 1st. It mentions the Federal Reserve's decision to raise interest rates, economic data releases, stock market performance, euro zone inflation, OPEC+ production quotas, and U.S. crude futures. The sentiment is neutral, as it simply provides an overview of the day's events.\n",
      "\n",
      "<한글 요약본>\n",
      "이 단락은 2월 1일의 금융 시장에 대한 개요를 제공한다. 금리 인상, 경제 데이터 공개, 증시 성과, 유로존 인플레이션, OPEC+ 생산 쿼터, 미국 원유 선물 등을 언급하고 있다. 그 정서는 단지 그날의 사건들에 대한 개요를 제공하기 때문에 중립적이다.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 5개의 신문기사 링크에 대한 본문 크롤링+요약+번역 을 진행합니다.\n",
    "top5_summarize = []\n",
    "\n",
    "for link in top5_links:\n",
    "    output = summarize_news(link)\n",
    "    top5_summarize.append(output)\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}