07_기계학습_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 07_기계학습_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train , X_test , y_train , y_test = train_test_split(X , y ,test_size=0.3, random_state=0)\n",
    "\n",
    "# y의 비율이 불균형할 때 사용하는 층화추출\n",
    "X_train, X_test, y_train, y_test =  train_test_split(X, y, test_size=0.3, random_state=0, stratify=y_target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-15T12:59:42.987784Z",
     "start_time": "2020-09-15T12:59:42.983794Z"
    }
   },
   "source": [
    "# 정규화, 표준화 - StandardScaler, MinMaxScaler, Log\n",
    "**train 데이터에는 .fit_transform(X_train) <br>\n",
    "test 데이터에는 .transform(X_test)다 명심**\n",
    "- 0단계 : 그냥 씀\n",
    "- 1단계 : 로그변환을 해본다\n",
    "- 2단계 : 정규화, 표준화를 해본다\n",
    "- 3단계 : 성능이 별 차이 없으면 다항 변수를 만들어본다"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-15T16:13:52.585174Z",
     "start_time": "2020-09-15T16:13:52.579190Z"
    }
   },
   "source": [
    "- y(타겟변수)는 hist를 그려봐서 분포가 한쪽으로 쏠려있으면 로그 변환을 해서 왜곡을 낮춘다.\n",
    "- y(타겟변수)는 변환을 한다면 일반적으로 로그 변환을 적용한다.\n",
    "\n",
    "- np.log1p(x)는 에러를 방지하기 위해 x에 1을 자동으로 더해서 계산해주는 것\n",
    "- np.log1p(x)의 역함수는 np.expm1(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log 씌우고\n",
    "y_log = np.log1p(y)\n",
    "# train, test split 하고\n",
    "# 모델링 하고\n",
    "\n",
    "# 평가는 test, pred 모두 expm1으로 풀어서 비교\n",
    "y_test_exp = np.expm1(y_test)\n",
    "y_pred_exp = np.expm1(y_pred)\n",
    "\n",
    "(y_test_exp, y_pred_exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 개별 정규화/표준화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-15T13:09:51.444017Z",
     "start_time": "2020-09-15T13:09:51.428060Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler, PolynomialFeatures\n",
    "\n",
    "scaled_data = StandardScaler().fit_transform(X_train)\n",
    "scaled_data = MinMaxScaler().fit_transform(X_train)\n",
    "scaled_data = np.log1p(input_data)\n",
    "scaled_data = PolynomialFeatures(degree=p_degree, include_bias=False).fit_transform(scaled_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-15T13:10:14.205072Z",
     "start_time": "2020-09-15T13:10:14.201119Z"
    }
   },
   "source": [
    "## 한 번에 다 비교해보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-15T12:58:12.288116Z",
     "start_time": "2020-09-15T12:58:10.871767Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler, PolynomialFeatures\n",
    "\n",
    "# method는 표준 정규 분포 변환(Standard), 최대값/최소값 정규화(MinMax), 로그변환(Log) 결정\n",
    "# p_degree는 다향식 특성을 추가할 때 적용. p_degree는 2이상 부여하지 않음. \n",
    "def get_scaled_data(method='None', p_degree=None, input_data=None):\n",
    "    if method == 'Standard':\n",
    "        scaled_data = StandardScaler().fit_transform(input_data)\n",
    "    elif method == 'MinMax':\n",
    "        scaled_data = MinMaxScaler().fit_transform(input_data)\n",
    "    elif method == 'Log':\n",
    "        scaled_data = np.log1p(input_data)\n",
    "    else:\n",
    "        scaled_data = input_data\n",
    "\n",
    "    if p_degree != None:\n",
    "        scaled_data = PolynomialFeatures(degree=p_degree, \n",
    "                                         include_bias=False).fit_transform(scaled_data)\n",
    "    \n",
    "    return scaled_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ridge의 alpha값을 다르게 적용하고 다양한 데이터 변환방법에 따른 RMSE 추출. \n",
    "alphas = [0.1, 1, 10, 100]\n",
    "#변환 방법은 모두 6개, 원본 그대로, 표준정규분포, 표준정규분포+다항식 특성\n",
    "# 최대/최소 정규화, 최대/최소 정규화+다항식 특성, 로그변환 \n",
    "scale_methods=[(None, None), ('Standard', None), ('Standard', 2), \n",
    "               ('MinMax', None), ('MinMax', 2), ('Log', None)]\n",
    "for scale_method in scale_methods:\n",
    "    X_data_scaled = get_scaled_data(method=scale_method[0], p_degree=scale_method[1], \n",
    "                                    input_data=X_data)\n",
    "    print('\\n## 변환 유형:{0}, Polynomial Degree:{1}'.format(scale_method[0], scale_method[1]))\n",
    "    get_linear_reg_eval('Ridge', params=alphas, X_data_n=X_data_scaled, \n",
    "                        y_target_n=y_target, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 오버샘플링 / 언더샘플링"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- train set에만 쓰는 것이지, 당연히 test set에는 쓰는 거 아니다!\n",
    "- 오버샘플링이 1옵션이고, 오버샘플링 방식은 일단 SMOTE 방식을 써라\n",
    "- 만약 다른 것도 해야하면 아래 참조"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.over_sampling import SMOTE\n",
    "\n",
    "smote = SMOTE(random_state=0)\n",
    "X_train_over, y_train_over = smote.fit_sample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 오버샘플링\n",
    "import imblearn.over_sampling as osam \n",
    "\n",
    "# 오버샘플링 전\n",
    "model = SVC(kernel = 'rbf').fit(train_X, train_Y)\n",
    "predicted = model.predict(test_X)\n",
    "\n",
    "# 랜덤 오버샘플링\n",
    "ros = osam.RandomOverSampler()\n",
    "oversampled_X, oversampled_Y = ros.fit_sample(train_X, train_Y)\n",
    "model = SVC(kernel = 'rbf').fit(oversampled_X, oversampled_Y)\n",
    "predicted = model.predict(test_X)\n",
    "\n",
    "# SMOTE\n",
    "SMOTEos = osam.SMOTE(k_neighbors = 5)\n",
    "oversampled_X, oversampled_Y = SMOTEos.fit_sample(train_X, train_Y)\n",
    "model = SVC(kernel = 'rbf').fit(oversampled_X, oversampled_Y)\n",
    "predicted = model.predict(test_X)\n",
    "\n",
    "# Borderline SMOTE\n",
    "B_SMOTEos = osam.SMOTE(k_neighbors = 5, m_neighbors = 5)\n",
    "oversampled_X, oversampled_Y = B_SMOTEos.fit_sample(train_X, train_Y)\n",
    "\n",
    "# ADASYN\n",
    "A_SMOTEos = osam.ADASYN(n_neighbors = 5)\n",
    "oversampled_X, oversampled_Y = A_SMOTEos.fit_sample(train_X, train_Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 언더샘플링\n",
    "\n",
    "# 랜덤 언더샘플링\n",
    "import imblearn.under_sampling as usam\n",
    "rus = usam.RandomUnderSampler()\n",
    "undersampled_X, undersampled_Y = rus.fit_sample(train_X, train_Y)\n",
    "\n",
    "# k-NN 기반의 언더샘플링\n",
    "krus = usam.CondensedNearestNeighbour(n_neighbors = 3)\n",
    "undersampled_X, undersampled_Y = krus.fit_sample(train_X, train_Y)\n",
    "\n",
    "# 수정된 k-NN 기반의 언더샘플링\n",
    "erus = usam.CondensedNearestNeighbour(n_neighbors = 3, n_jobs = 1)\n",
    "undersampled_X, undersampled_Y = erus.fit_sample(train_X, train_Y)\n",
    "\n",
    "# NearMiss 언더샘플링 Type 1\n",
    "nrus1 = usam.NearMiss(n_neighbors = 3, version = 1)\n",
    "undersampled_X, undersampled_Y = nrus1.fit_sample(train_X, train_Y)\n",
    "\n",
    "# NearMiss 언더샘플링 Type 2\n",
    "nrus2 = usam.NearMiss(n_neighbors = 3, version = 2)\n",
    "undersampled_X, undersampled_Y = nrus2.fit_sample(train_X, train_Y)\n",
    "\n",
    "# NearMiss 언더샘플링 Type 3\n",
    "nrus3 = usam.NearMiss(n_neighbors = 3, version = 3)\n",
    "undersampled_X, undersampled_Y = nrus3.fit_sample(train_X, train_Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 모델 성능 평가"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 회귀 모델 성능 평가\n",
    "- R2\n",
    "- MAE Mean absolute error\n",
    "- MSE Mean Squared Error\n",
    "- RMSE Root MSE\n",
    "- RMSLE (Root Mean Square Log Error) : 오류값의 로그에 대한 RMSE\n",
    "- MAPE mean absolute percentage error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-17T06:14:50.091615Z",
     "start_time": "2020-09-17T06:14:49.841280Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_error, mean_squared_error , r2_score\n",
    "\n",
    "r2 = r2_score(y_test, y_preds) # R2 : 0~1, 1에 가까울수록 설명력이 높은 것, 음수가 나오는 경우도 있는데, 그건 망한 거.\n",
    "mae = mean_absolute_error(true, pred)\n",
    "mse = mean_squared_error(y_test, y_preds) # MSE : 낮을 수록 좋겠지\n",
    "rmse = np.sqrt(mse) # RMSE : 얘도 당연히 낮을 수록 좋겠지\n",
    "def rmsle(y, pred): # RMSLE : 얘도 당연히 낮을 수록 좋겠지\n",
    "    log_y = np.log1p(y)\n",
    "    log_pred = np.log1p(pred)\n",
    "    squared_error = (log_y - log_pred) ** 2\n",
    "    rmsle = np.sqrt(np.mean(squared_error))\n",
    "    return rmsle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mape\n",
    "def mape(y_test, y_preds): \n",
    "    y_test, y_preds = np.array(y_test), np.array(y_preds)\n",
    "    return np.mean(np.abs((y_test - y_preds) / y_test)) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross Validation 시 \n",
    "- Negative MSE\n",
    "- RMSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "neg_mse_scores = cross_val_score(model, X_data, y_target, scoring=\"neg_mean_squared_error\", cv = 5) # Negative MSE\n",
    "rmse_scores  = np.sqrt(-1 * neg_mse_scores) # RMSE\n",
    "avg_rmse = np.mean(rmse_scores) # RMSE 평균"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 분류 모델 성능 평가\n",
    "- 오차행렬\n",
    "- 정확도\n",
    "- 정밀도\n",
    "- 재현율 = 민감도\n",
    "- F1\n",
    "- AUC\n",
    "- 특이도\n",
    "- ROC - x축 : 1-특이성 = FP / (FP+TN) // y축 : 민감도(재현율) = TP / (FN+TP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import precision_score, recall_score\n",
    "from sklearn.metrics import f1_score, roc_auc_score\n",
    "\n",
    "confusion = confusion_matrix( y_test, pred)\n",
    "accuracy_score(y_test, lr_preds) # 정확도 : 0~1, 1에 가까울수록 좋다\n",
    "precision_score(y_test , pred) # 정밀도 : TP/ (FP+TP) 예측 Positive 중에 실제 Positive\n",
    "recall_score(y_test , pred) # 재현율 : TP/(FN+TP) 실제 Positive 중에 예측이 Positive 암환자는 재현율이 중요\n",
    "f1_score(y_test,pred) # F1\n",
    "roc_auc_score(y_test , lr_preds) # AUC : 0~1, 1에 가까울수록 좋다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-17T06:08:26.436242Z",
     "start_time": "2020-09-17T06:08:26.421283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6666666666666666"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 특이도는 직접 짜야함\n",
    "from sklearn.metrics import confusion_matrix\n",
    "tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()\n",
    "specificity = tn / (tn+fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ROC curve는 길어서 별도로 정리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# 레이블 값이 1일때의 예측 확률을 추출 \n",
    "pred_proba_class1 = lr_clf.predict_proba(X_test)[:, 1] \n",
    "\n",
    "fprs , tprs , thresholds = roc_curve(y_test, pred_proba_class1)\n",
    "# fprs , tprs , thresholds = roc_curve(y_test, pred_proba_class1, pos_label=2) # y에 음수가 있으면 에러나나? 그래서 이렇게\n",
    "\n",
    "# 5 step에 하나씩 쓰겠다는 소리\n",
    "# 반환된 임곗값 배열 로우가 47건이므로 샘플로 10건만 추출하되, 임곗값을 5 Step으로 추출. \n",
    "thr_index = np.arange(0, thresholds.shape[0], 5)\n",
    "print('샘플 추출을 위한 임곗값 배열의 index 10개:', thr_index)\n",
    "print('샘플용 10개의 임곗값: ', np.round(thresholds[thr_index], 2))\n",
    "# 5 step 단위로 추출된 임계값에 따른 FPR, TPR 값\n",
    "print('샘플 임곗값별 FPR: ', np.round(fprs[thr_index], 3))\n",
    "print('샘플 임곗값별 TPR: ', np.round(tprs[thr_index], 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def roc_curve_plot(y_test , pred_proba_c1):\n",
    "    # 임곗값에 따른 FPR, TPR 값을 반환 받음. \n",
    "    fprs , tprs , thresholds = roc_curve(y_test ,pred_proba_c1)\n",
    "\n",
    "    # ROC Curve를 plot 곡선으로 그림. \n",
    "    plt.plot(fprs , tprs, label='ROC')\n",
    "    # 가운데 대각선 직선을 그림. \n",
    "    plt.plot([0, 1], [0, 1], 'k--', label='Random')\n",
    "    \n",
    "    # FPR X 축의 Scale을 0.1 단위로 변경, X,Y 축명 설정등   \n",
    "    start, end = plt.xlim()\n",
    "    plt.xticks(np.round(np.arange(start, end, 0.1),2))\n",
    "    plt.xlim(0,1); plt.ylim(0,1)\n",
    "    plt.xlabel('FPR( 1 - Sensitivity )'); plt.ylabel('TPR( Recall )')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "roc_curve_plot(y_test, lr_clf.predict_proba(X_test)[:, 1] )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 군집화 모델 성능 평가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 분류"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 로지스틱 회귀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 데이터 로딩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-17T08:55:55.728259Z",
     "start_time": "2020-09-17T08:55:52.738526Z"
    }
   },
   "outputs": [],
   "source": [
    "# 데이터 로딩\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "cancer = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StandardScaler, Train Test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-17T08:55:55.758894Z",
     "start_time": "2020-09-17T08:55:55.731207Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# StandardScaler( )로 평균이 0, 분산 1로 데이터 분포도 변환\n",
    "scaler = StandardScaler()\n",
    "data_scaled = scaler.fit_transform(cancer.data) \n",
    "# 내 생각엔 Split 먼저 하고, fit_transform(X_train)해서 모델링, 그 후에 transform(X_test)가 맞는 것 같음\n",
    "\n",
    "X_train , X_test, y_train , y_test = train_test_split(data_scaled, cancer.target, test_size=0.3, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-17T08:55:55.807797Z",
     "start_time": "2020-09-17T08:55:55.761883Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.982\n",
      "roc_auc: 0.979\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\50008313\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, roc_auc_score\n",
    "\n",
    "# 로지스틱 회귀를 이용하여 학습 및 예측 수행. \n",
    "lr_clf = LogisticRegression()\n",
    "lr_clf.fit(X_train, y_train)\n",
    "lr_preds = lr_clf.predict(X_test)\n",
    "\n",
    "# accuracy와 roc_auc 측정\n",
    "print('accuracy: {:0.3f}'.format(accuracy_score(y_test, lr_preds)))\n",
    "print('roc_auc: {:0.3f}'.format(roc_auc_score(y_test , lr_preds)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-17T08:55:55.977753Z",
     "start_time": "2020-09-17T08:55:55.809753Z"
    }
   },
   "outputs": [],
   "source": [
    "# 파라미터 튜닝하는 거\n",
    "# penalty는 규제 종류\n",
    "# C는 1/alpha로 작을수록 규제 강도가 커짐\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params={'penalty':['l2', 'l1'],\n",
    "        'C':[0.01, 0.1, 1, 1, 5, 10]}\n",
    "\n",
    "grid_clf = GridSearchCV(lr_clf, param_grid=params, scoring='accuracy', cv=3 )\n",
    "grid_clf.fit(data_scaled, cancer.target)\n",
    "print('최적 하이퍼 파라미터:{0}, 최적 평균 정확도:{1:.3f}'.format(grid_clf.best_params_, \n",
    "                                                  grid_clf.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T04:59:13.254512Z",
     "start_time": "2020-09-16T04:59:11.347924Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "                       max_features=None, max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, presort=False,\n",
       "                       random_state=156, splitter='best')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# DecisionTree Classifier 생성\n",
    "dt_clf = DecisionTreeClassifier(random_state=156)\n",
    "\n",
    "# 붓꽃 데이터를 로딩하고, 학습과 테스트 데이터 셋으로 분리\n",
    "iris_data = load_iris()\n",
    "X_train , X_test , y_train , y_test = train_test_split(iris_data.data, iris_data.target,\n",
    "                                                       test_size=0.2,  random_state=11)\n",
    "\n",
    "# DecisionTreeClassifer 학습. \n",
    "dt_clf.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T05:18:44.390023Z",
     "start_time": "2020-09-16T05:18:44.384048Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 2, 1, 1, 2, 0, 1, 0, 0, 1, 1, 1, 1, 2, 2, 0, 2, 1, 2, 2, 1, 0,\n",
       "       0, 1, 0, 0, 2, 1, 0, 1])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = dt_clf.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 트리 시각화\n",
    "- `sklearn.tree.export_graphviz`를 이용하여 **.dot** 파일을 생성\n",
    "- 생성된 dot 파일을 텍스트 편집기에서 불러들인 후, 이를 복사하여 [WebGraphviz.com](http://webgraphviz.com/)에 붙여넣으면 의사결정나무 그림을 얻을 수 있음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T04:59:13.316616Z",
     "start_time": "2020-09-16T04:59:13.305614Z"
    }
   },
   "outputs": [],
   "source": [
    "# 시각화를 위한 부분 - tree.dot 파일을 생성해줌\n",
    "\n",
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "# export_graphviz()의 호출 결과로 out_file로 지정된 tree.dot 파일을 생성함. \n",
    "export_graphviz(dt_clf, out_file=\"tree.dot\", class_names=iris_data.target_names , \\\n",
    "feature_names = iris_data.feature_names, impurity=True, filled=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T04:59:24.736102Z",
     "start_time": "2020-09-16T04:59:24.707184Z"
    }
   },
   "outputs": [],
   "source": [
    "# 에러나면 망한 거 포기. 인터넷 되어야 해결 가능. --> tree.dot 내용 복사해서  WebGraphviz.com에 넣으면 보임\n",
    "\n",
    "import graphviz\n",
    "\n",
    "# 위에서 생성된 tree.dot 파일을 Graphviz 읽어서 Jupyter Notebook상에서 시각화 \n",
    "with open('tree.dot') as f:\n",
    "    dot_graph = f.read()\n",
    "graphviz.Source(dot_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### feature importance 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T05:15:41.977856Z",
     "start_time": "2020-09-16T05:15:39.854094Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature importances:\n",
      "[0.025 0.    0.555 0.42 ]\n",
      "sepal length (cm) : 0.025\n",
      "sepal width (cm) : 0.000\n",
      "petal length (cm) : 0.555\n",
      "petal width (cm) : 0.420\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x189b224df88>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbsAAAD4CAYAAAB10khoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXRklEQVR4nO3de5SlVX3m8e8jjTaXFkXICBpsg4IDyMVumQEE0TgzLpJBXfZoHAKiZlxKBB0HL8t7ghpRV0wWXpjGYfBCIsoSB2HkptyEKHRr3xAaRZmgsgJGBDIgF/nNH2f3eCiru86pqq6iN9/PWr3qPfvsd+/froJ+er/vW1WpKiRJ6tlj5rsASZI2N8NOktQ9w06S1D3DTpLUPcNOktS9BfNdgCa300471eLFi+e7DEnaoqxcufIXVbXzxHbD7hFq8eLFrFixYr7LkKQtSpL/M1m7lzElSd0z7CRJ3TPsJEndM+wkSd3zAZVHqOt/+s8sedvnxz5v5ceO2QzVSNKWzZ2dJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXvzGnZJDk9y3qjtszDfS5PsNfT6siRLRzhvl9moJ8nOSS6Y6TiSpPE82nZ2LwX2mrLX73orcNpMJ6+q24Fbkxwy07EkSaPbZNgl2S7J+UlWJ1mX5JWtfUmSy5OsTHJhkl1a+2VJ/ibJ1a3/ga39wNb2/fZxz1ELbDWcnuTadv5LWvuxSb6a5IIkP0zy0aFzXpfkxlbPaUk+meRg4EjgY0lWJdm9df9PSa5p/Q/dSBkvBy5oY2+V5ONJ1iZZk+T41n5zkg8n+YckK5I8p31ubkryhqGxvgYcNer6JUkzt2CK918M/Lyq/gggyQ5JtgZOAV5SVbe3APwQ8Np2znZVdXCSw4DTgX2AG4DDqurBJC8CPswgQEbxbuBbVfXaJE8ArklySXtvf+AA4D5gfZJTgN8A7wWeA9wNfAtYXVVXJzkXOK+qzm7rAVhQVQcmOQJ4P/Ci4cmTPB24o6rua02vB54OHNDWs+NQ91uq6qAknwDOAA4BFgLXAae2PiuAD464dknSLJgq7NYCH09yMoOQuDLJPgwC7OIWFlsBtw6d8/cAVXVFkse3gFoEfC7JM4ECth6jxn8PHJnkxPZ6IbBbO/5mVd0JkOQHwNOAnYDLq+qXrf0rwB6bGP+r7eNKYPEk7+8C3D70+kXAqVX1YFvnL4feO7d9XAtsX1V3A3cn+XWSJ1TVr4DbgF0nKyTJ6xmEKY9d9KRNlCxJGscmw66qbkyyBDgC+KskFwHnANdV1UEbO22S1ycBl1bVy5IsBi4bo8YAL6+q9Q9rTP4Ngx3dBr9hsJ6MMTZDY2w4f6J7GQTscD0T1zhxrIcm1PbQ0NgL25i/o6qWA8sBtnvy0zc2hyRpTFPds9sVuKeqvgh8nMGlwfXAzkkOan22TrL30Gkb7us9D7iz7bx2AH7W3j92zBovBI5P20YmOWCK/tcAz0/yxCQLePjl0rsZ7DLHcSMP3/FdBLyhjc2Ey5ij2ANYN+Y5kqQZmOppzGczuEe2isG9sw9W1f3AMuDkJKuBVcDBQ+fckeRqBveoXtfaPspgZ3gVg8ue4ziJwWXPNUnWtdcbVVU/Y3BP8LvAJcAPgDvb218C3tYedNl9I0NMHO//AjcleUZr+izwj62e1cB/HnM9LwDOH/McSdIMpGr2rpYluQw4sapWzNqg06tj+6r6l7b7Ogc4varOmcF4LwOWVNV7ZqG2Kxg83HPHpvpt9+Sn17OO/ouxx1/5sWOmW5okbfGSrKyq3/n+6V6/z+4DbTe6DvgJg8f9p60F5c0zLSrJzsBfTxV0kqTZNdXTmGOpqsNnc7zpqqoTp+419pifnYUxbmeGwStJGl+vOztJkv4/w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1D3DTpLUPcNOktQ9w06S1L1Z/RU/mj3/+qlPYoW/iFWSZoU7O0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3HnFhl+TwJOdN47xdk5y9kfcuS7K0Hb9rqH1xknUjjv+WJMeMW9ck47wpyWtmOo4kaXSPuLCbrqr6eVUtG6Hru6bu8nBJFgCvBf5u7MJ+1+nACbMwjiRpRGOHXZLtkpyfZHWSdUle2dqXJLk8ycokFybZpbVfluRvklzd+h/Y2g9sbd9vH/ecYt7/nWTfdvz9JO9rxycl+bPhXVqSbZJ8KcmaJGcB27T2jwDbJFmV5Mw29FZJTktyXZKLkmwzyfQvBL5XVQ+2cZ6R5JL2Ofhekt3bjvTyJF9OcmOSjyQ5Ksk1SdYm2R2gqu4Bbt7weZAkbX7T2dm9GPh5Ve1XVfsAFyTZGjgFWFZVSxjsXj40dM52VXUwcFx7D+AG4LCqOgB4H/DhKea9Ajg0yeOBB4FDWvvzgCsn9H0jcE9V7dvqWAJQVe8E7q2q/avqqNb3mcCnqmpv4FfAyyeZ+xBg5dDrM9s5+wEHA7e29v2ANwPPBo4G9qiqA4HPAscPnb8COHTiJElen2RFkhW33377Jj8ZkqTRTSfs1gIvSnJykkOr6k5gT2Af4OIkq4D3AE8dOufvAarqCuDxSZ4A7AB8pe3GPgHsPcW8VwKHMQi384Htk2wLLK6q9RP6HgZ8sc25BliziXF/UlWr2vFKYPEkfXYBbgdIsgh4SlWd08b/ddutAVxbVbdW1X3ATcBFrX3thHFvA3adOElVLa+qpVW1dOedd95EyZKkcSwY94SqujHJEuAI4K+SXAScA1xXVQdt7LRJXp8EXFpVL0uyGLhsiqmvBZYCPwYuBnYC/gsP33Ftas6NuW/o+De0S54T3AssbMcZcayHhl4/xMM/1wvbmJKkOTCde3a7MrhE+EXg48BzgPXAzkkOan22TjK8U9twX+95wJ1tN7gD8LP2/rFTzVtV9wO3AK8AvsNgp3civ3sJEwaXPI9qc+4D7Dv03gPtsus4rgee0eq4C/hpkpe28R/Xdpjj2AMY6SlQSdLMTecy5rOBa9rlyncDH2xBtAw4OclqYBWDe1kb3JHkauBU4HWt7aMMdoZXAVuNOPeVwD+1y4ZXMrhUOlnYfYbBZc41wNuBa4beWw6sGXpAZRTfYHBpdIOjgRPa+FcDTx5jLBjcA7xkzHMkSdOUqlGv9k1zguQy4MSqWrFZJ9rMkpwDvL2qfjjDcQ4A3lpVR2+q39KlS2vFii36UyZJcy7JyqpaOrG9m++zmwPvZPCgykztBLx3FsaRJI1o7AdUxlVVh2/uOeZCe+Jz4lOf0xnn4lkoR5I0Bnd2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuLZjvAjS5G267gUNOOWS+y5CkOXXV8VdtlnHd2UmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6Z9hJkrpn2EmSumfYSZK6t9nCLsmxSXYdod8ZSZaN2j4Ldb1r6HhxknUjnveWJMfMwvxvSvKamY4jSRrd5tzZHQtMGXbz4F1Td3m4JAuA1wJ/Nwvznw6cMAvjSJJGNFLYtR3QDUk+l2RNkrOTbNveW5Lk8iQrk1yYZJe2I1sKnJlkVZJtkrwvybVJ1iVZniSjFjnZHK39siQnJ7kmyY1JDm3t2yb5cqv1rCTfTbI0yUeAbVpNZ7bht0pyWpLrklyUZJtJSngh8L2qerCN/4wklyRZneR7SXZPcnir8cutlo8kOarVtjbJ7gBVdQ9wc5IDR12/JGlmxtnZ7Qksr6p9gbuA45JsDZwCLKuqJQx2LR+qqrOBFcBRVbV/Vd0LfLKqnltV+wDbAH88yqQbm2Ooy4KqOhB4C/D+1nYccEer9SRgCUBVvRO4t9V0VOv7TOBTVbU38Cvg5ZOUcQiwcuj1me2c/YCDgVtb+37Am4FnA0cDe7TaPgscP3T+CuDQSdb6+iQrkqx44F8emOIzI0ka1YIx+t5SVVe14y8yuBR3AbAPcHHbqG3Fb//in+gFSd4ObAvsCFwHfH2EefecYo6vto8rgcXt+HnA3wJU1bokazYx/k+qatUkYwzbBbgeIMki4ClVdU4b/9etHeDaqrq1vb4JuKidvxZ4wdB4twHPmjhJVS0HlgNsv9v2tYmaJUljGCfsJv7lW0CA66rqoE2dmGQh8GlgaVXdkuQDwMIR551qjvvax9/w2/WMfIl06PwNY0x2GfNeflvvpsYeHuuhodcP8fDP9cI2piRpDoxzGXO3JBsC51XAt4H1wM4b2pNsnWTv1uduYFE73hAUv0iyPTDOU5abmmNjvg28ovXfi8FlxQ0eaJdGx3E98AyAqroL+GmSl7bxH7fh/uUY9gBGegpUkjRz44Td9cCr2yXBHYHPVNX9DILr5CSrgVUM7mEBnAGcmmQVgx3OaQwu530NuHbUSaeYY2M+zSAg1wDvANYAd7b3lgNrhh5QGcU3gMOGXh8NnNDGvxp48hhjweAe4CVjniNJmqZUTX1rKMli4Lz2cMkjXpKtgK2r6tftKchvMnhY5P4ZjHkO8Paq+uEMazsAeGtVHb2pftvvtn3t97b9ZjKVJG1xrjr+qqk7bUKSlVW1dGL7OPfstiTbApe2y5UB3jiToGveyeBBlRmFHbAT8N4ZjiFJGsNIYVdVNzN4InKLUFV3M/g+v9kccz2D+4czHefiWShHkjQGfzamJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXu9/j67Ld6zfu9ZM/4lhpKkAXd2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTuGXaSpO4ZdpKk7hl2kqTu+ePCHqHuXr+eyw97/nyXIWmann/F5fNdgoa4s5Mkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1z7CTJHXPsJMkdc+wkyR1b87CLsmxSXYdod8ZSZZNY/w3JDlmkvbFSda14/2THDH03geSnDjC2EnyrSSPH7euSca6JMkTZzqOJGl0c7mzOxaYMuymq6pOrarPT9Ftf+CIKfpM5ghgdVXdNY1zJ/oCcNwsjCNJGtG0wq7tlm5I8rkka5KcnWTb9t6SJJcnWZnkwiS7tJ3aUuDMJKuSbJPkfUmuTbIuyfIk2cR8v5dkZTveL0kl2a29vinJtsO7tFbD6iT/APx5a3ss8JfAK1sNr2zD75XksiQ/TnLCRko4CvhfQ/Uc09a9OskXWtsZST6T5NI21vOTnJ7k+iRnDI11LvCqMT/lkqQZmMnObk9geVXtC9wFHJdka+AUYFlVLQFOBz5UVWcDK4Cjqmr/qroX+GRVPbeq9gG2Af54YxNV1W3AwnYZ8dA21qFJngbcVlX3TDjlfwInVNVBQ2PcD7wPOKvVcFZ761nAfwAOBN7f1jDRIcCGsN0beDfwwqraD3jzUL8nAi8E/ivwdeATwN7As5Ps3+q4A3hckidtbL2SpNk1k7C7paquasdfBJ7HIAD3AS5Osgp4D/DUjZz/giTfTbKWQUDsPcV8VzMIncOAD7ePhwJXDndKsgPwhKq6vDV9YYpxz6+q+6rqF8BtwL+apM+OVXV3O34hcHbrT1X9cqjf16uqgLXAP1XV2qp6CLgOWDzU7zYmuaSb5PVJViRZcecDD0xRtiRpVAtmcG5N8jrAdcM7qskkWQh8GlhaVbck+QCwcIr5rmQQbk9jcEnxHW3O8yYOP0ltm3Lf0PFvmPxz8mCSx7Tg2tT4G8Z6aMK4D00YdyFw78STq2o5sBxgz0WLxlmDJGkTZrKz2y3JhlB7FfBtYD2w84b2JFu3y34AdwOL2vGGYPtFku2BUZ6+vAL4U+CHLXR+yeDBkauGO1XVr4A7kzyvNR019PZwDeNYD/xBO/4m8IoNlyGT7DjOQO3e5JOBm6dRhyRpGmYSdtcDr06yBtgR+Ey7L7YMODnJamAVcHDrfwZwaru8eR9wGoPLfV8Drp1qsqq6uR1e0T5+G/hVuwc20WuAT7UHVIZ3UJcyeCBl+AGVUZwPHN7quA74EHB5W+NfjzEOwBLgO1X14JjnSZKmKYNbTGOelCwGzmsPl3QvyS7A56vq383CWH8LnFtV39xUvz0XLarlBzxnptNJmifPv+LyqTtp1iVZWVVLJ7b7E1RGUFW3AqfNxjeVA+umCjpJ0uya1gMq7ZLio2JXt0FVfXmWxjltNsaRJI3OnZ0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe4adJKl7hp0kqXuGnSSpe9P6FT/a/Bbtuae//FGSZok7O0lS9ww7SVL3DDtJUvcMO0lS9ww7SVL3UlXzXYMmkeRuYP1817GZ7QT8Yr6LmAOPhnU+GtYIj451bulrfFpV7Tyx0W89eORaX1VL57uIzSnJit7XCI+OdT4a1giPjnX2ukYvY0qSumfYSZK6Z9g9ci2f7wLmwKNhjfDoWOejYY3w6Fhnl2v0ARVJUvfc2UmSumfYSZK6Z9jNsyQvTrI+yY+SvHOS9x+X5Kz2/neTLJ77KmdmhDUeluR7SR5Msmw+apwNI6zzrUl+kGRNkm8medp81DkTI6zxDUnWJlmV5NtJ9pqPOmdqqnUO9VuWpJJscY/qj/C1PDbJ7e1ruSrJn81HnbOmqvwzT3+ArYCbgD8AHgusBvaa0Oc44NR2/CfAWfNd92ZY42JgX+DzwLL5rnkzrvMFwLbt+I2dfi0fP3R8JHDBfNe9OdbZ+i0CrgC+Ayyd77o3w9fyWOCT813rbP1xZze/DgR+VFU/rqr7gS8BL5nQ5yXA59rx2cAfJskc1jhTU66xqm6uqjXAQ/NR4CwZZZ2XVtU97eV3gKfOcY0zNcoa7xp6uR2wJT4BN8r/lwAnAR8Ffj2Xxc2SUdfYDcNufj0FuGXo9U9b26R9qupB4E7gSXNS3ewYZY09GHedrwO+sVkrmn0jrTHJnye5iUEQnDBHtc2mKdeZ5ADg96vqvLksbBaN+t/ry9tl97OT/P7clLZ5GHbza7Id2sR/CY/S55FsS69/VCOvM8mfAkuBj23WimbfSGusqk9V1e7AO4D3bPaqZt8m15nkMcAngP82ZxXNvlG+ll8HFlfVvsAl/PYK0xbJsJtfPwWG/7X0VODnG+uTZAGwA/DLOaludoyyxh6MtM4kLwLeDRxZVffNUW2zZdyv5ZeAl27WijaPqda5CNgHuCzJzcC/Bc7dwh5SmfJrWVX/PPTf6GnAkjmqbbMw7ObXtcAzkzw9yWMZPIBy7oQ+5wKvbsfLgG9Vu3u8hRhljT2Ycp3t0td/ZxB0t81DjTM1yhqfOfTyj4AfzmF9s2WT66yqO6tqp6paXFWLGdx/PbKqVsxPudMyytdyl6GXRwLXz2F9s87fejCPqurBJG8CLmTwdNTpVXVdkr8EVlTVucD/AL6Q5EcMdnR/Mn8Vj2+UNSZ5LnAO8ETgPyb5i6raex7LHtuIX8uPAdsDX2nPGP1jVR05b0WPacQ1vqntXh8A7uC3/1DbYoy4zi3aiGs8IcmRwIMM/u45dt4KngX+uDBJUve8jClJ6p5hJ0nqnmEnSeqeYSdJ6p5hJ0nqnmEnSeqeYSdJ6t7/A11GRToDoWrCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "# feature importance 추출 \n",
    "print(\"Feature importances:\\n{0}\".format(np.round(dt_clf.feature_importances_, 3)))\n",
    "\n",
    "# feature별 importance 매핑\n",
    "for name, value in zip(iris_data.feature_names , dt_clf.feature_importances_):\n",
    "    print('{0} : {1:.3f}'.format(name, value))\n",
    "\n",
    "# feature importance를 column 별로 시각화 하기 \n",
    "sns.barplot(x=dt_clf.feature_importances_ , y=iris_data.feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DT GridSearch CV\n",
    "- max_depth, min_sample_split 조정하며 비교\n",
    "- train도 해보고, test도 해 보면 됨"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- train으로 GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {\n",
    "    'max_depth' : [ 8 , 12, 16 ,20], \n",
    "    'min_samples_split' : [16,24],\n",
    "}\n",
    "\n",
    "grid_cv = GridSearchCV(dt_clf, param_grid=params, scoring='accuracy', cv=5, verbose=1 )\n",
    "grid_cv.fit(X_train , y_train)\n",
    "print('GridSearchCV 최고 평균 정확도 수치: {0:.4f}'.format(grid_cv.best_score_))\n",
    "print('GridSearchCV 최적 하이퍼 파라미터:', grid_cv.best_params_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- test로 예측하고 정확도 계산"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_df_clf = grid_cv.best_estimator_\n",
    "\n",
    "pred1 = best_df_clf.predict(X_test)\n",
    "accuracy = accuracy_score(y_test , pred1)\n",
    "print('결정 트리 예측 정확도:{0:.4f}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- GridSearch CV의 세부 주요 내역"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GridSearchCV객체의 cv_results_ 속성을 DataFrame으로 생성. \n",
    "cv_results_df = pd.DataFrame(grid_cv.cv_results_)\n",
    "\n",
    "# max_depth 파라미터 값과 그때의 테스트(Evaluation)셋, 학습 데이터 셋의 정확도 수치 추출\n",
    "# 사이킷런 버전이 업그레이드 되면서 아래의 GridSearchCV 객체의 cv_results_에서 mean_train_score는 더이상 제공되지 않습니다\n",
    "# cv_results_df[['param_max_depth', 'mean_test_score', 'mean_train_score']]\n",
    "\n",
    "# max_depth 파라미터 값과 그때의 테스트(Evaluation)셋, 학습 데이터 셋의 정확도 수치 추출\n",
    "cv_results_df[['param_max_depth', 'mean_test_score']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- feature importance 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "ftr_importances_values = best_df_clf.feature_importances_\n",
    "\n",
    "# Top 중요도로 정렬을 쉽게 하고, 시본(Seaborn)의 막대그래프로 쉽게 표현하기 위해 Series변환\n",
    "ftr_importances = pd.Series(ftr_importances_values, index=X_train.columns  )\n",
    "\n",
    "# 중요도값 순으로 Series를 정렬\n",
    "ftr_top20 = ftr_importances.sort_values(ascending=False)[:20]\n",
    "plt.figure(figsize=(8,6))\n",
    "plt.title('Feature importances Top 20')\n",
    "sns.barplot(x=ftr_top20 , y = ftr_top20.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 앙상블 분류"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Voting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Logistic과 KNN을 Voting해보겠음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T06:15:48.066088Z",
     "start_time": "2020-09-16T06:15:47.928557Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.8</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.6</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.9</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.8</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.0</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.5</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38           122.8     1001.0          0.11840   \n",
       "1        20.57         17.77           132.9     1326.0          0.08474   \n",
       "2        19.69         21.25           130.0     1203.0          0.10960   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0           0.27760          0.3001              0.14710         0.2419   \n",
       "1           0.07864          0.0869              0.07017         0.1812   \n",
       "2           0.15990          0.1974              0.12790         0.2069   \n",
       "\n",
       "   mean fractal dimension  ...  worst radius  worst texture  worst perimeter  \\\n",
       "0                 0.07871  ...         25.38          17.33            184.6   \n",
       "1                 0.05667  ...         24.99          23.41            158.8   \n",
       "2                 0.05999  ...         23.57          25.53            152.5   \n",
       "\n",
       "   worst area  worst smoothness  worst compactness  worst concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "\n",
       "   worst concave points  worst symmetry  worst fractal dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "\n",
       "[3 rows x 30 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "\n",
    "data_df = pd.DataFrame(cancer.data, columns=cancer.feature_names)\n",
    "data_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 개별 모델을 VotingClassifier로 먼저 결합하고 데이터를 train\n",
    "- parameter에서 'hard', 'soft' 미리 결정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T06:17:25.594905Z",
     "start_time": "2020-09-16T06:17:25.536976Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Voting 분류기 정확도: 0.9561\n",
      "LogisticRegression 정확도: 0.9474\n",
      "KNeighborsClassifier 정확도: 0.9386\n"
     ]
    }
   ],
   "source": [
    "# 개별 모델은 로지스틱 회귀와 KNN 임. \n",
    "lr_clf = LogisticRegression()\n",
    "knn_clf = KNeighborsClassifier(n_neighbors=8)\n",
    "\n",
    "# 개별 모델을 소프트 보팅 기반의 앙상블 모델로 구현한 분류기 \n",
    "vo_clf = VotingClassifier( estimators=[('LR',lr_clf),('KNN',knn_clf)] , voting='soft' )\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, \n",
    "                                                    test_size=0.2 , random_state= 156)\n",
    "\n",
    "# VotingClassifier 학습/예측/평가. \n",
    "vo_clf.fit(X_train , y_train)\n",
    "pred = vo_clf.predict(X_test)\n",
    "print('Voting 분류기 정확도: {0:.4f}'.format(accuracy_score(y_test , pred)))\n",
    "\n",
    "# 개별 모델의 학습/예측/평가.\n",
    "classifiers = [lr_clf, knn_clf]\n",
    "for classifier in classifiers:\n",
    "    classifier.fit(X_train , y_train)\n",
    "    pred = classifier.predict(X_test)\n",
    "    class_name= classifier.__class__.__name__\n",
    "    print('{0} 정확도: {1:.4f}'.format(class_name, accuracy_score(y_test , pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T06:21:48.873959Z",
     "start_time": "2020-09-16T06:21:48.781317Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import pandas as pd\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# 결정 트리에서 사용한 get_human_dataset( )을 이용해 학습/테스트용 DataFrame 반환\n",
    "X_train, X_test, y_train, y_test = get_human_dataset()\n",
    "\n",
    "# 랜덤 포레스트 학습 및 별도의 테스트 셋으로 예측 성능 평가\n",
    "rf_clf = RandomForestClassifier(random_state=0)\n",
    "rf_clf.fit(X_train , y_train)\n",
    "pred = rf_clf.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test , pred)\n",
    "print('랜덤 포레스트 정확도: {0:.4f}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- GridSearch CV로 파라미터 튜닝 (train으로 튜닝)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {\n",
    "    'n_estimators':[100],\n",
    "    'max_depth' : [6, 8, 10, 12], \n",
    "    'min_samples_leaf' : [8, 12, 18 ],\n",
    "    'min_samples_split' : [8, 16, 20]\n",
    "}\n",
    "# RandomForestClassifier 객체 생성 후 GridSearchCV 수행\n",
    "rf_clf = RandomForestClassifier(random_state=0, n_jobs=-1)\n",
    "grid_cv = GridSearchCV(rf_clf , param_grid=params , cv=2, n_jobs=-1 )\n",
    "grid_cv.fit(X_train , y_train)\n",
    "\n",
    "print('최적 하이퍼 파라미터:\\n', grid_cv.best_params_)\n",
    "print('최고 예측 정확도: {0:.4f}'.format(grid_cv.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- GridSearch CV 후 test로 성능 평가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_clf1 = RandomForestClassifier(n_estimators=300, max_depth=10, min_samples_leaf=8, \\\n",
    "                                 min_samples_split=8, random_state=0)\n",
    "rf_clf1.fit(X_train , y_train)\n",
    "pred = rf_clf1.predict(X_test)\n",
    "print('예측 정확도: {0:.4f}'.format(accuracy_score(y_test , pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Feature Importance 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "ftr_importances_values = rf_clf1.feature_importances_\n",
    "ftr_importances = pd.Series(ftr_importances_values,index=X_train.columns  )\n",
    "ftr_top20 = ftr_importances.sort_values(ascending=False)[:20]\n",
    "\n",
    "plt.figure(figsize=(8,6))\n",
    "plt.title('Feature importances Top 20')\n",
    "sns.barplot(x=ftr_top20 , y = ftr_top20.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GBM (Gradient Boosting Machine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "import time\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "X_train, X_test, y_train, y_test = get_human_dataset()\n",
    "\n",
    "# GBM 수행 시간 측정을 위함. 시작 시간 설정.\n",
    "start_time = time.time()\n",
    "\n",
    "gb_clf = GradientBoostingClassifier(random_state=0)\n",
    "gb_clf.fit(X_train , y_train)\n",
    "gb_pred = gb_clf.predict(X_test)\n",
    "gb_accuracy = accuracy_score(y_test, gb_pred)\n",
    "\n",
    "print('GBM 정확도: {0:.4f}'.format(gb_accuracy))\n",
    "print(\"GBM 수행 시간: {0:.1f} 초 \".format(time.time() - start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- GridSearch CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T06:23:16.052747Z",
     "start_time": "2020-09-16T06:23:16.035328Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {\n",
    "    'n_estimators':[100, 500],\n",
    "    'learning_rate' : [ 0.05, 0.1]\n",
    "}\n",
    "grid_cv = GridSearchCV(gb_clf , param_grid=params , cv=2 ,verbose=1)\n",
    "grid_cv.fit(X_train , y_train)\n",
    "print('최적 하이퍼 파라미터:\\n', grid_cv.best_params_)\n",
    "print('최고 예측 정확도: {0:.4f}'.format(grid_cv.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- GridSearch CV 상세"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_df = pd.DataFrame(grid_cv.cv_results_)\n",
    "scores_df[['params', 'mean_test_score', 'rank_test_score',\n",
    "'split0_test_score', 'split1_test_score']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 최적 모델로 test 수행"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GridSearchCV를 이용하여 최적으로 학습된 estimator로 predict 수행. \n",
    "gb_pred = grid_cv.best_estimator_.predict(X_test)\n",
    "gb_accuracy = accuracy_score(y_test, gb_pred)\n",
    "print('GBM 정확도: {0:.4f}'.format(gb_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T06:25:08.758376Z",
     "start_time": "2020-09-16T06:25:08.744312Z"
    }
   },
   "source": [
    "### XGB (eXtra Gradient Boost)\n",
    "- early_stopping_rounds(조기 종료) 숫자를 조정하며 튜닝, 너무 적으면 성능이 안 나오니 적당히 100 정도"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 사이킷런 래퍼 XGBoost 클래스인 XGBClassifier 임포트\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "evals = [(X_test, y_test)]\n",
    "xgb_wrapper = XGBClassifier(n_estimators=400, learning_rate=0.1, max_depth=3)\n",
    "xgb_wrapper.fit(X_train , y_train,  early_stopping_rounds=100,eval_set=evals, eval_metric=\"logloss\",  verbose=True)\n",
    "w_preds = xgb_wrapper.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 변수 중요도 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import plot_importance\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 12))\n",
    "# 사이킷런 래퍼 클래스를 입력해도 무방. \n",
    "plot_importance(xgb_wrapper, ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- XGB GridSearch CV "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "xgb_clf = XGBClassifier(n_estimators=100)\n",
    "params = {'max_depth':[5, 7] , 'min_child_weight':[1,3] ,'colsample_bytree':[0.5, 0.75] }\n",
    "\n",
    "# 하이퍼 파라미터 테스트의 수행속도를 향상 시키기 위해 cv 를 지정하지 않음. \n",
    "gridcv = GridSearchCV(xgb_clf, param_grid=params)\n",
    "gridcv.fit(X_train, y_train, early_stopping_rounds=30, eval_metric=\"auc\",\n",
    "           eval_set=[(X_train, y_train), (X_test, y_test)])\n",
    "\n",
    "print('GridSearchCV 최적 파라미터:',gridcv.best_params_) \n",
    "\n",
    "xgb_roc_score = roc_auc_score(y_test, gridcv.predict_proba(X_test)[:,1], average='macro')\n",
    "print('ROC AUC: {0:.4f}'.format(xgb_roc_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LGB (LightGBM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LightGBM의 파이썬 패키지인 lightgbm에서 LGBMClassifier 임포트\n",
    "from lightgbm import LGBMClassifier\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 전체 데이터 중 80%는 학습용 데이터, 20%는 테스트용 데이터 추출\n",
    "X_train, X_test, y_train, y_test=train_test_split(ftr, target, test_size=0.2, random_state=156 )\n",
    "\n",
    "# 앞서 XGBoost와 동일하게 n_estimators는 400 설정. \n",
    "lgbm_wrapper = LGBMClassifier(n_estimators=400)\n",
    "\n",
    "# y의 class 비율이 극도로 불균형일 때는 아래와 같이 boost_from_average=False로 설정\n",
    "# lgbm_clf = LGBMClassifier(n_estimators=1000, num_leaves=64, n_jobs=-1, boost_from_average=False)\n",
    "\n",
    "# LightGBM도 XGBoost와 동일하게 조기 중단 수행 가능. \n",
    "evals = [(X_test, y_test)] # 원래는 validation set으로 수행해야 함\n",
    "lgbm_wrapper.fit(X_train, y_train, early_stopping_rounds=100, eval_metric=\"logloss\", \n",
    "                 eval_set=evals, verbose=True)\n",
    "preds = lgbm_wrapper.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 변수 중요도 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot_importance( )를 이용하여 feature 중요도 시각화\n",
    "from lightgbm import plot_importance\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 12))\n",
    "plot_importance(lgbm_wrapper, ax=ax, feature_names=dataset.feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- LGB GridSearch CV "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "LGBM_clf = LGBMClassifier(n_estimators=200)\n",
    "params = {'num_leaves': [32, 64 ],\n",
    "          'max_depth':[128, 160],\n",
    "          'min_child_samples':[60, 100],\n",
    "          'subsample':[0.8, 1]}\n",
    "\n",
    "# 하이퍼 파라미터 테스트의 수행속도를 향상 시키기 위해 cv 를 지정하지 않습니다. \n",
    "gridcv = GridSearchCV(lgbm_clf, param_grid=params)\n",
    "gridcv.fit(X_train, y_train, early_stopping_rounds=30, eval_metric=\"auc\",\n",
    "           eval_set=[(X_train, y_train), (X_test, y_test)])\n",
    "\n",
    "print('GridSearchCV 최적 파라미터:', gridcv.best_params_)\n",
    "lgbm_roc_score = roc_auc_score(y_test, gridcv.predict_proba(X_test)[:,1], average='macro')\n",
    "print('ROC AUC: {0:.4f}'.format(lgbm_roc_score))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 기타 분류 모델"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- KNN clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = [[0], [1], [2], [3]]\n",
    "y = [0, 0, 1, 1]\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "neigh = KNeighborsClassifier(n_neighbors=3)\n",
    "neigh.fit(X, y)\n",
    "\n",
    "print(neigh.predict([[1.1]]))\n",
    "\n",
    "print(neigh.predict_proba([[0.9]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 가우시안 나이브 베이즈"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n",
    "Y = np.array([1, 1, 1, 2, 2, 2])\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "clf = GaussianNB()\n",
    "clf.fit(X, Y)\n",
    "\n",
    "print(clf.predict([[-0.8, -1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- LDA clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n",
    "y = np.array([1, 1, 1, 2, 2, 2])\n",
    "clf = LinearDiscriminantAnalysis()\n",
    "clf.fit(X, y)\n",
    "\n",
    "print(clf.predict([[-0.8, -1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MLP clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = make_classification(n_samples=100, random_state=1)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,\n",
    "                                                    random_state=1)\n",
    "clf = MLPClassifier(random_state=1, max_iter=300).fit(X_train, y_train)\n",
    "clf.predict_proba(X_test[:1])\n",
    "\n",
    "clf.predict(X_test[:5, :])\n",
    "\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "model = SVC(kernel = 'rbf').fit(train_X, train_Y)\n",
    "predicted = model.predict(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 차원축소"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PCA\n",
    "- 스케일링 필요 - StandardScaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-11T07:57:59.272202Z",
     "start_time": "2020-12-11T07:57:58.059318Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('./data/iris.csv')\n",
    "irisDF = data.iloc[:,1:5]\n",
    "iris=pd.DataFrame()\n",
    "iris['target'] = data['Species']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-11T07:58:00.315468Z",
     "start_time": "2020-12-11T07:58:00.288540Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sepal.Length</th>\n",
       "      <th>Sepal.Width</th>\n",
       "      <th>Petal.Length</th>\n",
       "      <th>Petal.Width</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Sepal.Length  Sepal.Width  Petal.Length  Petal.Width\n",
       "0             5.1          3.5           1.4          0.2\n",
       "1             4.9          3.0           1.4          0.2\n",
       "2             4.7          3.2           1.3          0.2\n",
       "3             4.6          3.1           1.5          0.2\n",
       "4             5.0          3.6           1.4          0.2\n",
       "..            ...          ...           ...          ...\n",
       "145           6.7          3.0           5.2          2.3\n",
       "146           6.3          2.5           5.0          1.9\n",
       "147           6.5          3.0           5.2          2.0\n",
       "148           6.2          3.4           5.4          2.3\n",
       "149           5.9          3.0           5.1          1.8\n",
       "\n",
       "[150 rows x 4 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "irisDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-11T07:58:52.149375Z",
     "start_time": "2020-12-11T07:58:49.435939Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "iris_scaled = StandardScaler().fit_transform(irisDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- PCA로 변환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-11T07:59:08.042939Z",
     "start_time": "2020-12-11T07:59:07.719805Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 3)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "pca = PCA(n_components=3)\n",
    "\n",
    "#fit( )과 transform( ) 을 호출하여 PCA 변환 데이터 반환\n",
    "pca.fit(iris_scaled)\n",
    "iris_pca = pca.transform(iris_scaled)\n",
    "print(iris_pca.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-11T08:03:34.505770Z",
     "start_time": "2020-12-11T08:03:34.492804Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pca_component_1</th>\n",
       "      <th>pca_component_2</th>\n",
       "      <th>pca_component_3</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.264703</td>\n",
       "      <td>0.480027</td>\n",
       "      <td>-0.127706</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-2.080961</td>\n",
       "      <td>-0.674134</td>\n",
       "      <td>-0.234609</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2.364229</td>\n",
       "      <td>-0.341908</td>\n",
       "      <td>0.044201</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pca_component_1  pca_component_2  pca_component_3  target\n",
       "0        -2.264703         0.480027        -0.127706  setosa\n",
       "1        -2.080961        -0.674134        -0.234609  setosa\n",
       "2        -2.364229        -0.341908         0.044201  setosa"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# PCA 환된 데이터의 컬럼명을 각각 pca_component_1, pca_component_2로 명명\n",
    "pca_columns=['pca_component_1','pca_component_2', 'pca_component_3']\n",
    "irisDF_pca = pd.DataFrame(iris_pca,columns=pca_columns)\n",
    "irisDF_pca['target']=iris.target\n",
    "irisDF_pca.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- PCA 변환된 데이터 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-11T08:04:09.554976Z",
     "start_time": "2020-12-11T08:04:09.206540Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "\n",
    "#setosa를 세모, versicolor를 네모, virginica를 동그라미로 표시\n",
    "markers=['^', 's', 'o']\n",
    "\n",
    "#pca_component_1 을 x축, pc_component_2를 y축으로 scatter plot 수행. \n",
    "for i, marker in enumerate(markers):\n",
    "    x_axis_data = irisDF_pca[irisDF_pca['target']==i]['pca_component_1']\n",
    "    y_axis_data = irisDF_pca[irisDF_pca['target']==i]['pca_component_2']\n",
    "    plt.scatter(x_axis_data, y_axis_data, marker=marker,label=iris.target_names[i])\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('pca_component_1')\n",
    "plt.ylabel('pca_component_2')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- PCA 변환을 수행한 PCA 컴포넌트별로 차지하는 변동성 비율을 나타냄\n",
    "- 첫 번째 PCA 변환 요소가 전체 변동성의 75%, 두 번째가 20%를 차지하면 두 개의 요소만으로도 원본 데이터의 90% 설명 가능"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.72962445 0.22850762 0.03668922]\n"
     ]
    }
   ],
   "source": [
    "print(pca.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 원본 데이터와 PCA 데이터의 CV 분류 결과"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "rcf = RandomForestClassifier(random_state=156)\n",
    "scores = cross_val_score(rcf, iris.data, iris.target,scoring='accuracy',cv=3)\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PCA 요소 두 개로 거의 커버 가능하니 두 개 데이터만 가져다 씀\n",
    "pca_X = irisDF_pca[['pca_component_1', 'pca_component_2']]\n",
    "scores_pca = cross_val_score(rcf, pca_X, iris.target, scoring='accuracy', cv=3 )\n",
    "print(scores_pca)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LDA 선형 판별분석 Linear Discriminant Analysis\n",
    "- 특히 분류에 유리"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 스케일링 필요 - StandardScaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "iris_scaled = StandardScaler().fit_transform(iris.data) # X를 넣어줌"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- LDA는 PCA와 달리 지도학습이기 때문에 y가 필요함\n",
    "- 아래는 2개 컴포넌트로 데이터를 LDA 변환함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda = LinearDiscriminantAnalysis(n_components=2)\n",
    "lda.fit(iris_scaled, iris.target) # X들과 y를 넣어줌\n",
    "iris_lda = lda.transform(iris_scaled) # Scaled X를 넣어줌\n",
    "print(iris_lda.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LDA 변환된 데이터의 컬럼명을 각각 lda_component_1, lda_component_2로 명명\n",
    "lda_columns=['lda_component_1','lda_component_2']\n",
    "irisDF_lda = pd.DataFrame(iris_lda,columns=lda_columns)\n",
    "irisDF_lda['target']=iris.target\n",
    "irisDF_lda.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_X = irisDF_lda[['lda_component_1', 'lda_component_2']]\n",
    "scores_lda = cross_val_score(rcf, lda_X, iris.target, scoring='accuracy', cv=3 )\n",
    "print(scores_lda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVD Singular Value Decomposition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 일반적인 SVD는 책 p.398 참고\n",
    "### TruncatedSVD 잘린 SVD?\n",
    "- 스케일링 하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 데이터 로드\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "\n",
    "# TruncatedSVD 수행\n",
    "from sklearn.decomposition import TruncatedSVD, PCA\n",
    "iris_ftrs = iris.data # X\n",
    "\n",
    "# iris 데이터를 StandardScaler로 변환\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "iris_scaled = scaler.fit_transform(iris_ftrs) # X\n",
    "\n",
    "# 스케일링된 데이터를 기반으로 2개의 주요 component로 TruncatedSVD 변환 수행 \n",
    "tsvd = TruncatedSVD(n_components=2)\n",
    "tsvd.fit(iris_scaled) # Scaled X\n",
    "iris_tsvd = tsvd.transform(iris_scaled) # Scaled X\n",
    "\n",
    "# 시각화\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# Scatter plot 2차원으로 TruncatedSVD 변환 된 데이터 표현. 품종은 색깔로 구분\n",
    "plt.scatter(x=iris_tsvd[:,0], y= iris_tsvd[:,1], c= iris.target)\n",
    "plt.xlabel('TruncatedSVD Component 1')\n",
    "plt.ylabel('TruncatedSVD Component 2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NMF Non-Negative Matrix Factorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import NMF\n",
    "from sklearn.datasets import load_iris\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "iris = load_iris()\n",
    "iris_ftrs = iris.data # X\n",
    "nmf = NMF(n_components=2) \n",
    "nmf.fit(iris_ftrs) # X\n",
    "iris_nmf = nmf.transform(iris_ftrs) # X\n",
    "plt.scatter(x=iris_nmf[:,0], y= iris_nmf[:,1], c= iris.target)\n",
    "plt.xlabel('NMF Component 1')\n",
    "plt.ylabel('NMF Component 2')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "389px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}