NLP/naive-bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 朴素贝叶斯\n",
    "\n",
    "朴素贝叶斯是文本分类常用的算法，本文将详细叙述朴素贝叶斯算法的数学原理。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "符号定义：  \n",
    "* $n$：词汇表大小  \n",
    "* $X$：训练语料库，由$m$篇文档组成，$X=\\{x^{(1)},x^{(2)},...,x^{(m)}\\}$    \n",
    "* $x=(x_1,x_2,...,x_n)$：一篇文章的特征向量，其中$x_i$表示词汇表中第$i$个词$w_i$的词频或者tfidf。这里为了简化数学推导，我们假设$d$是二进制变量构成的向量，即$x_i\\in\\{0,1\\}$，1表示$w_i$出现，0表示$w_i$未出现   \n",
    "* $K$：文档的类别数，为了简化推导，假设K=2即二元分类的情形  \n",
    "* $y$：文档的类别  \n",
    "* $p(y=k|x)$：表示文档$x$属于类别$k$的后验概率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 贝叶斯分类器\n",
    "\n",
    "分类器可以理解为一个从特征$x$到类别$y$的映射关系：\n",
    "$$f:x\\to y$$\n",
    "而贝叶斯分类器的原理是将样本分到后验概率最大的类\n",
    "$$ \\hat{y}=\\arg\\max_{k} p(y=k|x) $$\n",
    "\n",
    "为了数学上的简便，我将概率$p(y=k|x)$简记为$p(y_k|x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 伯努利朴素贝叶斯\n",
    "我们以在$k=2$的情况为例，推导朴素贝叶斯的参数估计，这种情况下的朴素贝叶斯模型称为伯努利朴素贝叶斯。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "回想贝叶斯公式：\n",
    "$$p(y|x)=\\frac{p(x|y)p(y)}{p(x)}$$\n",
    "其中\n",
    "* $p(y)$称为先验概率\n",
    "* $p(x|y)$称为似然概率\n",
    "* $p(y|x)$称为后验概率\n",
    "* $p(x)$是归一化因子，保证概率之和为1\n",
    "\n",
    "由于训练样例$x$是事先已知的，因此$p(x)$可以视为常量，我们只需要考虑上式的分子，因此目标函数可以简化为\n",
    "$$p(y|x)\\propto p(x|y)p(y)$$\n",
    "\n",
    "因此我们可以将贝叶斯分类器的决策函数改写为\n",
    "\n",
    "$$ \\begin{aligned}\\hat{y}&=\\arg\\max_{k} p(y_k|x) \\\\&=\\arg\\max_{k} p(x,y_k)\\\\&=\\arg\\max_{k} p(x|y_k)p(y_k)\\\\&=\\arg\\max_{k} p(x_1,x_2,...,x_n|y_k)p(y_k)\\end{aligned}$$\n",
    "\n",
    "那么问题来了，似然概率$p(x_1,x_2,...,x_n|y_k)$怎么计算？  \n",
    "\n",
    "如果根据条件概率的公式，我们可以做如下的拆解：\n",
    "\n",
    "$$ p(x_1,x_2,...,x_n|y_k)=p(x_1|y_k)p(x_2|y_k,x_1)\\cdots p(x_n|y_k,x_1,...,x_{n-1})$$\n",
    "\n",
    "朴素贝叶斯作了一个很巧妙也很naive的假设，如果在给定$y$的前提下，特征之间相互独立。那么在给定$y$的情况下，$x_n$与$x_1,x_2,...,x_{n-1}$都无关，即$p(x_n|y_k,x_1,...,x_{n-1})=p(x_n|y_k)$，于是我们有\n",
    "\n",
    "$$\\begin{aligned}p(x_1,x_2,...,x_n|y_k)&=\\prod_{j=1}^n p(x_j|y_k)\\\\&=\\prod_{j=1}^n (p(x_j=1|y_k))^{x_j}\\cdot (1-p(x_j=1|y_k)^{(1-x_j)}\\\\&=\\prod_{j=1}^n \\phi_{jk}^{x_j}(1-\\phi_{jk})^{(1-x_j)}\\end{aligned}$$\n",
    "\n",
    "\n",
    "其中$\\phi_{jk}=p(x_j=1|y=k)$，表示类别为$k$的情况下，第$j$个词出现的概率  \n",
    "定义$\\phi_k=p(y=k)$，那么决策函数进一步简化为\n",
    "\n",
    "$$ \\hat{y}=\\arg\\max_{k} p(y_k|x) =\\arg\\max_{k} \\phi_k \\prod_{i=1}^n \\phi_{jk}^{x_j}(1-\\phi_{jk})^{(1-x_j)}$$\n",
    "\n",
    "如果我们知道了$\\phi_{jk}$，$\\phi_k$就可以预测每篇文章的类别。对于$\\phi_{jk}$来说，参数个数一共是$K*n$，其中$n$是词汇表大小，$K$是类别数，对于二分类的情形来说，我们只需要估计$2n$个参数；对于$\\phi_k$来说，参数一共有$K$个，二分类的情况下该参数一共就只有2个（$\\phi_0$和$\\phi_1$）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P(X|Y)=P(X)$$\n",
    "$$P(X|Y)=P(X,Y)/P(Y)$$\n",
    "$$P(X,Y)=P(X)P(Y)$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MAP目标函数\n",
    "\n",
    "最大后验估计（MAP）是朴素贝叶斯的基石，本小节将详细叙述如何用MAP估计朴素贝叶斯模型的参数。    \n",
    "透过上一小节的推导，我们知道后验概率与$(x,y)$的联合概率成正比\n",
    "$$p(y|x)\\propto p(x,y)$$\n",
    "\n",
    "现在让我们假设训练集的样本之间是独立同分布的，那么我们可以把生成该训练集的概率表示为\n",
    "\n",
    "$$ \\begin{aligned}P(X,Y)&=\\prod_{i=1}^m p(x^{(i)}, y^{(i)})\\\\&=\\prod_{i=1}^m p(y^{(i)})p(x^{(i)}|y^{(i)})\\\\&=\\prod_{i=1}^m \\phi_{y^{(i)}} \\prod_{j=1}^n \\phi_{jy^{(i)}}^{x^{(i)}_j}(1-\\phi_{jy^{(i)}})^{(1-x^{(i)}_j)}\\end{aligned}$$\n",
    "\n",
    "\n",
    "其对数似然为\n",
    "$$ \\begin{aligned}\\log P(X,Y) &=\\log \\prod_{i=1}^m \\phi_{y^{(i)}} \\prod_{j=1}^n \\phi_{jy^{(i)}}^{x^{(i)}_j}(1-\\phi_{jy^{(i)}})^{(1-x^{(i)}_j)}\\\\ &= \\sum_{i=1}^m \\bigg\\{  \\log(\\phi_{y^{(i)}}) + \\sum_{j=1}^n x^{(i)}_j\\log  \\phi_{jy^{(i)}} + (1-x^{(i)}_j)\\log(1-\\phi_{jy^{(i)}})\\big)\\bigg\\} \\end{aligned}$$\n",
    "综合$\\sum_{k=1}^K \\phi_k=1$的约束条件，我们可以得到如下的目标函数\n",
    "\n",
    "$$\\begin{aligned}& \\max \\sum_{i=1}^m \\bigg\\{  \\log(\\phi_{y^{(i)}}) + \\sum_{j=1}^n x^{(i)}_j\\log  \\phi_{jy^{(i)}} + (1-x^{(i)}_j)\\log(1-\\phi_{jy^{(i)}})\\big)\\bigg\\}\\\\&s.t. \\sum_{k=1}^K \\phi_k=1\\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MAP和MLE都属于频率学派的经典优化算法，它认为最优的参数应该能使似然函数取得最大值，因为在这样一组参数下，训练样本产生的可能性才是最大的。  \n",
    "\n",
    "估计参数的过程实际上是一个带等式约束的最优化问题，可以采用拉格朗日乘数法（Lagrangian multiplier）来求解。\n",
    "\n",
    "### 先验概率的MAP估计\n",
    "\n",
    "我们首先考察先验概率$\\phi_k$，如果我们要优化它，那么其他参数$\\phi_{jk}$就可以视为常数，从而可以把他们忽略，这时目标函数可以表示为\n",
    "\n",
    "$$\\begin{aligned} &\\phi_k^{MAP}=\\arg\\max_{\\phi_k} \\sum_{i=1}^m \\log(\\phi_{y^{(i)}}) \\\\&s.t. \\sum_{k=1}^K \\phi_k=1 \\end{aligned}$$\n",
    "\n",
    "拉格朗日函数为\n",
    "$$\\mathcal{L}(\\phi_k,\\lambda)=\\sum_{i=1}^m \\log(\\phi_{y^{(i)}})-\\lambda (\\sum_{k=1}^K \\phi_k-1)$$\n",
    "\n",
    "其中$\\lambda$为拉格朗日乘子\n",
    "\n",
    "$\\mathcal{L}(\\phi_k,\\lambda)$分别关于$\\phi_j$求偏导，并令导数为0  \n",
    "$$\\frac{\\partial \\mathcal{L}(\\phi_k,\\lambda)}{\\partial \\phi_k}=\\frac{1}{\\phi_k}\\sum_{i=1}^m \\mathbb{1}(y^{(i)}=k)  - \\lambda =0$$\n",
    "\n",
    "由此得\n",
    "\n",
    "$$\\phi_k= \\frac{\\mathbb{1}(y^{(i)}=k)}{\\lambda}=\\frac{m_k}{\\lambda} $$\n",
    "\n",
    "其中$m_k$表示训练集中类别为$k$的样例总数。\n",
    "\n",
    "结合$ \\sum_{k=1}^K \\phi_k=1 $的事实，有\n",
    "$$\\sum_{k=1}^K \\phi_k=\\frac{1}{\\lambda} \\sum_{k=1}^n m_k=\\frac{1}{\\lambda} m=1$$\n",
    "则$\\lambda=m$\n",
    "\n",
    "因此$\\phi_k$的MAP估计为\n",
    "$$\\phi_k=\\frac{m_k}{m}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 似然概率的MAP估计\n",
    "\n",
    "接着我们考虑似然概率，即关于$\\phi_{jk}$优化$\\log P(X,Y)$，这时$\\phi_k$可以看作常数，这时目标函数可以写为\n",
    "\n",
    "$$ \\ell(\\phi_{jk})=\\sum_{j=1}^n x^{(i)}_j\\log  \\phi_{jy^{(i)}} + (1-x^{(i)}_j)\\log(1-\\phi_{jy^{(i)}}) $$\n",
    "\n",
    "$$\\phi_{jk}^{MAP}=\\arg\\max_{\\phi_{jk}} \\ell(\\phi_{jk})$$\n",
    "\n",
    "关于$\\phi_{jk}$求导，令其导数为0\n",
    "$$\\frac{\\partial \\log \\ell(\\phi_{jk})}{\\partial \\phi_{jk} } = \\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k)  \\bigg(\\frac{ x^{(i)}_j}{\\phi_{jk}}-\\frac{1- x^{(i)}_j}{1-\\phi_{jk}}\\bigg)=0$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们在等式两边同时乘以$\\phi_{jk}(1-\\phi_{jk})$，则有\n",
    "$$\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k)\\big\\{(1-\\phi_{jk}) x_j^{(i)}-\\phi_{jk}  (1-x_j^{(i)})\\big\\}=0$$\n",
    "\n",
    "$$\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k)\\big( x_j^{(i)}-\\phi_{jk} x_j^{(i)}-\\phi_{jk}+\\phi_{jk} x_j^{(i)}\\big)=0$$\n",
    "\n",
    "$$\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k)\\big(x_j^{(i)}-\\phi_{jk}\\big)=0$$\n",
    "\n",
    "得\n",
    "\n",
    "$$\\phi_{jk}=\\frac{\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k) x_j^{(i)}}{\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k)}=\\frac{m_{jk}}{m_k}$$\n",
    "其中，$m_{jk}=\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k \\bigwedge x^{(i)}_j=1 ) $，$m_k=\\sum_{i=1}^m\\mathbb{1}(y^{(i)}=k)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\phi_k=\\sum_{i=1}^m \\frac{\\mathbb{1}(y^{(i)}=k)}{m}\\\\\\phi_{jk}=\\sum_{i=1}^m \\frac{\\mathbb{1}(y^{(i)}=k \\bigwedge x^{(i)}_j=1 )}{\\mathbb{1}(y^{(i)}=k)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 拉普拉斯平滑\n",
    "\n",
    "### 加1平滑\n",
    "\n",
    "朴素贝叶斯的一个缺点是容易过拟合，当一个词没有在某个类别中出现时，概率$\\phi_{jk}$将会是0，这个问题在训练语料比较小时尤为明显。拉普拉斯平滑可以解决这个问题，最简单的是`加1`平滑\n",
    "\n",
    "$$\\phi_{jk}=\\frac{m_{jk}+1}{m_k+n}$$\n",
    "\n",
    "在分母上加$n$，是为了保证$\\sum_{j=1}^n \\phi_{jk}=1$\n",
    "\n",
    "### 加$\\alpha$平滑\n",
    "\n",
    "加1平滑比较粗糙，一种比较好优化的方式是加$\\alpha$平滑\n",
    "\n",
    "$$ \\phi_{jk}=\\frac{m_{jk}+\\alpha}{m_k+n\\alpha}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实例\n",
    "## 1.虚假账号分类\n",
    "例子选自[算法杂货铺——分类算法之朴素贝叶斯分类(Naive Bayesian classification](http://www.cnblogs.com/leoo2sk/archive/2010/09/17/naive-bayesian-classifier.html)  \n",
    "根据某社区网站的抽样统计，该站10000个账号中有89%为真实账号（设为C0），11%为虚假账号（设为C1）。  \n",
    "接下来，要设计特征判断一个账号的真实性，我们选择了以下三个特征：  \n",
    "a1: 日志数量/注册天数  \n",
    "a2: 好友数量/注册天数   \n",
    "a3: 是否使用真实头像（真实头像为1，非真实头像为0）  \n",
    "\n",
    "假定某一个账号有以下三个特征：  \n",
    "a1 = 0.1  \n",
    "a2 = 0.2  \n",
    "a3 = 0  \n",
    "请问该账号是真实账号还是虚假账号？  \n",
    "\n",
    "这里有一个问题：a1和a2是连续变量，不适宜按照某个特定值计算概率。 一个技巧是将连续值变为离散值，计算区间的概率。比如将F1分解成[0, 0.05]、(0.05, 0.2)、[0.2, +∞]三个区间，然后计算每个区间的概率。在我们这个例子中，a1等于0.1，落在第二个区间，所以计算的时候，就使用第二个区间的发生概率。  \n",
    "下面给出划分：  \n",
    "`a1：{a<=0.05, 0.05<a<0.2, a>=0.2}，a1：{a<=0.1, 0.1<a<0.8, a>=0.8}，a3：{a=0(不是),a=1(是)}`  \n",
    "基于历史数据，我们统计出每个类别下各个特征的条件概率：  \n",
    "![](http://7xikew.com1.z0.glb.clouddn.com/naive_bayes_example1_condprob.png)\n",
    "\n",
    "接着我们计算先验概率：  \n",
    "$$P(C=0)=8900/10000=0.89\\\\P(C=1)=1100/10000=0.11$$\n",
    "\n",
    "最后计算决策函数：\n",
    "$$\\begin{aligned}P(C=0|a_1=0.1,a_2=0.2,a_3=0)&=P(a_1=0.1|C=0)P(a_2=0.2|C=0)P(a_3=0|C=0)P(C=0)\\\\&=P(0.05<a_1<0.2|C=0)P(0.1<a_2<0.8|C=0)P(a_3=0|C=0)P(C=0)\\\\&=0.5*0.7*0.2*0.89=0.0623\\end{aligned}$$\n",
    "\n",
    "\n",
    "$$\\begin{aligned}P(C=1|a_1=0.1,a_2=0.2,a_3=0)&=P(a_1=0.1|C=1)P(a_2=0.2|C=1)P(a_3=0|C=1)P(C=1)\\\\&=P(0.05<a_1<0.2|C=1)P(0.1<a_2<0.8|C=1)P(a_3=0|C=1)P(C=1)\\\\&=0.1*0.2*0.9*0.11=0.00198\\end{aligned}$$\n",
    "\n",
    "可以看到，虽然这个用户没有使用真实头像，但是通过分类器的鉴别，更倾向于将此账号归入真实账号类别。这个例子也展示了当特征属性充分多时，朴素贝叶斯分类对个别属性的抗干扰性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.MNIST手写数字分类\n",
    "\n",
    "### 首先从网络上下载数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "# Fetch the MNIST handwritten digit dataset\n",
    "from sklearn.datasets import fetch_mldata\n",
    "mnist = fetch_mldata('MNIST original', data_home=\"~/Data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一些样本的可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Number of samples, No. of pixels) =  (70000, 784)\n",
      "labels are:  set([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0])\n",
      "total number of classes = 10\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVEAAAEICAYAAADm98d9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUFNXZx/HvFVA22VFBNnEHEVwOkCiBiBsao4KiBkRF\n3Imao8irhoCKGhaXYzSKykGCoqiArG4YRgIuSEgw4otEXwURUBZh2ESU+/7Rfauqe7pnerp6mW5+\nn3P6TE3Vra4788zcfqrq1r3GWouIiKRnv3xXQESkkKkRFREJQY2oiEgIakRFREJQIyoiEoIaURGR\nEKp0I2qMKTHGDMr1vpJdimvx2hdjm5NG1BjzlTHm9FwcK5tMxChjzKboa5QxxpRTvqcxZoUxZqcx\nZr4xpnUu65ttRRTXIcaYT4wx24wxXxpjhlRQvqjjCsUTW8cYs78x5n+NMWsqKFfp2FbpTLQKuha4\nAOgIHA+cB1yXqKAxpgkwDRgGNAKWAFNyU02pJAMMABoCZwODjTGXJiyouBaqIcCG8gqkG9u8NqLG\nmIbGmNnGmA3GmO+jyy3iih1ujFlsjCk1xswwxjQK7N/VGPOeMWaLMWaZMaZHlqt8BfCQtXaNtfYb\nYCxwZZKyvYHl1tpXrLU/ACOAjsaYY7Jcx7wrtLhaa0dba5daa3+y1n4GzABOSVJ8n40rFF5so8c8\nDOgPPFhB0bRim+9MdD9gAtAaaAXsAh6PKzMAGAg0A34CHgMwxhwKzAFGEvnUuB2YaoxpWtFBjTG/\niwYx2atVkl3bA8sC3y+LrquwrLV2B/B5OeWLSaHFNfgeBugGLE9SZF+OKxRmbP8C3BWta3nSim1e\nG1Fr7SZr7VRr7U5r7TbgfqB7XLFJ1tpPoj/QMKCvMaYakU+Wudbaudbavdbat4mk3+ekcNzJ1toG\n5bxWJ9m1LrA18H0pUDfJddH4sq78gRXVr9AVYFyDRuA3FInss3GFwoutMeZCoJq1dnoKP15asa2e\nwhtnjTGmNvAIketQDaOrDzTGVLPW/hz9/uvALquAGkATIp+EFxtjzgtsrwHMz2KVtwP1At/XB7bb\nxKO4xJd15bdlqW5VRgHGFQBjzGAiWVQ3a+3uJMX22bhCYcXWGFMHGE0KjXRUWrHN9+n8bcDRQBdr\nbT3gV9H1wcyuZWC5FbAH2EgkUJPiPo3qWGv/XNFBjTH9jDHby3klOzVYTuSmktOR5Kd9MWWjAT28\nnPLFpNDiijFmIPA/QE9rbXl3cPfluEJhxfZIoA3wD2PMeiI3jZoZY9YbY9okKJ9ebK21WX8BXwG9\ngJqBV3UinxKvR79vBEwHLFA9ul8JsAZoB9QGXgEmR7e1BNYDZwHVou/RA2gR2HdQhn+O64H/BQ6N\nvj4Frk9StimRU4M+0bqNBj7Ixe87V68iimu/6DGPTaFs0ce1WGIbre8hgVdvYG10uVqmYpvLgNi4\n10igefQXtx1YSaS7UHxAHgQWE7k2MQtoEnjfLsC7wGYi3RfmAK2yEZDoe5roL3Zz9DUaMIHty4F+\nge9PB1YQuaBdArTJ9z+H4prw5/iSSLa0PfB6al+NazHFNu5n6gGsiVsXOrYmuqOIiKQh39dERUQK\nmhpREZEQ1IiKiISgRlREJIScdrY3xuzTd7GstUlHfCpkiqviWoxSjasyURGRENSIioiEoEZURCQE\nNaIiIiGoERURCUGNqIhICHkdT1QkqE2bNt7yO++8A0Dbtm1T3n/BggXe8htvvAHA2LFjAdizZ08G\naihSljJREZEQcjqKUy477/br189bnjRpEgBvvvkmAKNGjfK2lZSU5KpK6pRdgZdfftlbvuiii9Kp\nh7fs/q5fffVVAK644goAdu2qaJqdylNcU9exY2TM41/9KjKW88yZMwFYtWpVxo7RvXv3mGNNnDjR\n27Z1a/zsH8mps72ISA4UXSY6efJkAPr06eOtq1499tLvTz/95C3Pnx+Z3sVdO5s3b17W6qaMpXzu\njAGgS5cuMeu+/vrrhPsEPfzww95ygwYNYrZdf/31ADz99NOh6xlPca3wfbzldevWAdCkSRMA3n//\nfQC6desW6hi9evXylp9//nnA/xv429/+5m276qqrUn5PZaIiIjmgRlREJISi6eLk0vTzzz8fiD2F\nX7p0KQAnnngiADVq1PC2nXHGGQCccsopAIwYMQKATp06eWXuuOMOwD8Vkey4//77veUVK1ZUev8l\nS5Z4y6NHjwbg7LPPBuDPf45MKLl6tT89uesGJbnjTuMzbfDgwd5y/fr1Y7Zl+5KlMlERkRAK/sbS\nQQcdBMBbb70FQIcOHQD/hhHAJZdcAvifgg8++KC3zf387dq1A6B58+YA1K1b1yvz3nvvxbzP2rVr\n06qrbkDkzsknnwzArFmzADj44IMBWLhwoVfGdbMJS3EtX7C72ksvvRSz7dxzzwX87oeV5f5vXZwB\nWrduDfg3tI466ihv2xdffJHye+vGkohIDhT8NdELL7wQ8DPQoUOHAvDcc895ZTZt2hTztXfv3mXe\np0WLFoDfLeLFF1/0tv3yl78E4LLLLgPgoYceylj9JTu+/PJLILZ7jeSW+1+66aabymx7/fXXgfQz\nUMdlne5rUN++fYHKZZ/pUCYqIhKCGlERkRAK8nT+1FNP9Zbdc/A7duwA4KOPPgJg48aNlXrPNWvW\nxHzduXNnmTIDBw4EdDpfCEpLSwG45pprAJgxY0Y+q7NPcjdywz6NlIgb8euBBx5IWmbq1KkZP24i\nykRFREIoyEz01ltv9ZYPPPBAAObMmQPAu+++m5FjvPDCC96y6y7jbj716NHD25bLUaAkdW780EaN\nGsWsd7EE/wGNCRMm5K5i+xA3XkE2nHfeeYB/Qzlow4YNWTtuIspERURCKKhM1F0L7dmzp7du9+7d\nQOwYoZnw6aefllnnOuC3bNkyo8eS7Fm0aFHM9zVr1vSW031oQlJz7LHH5uW4wceHc0GZqIhICAWR\nie6///4A/OEPfwCgXr163jY3F098xhHWZ5995i2vXLkSiH18TArDkCFDkm5zPTokO9yDDtl44OGx\nxx4DYO/evQBs2bLF2/bxxx9n/HjlUSYqIhKCGlERkRAK4nT+2muvBeCCCy4os82N3pRpwekoXAd8\nnc4XDtf17Re/+AXgn1K60z+InSZGMs+NkJZopLiRI0fGfO9GYwL/OXg3xsW0adPK7O/i6N77u+++\n87YFp87OBWWiIiIhFEQm6rhswt3ogbLjE2aKe6wM4LDDDgNgv/0inznBR82Ck6tJ9tSqVQuInYDO\nZR8///xzmfJu/NDjjjsO8DOW4A3IDz74IDuV3ce5BxpOOumkpGXc//K9994LxE5x7v73XMxSmVzO\n3XwGOOKIIyos78Y4dTMehKFMVEQkhILKRN0n0/Lly7117nplpvXv399bdpmouw7j5mGSzHBZRPBx\n3mOOOSamjMtOgo/cTpkyBYBdu3YBsV1pOnfunI2qSgqaNm0KlD+fUvCxakg8HmhlBPd383O5v4dE\n12RfffXVUMcLUiYqIhKCGlERkRAK6nQ+l9woMUFbt24F0pvOV8pq2LAhAJMnTwbgrLPOqtT+buJA\nJ3g6n2wCxjp16njLblpsd6My0RiykpqOHTt6y25qnvKeVAreuI3nbuC6KV6CXZYuv/zymDLBLmvJ\n3idYZubMmQD89a9/TbpfZSkTFREJQZlonFtuuQXwu8YEzZs3D8j8c/r7qiOPPBIoPwN1z0E/8sgj\nAHz77bdlyriJBIcNG5b0fdzDEyeccIK3bunSpQCMHz8egDFjxnjbgt3opGLB6acbN24MJD8bCHLj\nFwSzTXeD0Z0ZBM8e3A3f+M72S5Ys8cq48USDNyqddevWxbx3JigTFREJoSAzUdeFAvwRndycOuk6\n99xzAb8jfXDcSTe26HXXXRfqGBIr2I0snstAXYaTKL6uw/Rdd92V9H3+85//AP5U2q67GsATTzwB\nwNVXXw3EXgcfNGgQALNnz67gp5B0uAzUjbL19NNPJy2bSuf54cOHe8thp2GuLGWiIiIhFGQmesop\np3jLTz75JAA33HADUPmMdPDgwYCfZboMNPg+brCE77//Ps0aS2W5Dvhnn312zPrgnDpXXnkl4N+F\nDVq4cCHgD1qzefNmwJ+dAPzHRvv27QvEjkOZ62xmX+OuVxbD/FbKREVEQlAjKiISQkGczs+dOxeA\nG2+8EYCjjz7a23bppZcC0KpVKwAeffRRIPb5etctpkaNGgD06dPH2/bQQw8BUL165Fexbds2IPYm\n0ssvv5ypH0UCypvGwT07X5lRuv75z396y/fccw/gn8Y727dv95anTp0a81XSF+ziVF4ne3fpJN3f\nuXvv+I70wePrxpKISAExqXSIzdjBjAl1sMMPPxyAkpISb13z5s0r3G/x4sWA32k3eHPCfZK5zrdu\n7MJMjvLiWGszP2NXFZBuXF33tPfffx9If4rdAQMGAPDaa69564IZZ7Yprn5XMigbx+D048cff3yl\n6xHs4pRshKY9e/Z4ZVavXg3EnrGmI9W4KhMVEQmhoDJR57LLLvOWXUfrli1bAv4I6O4aZyLBT635\n8+cDMHbsWMCfgjkblLEk1qxZM8Dvbgb+4CJt27YF/Axk+vTpXhn3qJ/LQHP5txykuPpnAwB33nkn\nAOPGjQNiY7Zq1apK1yP42Kd7/Nc9IOFivnbtWq/MxIkTgfIfA06FMlERkRxQIyoiEkJBns6Xx3Vf\nuummm7x18dMqBycoC96kyjad9hUnxTX3gpd+AGbNmuUtp3PJIBGdzouI5EDRZaJVmTKW4qS4Fidl\noiIiOaBGVEQkBDWiIiIhqBEVEQlBjaiISAhqREVEQlAjKiISQk4aUWPMV8aY03NxrKrMGNM633XI\nJMU1otjiCpHY5rsO+WaMmZ9KbHOaiVprTWVewLvANZXdL+y+5bznc8CrQB3gMOD/gIFJyjYFSoG+\nQC1gLDAll7/vXFFcizOuwBlFENsuwADgOmBVBWXjY7uEFGKb19N5Y0xDY8xsY8wGY8z30eUWccUO\nN8YsNsaUGmNmGGMaBfbvaox5zxizxRizzBjTI8tVPg8YY63daa39ChgPDExStjew3Fr7irX2B2AE\n0NEYc0yW65h3imvxKrTYWmsXW2snEflgrEhasc33NdH9gAlAa6AVsAt4PK7MACJ/0M2An4DHAIwx\nhwJzgJFAI+B2YKoxpmlFBzXG/C4axGSvVinW3wDHJdnWHljmvrHW7gA+j64vdopr8Sr02JYnrdjm\ntRG11m6y1k6NZgDbgPuB7nHFJllrP4n+QMOAvsaYakB/YK61dq61dq+19m0i6fc5KRx3srW2QTmv\n1Ul2fQMYaow50BhzBJE/lNpJytYFtsatKwUOrKh+hU5xLV4FGNvKSCu2+T6dr22MGWeMWWWMKQUW\nAA2iv3Dn68DyKqAG0ITIJ+HFwU8j4FQin37ZcjPwA/BfYAbwIrAmSdntQL24dfWBbVmrXRWhuBav\nAoxtZaQV23yfzt8GHA10sdbWA9y8p8HRU1oGllsBe4CNRAI1Ke7TqI619s8VHdQY088Ys72cV8JT\nA2vtZmttP2vtIdba9kR+f4uTHGY50DFwzDrA4dH1xU5xLV4FFdtKSi+21tqsv4CvgF5AzcCrOjAa\neD36fSNgOmCB6tH9SohkBO2InF69AkyObmsJrAfOAqpF36MH0CKw76AM/xyHA42jx+tF5A+jfZKy\nTYmcGvSJ1m008EEuft+5eimuxRnXIovtftHj9CKSFdcE9s9kbHMZEBv3Ggk0j/7itgMriXRDiA/I\ng0SyglJgFtAk8L5diHSL2AxsIHLRulUWA9IXWAvsBP4NnBW3fTnQL/D96cAKIhffS4A2+f7nUFwV\n130stj0S/BwlmYytie4oIiJpyPc1URGRgqZGVEQkBDWiIiIhqBEVEQmhei4PZjR7oGaFLEKKa3FK\nNa7KREVEQlAjKiISghpREZEQ1IiKiISgRlREJAQ1oiIiIagRFREJQY2oiEgIOe1sLyKSCZ06dQLg\n7bffBuC5557ztg0ZMiSndVEmKiISgjJRKSjXXXedtzxy5EgAN5gu1av7f84vvPACAGPHjgVg1apV\nuaqiZFGrVpFZQObOnQtA48aNAf9vIB+UiYqIhKBGVEQkhJxOD5KvUWGaN28OwLXXXgvA9ddf7207\n+OCDARg+fDgA9957b9bqodF+wuvSpYu3/I9//AOAatWqJSvODz/8APgxd6f5AHv37s1InRTX3OnY\nMTIZ57/+9S8Axo8fD8CYMWO8MitXrszIsTSKk4hIDhRtJtqjRw9v+cUXXwRg69atADzzzDPetnPO\nOQeAzp07A9CuXTsAvv7664zXSRlLeA0aNPCWP/roIwDatm0LwMSJE71tLgO9/PLLAahTpw4AF154\noVdmxowZGamT4po7Dz/8MACXXHIJAF27dgXy+/+qTFREJISiy0SPP/54ABYtWuSt++677wA488wz\nAfjiiy+8bb169QJgzpw5gJ+ZvvXWW14ZXTsrXy7i6rovBa9pXnzxxQAsXboUiL1e+vPPPwNw8803\nA/Doo48CsGnTJq9M06ZNM1I3xTW7DjroIG/5q6++Avzr4WeddVbWjqtMVEQkB9SIioiEUDRPLLmu\nSjNnzgTg3//+t7ft0ksvBeCbb74ps1/37t1jvndPQjRp0sRbt3nz5sxWVirN3VC46KKLvHXbt2+P\nWedO4YMee+wxwO/i1KxZs6zWUzLvjjvu8JZr1aoFQElJSZ5qU5YyURGREIomEx01ahQANWvWBGIz\nlm+//TambLBztrux5LhRYXbs2JGVekrlDBo0CIDBgweX2eZuArqbDeVZvXo1AEcccYS37phjjgFg\nxYoVYaspWeTiDH4cg6M25ZsyURGREAo+E/3tb38LwGWXXQb43V7is8+gYEf8Dh06ALBnzx4Abrnl\nFgB2796d8bpK6ho2bAjAX/7yF8Afpcdd4wRYuHBhyu/n9t9vPz9vqFu3buh6Sva4cUGPPfZYb517\nQGLdunV5qVMiykRFREIoyM72xvh9YKdOnQr4d9NPP/10AH788ccy+7kO2y+//LK37oILLgBg2bJl\nAJxwwgmZqGJC6pSduunTpwP+mcZ7770HxHau3rlzZ8rv53pYBP92XLYbluKaHevXrwegfv363jr3\n/x18mCZb1NleRCQH1IiKiIRQkDeWgl0e3On4wIEDgcSn8fFl3degp556KpNVlDQET6+7desWs23K\nlClA7Cl8fBelAw44wNvmnrd2fxdu9KcFCxZkutqSYe3btwegdu3aAMybN8/blovT+MpSJioiEkJB\nZqKNGjUqs+7vf/970vKHHHII4HeXCdq4cSPgZzqSP8EsM/6MwmWUwYco3AMRn3/+OQC/+c1vvG1t\n2rRJeIwHH3wwI3WVzAqeRbjHPA888EDAfxS7qlImKiISQkFmom7giaAjjzwSgP333x+IfbTTjWTv\nBikJclPqbtmyJeP1lMoJPuDgupy5Lk2dOnUqU951V3IzFgRHN9+2bRsA9erVA2D58uUAvPvuu5mu\ntmRAcCAgNxuBGwf4ySefzEudUqVMVEQkBDWiIiIhFOTpvBszFPxJ6F577TXAn5As+CSWmxpi1qxZ\ngD/iE+j0rqpyE5G5sRBOO+00AD788EOvjOuu5E7jgyM0uW4xbkwENxqUm8BOqpZENwLPO++83Fck\nDcpERURCKMhn54PcqDxnnHEGAKeeeioQ20HXPXftMhf3DD1Az549ASgtLc101crQM9bZ4caQXbx4\nsbfOddh+4IEHABg2bFjWjq+4ps89YBF8CMLFzs1CUN6IbNmkZ+dFRHKgIK+JBrnpjN98882Yr0Fd\nu3YF/Cl177//fm9bLjJQyQ6XxbgHJVwGAzB58mQARowYkfN6Ser69OkDxMbOdUcrlNkllImKiIRQ\n8JloMsFxI2+++WbAn+1Tg40ULnfNG2DMmDEAdO7cGYB33nnH23bllVcCiWcAlfxzD8OcffbZgN+L\nAvyeGYkeqqmKlImKiISgRlREJISiPZ0PPjt/6aWXAv4NCHdaL4XDTS4YfNDCTTTnRudykwxK1Tdg\nwAAAevfuDfhTlQN8+umnealTupSJioiEULSZaMeOHcusmzZtWh5qIumoVasWAH/6058AGDx4MBA7\n5fHvf/97QDcKC9Hw4cPzXYWMUSYqIhJC0WaiQ4cO9ZbdABWzZ8/OV3UkBSeffLK3/OyzzwLQoUMH\nAFavXg34j/eCP6K9FJ7WrVsDsQMFFSploiIiIagRFREJoehO5+vXrw/4XWIASkpKANi1a1ceaiTJ\nuC5Kt912GwB33323t83dQJoxYwbgd4kplKdYJDVz5swBsjvKVrYpExURCaHoMtFzzjkHiJ1WuZA/\n5YqNey4a/EkCmzdvDsCqVau8ba77kstUpLgEx7YodMpERURCKPiR7eO58SOvueYab92hhx6a7cOm\nRCOgFyfFtThpZHsRkRwouky0KlPGUpwU1+KkTFREJAfUiIqIhKBGVEQkhJw0osaYr4wxp+fiWFWZ\nMaZ1vuuQSYprRLHFFSKxzXcd8s0YMz+V2OY0E7XWmsq8gHeBayq7X9h9y3nPocBJQA3gGGA1cFmS\nsk2BUqAvUAsYC0zJ5e87VxTX4owrcEaRxPZj4BAi8ZoETE8xtktIIbZ5PZ03xjQ0xsw2xmwwxnwf\nXW4RV+xwY8xiY0ypMWaGMaZRYP+uxpj3jDFbjDHLjDE9sllfa+1oa+1Sa+1P1trPgBnAKUmK9waW\nW2tfsdb+AIwAOhpjjslmHasCxbV4FVpsgcOAN62130bjNQVon6RsWrHN9zXR/YAJQGugFbALeDyu\nzABgINAM+Al4DMAYcygwBxgJNAJuB6YaY5pWdFBjzO+iQUz2apXCexigG7A8SZH2wDL3jbV2B/A5\nyQNYTBTX4lVosR0PnGKMaW6MqQ30A15PUjat2Oa1EbXWbrLWTrXW7rTWbgPuB7rHFZtkrf0k+gMN\nA/oaY6oB/YG51tq51tq91tq3iaTf56Rw3MnW2gblvFanUP0R+H9QidQFtsatKwUOTOG9C5riWrwK\nMLb/Bb4GviESp2OBe5OUTSu2+T6dr22MGWeMWWWMKQUWAA2iv3Dn68DyKiLXrZoQ+SS8OPhpBJxK\n5NMv2/UeTOTT9lxr7e4kxbYD9eLW1Qe2ZbNuVYHiWrwKMLZPADWBxkAdYBrJM9G0Ypvv0/nbgKOB\nLtbaesCvouuDTwq0DCy3AvYAG4kEalLcp1Eda+2fKzqoMaafMWZ7Oa+kp33GmIHA/wA9rbVryjnM\ncsCbLc8YUwc4nOSnicVEcS1ehRbbTsAEa+3m6AfjX4DOxpgmCcqmF1trbdZfwFdALyKfCO5VHRhN\n5FOhJpFrJNMBC1SP7lcCrAHaAbWBV4DJ0W0tgfXAWUC16Hv0AFoE9h2U4Z+jX/SYx6ZQtimRU4M+\n0bqNBj7Ixe87Vy/FtTjjWmSxnQBMJZJR1gDuAr7JZGxzGRAb9xoJNI/+4rYDK4HrEgTkQWAxkWsT\ns4AmgfftQqRbxGZgA5GL1q2yGJAviXyqbg+8ngpsXw70C3x/OrCCyMX3EqBNvv85FFfFdR+LbWPg\nBeA7YAuwEOicydia6I4iIpKGfF8TFREpaGpERURCUCMqIhKCGlERkRByOtun0UjZGgG9CCmuxSnV\nuCoTFREJQY2oiEgIakRFREJQIyoiEoIaURGRENSIioiEoEZURCQENaIiIiHktLN9vlSvHvkx27Zt\nC0C7du3KlHnnnXcA2LZtnxigXKQonHnmmQBcccUV3rqbbroJgC1btuSkDspERURCKLpM1GWZN9xw\ng7fu/PPPB6BFi/iZXX0bN24E4MILLwRg0aJF2aqiZMiIESMA6N49Mk9ajx49vG0lJSUA3HPPPTHf\nS3E44IADABgyZAgAv/71r71tCxYsAGDcuHE5qYsyURGRENSIioiEkNPpQbI5KszgwYMBuPPOOwFo\n1syfhXX37sjsty+99BLgp/sAJ598MgD9+/cHoLS0FICOHb1J/9i8eXNG6qjRfirPnbIPHz48rf3d\naXzwdC/TFNfs6Nq1KwAtW7Yss23o0KEAdOrUKel+S5YsCXV8jeIkIpIDBZmJui5LABdffDEA48eP\nB/wbRFOnTvXKjBkzBoC1a9cmfc/bbrstpqz7HuCRRx7JRLWVsVQgeGNo/vz5ld4/ePMo+F7g32By\nmW0mKa7pO+qoowAYNmyYt87dKGzYsCEAtWrVSlQ3ABK1X23atAFgzZo1oeqmTFREJAcKsotT7dq1\nveU//vGPAHz88ceA353p22+/rdR7nnbaaTHfuy4Ukn0ua0wl+3QZJZSfVcZnKOriVDW4s8hzzz0X\ngGnTppUps99+kdxu7969Sd/HldmwYQMAy5Yt87Zt3bo1M5VNkTJREZEQ1IiKiIRQkKfzrhsSQPv2\n7UO9V506dQC/q1Om3ldSV95pvOualMrpePzNpCCdzlcNgwYNAuDxxx8HEt8Y+vLLLwH///y4444r\nU+a5554D/BtSYW8ihaFMVEQkhILMRDPp5ptvBqBp06YAvPDCCwDceOONeauTpJeBJspogzeiJP+O\nPPLIhOvvu+8+b/n5558HYNSoUUBsJrpy5UoA7rrrLgDWrVuXlXpWhjJREZEQ9slM1HXUBbjooosA\n2LNnD+A/NqpxRXMvmDWGzUDd/roWWrUsXbo04frg/+Sjjz4KQK9evcqU6927N1A1MlBHmaiISAgF\n+dhnWLfffru3PHr0aAAGDBgA+NdjskGPB2ZWeX+7wcwmB/VQXCvJPaZ95ZVXJjou4Md3woQJ3jZ3\ndz8X9NiniEgOqBEVEQlhnzqdd6O7uG4SAB9++CHgd6n56aefsnZ8nfalz91ECo4rGt+5PjhmaC5v\nKCmulee6LS1cuBCAunXrBo8LwGeffQZAt27dvG2bNm3KVpXK0Om8iEgO7BNdnNyn3EMPPQTA6tWr\nvW1uoqtsZqASnstAEz3aWZmO+VI1fPLJJwBMnz4dgMsvv7xMGZd15jL7TIcyURGREPaJTPTuu+8G\n/OmQ3eM9Zhw8AAADvklEQVRkAB988EFe6iSpcR3plYEWp+bNmyfddtJJJwGx3ZqeffbZrNepspSJ\nioiEoEZURCSEouvidOihhwKx3SKeeeYZwB879IsvvvC2zZw5Eyg7LXLjxo295VQubLuuUvPmzUta\nRl1hUpfsufhUpwdJR/D9KvPsveJaeW5a40WLFpXZtn79egAOOeSQMtvcOL8rVqzIVtU86uIkIpID\nBZmJurE/Ac4880wAOnToAPjPwCf6FPv+++8BqFGjhrcu2MkX/NGcgmXiBZ/Ldr8/t195E9wpYylf\n8OZRfJemVKY8TnTzKX5dsLN+ZZT3LL7iWnkuE3Wd7d3ZIvjTlU+cOBHwbwgD/Pe//wWgZ8+eQPnT\noIelTFREJAcKKhN1n15vvfWWty4+k0zkiSeeAPxR7OvXr+9tO+qoo2LKuulWg2VS4a6hvv7660nL\nKGNJzGWX3bt399Yle6QzUbbp9itvjqVMSZSRKq6V58YMHTx4MOA/kg3+fEnuf9NlqwCNGjUC/Idk\nHnnkkWxVUZmoiEguVNnO9ieeeKK3PG7cOAA6deoEQLVq1ZLu9+OPPwLQv39/b517tMxl3Vu2bPG2\nLV68OEM1lspyGWgq1ynLmxE0FYnmWnJ33hNlsPF1Uof+zHCd66+++uoKy7qBgoYOHeqtC147rSqU\niYqIhKBGVEQkhCp7Y6m0tNRbTuXmketidMsttwDw1FNPVbZ6WacbELHC/u25U2x3qp6vU27FNXVu\nJLVbb701Zn3r1q29ZXdjyXUXvOqqq7xt7ibxN998A0CrVq0yXUWPbiyJiORAlb2xlGh8wfLs3r0b\ngDfeeCMb1ZE8ic8ydYOnOMSfhQRvJLubwq5DfXDGArdfVbrBpExURCSEKntNtBjp2lks17XIfQ1m\nmYWUcSquqXNzK82ePRuAFi1aJDoukPia+X333Qck7rKWabomKiKSA2pERURC0Ol8Dum0rzgprpXn\nTuvdWBPNmjXztu3YsQOAe++9F4ApU6Z429atWwfAzz//nK2qeXQ6LyKSA8pEc0gZS3FSXIuTMlER\nkRxQIyoiEoIaURGRENSIioiEoEZURCQENaIiIiGoERURCUGNqIhICDntbC8iUmyUiYqIhKBGVEQk\nBDWiIiIhqBEVEQlBjaiISAhqREVEQlAjKiISghpREZEQ1IiKiISgRlREJAQ1oiIiIagRFREJQY2o\niEgIakRFREJQIyoiEoIaURGRENSIioiEoEZURCQENaIiIiGoERURCUGNqIhICGpERURCUCMqIhLC\n/wMnY22VVyuuowAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1170987d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the number of samples \n",
    "print \"(Number of samples, No. of pixels) = \", mnist.data.shape\n",
    "print 'labels are: ',set(mnist.target)\n",
    "print 'total number of classes = %d'%(len(set(mnist.target)))\n",
    "\n",
    "# Display 9 number randomly selectly\n",
    "for c in range(1, 10):\n",
    "    subplot(3, 3,c)\n",
    "    i = randint(mnist.data.shape[0])\n",
    "    im = mnist.data[i].reshape((28,28))\n",
    "    axis(\"off\")\n",
    "    title(\"Label = {}\".format(mnist.target[i]))\n",
    "    imshow(im, cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 划分训练集、测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num of train examples:66500, num of test examples:3500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/chenhongsheng/.pyenv/versions/anaconda2-4.3.1/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# Split the data into training and test data\n",
    "from sklearn.cross_validation import train_test_split\n",
    "x_train, x_test, y_train, y_test = train_test_split(mnist.data, mnist.target, test_size=0.05, random_state=42)\n",
    "print 'num of train examples:%d, num of test examples:%d'%(len(x_train), len(x_test))\n",
    "# Which is same as \n",
    "# x_train = mnist.data[:split]\n",
    "# y_train = mnist.target[:split]\n",
    "# x_test = mnist.data[split:]\n",
    "# y_test = mnist.target[split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载并初始化分类器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create the Multinomial Naive Bayes Classifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "clf = MultinomialNB()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练并计算测试集上的准确率、PRF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy = 81.7142857143 %\n",
      "Classification Report \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          0       0.91      0.89      0.90       320\n",
      "          1       0.85      0.95      0.90       387\n",
      "          2       0.89      0.84      0.87       352\n",
      "          3       0.78      0.81      0.80       356\n",
      "          4       0.83      0.73      0.78       329\n",
      "          5       0.85      0.67      0.75       342\n",
      "          6       0.90      0.91      0.91       377\n",
      "          7       0.94      0.82      0.87       360\n",
      "          8       0.63      0.72      0.67       337\n",
      "          9       0.66      0.80      0.72       340\n",
      "\n",
      "avg / total       0.83      0.82      0.82      3500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Perform the predictions\n",
    "clf.fit(x_train,y_train)\n",
    "# Perform the predictions\n",
    "y_predicted = clf.predict(x_test)\n",
    "# Calculate the accuracy of the prediction\n",
    "from sklearn.metrics import accuracy_score\n",
    "print \"Accuracy = {} %\".format(accuracy_score(y_test, y_predicted)*100)\n",
    "# Cross validate the scores\n",
    "from sklearn.metrics import classification_report\n",
    "print \"Classification Report \\n {}\".format(classification_report(y_test, y_predicted, labels=range(0,10)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}