YidaXu-ML/exponential-family-variational-inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 指数分布族的变分推理\n",
    "回顾一下上节课*变分推断*讲的内容。  \n",
    "与MCMC，Gibbs Sampling等基于抽样来估计未知分布的方法不同，变分推断通过一个优化过程来近似隐变量的后验概率分布。给定一组隐变量 $Z=\\{z_1,...,z_M\\}$，其中$M$表示$Z$可以分解为$M$个独立成分。变分推断用一个变分分布$q(Z)$估计后验分布$P(Z|X)$，通过最大化如下的ELBO来完成近似过程：\n",
    "$$ \\mathcal{L}(q(Z))=\\mathbb{E}_{q(Z)}[ln\\ p(X,Z)]-\\mathbb{E}_{q(Z)}[ln\\ q(Z)]$$\n",
    "最优的近似分布为\n",
    "$$ q^*(Z)=\\arg\\max_{q(Z)}\\mathcal{L}(q(Z))$$\n",
    "\n",
    "根据 meanfield假设，我们可以对$q(Z)$进行分解：\n",
    "$$q(Z)=\\prod_{i=1}^M q_i(Z_i)$$\n",
    "\n",
    "与上一堂课所用方法不同，这次我们假设$q_i(Z_i),(i=1,2,...,M)$属于指数分布族 ，这些分布的超参数是超参数$\\lambda_i,(i=1,2,...,M)$，这样做的好处是将泛函优化问题简化为了超参数优化问题。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以用一个概率图模型来表示上一节课的模型（原谅我图画的丑）：\n",
    "<img src=\"http://7xikew.com1.z0.glb.clouddn.com/QQ%E6%88%AA%E5%9B%BE20160808135828.jpg\" height=\"300\" width=\"300\" />\n",
    "由于$Z_1,Z_2,...,Z_M$都是$X$的co-parents，根据概率图模型理论，要估计$Z_i$的后验，我们不仅需要知道$X$的信息，还需要知道$\\{Z_1,...,Z_{i-1},Z_{i+1},...,Z_M\\}$的信息，即$Z_i$的后验同时取决于$X$和其他的$Z_j$，可以将其表示为$P(Z_i|X,Z_1,...,Z_{i-1},Z_{i+1},...,Z_M)$  \n",
    "为了数学上表示的方便，我们将$\\{Z_1,...,Z_{i-1},Z_{i+1},...,Z_M\\}$简记为$Z_{-i}$。我们假设$Z_i$的后验属于指数分布族：\n",
    "$$p(Z_i|X,Z_{-i})=h(Z_i)\\exp(T(Z_i)^T\\eta(X,Z_{-i})-A_g(\\eta(X,Z_{-i}))$$\n",
    "其中$T(Z_i)$是$Z_i$的充分统计量，$\\eta(X,Z_{-i})$是其自然参数，它是关于$X,Z_{-i}$的函数，特别地，对于广义线性模型来说这个函数是线性的。\n",
    "\n",
    "同样地，我们假设每个近似分布$q_i(Z_i)$也是指数分布族：\n",
    "$$q_i(Z_i|\\lambda_i)=h(Z_i)\\exp(T(Z_i)^T\\lambda_i-A_\\ell(\\lambda_i))$$\n",
    "其中$\\lambda_i$是$Z_i$的超参数。为了方便，记$\\lambda_{-i}=\\{\\lambda_1,...,\\lambda_{i-1},\\lambda_{i+1},...,\\lambda_{M}\\}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过引入超参我们可以将问题转化为一个参数优化问题：\n",
    "$$  \\lambda^*_1,...,\\lambda^*_M=\\arg\\max_{\\lambda_1,...,\\lambda_M} \\mathcal{L}(\\lambda_1,...,\\lambda_M)$$\n",
    "\n",
    "用贝叶斯公式给目标函数做个变形：\n",
    "$$ \\mathcal{L}(q(Z))=\\mathbb{E}_{q(Z)}[\\ln P(Z_i|X,Z_{-i}) +\\ln P(Z_{-i}|X) + \\ln P(X)]-\\mathbb{E}_{q(Z)}[\\ln q(Z)]$$\n",
    "其参数化形式为：\n",
    "$$ \\begin{aligned}\\mathcal{L}(\\lambda_1,...,\\lambda_M)&= \\mathbb{E}_{q(Z|\\lambda)}[\\ln P(Z_i|X,Z_{-i}) +\\ln P(Z_{-i}|X) + \\ln P(X)]-\\mathbb{E}_{q(Z|\\lambda)}[\\ln q_i(Z_i|\\lambda_i)]-\\mathbb{E}_{q(Z|\\lambda)}[\\sum_{j\\neq i} \\ln q_j(Z_j|\\lambda_j)]\\\\&= \\mathbb{E}_{q(Z|\\lambda)}[\\ln P(Z_i|X,Z_{-i})]-\\mathbb{E}_{q(Z|\\lambda)}[\\ln q_i(Z_i|\\lambda_i)]+const\\end{aligned}$$\n",
    "其中$\\mathbb{E}_{q(Z|\\lambda)}[\\ln P(Z_{-i}|X)],\\mathbb{E}_{q(Z|\\lambda)}[\\ln P(X)]$和$\\mathbb{E}_{q(Z|\\lambda)}[\\sum_{j\\neq i} \\ln q_j(Z_j|\\lambda_j)]$都是常数。一开始不太理解为什么$\\mathbb{E}_{q(Z|\\lambda)}[\\ln P(Z_{-i}|X)]$是常数，后来经过如下的计算想通了。\n",
    "$$\\begin{aligned}\\mathbb{E}_{q(Z|\\lambda)}[\\ln P(Z_{-i}|X)]&= \\int_{Z_i} \\bigg[\\int_{Z_{j\\neq i}} \\ln P(Z_{-i}|X)\\prod_{j\\neq i}q_j(Z_j|\\lambda_j) dZ_j\\bigg] q_i(Z_i|\\lambda_i) dZ_i\\\\&=\\int_{Z_{j\\neq i}} \\ln P(Z_{-i}|X)\\prod_{j\\neq i}q_j(Z_j|\\lambda_j) dZ_j \\int_{Z_i} q_i(Z_i|\\lambda_i) dZ_i\\\\&=\\int_{Z_{j\\neq i}} \\ln P(Z_{-i}|X)\\prod_{j\\neq i}q_j(Z_j|\\lambda_j) dZ_j =constant(\\mbox{因为$\\lambda_{-i}$已知})\\end{aligned}$$\n",
    "\n",
    "\n",
    "\n",
    "代入指数分布式子有\n",
    "$$ \\begin{aligned}\\mathcal{L}(\\lambda_1,...,\\lambda_M)&= \\mathbb{E}_{q(Z|\\lambda)}[\\ln P(Z_i|X,Z_{-i})]-\\mathbb{E}_{q(Z|\\lambda)}[\\ln q_i(Z_i|\\lambda_i)]+const\\\\&= \\mathbb{E}_{q(Z|\\lambda)}[\\ln h(Z_i)]+\\mathbb{E}_{q(Z|\\lambda)}[T(Z_i)^T\\eta(X,Z_{-i})]-\\underbrace{\\mathbb{E}_{q(Z|\\lambda)}[A_g(\\eta(X,Z_{-i}))]}_{const}\\\\&\\quad-\\mathbb{E}_{q(Z|\\lambda)}[\\ln h(Z_i)]-\\mathbb{E}_{q(Z|\\lambda)}[T(Z_i)^T\\lambda_i]+\\mathbb{E}_{q(Z|\\lambda)}[A_\\ell(\\lambda_i)]+const\\\\&=\\mathbb{E}_{q(Z|\\lambda)}[T(Z_i)^T(\\eta(X,Z_{-i})-\\lambda_i)]+\\mathbb{E}_{q(Z|\\lambda)}[A_\\ell(\\lambda_i)]+const\\\\\n",
    "&=\\mathbb{E}_{q_i(Z_i|\\lambda_i)}[T(Z_i)]^T\\cdot \\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})-\\lambda_i]+\\mathbb{E}_{q(Z|\\lambda)}[A_\\ell(\\lambda_i)]+const\\\\&=A'_\\ell(\\lambda_i)^T\\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})-\\lambda_i]+\\mathbb{E}_{q(Z|\\lambda)}[A_\\ell(\\lambda_i)]+const\\\\&=A'_\\ell(\\lambda_i)^T(\\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})]-\\lambda_i)+A_\\ell(\\lambda_i)+const\\end{aligned}$$\n",
    "这里用了一个trick:\n",
    "$\\mathbb{E}_{q(Z|\\lambda)}[T(Z_i)^T(\\eta(X,Z_{-i})-\\lambda_i)]=\\mathbb{E}_{q_i(Z_i|\\lambda_i)}[T(Z_i)]^T\\cdot \\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})-\\lambda_i]$  \n",
    "来说明一下这是为什么：\n",
    "$$\\begin{aligned}\\mathbb{E}_{q(Z|\\lambda)}[T(Z_i)^T(\\eta(X,Z_{-i})-\\lambda_i)]&=\\int_{Z_1,...,Z_M}T(Z_i)^T(\\eta(X,Z_{-i})-\\lambda_i)\\prod_{i=1}^M q_i(Z_i|\\lambda_i) dZ_i\\\\&=\\int_{Z_i}T(Z_i)^T q_i(Z_i|\\lambda_i) dZ_i\\cdot \\int_{Z_{-i}}(\\eta(X,Z_{-i})-\\lambda_i) \\prod_{j\\neq i} q_j(Z_j|\\lambda_j) dZ_j \\\\&=\\mathbb{E}_{q_i(Z_i|\\lambda_i)}[T(Z_i)]^T\\cdot \\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})-\\lambda_i]\\end{aligned}$$\n",
    "回归正题，接着我们对$\\lambda_i$求导：\n",
    "\n",
    "$$\\frac{\\partial \\mathcal{L}}{\\partial \\lambda_i}=A_\\ell''(\\lambda_i)(\\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})]-\\lambda_i)-A'_\\ell(\\lambda_i)+A'_\\ell(\\lambda_i)=0$$\n",
    "一般来说$A_\\ell''(\\lambda_i)\\neq 0$，于是我们有\n",
    "$$\\lambda^*_i=\\mathbb{E}_{q_{-i}(Z_{-i}|\\lambda_{-i})}[\\eta(X,Z_{-i})]$$\n",
    "有了这个更新式，只要我们遍历$\\lambda_i$固定其他参数，就可以迭代地对ELBO进行优化，获得后验概率分布的估计。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}