生成学习算法（Generative Learning algorithms）

看了一下斯坦福大学公开课：机器学习教程（吴恩达教授），记录了一些笔记，写出来以便以后有用到。笔记如有误，还望告知。
本系列其它笔记：
线性回归（Linear Regression）
分类和逻辑回归（Classification and logistic regression）
广义线性模型（Generalized Linear Models）
生成学习算法（Generative Learning algorithms）

生成学习算法（Generative Learning algorithms）

之前我们学习的算法$p(y|x;\theta )$给定x的y的条件分布，我们称之为判别学习算法（discriminative learning algorithms）；现在我们学习相反的算法$p(x|y)(p(y))$,称之为生成学习算法（generative
learning algorithms）。

使用贝叶斯定理，我们可以得到给定x后y的分布：
$$\begin{gathered} p(y|x)=\frac{p(x|y)p(y)}{p(x)} \\ p(x)=p(x|y=1)p(y=1)+p(x|y=0)p(y=0) \end{gathered}$$

1 高斯判别分析（Gaussian discriminant analysis）

1.1 多元高斯分布（多元正态分布）

假设输入特征$x\in {\mathbb{R}}^n$,且是连续的；p(x|y)满足高斯分布。

假设z符合多元高斯分布$z\backsim \mathcal{N}(\mu ,\Sigma )$
$$p(z)=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))$$

1.2 高斯判别分析模型（The Gaussian Discriminant Analysis model）

$$\begin{gathered} y\backsim Bernoulli(\varphi ) \\ x|y=0\backsim \mathcal{N}({\mu }_0,\Sigma ) \\ x|y=1\backsim \mathcal{N}({\mu }_1,\Sigma ) \\ p(y)={\varphi }^y(1-\varphi {)}^{1-y} \\ p(x|y=0)=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x-{\mu }_0{)}^T{\Sigma }^{-1}(x-{\mu }_0)) \\ p(x|y=1)=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }^{-1}(x-{\mu }_1)) \end{gathered}$$

$$\begin{gathered} \ell (\varphi ,{\mu }_0,{\mu }_1,\Sigma )=\log \prod _{i=1}^{m}p(x^{(i)},y^{(i)};\varphi ,{\mu }_0,{\mu }_1,\Sigma ) \\ =\log \prod _{i=1}^{m}p(x^{(i)}|y^{(i)};{\mu }_0,{\mu }_1,\Sigma )\cdot p(y^{(i)};\varphi )\to jointLikelihood \\ =\sum _{i=1}^m(\log p(x^{(i)}|y^{(i)};{\mu }_0,{\mu }_1,\Sigma )+\log p(y^{(i)};\varphi )) \end{gathered}$$

$$\begin{gathered} \frac{\partial }{\partial \varphi }\ell (\varphi ,{\mu }_0,{\mu }_1,\Sigma )=\frac{\partial }{\partial \varphi }\sum _{i=1}^m(\log p(x^{(i)}|y^{(i)};{\mu }_0,{\mu }_1,\Sigma )+\log p(y^{(i)};\varphi )) \\ =\sum _{i=1}^m\frac{\partial }{\partial \varphi }\log p(y^{(i)};\varphi ) \\ =\sum _{i=1}^m\frac{\partial }{\partial \varphi }(y^{(i)}\log (\varphi )+(1-y(i))\log (1-\varphi )) \\ =\sum _{i=1}^m(\frac{y^{(i)}-\varphi }{\varphi (1-\varphi )}) \\ 令\frac{\partial }{\partial \varphi }\ell (\varphi ,{\mu }_0,{\mu }_1,\Sigma )=0\Rightarrow \varphi =\frac{{\sum }_{i=1}^m{y}^{(i)}}{m}=\frac{1}{m}\sum _{i=1}^{m}1\{y^{(i)}=1\} \end{gathered}$$

logistic回归中
$$\ell (\theta )=\log \prod _{i=1}^{m}p(y^{(i)}|x^{(i)};\theta )\to conditionalLikelihood$$
最大化$\ell$，得出下面结果：
$$\begin{gathered} \varphi =\frac{1}{m}\sum _{i=1}^{m}1\{y^{(i)}=1\} \\ {\mu }_0=\frac{{\sum }_{i=1}^{m}1\{y^{(i)}=0\}{x}^{(i)}\to 标签为0的所有样本x^{(i)}求和}{{\sum }_{i=1}^{m}1\{y^{(i)}=0\}\to 标签为0的样本数目} \\ {\mu }_1=\frac{{\sum }_{i=1}^{m}1\{y^{(i)}=1\}{x}^{(i)}}{{\sum }_{i=1}^{m}1\{y^{(i)}=1\}} \\ \Sigma =\frac{1}{m}\sum _{i=1}^m(x^{(i)}-{\mu }_{y^{(i)}})(x^{(i)}-{\mu }_{y^{(i)}}{)}^T \end{gathered}$$
得到$\varphi ,{\mu }_0,{\mu }_1,\Sigma$之后，我们需要预测给定x的情况下最可能的y
$$\begin{gathered} {\mathrm{argmax}}_{y}P(y|x)={\mathrm{argmax}}_y\frac{P(x|y)P(y)}{P(x)} \\ ={\mathrm{argmax}}_{y}P(x|y)P(y)\to P(x)独立于y，所以P(x)值不会变 \end{gathered}$$

$$\begin{gathered} {\mathrm{argmax}}_y表达式\to 表达式中最大的y的值 \\ min(x-5{)}^2=0\to {\mathrm{argmin}}_x(x-5{)}^2=5 \end{gathered}$$

2 朴素贝叶斯（Naive Bayes）

假定给定y,$x_i$是条件独立的：
$$\begin{gathered} p(x_1,x_2,\dots ,x_{50000}|y)=p(x_1|y)p(x_2|y,x_1)p(x_3|y,x_1,x_2)\dots p(x_{50000}|y,x_1,x_2,\dots ,x_{49999}) \\ =p(x_1|y)p(x_2|y)p(x_3|y)\dots p(x_{50000}|y) \\ =\prod _{i=1}^{50000}p(x_i|y) \end{gathered}$$
模型参数：
$$\begin{gathered} {\varphi }_{j|y=1}=p(x_j=1|y=1) \\ {\varphi }_{j|y=0}=p(x_j=1|y=0) \\ {\varphi }_y=p(y=1) \end{gathered}$$
joint likelihood:
$$\ell ({\varphi }_y,{\varphi }_{j|y=0},{\varphi }_{j|y=1})=\prod _{i=1}^{m}p(x_i,y_i)$$
参数推导过程
①、
$$\begin{gathered} \mathcal{L}({\varphi }_y,{\varphi }_{j|y=0},{\varphi }_{j|y=1})=\prod _{i=1}^{m}p(x^{(i)},y^{(i)}) \\ =\prod _{i=1}^m\{\prod _{j=1}^{n}p(x_j^{(i)}|y^{(i)};{\varphi }_{j|y=0},{\varphi }_{j|y=1})\}p(y^{(i)};{\varphi }_y) \end{gathered}$$
②、
$$\begin{gathered} \ell ({\varphi }_y,{\varphi }_{j|y=0},{\varphi }_{j|y=1})=\log \mathcal{L}({\varphi }_y,{\varphi }_{j|y=0},{\varphi }_{j|y=1}) \\ =\log \prod _{i=1}^m\{\prod _{j=1}^{n}p(x_j^{(i)}|y^{(i)};{\varphi }_{j|y=0},{\varphi }_{j|y=1})\}p(y^{(i)};{\varphi }_y) \\ =\sum _{i=1}^m\{\log (\prod _{j=1}^{n}p(x_j^{(i)}|y^{(i)};{\varphi }_{j|y=0},{\varphi }_{j|y=1}))+\log (p(y^{(i)};{\varphi }_y))\} \end{gathered}$$
③、
$$\begin{gathered} \frac{\partial }{\partial {\varphi }_y}\ell ({\varphi }_y,{\varphi }_{j|y=0},{\varphi }_{j|y=1})=\sum _{i=1}^m\frac{\partial }{\partial {\varphi }_y}\log (p(y^{(i)};{\varphi }_y) \\ =\sum _{i=1}^m\frac{\partial }{\partial {\varphi }_y}\log ({\varphi }_y^{1\{y^{(i)}=1\}}(1-{\varphi }_y{)}^{(1-1\{y^{(i)}=1\})} \\ =\sum _{i=1}^m\frac{\partial }{\partial {\varphi }_y}\{(1\{y^{(i)}=1\}\log {\varphi }_y)+(1-1\{y^{(i)}=1\})\log (1-{\varphi }_y)\} \\ =\sum _{i=1}^m\frac{1\{y^{(i)}=1\}-{\varphi }_y}{{\varphi }_y(1-{\varphi }_y)} \\ 令\frac{\partial }{\partial {\varphi }_y}\ell ({\varphi }_y,{\varphi }_{j|y=0},{\varphi }_{j|y=1})=0\Rightarrow {\varphi }_y=\frac{{\sum }_{i=1}^{m}1\{y^{(i)}=1\}}{m} \end{gathered}$$
④、

最大化$\ell$，得出下面结果：

2.1 拉普拉斯平滑（Laplace smoothing）

2.2 文本分类的事件模型（Event models for text classi cation）

多项式事件模型（Multinomial Event Model）

对于第i个训练样本邮件，特征向量$x^{(i)}=(x_1^{(i)},x_2^{(i)},\dots ,x_{ni}^{(i)}),ni=邮件中词的数量$,特征向量的每个元素$x_j=\{1,2,\dots ,50000\}字典中的一个索引$。

生成模型$p(x,y)=\{{\prod }_{i=1}^{n}p(x_i,y)\}p(y)$

模型参数
$$\begin{gathered} {\varphi }_{k|y=1}=p(x_j=k|y=1) \\ {\varphi }_{k|y=0}=p(x_j=k|y=0) \\ {\varphi }_y=p(y=1) \end{gathered}$$
极大似然参数推导过程
①、
$$\begin{gathered} \mathcal{L}({\varphi }_y,{\varphi }_{k|y=0},{\varphi }_{k|y=1})=\prod _{i=1}^{m}p(x^{(i)},y^{(i)}) \\ =\prod _{i=1}^m\{\prod _{j=1}^{n_i}p(x_j^{(i)}|y^{(i)};{\varphi }_{k|y=0},{\varphi }_{k|y=1})\}p(y^{(i)};{\varphi }_y) \end{gathered}$$
②、
$$\begin{gathered} \ell ({\varphi }_y,{\varphi }_{k|y=0},{\varphi }_{k|y=1})=\log \mathcal{L}({\varphi }_y,{\varphi }_{k|y=0},{\varphi }_{k|y=1}) \\ =\log \prod _{i=1}^m\{\prod _{j=1}^{n_i}p(x_j^{(i)}|y^{(i)};{\varphi }_{k|y=0},{\varphi }_{k|y=1})\}p(y^{(i)};{\varphi }_y) \\ =\sum _{i=1}^m\{\log (\prod _{j=1}^{n_i}p(x_j^{(i)}|y^{(i)};{\varphi }_{k|y=0},{\varphi }_{k|y=1}))+\log (p(y^{(i)};{\varphi }_y))\} \end{gathered}$$
③、
$$\begin{gathered} \frac{\partial }{\partial {\varphi }_y}\ell ({\varphi }_y,{\varphi }_{k|y=0},{\varphi }_{k|y=1})=\sum _{i=1}^m\frac{\partial }{\partial {\varphi }_y}\log (p(y^{(i)};{\varphi }_y) \\ =\sum _{i=1}^m\frac{\partial }{\partial {\varphi }_y}\log ({\varphi }_y^{1\{y^{(i)}=1\}}(1-{\varphi }_y{)}^{(1-1\{y^{(i)}=1\})} \\ =\sum _{i=1}^m\frac{\partial }{\partial {\varphi }_y}\{(1\{y^{(i)}=1\}\log {\varphi }_y)+(1-1\{y^{(i)}=1\})\log (1-{\varphi }_y)\} \\ =\sum _{i=1}^m\frac{1\{y^{(i)}=1\}-{\varphi }_y}{{\varphi }_y(1-{\varphi }_y)} \\ 令\frac{\partial }{\partial {\varphi }_y}\ell ({\varphi }_y,{\varphi }_{i|y=0},{\varphi }_{i|y=1})=0\Rightarrow {\varphi }_y=\frac{{\sum }_{i=1}^{m}1\{y^{(i)}=1\}}{m} \end{gathered}$$
④、