Naive Bayes 朴素贝叶斯

Naive Bayesian 朴素贝叶斯

内容来自于CS229,浙江大学机器学习听课笔记以及百度百科，blog补充

---

We would use an example to show this algorithm, this examples is still using in practice now.

Spam email classifier
$X$ is 1/0 vector correspond to dictionary and each dimension represents a word. If a word shows up in a email, then its corresponding value equals to 1.
We assume that $X_i$ are independently and identically distribution(IID), although it is obvious that they are not for the meaning of email.
We choose top 10,000 common used words as the dictionary.
$P(x_1...x_{10000}|y)=P(x_1|y)P(x_2|x_1,y)...P(x_{10000}|x_{9999},x_{9998}...x_1,y)$
$P(x_1...x_{10000}|y)=\prod _{i=1}^{10000}P(x_i|y)$
Parameters:
${\varphi }_{j|y=1}=P(x_j=1|y=1)$
${\varphi }_{j|y=0}=P(x_j=1|y=0)$
${\varphi }_y=P(y=1)$
$y=1$ means the email is spam email.

Joint likelihood:
$L({\varphi }_y,{\varphi }_{j|y})=\prod _{i=1}^m(x^{(i)},y^{(i)};{\varphi }_y,{\varphi }_{j|y})$
MLE:
${\varphi }_y=\frac{\sum _{i=1}^{m}1\{y^{(i)}=1\}}{m}$
${\varphi }_{j|y=1}=\frac{\sum _{i=1}^{m}1\{x^{(i)}=1,y^{(i)}=1\}}{\sum _{i=1}^{m}1\{y^{(i)}=1\}}$
${\varphi }_{j|y=0}=\frac{\sum _{i=1}^{m}1\{x^{(i)}=1,y^{(i)}=0\}}{\sum _{i=1}^{m}1\{y^{(i)}=0\}}$

Laplace moving
If a new word(not in the top 10000 common words) shows up in a email, $\sum _{i=1}^{m}1\{x^{(i)}=1,y^{(i)}=1\}=0$, then ${\varphi }_{j|j=0}=0$. This seems not robust, so we use Laplace moving to optimize this equation.
${\varphi }_{j|y=1}=\frac{\sum _{i=1}^{m}1\{x^{(i)}=1,y^{(i)}=1\}+1}{\sum _{i=1}^{m}1\{y^{(i)}=1\}+10000}$
${\varphi }_{j|y=0}=\frac{\sum _{i=1}^{m}1\{x^{(i)}=1,y^{(i)}=0\}+1}{\sum _{i=1}^{m}1\{y^{(i)}=0\}+10000}$