缺失数据的极大似然估计：《Statistical Analysis with Missing Data》习题7.16

一、题目

a）极大似然估计

$X$为伯努利分布，并且$\text{Pr}(X=1)=1-\text{Pr}(X=0)=\pi$，并且在给定$X=j(j=0，1)$时，$Y$的分布为均值${\mu }_j$，方差${\sigma }^2$。

针对一份完整随机样本$(x_i，y_i)，i=1，...，n$，计算$(\pi ，{\mu }_0，{\mu }_1，{\sigma }^2)$的极大似然估计并计算$Y$的边际均值与方差。

b）缺失数据的极大似然估计

假设现在$X$是完整的观测，但$Y$有$n-r$个值缺失，请使用第七章的方法，计算$Y$的边际均值与方差。

c）从后验分布中生成参数

当先验分布表现为$p(\pi ，{\mu }_0，{\mu }_1，\text{log}{\sigma }^2)\propto {\pi }^{1/2}(1-\pi {)}^{1/2}$的形式，描述如何从参数为$(\pi ，{\mu }_0，{\mu }_1，{\sigma }^2)$的后验分布中抽出参数。

（注：前面的逗号均使用全角，后面公式中的逗号为半角，中文字中间的逗号为全角。）

二、解答

a）极大似然估计

写出联合密度函数，首先列出一个样本时的密度：
$$\begin{array}{rl}& f(x_i,y_i|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& f(x_i|\pi )\cdot f(y_i|x_i,{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& {\pi }^{x_i}\cdot (1-\pi {)}^{1-x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_1{)}^2}{2{\sigma }^2}\}{)}^{x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}{)}^{1-x_i}\end{array}$$

$n$个样本的联合密度函数：
$$\begin{array}{rl}& f(X,Y|{\mu }_0,{\mu }_0,{\sigma }^2,\pi )\\ =& \prod _{i=1}^{n}f(x_i|\pi )\cdot f(y_i|x_i,{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& \prod _{i=1}^n{\pi }^{x_i}\cdot (1-\pi {)}^{1-x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_1{)}^2}{2{\sigma }^2}\}{)}^{x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}{)}^{1-x_i}\end{array}$$

对数似然函数：
$$\begin{array}{rl}& \text{ln}f(X,Y|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& \text{ln}\prod _{i=1}^{n}f(x_i|\pi )\cdot f(y_i|x_i,{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& \text{ln}\prod _{i=1}^n{\pi }^{x_i}\cdot (1-\pi {)}^{1-x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_1{)}^2}{2{\sigma }^2}\}{)}^{x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}{)}^{1-x_i}\\ =& \sum _{i=1}^n\{x_i\text{ln}\pi +(1-x_i)\text{ln}(1-\pi )-\frac{x_i}{2}\text{ln}(2\pi {\sigma }^2)-\frac{x_i(y_i-{\mu }_1{)}^2}{2{\sigma }^2}-\frac{1-x_i}{2}\text{ln}(2\pi {\sigma }^2)-\frac{(1-x_i)(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}\\ =& \sum _{i=1}^n{x}_i\text{ln}\pi +(n-\sum _{i=1}^n{x}_i)\text{ln}(1-\pi )-\frac{n}{2}\text{ln}(2\pi {\sigma }^2)-\sum _{i=1}^n\frac{x_i(y_i-{\mu }_1{)}^2}{2{\sigma }^2}-\sum _{i=1}^n\frac{(1-x_i)(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}\end{array}$$

对上式求偏导，使其等于$0$即可得极大似然估计
$$\begin{array}{rl}\frac{\partial \text{ln}f(X,Y|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )}{\partial {\mu }_1}=& \frac{\partial \text{ln}f(X,Y|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )}{\partial {\mu }_0}\\ =& \frac{\partial \text{ln}f(X,Y|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )}{\partial {\sigma }^2}\\ =& \frac{\partial \text{ln}f(X,Y|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )}{\partial \pi }\\ =& 0\end{array}$$

可解得：
$$\begin{array}{rl}\hat{\pi }=& \frac{{\sum }_{i=1}^n{x}_i}{n}\\ \widehat{{\mu }_0}=& \frac{{\sum }_{i=1}^n(1-x_i)y_i}{{\sum }_{i=1}^n(1-x_i)}\\ \widehat{{\mu }_1}=& \frac{{\sum }_{i=1}^n{x}_i{y}_i}{{\sum }_{i=1}^n{x}_i}\\ \widehat{{\sigma }^2}=& \frac{{\sum }_{i=1}^n{y}_i^2}{n}-\frac{[{\sum }_{i=1}^n(1-x_i)y_i{]}^2}{n{\sum }_{i=1}^n(1-x_i)}-\frac{({\sum }_{i=1}^n{x}_i{y}_i{)}^2}{n{\sum }_{i=1}^n{x}_i}\end{array}$$

由于$\widehat{{\sigma }^2}$的计算相对麻烦，这里将其详细的计算过程写出：
$$\begin{array}{rl}\frac{\partial \text{ln}f(X,Y|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )}{\partial {\sigma }^2}=& 0\\ -\frac{n}{2}\cdot -\frac{1}{2\pi \widehat{{\sigma }^2}}\cdot 2\pi +\frac{{\sum }_{i=1}^n{x}_i(y_i-\widehat{{\mu }_1}{)}^2}{2(\widehat{{\sigma }^2}{)}^2}+\frac{{\sum }_{i=1}^n(1-x_i)(y_i-\widehat{{\mu }_0}{)}^2}{2(\widehat{{\sigma }^2}{)}^2}\}=& 0\\ \sum _{i=1}^n{x}_i(y_i-\frac{{\sum }_{i=1}^n{x}_i{y}_i}{{\sum }_{i=1}^n{x}_i}{)}^2+\sum _{i=1}^n(1-x_i)(y_i-\frac{{\sum }_{i=1}^n(1-x_i)y_i}{{\sum }_{i=1}^n(1-x_i)}{)}^2=& n\widehat{{\sigma }^2}\\ \frac{{\sum }_{i=1}^n{y}_i^2}{n}-\frac{[{\sum }_{i=1}^n(1-x_i)y_i{]}^2}{n{\sum }_{i=1}^n(1-x_i)}-\frac{({\sum }_{i=1}^n{x}_i{y}_i{)}^2}{n{\sum }_{i=1}^n{x}_i}=& \widehat{{\sigma }^2}\end{array}$$

均值与方差为：
将随机变量$X$求和掉，可求得$Y$的边际分布：
$$Y=(1-\pi )Y_0+\pi Y_1$$
其中：
$$\begin{array}{rl}{Y}_0\sim & N({\mu }_0,{\sigma }^2)\\ Y_1\sim & N({\mu }_1,{\sigma }^2)\end{array}$$

对其求期望与方差：
$$EY=(1-\pi ){\mu }_0+\pi {\mu }_1$$
$$Var(Y)=(1-\pi {)}^2{\sigma }^2+{\pi }^2{\sigma }^2$$
则$Y$边际均值的估计为：
$$(1-\hat{\pi })\widehat{{\mu }_0}+\hat{\pi }\widehat{{\mu }_1}$$
边际方差的估计为：
$$(1-\hat{\pi }{)}^2\widehat{{\sigma }^2}+{\hat{\pi }}^2\widehat{{\sigma }^2}$$
将前面的估计得到的参数带入即可。

b）带缺失数据的极大似然估计

联合密度函数：
$$\begin{array}{rl}& f(X,Y_{obs}|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& \prod _{i=1}^{r}f(x_i,y_i|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\cdot \prod _{i=r+1}^{n}f(x_i|\pi )\\ =& \prod _{i=1}^{r}f(x_i|\pi )f(y_i|x_i,{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\cdot \prod _{i=r+1}^{n}f(x_i|\pi )\\ =& \prod _{i=1}^{n}f(x_i|\pi )\cdot \prod _{i=1}^{r}f(y_i|x_i,{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ =& \prod _{i=1}^n{\pi }^{x_i}(1-\pi {)}^{1-x_i}\cdot \prod _{i=1}^r(\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_1{)}^2}{2{\sigma }^2}\}{)}^{x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}{)}^{1-x_i}\end{array}$$

同样对上式求对数与偏导，使其等于$0$，可解得：
$$\begin{array}{rl}\hat{\pi }=& \frac{{\sum }_{i=1}^n{x}_i}{n}\\ \widehat{{\mu }_1}=& \frac{{\sum }_{i=1}^r{x}_i{y}_i}{{\sum }_{i=1}^r{x}_i}\\ \widehat{{\mu }_0}=& \frac{{\sum }_{i=1}^r(1-x_i)y_i}{{\sum }_{i=1}^r(1-x_i)}\\ \widehat{{\sigma }^2}=& \frac{{\sum }_{i=1}^r{y}_i^2}{r}-\frac{[{\sum }_{i=1}^r(1-x_i)y_i{]}^2}{r{\sum }_{i=1}^r(1-x_i)}-\frac{({\sum }_{i=1}^r{x}_i{y}_i{)}^2}{r{\sum }_{i=1}^r{x}_i}\end{array}$$

同前面无缺失的情况，$Y$边际均值的估计为：
$$(1-\hat{\pi })\widehat{{\mu }_0}+\hat{\pi }\widehat{{\mu }_1}$$
边际方差的估计为：
$$(1-\hat{\pi }{)}^2\widehat{{\sigma }^2}+{\hat{\pi }}^2\widehat{{\sigma }^2}$$
同样将前面的带缺失数据的极大似然估计得到的参数带入即可。

c）从后验分布中生成参数

后验分布：
$$\begin{array}{rl}& f({\mu }_0,{\mu }_1,{\sigma }^2,\pi |X,Y_{obs})\\ \propto & f(X,Y_{obs}|{\mu }_0,{\mu }_1,{\sigma }^2,\pi )\cdot f({\mu }_0,{\mu }_1,{\sigma }^2,\pi )\\ \propto & \prod _{i=1}^n{\pi }^{\frac{1}{2}+x_i}(1-\pi {)}^{\frac{3}{2}-x_i}\cdot \prod _{i=1}^r(\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_1{)}^2}{2{\sigma }^2}\}{)}^{x_i}\cdot (\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_0{)}^2}{2{\sigma }^2}\}{)}^{1-x_i}\end{array}$$

我们可以类似书上的141页进行参数任意函数$g_d$的生成：

1. 从参数为$(\frac{n}{2}+{\sum }_{i=i}^n{x}_i+1,\frac{3n}{2}-{\sum }_{i=i}^n{x}_i+1)$的Beta分布中抽取$b_t$；从自由度为$2n-2$的卡方分布中抽取$x_t$；从标准正态分布中抽取相互独立的$z_0$与$z_1$；
2. 计算${\varphi }^{(d)}=({\pi }^{(d)},{\mu }_1^{(d)},{\mu }_0^{(d)},{{\sigma }^2}^{(d)})$：（其中$\widehat{{\sigma }^2},\widehat{{\mu }_1},\widehat{{\mu }_0}$均为上一问所求）
   $$\begin{array}{rl}{\pi }^{(d)}& =b_t\\ {{\sigma }^2}^{(d)}& =n\widehat{{\sigma }^2}/x_t\\ {\mu }_1^{(d)}& =\widehat{{\mu }_1}+z_0({{\sigma }^2}^{(d)}/\sum _{i=1}^r{x}_i{)}^{1/2}\\ {\mu }_0^{(d)}& =\widehat{{\mu }_0}+z_1({{\sigma }^2}^{(d)}/\sum _{i=1}^r(1-x_i){)}^{1/2}\end{array}$$