logistics regression

逻辑回归

回归：输入输出均为连续变量；
分类：输出为离散变量；
联合概率计算最大似然函数，即调整当前超参数，使之符合训练数据的概率最大。

评价回归函数

设置超参数，描述联合概率：
$$\begin{array}{rl}& g(w^{T}x)=\frac{1}{1+e^{-z}}=\frac{1}{1+e^{w^{T}x}}\\ & \{\begin{array}{ll}P(y=1)& =g(w^{T}x)\\ P(y=0)& =1-g(w^{T}x)\end{array}\\ \Rightarrow & P(True)=(g(w,x_i){)}^{y_i}\ast (1-g(w_i,xi){)}^{1-y_i}\\ \Rightarrow & L(w)=\prod _{i=1}^{m}P(True)\\ \Rightarrow & Loss(w)=-\frac{1}{m}L(w)\end{array}$$
其中，y是真实值。P表示当前超参数时，各情况概率，用以评价当前超参数。此时损失函数描述了变量w的变化规律。
推导似然函数 L 及损失函数：
$$\begin{array}{rl}{h}_{\theta }(x)& =g(\theta ;x)=\frac{1}{1+e^{{\theta }^{T}x}}\\ L(\theta )& =\prod _{i=1}^m(h_{\theta }(x_i){)}^{y_i}(1-h_{\theta }(x_i){)}^{1-y_i}\\ \Rightarrow logL(\theta )& =\sum _{i=1}^m{y}_{i}log(h_{\theta }(x_i))+(1-y_i)log(1-h_{\theta }(x_i))\\ \Rightarrow \frac{\delta }{{\delta }_{{\theta }_j}}logL(\theta )& =-\frac{1}{m}\sum _{i=1}^m(y_i\frac{1}{h_{\theta }(x_i)}\frac{\delta }{{\delta }_{\theta }}{h}_{\theta }(x_i)-(1-y_i)\frac{1}{1-h_{\theta }(x_i)}\frac{\delta }{{\delta }_{{\theta }_j}}{h}_{\theta }(x_i))\\ & =-\frac{1}{m}\sum _{i=1}^m[y_i\frac{1}{h_{\theta }(x_i)}-(1-y_i)\frac{1}{1-h_{\theta }(x_i)}]\frac{\delta }{{\delta }_{{\theta }_j}}{h}_{\theta }(x_i)\\ & =-\frac{1}{m}\sum _{i=1}^m[y_i\frac{1}{h_{\theta }(x_i)}-(1-y_i)\frac{1}{1-h_{\theta }(x_i)}]h_{\theta }(x_i)(1-h_{\theta }(x_i))\frac{\delta }{{\delta }_{{\theta }_j}}{\theta }^T{x}_i\\ & =-\frac{1}{m}\sum _{i=1}^m[y_i(1-h_{\theta }(x_i))-(1-y_i)h_{\theta }(x_i)]\frac{\delta }{{\delta }_{{\theta }_j}}{\theta }^T{x}_i\\ & =-\frac{1}{m}\sum _{i=1}^m[y_i(1-h_{\theta }(x_i))-(1-y_i)h_{\theta }(x_i)]{x_i}_j\\ & =\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i){x_i}_j\end{array}$$

更新超参数

上例中求得了针对变量的**偏导数**，实际变量变化时候，更新方向也要依据偏导数进行更新：
$${\theta }_j={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i){x_i}_j$$

多分类的softmax

其中的概率函数表示：
$$\begin{array}{rlr}{h}_{\theta }(x^{(i)})& =\left[\begin{array}{c}p(y^{(i)}=1|x^{(i)};\theta )\\ p(y^{(i)}=2|x^{(i)};\theta )\\ .\\ .p(y^{(i)}=k|x^{(i)};\theta );\end{array}\right]& =\frac{1}{{\sum }_{j=1}^k{e}^{{\theta }_j^T{x}^{(i)}}}\left[\begin{array}{c}{e}^{{\theta }_1^T{x}^{(i)}}\\ e^{{\theta }_2^T{x}^{(i)}}\\ .\\ e^{{\theta }_k^T{x}^{(i)}}\end{array}\right]\end{array}$$