吴恩达机器学习笔记（五）正则化Regularization

正则化（regularization）

过拟合问题（overfitting)

• Underfitting（欠拟合）–>high bias(高偏差)
• Overfitting（过拟合）–>high variance(高方差)
• Overfitting:If we have too many features, the learned hypothesis
  may fit the training set very well , but fail to generalize to new examples (predict prices on new examples).模型泛化能力差
  addressing overfitting
  options:
  1)reduce number of features（减少特征数量）
  –Manually select which features to keep
  –Model selection algorithm(模型选择算法)
  2)regularization（正则化）
  –keep all the features but reduce magnitude/values(但减少参数的大小/值) of parameters.
  –Works well when we have a lot of features,each of which contributes a bit to predicting y.

代价函数Cost function（正则化代价函数）

the effect of penalizing two of the parameter values being large.
加入惩罚增大了两个参数带来的效果。
对 ${\theta }_j$ 加入惩罚项：
In regularized linear regression,we choose $\theta$ to minimize.
Regularization线性回归代价函数：
$$J(\theta )=\frac{1}{2m}\left[\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2+\lambda \sum _{j=1}^m{\theta }_j^2\right]$$
目标： $$\underset{\theta }{\min }J(\theta )$$
$\lambda$:regularization parameter（正则参数）

• λ很大的结果？

线性回归的正则化（Regularized linear regression）

梯度下降（Gradient descent)
梯度下降算法：
repeat：$${\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)}$$
$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\left[\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)}+\lambda {\theta }_j\right]$$
等价于：
$${\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)}$$
$${\theta }_j:={\theta }_j(1-\alpha \frac{1}{m})-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$$

正规方程（Normal equation）

正规方程：
假设： $m\le n(examples\le features)$
$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$
if λ>0，
$$\theta ={\left(X^{T}X+\lambda \underset{(n+1)\times (n+1)}{\left[\begin{array}{ccccc}0& & & & \\ & 1& & & \\ & & 1& & \\ & & & \ddots & \\ & & & & 1\end{array}\right]}\right)}^{-1}{X}^{T}y$$
只要λ>0，那么括号内的矩阵一定不是奇异矩阵，也就是可逆的。

逻辑回归的正则化（Regularization logistic regression）

逻辑回归代价函数：
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m(y^{(i)}\log (h_{\theta }(x^{(i)}))+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)})))+\frac{\lambda }{2m}\sum _{j=1}^m{\theta }_j^2$$