【机器学习】3 正则化

1 Addressing Overfitting

Reduce number of features	Regularization
Manually select which features to keep	Keep all the features but reduce magnitude / values of parameters ${\theta }_j$
Model selection algorithm（Automatically select）	Works well when we have a lot of features, each of which contributese a bit to predicting $y$
有可能会丢失信息

2 Regularization

2.1 Theory

Small values for parameters ${\theta }_0,{\theta }_1,\cdot \cdot \cdot ,{\theta }_n$

• “Simpler” hypothesis
• Less prone to overfitting

2.2 Model

• $$\underset{\theta }{\mathrm{minimize}}J(\theta )$$
  $$J(\theta )=\frac{1}{2m}\left[\sum _{i=1}^m(h_{\theta }{(x^{(i)})-y^{(i)})}^2+\lambda \sum _{j=1}^n{{\theta }_j}^2\right]$$

$\lambda$：Regularization Parameter
$\lambda {\sum }_{j=1}^n{{\theta }_j}^2$：to shrink every single parameter ${\theta }_j$
【注意】从${\theta }_1$开始（其实从${\theta }_0$开始影响不大）

2.3 $\lambda$ 设置值太大可能导致的问题

• Algorithm works fine; setting $\lambda$ to be very large can not hurt it
• Algorithm fails to eliminate overfitting
• Algorithm results in underfitting.（Fails to fit even training data well）
• Gradient descent will fail to converge

3 Regularized Linear Regession

3.1 Gradient Descent

• repeat until convergence{
    $$\begin{array}{rl}{\theta }_0& :={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)}\\ {\theta }_j& :={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}+\frac{\lambda }{m}{\theta }_j\right]\\ 变式：{\theta }_j& :={\theta }_j(1-\alpha \frac{\lambda }{m})-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}\end{array}$$
  （$j=1,2,3,\cdot \cdot \cdot ,n$）
  }

3.2 Normal Equation

$$\theta ={\left(X^{T}X+\lambda {\left[\begin{array}{ccccc}0& & & & \\ & 1& & & \\ & & 1& & \\ & & & \ddots & \\ & & & & 1\end{array}\right]}_{(n+1)\times (n+1)}\right)}^{-1}{X}^{T}y$$

• 只要$\lambda ＞0$，就不会non-invertibility

4 Regularized Logistic Regession

4.1 Gradient Descent

• $$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\text{log}{h}_{\theta }(x^{(i)})+(1-y^{(i)})\text{log}(1-h_{\theta }(x^{(i)})\right]+\frac{\lambda }{2m}\sum _{j=1}^n{{\theta }_j}^2$$
• repeat until convergence{
    $$\begin{array}{rl}{\theta }_0& :={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})\cdot x_0^{(i)})\\ {\theta }_j& :={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}-\frac{\lambda }{m}{\theta }_j\right]\end{array}$$
  （$j=1,2,3,\cdot \cdot \cdot ,n$）
  (simultaneously update all ${\theta }_j$）
  }

import numpy as np
def costReg(theta, X, y, learningRate):
	theta = np.matrix(theta)
	X = np.matrix(X)
	y = np.matrix(y)
	first = np.multiply(-y, np.log(sigmoid(X*theta.T)))
	second = np.multiply((1-y),np.log(1-sigmoid(X*theta*T)))
	reg = (learningRate / (2 * len(X))* np.sum(np.power(theta[:,1:theta.shape[1]],2))
	return np.sum(first - second) / (len(X)) + reg

4.2 Advanced optimization

Octave代码
function [jval, gradient] = costFunction(theta)
	jVal = [...code to compute J(theta)...];
	gradient = [... code to compute derivative of J(theta)...];
end
options = optimset('GradObj', 'on', 'MaxIter', '100');
initialTheta = zeros(2,1);

[optTheta, functionVal, exitFlag] = fminunc(@costFunction, initialTheta, options);

5 Reference

吴恩达 机器学习 coursera machine learning
黄海广 机器学习笔记