[Machinie Learning] 吴恩达机器学习课程笔记——Week3

Machine Learning by Andrew Ng

吴恩达机器学习课程学习笔记——Week 3
本人学习笔记汇总 合订本
✓ 课程网址 standford machine learning
参考资源

• 课程笔记
• python版作业

学习提纲

• Classification and Regression
• Logistic Regression Model
• Multi-class Classification
• Solving the Problem of Overfitting

Classification and Regression

1.Classification

label 0 denotes the negative class (the absence of sth)
label 1 denotes the positive class(the presence of sth)
but it is rather arbitrary to decide which label denotes the negative/positive class

---

Linear Regression is not a good idea as the values to predict take on a small number of discrete values and linear regression would exceed those values.

---

Logistic Regression is a classification algorithm 分类算法
(not a regression algorithm as its name may indicate)

---

2.Hypothesis Representation
we want our classifier
$0\le h_{\theta }(x)\le 1$

we turn the linear regression function
$h_{\theta }={\theta }^{T}x$
into
$h_{\theta }=g({\theta }^{T}x)$
where g is
$g(z)=\frac{1}{1+e^{-z}}$
then we get
$$h_{\theta }=\frac{1}{1+e^{-{\theta }^{T}x}}$$
g is called sigmoid function or logistic function。

Sigmoid函数的性质：

g asymptotes at 0 as z goes to minus infinity, g asymptotes at 1 as z goes to infinity

---

The look of the sigmoid function 函数曲线

---

malignant 恶性 benign 良性

Interpretation of Hypothesis Output 解读假设函数的输出
The probability that y = 1, given x, parameterized by $\theta$ 在$\theta$参数下，给定x，y=1的概率

---

3.Decision Boundary 决策边界

---

The Decision Boudary

---

The boundary is decided by parameters, not the training set

---

with more higher order polynomial terms, we can get more complex decision boundaries
通过构造高阶多项式函数，我们可以得到更加复杂的决策边界。

---

Logistic Regression Model

1.Cost Function
= optimization objective 优化目标

Given a training set, how to choose $\theta$

---

comma 逗号 ,

Cost function of linear regression

---

if we directly use the cost function of linear regression, it turn out to be a non-convex function. 不能直接套用线性回归的损失函数，因为用于分类问题的话，它是非凸的。

---

so, we have to find a new const function to make J convex

$J(\theta )=\frac{1}{m}{\Sigma }_{i=1}^{m}Cost(h_{\theta }(x^i,y))$

---

---

---

损失函数的性质：

---

2.Simplified Cost Function and Gradient Descent

We can compress the cost function’s two conditional cases in to one case
$Cost(h\theta (x),y)=-ylog(h\theta (x))-(1-y)log(1-h\theta (x))$

The full const function
$J(\theta )=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\log (h_{\theta }(x^{(i)}))+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))$

A vectorized implementation is
$h=g(X\theta )$
$J(\theta )=\frac{1}{m}\cdot (-y^{T}log(h)-(1-y{)}^{T}log(1-h))$

---

apply the template of gradient descent and take the the derivative of J

---

The look is identical to linear regression except that the definition of $h_{\theta }(x)$ is changed

---

A vectorized implementation

---

3.Advanced Optimization
There are some other optimization algorithm

---

Octave part of optimization, skip

Multi-class Classification

1.Multi-class Classification: 1 vs. all

---

Summary
use k classifiers to solve k-class problems

---

Solving the Problem of Overfitting

1.The Problem of Overfitting

过拟合问题：fit perfect on the training examples but bot good on the testing examples
left plot: underfitting
right plot: overfitting

---

Address overfitting

• 减少特征
• 正则（下讲）
  

---

2.Cost Function
${\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3+{\theta }_4{x}_4$

suppose we penalize and make θ3 and θ4 smaller, then we actually force to model to ‘simplify’ itself

---

Regularization
by convention it starts from θ1 (but it makes little difference if it starts from θ0)

---

$\lambda$ is called regularization parameter 正则参数
if $\lambda$ is set to an extremely large value, then all $\theta$s will be 0. “underfit”

---

3.Regularized Linear Regression

---

Normal Equation 中添加正则项

the dimension is (n + 1) * (n + 1)

when m < n, $X^{T}X$ is not invertible. But after adding the term $\lambda$ * L, it becomes invertible!

4.Regularized Logistic Regression
We can regularize logistic regression in a similar way that we regularize linear regression.

---

Recall that our cost function for logistic regression was:

$J(\theta )=-m_1{\sum }_{i=1}^m[y^{i}log(h_{\theta }(x^i))+(1-y^i)log(1-h_{\theta }(x^i))]$

regularize the equation

$J(\theta )=-\frac{1}{m}{\sum }_{i=1}^m[y^{i}log(h_{\theta }(x^i))+(1-y^i)log(1-h_{\theta }(x^i))]+\frac{\lambda }{2m}{\sum }_{j=1}^n{\theta }_j^2$

---

Advanced optimization