Machine Learning By Andrew Ng (5)

Notes on Machine Learning By Andrew Ng (5)

Click here to see previous note.

Neural Networks: Representation

Non-linear hypotheses

Non-linear classification

You may use polynomial features to find an ideal classifier, but when we have lots of features, it may comes to overfitting in the end.

Neurons and the brain

Model representation I

Neuron model: Logistic unit

Notation

$a_i^{(j)}=$ “activation” of unit $i$ in layer $j$.

$\Theta^{(j)} = $ matrix of weight controlling function mapping from layer $j$ to layer $j+1$.

$$\begin{gathered} a_1^{(2)}=g({\Theta }_{10}^{(1)}{x}_0+{\Theta }_{11}^{(1)}{x}_1+{\Theta }_{12}^{(1)}{x}_2+{\Theta }_{13}^{(1)}{x}_3) \\ {a}_2^{(2)}=g({\Theta }_{20}^{(1)}{x}_0+{\Theta }_{21}^{(1)}{x}_1+{\Theta }_{22}^{(1)}{x}_2+{\Theta }_{23}^{(1)}{x}_3) \\ {a}_3^{(2)}=g({\Theta }_{30}^{(1)}{x}_0+{\Theta }_{31}^{(1)}{x}_1+{\Theta }_{32}^{(1)}{x}_2+{\Theta }_{33}^{(1)}{x}_3) \\ {h}_{\Theta }(x)=a_1^{(3)}=g({\Theta }_{10}^{(2)}{a}_0^{(2)}+{\Theta }_{11}^{(2)}{a}_1^{(2)}+{\Theta }_{12}^{(2)}{a}_2^{(2)}+{\Theta }_{13}^{(2)}{a}_3^{(2)}) \end{gathered}$$
If network has $s_j$ units in layer $j$, $s_j$ units in layer $j+1$, then ${\Theta }^{(j)}$ will be of dimension $s_{j+1}\times (s_j+1)$.

Model representation II

Forward propagation（前向传播）: Vertorized implementation

Let ${\Theta }_{10}^{(1)}{x}_0+{\Theta }_{11}^{(1)}{x}_1+{\Theta }_{12}^{(1)}{x}_2+{\Theta }_{13}^{(1)}{x}_3=z_1^{(2)}$ and $a_1^{(2)}=g(z_1^{(2)})$.

Turn it to a vector!
$$\begin{gathered} \mathbf{x}=\left[\begin{array}{c}{x}_0\\ x_1\\ x_2\\ x_3\end{array}\right]{\mathbf{z}}^{(2)}=\left[\begin{array}{c}{z}_1^{(2)}\\ z_2^{(2)}\\ z_3^{(2)}\end{array}\right], \\ {z}^{(2)}={\Theta }^{(1)}x(x=a^{(1)}) \\ {a}^{(2)}=g(z^{(2)}). \end{gathered}$$
Add $a_0^{(2)}=1$, $z^{(3)}={\Theta }^{(2)}{a}^{(2)}$, $h_{\Theta }(x)=a^{(3)}=g(z^{(3)})$.

Examples and intuitions I

Examples and intuitions II

Multi-class classification

Multiple output units: One-vs-all

Training set: $(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),(x^{(m)},y^{(m)})$,

$y^{(i)}\in \left[\begin{array}{c}1\\ 0\\ 0\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\\ 0\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 0\\ 1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 0\\ 0\\ 1\end{array}\right]$.