深度学习笔记

Deep Learning

Basic

• 神经网络：

algorithm1

input1

output

input2

input3

input4

algorithm2

• 监督学习：1个x对应1个y；

• Sigmoid : 激活函数
  $$sigmoid=\frac{1}{1+e^{-x}}$$

• ReLU : 线性整流函数；

## Logistic Regression

-->binary classification / x-->y 0 1

### some sign

$$
(x,y) , x\in{\mathbb{R}^{n_{x}}},y\in{0,1}\\\\
M=m_{train}\quad m_{test}=test\\\\
M:{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)})...,(x^{(m)},y^{(m)})}\\\\
X =
\left[ 
\begin{matrix} 
x^{(1)} & x^{(2)} &\cdots & x^{(m)} 
\end{matrix} \right] \leftarrow n^{x}\times m\\\\
\hat{y}=P(y=1\mid x)\quad\hat{y}=\sigma(w^tx+b)\qquad
w\in \mathbb{R}^{n_x} \quad b\in \mathbb{R}\\
\sigma (z)=\frac{1}{1+e^{-z}}
$$

### Loss function

单个样本
$$
Loss\:function:\mathcal{L}(\hat{y},y)=\frac{1}{2}(\hat{y}-y)^2\\\\
p(y\mid x)=\hat{y}^y(1-\hat y)^{(1-y)}\\
min\;cost\rightarrow max\;\log(y\mid x)\\
\mathcal{L}(\hat{y},y)=-(y\log(\hat{y})+(1-y)\log(1-\hat{y}))\\\\
y=1:\mathcal{L}(\hat{y},y)=-\log\hat{y}\quad \log\hat{y}\leftarrow larger\quad\hat{y}\leftarrow larger\\
y=0:\mathcal{L}(\hat{y},y)=-\log(1-\hat{y})\quad \log(1-\hat{y})\leftarrow larger\quad\hat{y}\leftarrow smaller\\\\
$$

### cost function 

$$
\mathcal{J}(w,b)=\frac{1}{m}\sum_{i=1}^{m}\mathcal{L}(\hat{y}^{(i)},y^{(i)})
$$

### Gradient Descent

find w,b that minimiaze J(w,b) ;

Repeat:
$$
w:=w-\alpha \frac{\partial\mathcal{J}(w,b)}{\partial w}(dw)\\
b:=b-\alpha \frac{\partial\mathcal{J}(w,b)}{\partial b}(db)
$$

### Computation Grapha

example:
$$
J=3(a+bc)
$$

```mermaid
graph LR
	C[v=a+u]
	A(a)
	B(b)
	D(c)
	E[u=bc]
	F[J=3v]
	
	A --> C
	B --> E
	D --> E
	E --> C
	C --> F

one example gradient descent computer grapha:

recap:
$$\begin{gathered} z=w^{T}x+b \\ \hat{y}=a=\sigma (z)=\frac{1}{1+e^{-z}} \\ \mathcal{L}(a,y)=-(y\log (a)+(1-y)\log (1-a)) \end{gathered}$$
The grapha:

$$\begin{gathered} {}^{\prime }d{a}^{\prime }=\frac{d\mathcal{L}(a,y)}{da}=-\frac{y}{a}+\frac{1-y}{1-a}{ \\ }^{\prime }d{z}^{\prime }=\frac{d\mathcal{L}(a,y)}{dz}=\frac{d\mathcal{L}}{da}\cdot \frac{da}{dz}=a-y{ \\ }^{\prime }d{w}_1^{\prime }=x_1\cdot dz... \\ {w}_1:=w_1-\alpha d{w}_1... \end{gathered}$$
m example gradient descent computer grapha:

recap:
$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}(a^{(i)},y^{(1)})$$
The grapha: (two iterate)
$$\begin{gathered} \frac{\partial }{\partial w_1}\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\frac{\partial }{\partial w_1}\mathcal{L}(a^{(i)},y^{(1)}) \\ Fori=1tom:\{ \\ {a}^{(i)}=\sigma (w^T{x}^{(i)}+b) \\ \mathcal{J}+=-[y^{(i)}\log a^i+(1-y^{(i)}\log (1-a^{(i)}))] \\ d{z}^{(i)}=a^{(i)}-y^{(i)} \\ d{w}_1+=x_1^{(i)}d{z}^{(i)} \\ d{w}_2+=x_2^{(i)}d{z}^{(i)} \\ db+=d{z}^{(i)}\} \\ \mathcal{J}/=m;d{w}_1/=m;d{w}_2/=m;db/=m \\ d{w}_1=\frac{\partial \mathcal{J}}{\partial w_1} \\ {w}_1=w_1-\alpha d{w}_1 \end{gathered}$$

Vectorization

vectorized

$$z=np.dot(w,x)+b$$
logistic regression derivatives:

change:
$$\begin{gathered} d{w}_1=0,d{w}_2=0\to dw=np.zeros((n_x,1)) \\ \{\begin{array}{l}d{w}_1+=x_1^{(i)}d{z}^{(i)}\\ d{w}_2+=x_2^{(i)}d{z}^{(i)}\end{array}\to dw+=x^{(i)}d{z}^{(i)} \\ Z=\left(\begin{array}{cccc}{z}^{(1)}& z^{(2)}& ...& z^{(m)}\end{array}\right)=w^{T}X+b \\ A=\sigma (Z) \\ dz=A-Y=\left(\begin{array}{cccc}{a}^{(1)}-y^{(1)}& z^{(2)}-y^{(2)}& ...& z^{(m)}-y^{(m)}\end{array}\right) \\ db=\frac{1}{m}\sum _{i=1}^{m}d{z}^{(i)}=\frac{1}{m}np.sum(dz) \\ dw=\frac{1}{m}Xd{z}^T=\frac{1}{m}\left(\begin{array}{cccc}{x}^{(1)}\cdot d{z}^{(1)}& x^{(2)}\cdot d{z}^{(2)}& ...& x^{(m)}\cdot d{z}^{(m)}\end{array}\right) \end{gathered}$$

Implementing:

$$\begin{gathered} Z=w^{T}X+b=np.dot(w^T,X)+b \\ A=\sigma (Z) \\ J=-\frac{1}{m}\sum _{i=1}^m(y^{(i)}\log (a^{(i)})+(1-y^{(i)})\log (1-a^{(i)})) \\ dZ=A-Y \\ dw=\frac{1}{m}Xd{Z}^T \\ db=\frac{1}{m}np.sum(dZ) \\ w:=w-\alpha dw \\ b:=b-\alpha db \end{gathered}$$

broadcasting

$$np.dot(w^T,X)+b$$
A note on numpy
$$\begin{gathered} a=np.random.randn(5)//wrong\to a=a.reshape(5,1) \\ assert(a.shape==(5,1)) \\ a=np.random.randn(5,1)\to columvector \end{gathered}$$

Shallow Neural Network

Representation

2 layer NN:
$$\begin{gathered} Inputlayer\to hidden\to layer\to outlayer \\ {a}^{[0]}\to a^{[1]}\to a^{[2]} \\ {z}^{[1]}=W^{[1]}{a}^{[0]}+b^{[1]} \\ {a}^{[1]}=\sigma (z^{[1]}) \\ {z}^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]} \\ {a}^{[2]}=\sigma (z^{[2]})=\hat{y} \end{gathered}$$

computing:

z i [ 1 ] = w i [ 1 ] T x + b i [ 1 ] a i [ 1 ] = σ ( z i [ 1 ] ) [ w 1 [ 1 ] T w 2 [ 1 ] T w 3 [ 1 ] T w 4 [ 1 ] T ] ⋅ [ x 1 x 2 x 3 ] + [ b 1 [ 1 ] b 2 [ 1 ] b 3 [ 1 ] b 4 [ 1 ] ] = [ z 1 [ 1 ] z 2 [ 1 ] z 3 [ 1 ] z 4 [ 1 ] ] z_i^{[1]}=w_i^{[1]T}x+b_i^{[1]}\\ a_i^{[1]}=\sigma(z_i^{[1]})\\ \left[ \begin{matrix} w_1^{[1]T}\\w_2^{[1]T}\\w_3^{[1]T}\\w_4^{[1]T} \end{matrix} \right] \cdot \left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right]+\left[ \begin{matrix} b_1^{[1]}\\b_2^{[1]}\\b_3^{[1]}\\b_4^{[1]} \end{matrix} \right]=\left[ \begin{matrix} z_1^{[1]}\\z_2^{[1]}\\z_3^{[1]}\\z_4^{[1]} \end{matrix} \right] zi[1]​=wi[1]T​x+bi[1]​ai[1]​=σ(zi[1]​) ​w1[1]T​w2[1]T​w3[1]T​w4[1]T​​ ​⋅ ​x1​x2​x3​​ ​+ ​b1[1]​b2[1]​b3[1]​b4[1]​​ ​= ​z1[1]​z2[1]​z3[1]​z4[1]​​ ​

Vectorize:

$$\begin{gathered} x^{(i)}\to a^{[2](i)}={\hat{y}}^{(i)} \\ {Z}^{[1]}=W^{[1]}X+b^{[1]} \\ {A}^{[1]}=\sigma (Z^{[1]}) \\ {Z}^{[2]}=W^{[2]}{A}^{[1]}+b^{[2]} \\ {A}^{[2]}=\sigma (Z^{[2]}) \\ {W}^{[1]}\cdot \left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]+b=\left[\begin{array}{cccc}{z}^{[1](1)}& z^{[1](2)}& \cdots & z^{[1](m)}\end{array}\right]=Z^{[1]} \end{gathered}$$

Activation functions

$$\begin{gathered} a=\frac{1}{1+e^{-z}},a^{\prime }=a(1-a) \\ a=\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},a\in (-1,1),a^{\prime }=1-a^2 \\ a=max(0,z) \\ a=max(0.01z,z) \end{gathered}$$

Gradient descent

computation

$$\begin{gathered} z^{[1]}=W^{[1]}x+b^{[1]}\to \\ {a}^{[1]}=\sigma (z^{[1]})\to \\ {z}^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]}\to \\ {a}^{[2]}=\sigma (z^{[2]})\to \\ \mathcal{L}(a^{[2]},y) \\ d{z}^{[2]}=a^{[2]}-y \\ d{w}^{[2]}=d{z}^{[2]}{a}^{[1]T} \\ d{b}^{[2]}=d{z}^{[2]} \\ d{z}^{[1]}=w^{[2]T}d{z}^{[2]}\ast a^{{}^{\prime }[1]} \\ d{w}^{[1]}=d{z}^{[1]}\cdot x^T \\ d{b}^{[1]}=d{z}^{[1]} \end{gathered}$$

dz^[1]的推导涉及到了矩阵求导

the dimension

$$\begin{gathered} x:(n_0,m)W^{[1]}:(n_1,n_0)\to \\ {a}^{[1]}:(n_1,m)W^{[2]:}:(n_2,n_1)\to \\ {a}^{[2]}:(n_2,m) \end{gathered}$$

vectorize

$$\begin{gathered} d{Z}^{[2]}=A^{[2]}-Y \\ d{W}^{[2]}=\frac{1}{m}d{Z}^{[2]}{A}^{[1]T} \\ d{b}^{[2]}=np.sum(d{Z}^{[2]},axis=1,keepdims=True) \\ d{Z}^{[1]}=W^{[2]T}d{Z}^{[2]}\ast A^{{}^{\prime }[1]} \\ d{W}^{[1]}=\frac{1}{m}d{Z}^{[1]}{X}^T \\ d{b}^{[1]}=\frac{1}{m}np.sum(dZ[1],axis=1,keepdims=True) \end{gathered}$$

Random Initialization

$$\begin{gathered} w^{[1]}=np.random.randn((2,2))\ast 0.01 \\ {b}^{[1]}=np.zero((2,1)) \end{gathered}$$

Deep neural network

notation

$$\begin{gathered} example:LlayerNN \\ {a}^{[l]}\to activationfunction \\ {w}^{[l]}\to weightsfor{z}^{[l]} \\ \hat{y}=a^{[L]} \end{gathered}$$

Forward propagation

$$\begin{gathered} forl=1,2,\mathrm{3..} \\ {z}^{[l]}=w^{[l]}{a}^{[l-1]}+b^{[l]} \\ cache{z}^{[l]},w^{[l]},b^{[l]} \\ {a}^{[l]}=g^{[l]}(z^{[l]}) \end{gathered}$$

Backward propagation

$$\begin{gathered} d{a}^{[l]}\to d{a}^{[l-1]}(d{z}^{[l]},d{w}^{[l]},d{b}^{[l]}) \\ d{z}^{[l]}=d{a}^{[l]}\ast g^{[l{]}^{\prime }}(z^{[l]})=w^{[l+1]}d{z}^{[l+1]}\ast g^{[l{]}^{\prime }}(z^{[l]}) \\ d{w}^{[l]}=d{z}^{[l]}\cdot a^{[l-1]T} \\ d{b}^{[l]}=d{z}^{[l]} \\ d{a}^{[l-1]}=w^{[l]T}\cdot d{z}^{[l]} \end{gathered}$$

matrix dimensions

$$\begin{gathered} dw,w^{[l]}:(n^{[l]},n^{[l-1]}) \\ db,b^{[l]}:(n^{[l]},1) \\ {Z}^{[l]},A^{[l]}:(n^{[l]},m) \end{gathered}$$

Improve NN

train/dev/test set

0.7/0/0.3 0.6.0.2.0.2 -> 100-10000

0.98/0.01/0.01 … -> big data

Bias/Variance

偏差度量的是单个模型的学习能力，而方差度量的是同一个模型在不同数据集上的稳定性。

high variance ->high dev set error

high bias ->high train set error

basic recipe

high bias -> bigger network / train longer / more advanced optimization algorithms / NN architectures

high variance -> more data / regularization / NN architecture

Regularization

Logistic Regression

$$\begin{gathered} L2regularization: \\ min\mathcal{J}(w,b)\to J(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)})+\frac{\lambda }{2m}\Vert w{\Vert }_2^2 \end{gathered}$$

Neural network

$$\begin{gathered} Frobeniusnorm \\ \Vert w^{[l]}{\Vert }_F^2=\sum _{i=1}^{n^{[l]}}\sum _{j=1}^{n^{[l-1]}}(w_{i,j}^{[l]}{)}^2 \\ Dropoutregularization: \\ d3=np.randm.rand(a3.shape.shape[0],a3.shape[1]<keep.prob) \\ a3=np.multiply(a3,d3) \\ a3/=keep.prob \end{gathered}$$

other ways

• early stopping
• data augmentation

Optimization problem

speed up the training of your neural network

Normalizing inputs

1. subtract mean

$$\begin{gathered} \mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)} \\ x:=x-\mu \end{gathered}$$

2. normalize variance

$$\begin{gathered} {\sigma }^2=\frac{1}{m}\sum _{i=1}^m(x^{(i)}{)}^2 \\ x/=\sigma \end{gathered}$$

vanishing/exploding gradients

$$\begin{gathered} y=w^{[l]}{w}^{[l-1]}...w^{[2]}{w}^{[1]}x \\ {w}^{[l]}>I\to (w^{[l]}{)}^L\to \infty \\ {w}^{[l]}<I\to (w^{[l]}{)}^L\to 0 \end{gathered}$$

weight initialize

$$\begin{gathered} var(w)=\frac{1}{n^{(l-1)}} \\ {w}^{[l]}=np.random.randn(shape)\ast np.sqrt(\frac{1}{n^{(l-1)}}) \end{gathered}$$

gradient check

Numerical approximation

$$\begin{gathered} f(\theta )={\theta }^3 \\ {f}^{\prime }(\theta )=\frac{f(\theta +\epsilon )-f(\theta -\epsilon )}{2\epsilon } \end{gathered}$$

grad check

$$\begin{gathered} d{\theta }_{approx}[i]=\frac{J({\theta }_1,...{\theta }_i+\epsilon ...)-J({\theta }_1,...{\theta }_i-\epsilon ...)}{2\epsilon }=d\theta [i] \\ check:\frac{\Vert d{\theta }_{approx}-d\theta {\Vert }_2}{\Vert d{\theta }_{approx}{\Vert }_2+\Vert d\theta {\Vert }_2}<10^{-7} \end{gathered}$$

Optimize algorithm

mini-bach gradient

$$\begin{gathered} [x^{(1)}...x^{(m)}]\to [x^{\{1\}}...x^{\{m/u\}}] \\ (anepoch:Forwardpropon{x}^{\{t\}}: \\ {z}^{[l]}=w^{[l]}{X}^{\{t\}}+b^{[l]} \\ {A}^{[l]}=g^{[l]}(z^{[l]}) \\ {J}^{\{t\}}=\frac{1}{1000}\sum _{i=1}^l\mathcal{L}({\hat{y}}^{(i)},y^{(i)})+\frac{\lambda }{2\ast size}\sum _l\Vert w^{[l]}{\Vert }_F^2 \\ Backwardprop \end{gathered}$$

mini-batch size

size = m -> Batch gradient descent <- small train set (<2000)

size = 1 -> stochastic gradient descent

typical mini-batch size (62,128,256…)

exponential weighted averages

$$\begin{gathered} v_{\theta }=0 \\ {\theta }_t\to v_{\theta }:=\beta v_{\theta -1}+(1-\beta ){\theta }_{\theta } \end{gathered}$$

Bias correction

$$\frac{1}{1-\beta }\to \frac{v_t}{1-{\beta }^t}$$

Momentum

$$\begin{gathered} V_{dw}=\beta V_{dw}+(1-\beta )dw \\ {V}_{db}=\beta V_{db}+(1-\beta )db \\ w:=w-\alpha V_{dw} \end{gathered}$$

RMSprop

$$\begin{gathered} S_{dw}={\beta }_2{S}_{dw}+(1-{\beta }_2)d{w}^2 \\ {S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2 \\ w:=w-\alpha \frac{dw}{{\sqrt{S}}_{dw}+\epsilon } \end{gathered}$$

Adam algorithm

$$\begin{gathered} V_{dw}=0,S_{dw}=0 \\ {V}_{dw}={\beta }_1{V}_{dw}+(1-{\beta }_1)dw \\ {V}_{db}={\beta }_1{V}_{db}+(1-{\beta }_1)db \\ {S}_{dw}={\beta }_2{S}_{dw}+(1-{\beta }_2)d{w}^2 \\ {S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2 \\ {V}_{dw}^{correct}=\frac{v_{dw}}{1-{\beta }_1^t} \\ {S}_{dw}^{correct}=\frac{s_{dw}}{1-{\beta }_2^t} \\ W:=W-\alpha \frac{V_{dw}^{correct}}{\sqrt{S_{dw}^{correct}}+\epsilon } \\ {\beta }_1:0.9,{\beta }_2:0.999 \end{gathered}$$

Learning rate decay

$$\begin{gathered} \alpha =\frac{1}{1+decayRate\ast epochNumber}{\alpha }_0 \\ \alpha =\frac{k}{\sqrt{epochNum}}{\alpha }_0 \end{gathered}$$

Local optima

Tuning process

hyperparameter search

• learning_rate -> beta / hidden units / mini-batch size

• Try random values

• Coarse to fine

• babysitting one model / training many models in parallel

Pick at random

appropriate scale for hyparameters

• learning rate

$$\begin{gathered} a=0.0001\to 1 \\ r=-4\ast np.random.rand() \\ \alpha =10^r \end{gathered}$$

• exponentially weighted averages

$$\begin{gathered} \beta =\mathrm{0.9...0.999} \\ 1-\beta =\mathrm{0.1...0.001} \\ r\in [-3,-1] \\ \beta =10^r \end{gathered}$$

Narmalizing activations

$$\begin{gathered} \mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)} \\ x:=x-\mu \\ {\sigma }^2=\frac{1}{m}\sum _{i=1}^m(x^{(i)}{)}^2 \\ x/=\sigma \end{gathered}$$

implementing Batch Norm

$$\begin{gathered} \mu =\frac{1}{m}\sum _{i=1}^m{z}^{(i)} \\ z:=z-\mu \\ {\sigma }^2=\frac{1}{m}\sum _{i=1}^m(z^{(i)}{)}^2 \\ {z}_{norm}^{(i)}/=\sqrt{{\sigma }^2+\epsilon } \\ \tilde{z}=\gamma z_{norm}^{(i)}+\beta (if\gamma =\sqrt{{\sigma }^2+\epsilon },\beta =\mu ) \end{gathered}$$

fit in NN

$$\begin{gathered} x\to z^{[i]}\to {\tilde{z}}^{[i]}\to a^{[i+1]}\to z^{[i+1]}\to {\tilde{z}}^{[i+1]}\to a^{[i+1]} \\ {\beta }^{[i]}={\beta }^{[l]}-\alpha d{\beta }^{[i]} \end{gathered}$$

at test time

exponential weighted averages

softmax regression

$$\begin{gathered} Softmaxlayeractivationfunction: \\ t=e^{(z^{[l]})} \\ {a}^{[l]}=\frac{e^{z^{[l]}}}{{\sum }_{i=1}^4{t}_i},a_i^{[l]}=\frac{t_i}{{\sum }_{i=1}^4{t}_i} \\ \mathcal{L}(\hat{y},y)=-\sum _{j=1}^4{y}_j\log {\hat{y}}_j \end{gathered}$$

Structuring ML project

Ml strategy

Single Numble Evaluation Metric

Precision : n% actually are …

Recall : n% was correctly recognized

F1 score : $ \frac{2}{\frac{1}{P}+\frac{1}{R}} $

optimizing and satisficing metric

cost = accuracy - 0.5 * running time

N matrics : 1 optimizing , (N-1) reach threshold