Deeplearning

Numpy

Deep Learning

Basic

• 神经网络：

algorithm1

input1

output

input2

input3

input4

algorithm2

• 监督学习：1个x对应1个y；

• Sigmoid : 激活函数
  $$sigmoid=\frac{1}{1+e^{-x}}$$

• ReLU : 线性整流函数；

Logistic Regression

–>binary classification / x–>y 0 1

some sign

$$\begin{gathered} (x,y),x\in {\mathbb{R}}^{n_x},y\in 0,1 \\ M=m_{train}{m}_{test}=test \\ M:(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)})...,(x^{(m)},y^{(m)}) \\ X=\left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]\leftarrow n^x\times m \\ \hat{y}=P(y=1\mid x)\hat{y}=\sigma (w^{t}x+b)w\in {\mathbb{R}}^{n_x}b\in \mathbb{R} \\ \sigma (z)=\frac{1}{1+e^{-z}}\text{\hspace{1em}(3)} \end{gathered}$$

Loss function

单个样本
$$\begin{gathered} Lossfunction:\mathcal{L}(\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2 \\ \mathcal{L}(\hat{y},y)=-(y\log (\hat{y})+(1-y)\log (1-\hat{y})) \\ y=1:\mathcal{L}(\hat{y},y)=-\log \hat{y}\log \hat{y}\leftarrow larger\hat{y}\leftarrow larger \\ y=0:\mathcal{L}(\hat{y},y)=-\log (1-\hat{y})\log (1-\hat{y})\leftarrow larger\hat{y}\leftarrow smaller \end{gathered}$$

cost function

$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)})$$

Gradient Descent

find w,b that minimiaze J(w,b) ;

Repeat:
$$\begin{gathered} w:=w-\alpha \frac{\partial \mathcal{J}(w,b)}{\partial w}(dw) \\ b:=b-\alpha \frac{\partial \mathcal{J}(w,b)}{\partial b}(db) \end{gathered}$$

Computation Grapha

example:
$$J=3(a+bc)$$

v=a+u

a

b

c

u=bc

J=3v

one example gradient descent computer grapha:

recap:
$$\begin{gathered} z=w^{T}x+b \\ \hat{y}=a=\sigma (z)=\frac{1}{1+e^{-z}} \\ \mathcal{L}(a,y)=-(t\log (a)+(1-y)\log (1-a)) \end{gathered}$$
The grapha:

$$\begin{gathered} {}^{\prime }d{a}^{\prime }=\frac{d\mathcal{L}(a,y)}{da}=-\frac{y}{a}+\frac{1-y}{1-a}{ \\ }^{\prime }d{z}^{\prime }=\frac{d\mathcal{L}(a,y)}{dz}=\frac{d\mathcal{L}}{da}\cdot \frac{da}{dz}=a-y{ \\ }^{\prime }d{w}_1^{\prime }=x_1\cdot dz... \\ {w}_1:=w_1-\alpha d{w}_1... \end{gathered}$$
m example gradient descent computer grapha:

recap:
$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}(a^{(i)},y^{(1)})$$
The grapha: (two iterate)
$$\begin{gathered} \frac{\partial }{\partial w_1}\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\frac{\partial }{\partial w_1}\mathcal{L}(a^{(i)},y^{(1)}) \\ Fori=1tom:\{ \\ {a}^{(i)}=\sigma (w^T{x}^{(i)}+b) \\ \mathcal{J}+=-[y^{(i)}\log a^i+(1-y^{(i)}\log (1-a^{(i)}))] \\ d{z}^{(i)}=a^{(i)}-y^{(i)} \\ d{w}_1+=x_1^{(i)}d{z}^{(i)} \\ d{w}_2+=x_2^{(i)}d{z}^{(i)} \\ db+=d{z}^{(i)}\} \\ \mathcal{J}/=m;d{w}_1/=m;d{w}_2/=m;db/=m \\ d{w}_1=\frac{\partial \mathcal{J}}{\partial w_1} \\ {w}_1=w_1-\alpha d{w}_1 \end{gathered}$$

Vectorization

vectorized:
$$z=np.dot(w,x)+b$$
logistic regression derivatives:

change:
$$\begin{gathered} d{w}_1=0,d{w}_2=0\to dw=np.zeros((n_x,1)) \\ \{\begin{array}{l}d{w}_1+=x_1^{(i)}d{z}^{(i)}\\ d{w}_2+=x_2^{(i)}d{z}^{(i)}\end{array}\to dw+=x^{(i)}d{z}^{(i)} \\ Z=\left(\begin{array}{cccc}{z}^{(1)}& z^{(2)}& ...& z^{(m)}\end{array}\right)=w^{T}X+b \\ A=\sigma (Z) \\ dz=A-Y=\left(\begin{array}{cccc}{a}^{(1)}-y^{(1)}& z^{(2)}-y^{(2)}& ...& z^{(m)}-y^{(m)}\end{array}\right) \\ db=\frac{1}{m}\sum _{i=1}^{m}d{z}^{(i)}=\frac{1}{m}np.sum(dz) \\ dw=\frac{1}{m}Xd{z}^T=\frac{1}{m}\left(\begin{array}{cccc}{x}^{(1)}\cdot d{z}^{(1)}& x^{(2)}\cdot d{z}^{(2)}& ...& x^{(m)}\cdot d{z}^{(m)}\end{array}\right) \end{gathered}$$
Implementing:
$$\begin{gathered} Z=w^{T}X+b=np.dot(w^T,X)+b \\ A=\sigma (Z) \\ dZ=A-Y \\ dw=\frac{1}{m}Xd{Z}^T \\ db=\frac{1}{m}np.sum(dZ) \\ w:=w-\alpha dw \\ b:=b-\alpha db \end{gathered}$$
broadcasting in python:
$$np.dot(w^T,X)+b$$
A note on Numpy
$$\begin{gathered} a=np.random.randn(5)//wrong\to a=a.reshape(5,1) \\ assert(a.shape==(5,1)) \\ a=np.random.randn(5,1)\to columvector \end{gathered}$$
:

$$

$$