Deeplearning

Numpy

Deep Learning

Basic

  • 神经网络:
algorithm1
input1
output
input2
input3
input4
algorithm2
  • 监督学习:1个x对应1个y;

  • Sigmoid : 激活函数
    s i g m o i d = 1 1 + e − x sigmoid=\frac{1}{1+e^{-x}} sigmoid=1+ex1

  • ReLU : 线性整流函数;

Deeplearning_第1张图片

Logistic Regression

–>binary classification / x–>y 0 1

some sign

( x , y ) , x ∈ R n x , y ∈ 0 , 1 M = m t r a i n m t e s t = t e s t M : ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) . . . , ( x ( m ) , y ( m ) ) X = [ x ( 1 ) x ( 2 ) ⋯ x ( m ) ] ← n x × m y ^ = P ( y = 1 ∣ x ) y ^ = σ ( w t x + b ) w ∈ R n x b ∈ R σ ( z ) = 1 1 + e − z (3) (x,y) , x\in{\mathbb{R}^{n_{x}}},y\in{0,1}\\\\ M=m_{train}\quad m_{test}=test\\\\ M:{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)})...,(x^{(m)},y^{(m)})}\\\\ X = \left[ \begin{matrix} x^{(1)} & x^{(2)} &\cdots & x^{(m)} \end{matrix} \right] \tag{3}\leftarrow n^{x}\times m\\\\ \hat{y}=P(y=1\mid x)\quad\hat{y}=\sigma(w^tx+b)\qquad w\in \mathbb{R}^{n_x} \quad b\in \mathbb{R}\\ \sigma (z)=\frac{1}{1+e^{-z}} (x,y),xRnx,y0,1M=mtrainmtest=testM:(x(1),y(1)),(x(2),y(2))...,(x(m),y(m))X=[x(1)x(2)x(m)]nx×my^=P(y=1x)y^=σ(wtx+b)wRnxbRσ(z)=1+ez1(3)

Loss function

单个样本
L o s s   f u n c t i o n : L ( y ^ , y ) = 1 2 ( y ^ − y ) 2 L ( y ^ , y ) = − ( y log ⁡ ( y ^ ) + ( 1 − y ) log ⁡ ( 1 − y ^ ) ) y = 1 : L ( y ^ , y ) = − log ⁡ y ^ log ⁡ y ^ ← l a r g e r y ^ ← l a r g e r y = 0 : L ( y ^ , y ) = − log ⁡ ( 1 − y ^ ) log ⁡ ( 1 − y ^ ) ← l a r g e r y ^ ← s m a l l e r Loss\:function:\mathcal{L}(\hat{y},y)=\frac{1}{2}(\hat{y}-y)^2\\\\ \mathcal{L}(\hat{y},y)=-(y\log(\hat{y})+(1-y)\log(1-\hat{y}))\\\\ y=1:\mathcal{L}(\hat{y},y)=-\log\hat{y}\quad \log\hat{y}\leftarrow larger\quad\hat{y}\leftarrow larger\\ y=0:\mathcal{L}(\hat{y},y)=-\log(1-\hat{y})\quad \log(1-\hat{y})\leftarrow larger\quad\hat{y}\leftarrow smaller\\\\ Lossfunction:L(y^,y)=21(y^y)2L(y^,y)=(ylog(y^)+(1y)log(1y^))y=1:L(y^,y)=logy^logy^largery^largery=0:L(y^,y)=log(1y^)log(1y^)largery^smaller

cost function

J ( w , b ) = 1 m ∑ i = 1 m L ( y ^ ( i ) , y ( i ) ) \mathcal{J}(w,b)=\frac{1}{m}\sum_{i=1}^{m}\mathcal{L}(\hat{y}^{(i)},y^{(i)}) J(w,b)=m1i=1mL(y^(i),y(i))

Gradient Descent

find w,b that minimiaze J(w,b) ;

Repeat:
w : = w − α ∂ J ( w , b ) ∂ w ( d w ) b : = b − α ∂ J ( w , b ) ∂ b ( d b ) w:=w-\alpha \frac{\partial\mathcal{J}(w,b)}{\partial w}(dw)\\ b:=b-\alpha \frac{\partial\mathcal{J}(w,b)}{\partial b}(db) w:=wαwJ(w,b)(dw)b:=bαbJ(w,b)(db)

Computation Grapha

example:
J = 3 ( a + b c ) J=3(a+bc) J=3(a+bc)

v=a+u
a
b
c
u=bc
J=3v

one example gradient descent computer grapha:

recap:
z = w T x + b y ^ = a = σ ( z ) = 1 1 + e − z L ( a , y ) = − ( t log ⁡ ( a ) + ( 1 − y ) log ⁡ ( 1 − a ) ) z=w^Tx+b\\ \hat{y}=a=\sigma(z)=\frac{1}{1+e^{-z}} \\ \mathcal{L}(a,y)=-(t\log(a)+(1-y)\log(1-a)) z=wTx+by^=a=σ(z)=1+ez1L(a,y)=(tlog(a)+(1y)log(1a))
The grapha:

′ d a ′ = d L ( a , y ) d a = − y a + 1 − y 1 − a ′ d z ′ = d L ( a , y ) d z = d L d a ⋅ d a d z = a − y ′ d w 1 ′ = x 1 ⋅ d z        . . . w 1 : = w 1 − α d w 1      . . . 'da'=\frac{d\mathcal{L}(a,y)}{da}=-\frac{y}{a}+\frac{1-y}{1-a}\\ 'dz'=\frac{d\mathcal{L}(a,y)}{dz}=\frac{d\mathcal{L}}{da}\cdot\frac{da}{dz}=a-y\\ 'dw_1'=x_1\cdot dz\;\;\; ... \\w_1:=w_1-\alpha dw_1\;\;... da=dadL(a,y)=ay+1a1ydz=dzdL(a,y)=dadLdzda=aydw1=x1dz...w1:=w1αdw1...
m example gradient descent computer grapha:

recap:
J ( w , b ) = 1 m ∑ i = 1 m L ( a ( i ) , y ( 1 ) ) \mathcal{J}(w,b)=\frac{1}{m}\sum_{i=1}^m\mathcal{L}(a^{(i)},y^{(1)}) J(w,b)=m1i=1mL(a(i),y(1))
The grapha: (two iterate)
∂ ∂ w 1 J ( w , b ) = 1 m ∑ i = 1 m ∂ ∂ w 1 L ( a ( i ) , y ( 1 ) ) F o r i = 1 t o m : { a ( i ) = σ ( w T x ( i ) + b ) J + = − [ y ( i ) log ⁡ a i + ( 1 − y ( i ) log ⁡ ( 1 − a ( i ) ) ) ] d z ( i ) = a ( i ) − y ( i ) d w 1 + = x 1 ( i ) d z ( i ) d w 2 + = x 2 ( i ) d z ( i ) d b + = d z ( i ) } J / = m ; d w 1 / = m ; d w 2 / = m ; d b / = m d w 1 = ∂ J ∂ w 1 w 1 = w 1 − α d w 1 \frac{\partial}{\partial w_1}\mathcal{J}(w,b)=\frac{1}{m}\sum_{i=1}^m\frac{\partial}{\partial w_1}\mathcal{L}(a^{(i)},y^{(1)})\\\\ For \quad i=1 \quad to \quad m:\{\\ a^{(i)}=\sigma (w^Tx^{(i)}+b)\\ \mathcal{J}+=-[y^{(i)}\log a^{i}+(1-y^{(i)}\log(1-a^{(i)}))] \\ dz^{(i)}=a^{(i)}-y^{(i)}\\ dw_1+=x_1^{(i)}dz^{(i)}\\ dw_2+=x_2^{(i)}dz^{(i)}\\ db+=dz^{(i)}\}\\ \mathcal{J}/=m;dw_1/=m;dw_2/=m;db/=m\\ dw_1=\frac{\partial\mathcal{J}}{\partial w_1}\\ w_1=w_1-\alpha dw_1 w1J(w,b)=m1i=1mw1L(a(i),y(1))Fori=1tom:{a(i)=σ(wTx(i)+b)J+=[y(i)logai+(1y(i)log(1a(i)))]dz(i)=a(i)y(i)dw1+=x1(i)dz(i)dw2+=x2(i)dz(i)db+=dz(i)}J/=m;dw1/=m;dw2/=m;db/=mdw1=w1Jw1=w1αdw1

Vectorization

vectorized:
z = n p . d o t ( w , x ) + b z=np.dot(w,x)+b z=np.dot(w,x)+b
logistic regression derivatives:

change:
d w 1 = 0 , d w 2 = 0 → d w = n p . z e r o s ( ( n x , 1 ) ) { d w 1 + = x 1 ( i ) d z ( i ) d w 2 + = x 2 ( i ) d z ( i ) → d w + = x ( i ) d z ( i ) Z = (    z ( 1 ) z ( 2 ) . . . z ( m )    ) = w T X + b A = σ ( Z ) d z = A − Y = (    a ( 1 ) − y ( 1 ) z ( 2 ) − y ( 2 ) . . . z ( m ) − y ( m )    ) d b = 1 m ∑ i = 1 m d z ( i ) = 1 m n p . s u m ( d z ) d w = 1 m X d z T = 1 m (    x ( 1 ) ⋅ d z ( 1 ) x ( 2 ) ⋅ d z ( 2 ) . . . x ( m ) ⋅ d z ( m )    ) dw_1=0,dw_2=0\rightarrow dw=np.zeros((n_x,1))\\ \begin{cases}dw_1+=x_1^{(i)}dz^{(i)}\\ dw_2+=x_2^{(i)}dz^{(i)}\end{cases}\rightarrow dw+=x^{(i)}dz^{(i)}\\\\ Z=\left(\;\begin{matrix} z^{(1)} & z^{(2)} &... &z^{(m)}\end{matrix}\;\right)=w^TX+b\\ A=\sigma(Z)\\\\ dz=A-Y=\left(\;\begin{matrix} a^{(1)}-y^{(1)} & z^{(2)}-y^{(2)} &... &z^{(m)}-y^{(m)}\end{matrix}\;\right)\\ db=\frac{1}{m}\sum_{i=1}^mdz^{(i)}=\frac{1}{m}np.sum(dz)\\ dw=\frac{1}{m}Xdz^T=\frac{1}{m}\left(\;\begin{matrix} x^{(1)}\cdot dz^{(1)} & x^{(2)}\cdot dz^{(2)} &... &x^{(m)}\cdot dz^{(m)}\end{matrix}\;\right) dw1=0,dw2=0dw=np.zeros((nx,1)){dw1+=x1(i)dz(i)dw2+=x2(i)dz(i)dw+=x(i)dz(i)Z=(z(1)z(2)...z(m))=wTX+bA=σ(Z)dz=AY=(a(1)y(1)z(2)y(2)...z(m)y(m))db=m1i=1mdz(i)=m1np.sum(dz)dw=m1XdzT=m1(x(1)dz(1)x(2)dz(2)...x(m)dz(m))
Implementing:
Z = w T X + b = n p . d o t ( w T , X ) + b A = σ ( Z ) d Z = A − Y d w = 1 m X d Z T d b = 1 m n p . s u m ( d Z ) w : = w − α d w b : = b − α d b Z=w^TX+b=np.dot(w^T,X)+b\\ A=\sigma(Z)\\ dZ=A-Y\\ dw=\frac{1}{m}XdZ^T\\ db=\frac{1}{m}np.sum(dZ)\\ w:=w-\alpha dw\\ b:=b-\alpha db Z=wTX+b=np.dot(wT,X)+bA=σ(Z)dZ=AYdw=m1XdZTdb=m1np.sum(dZ)w:=wαdwb:=bαdb
broadcasting in python:
n p . d o t ( w T , X ) + b np.dot(w^T,X)+b np.dot(wT,X)+b
A note on Numpy
a = n p . r a n d o m . r a n d n ( 5 ) / / w r o n g → a = a . r e s h a p e ( 5 , 1 ) a s s e r t ( a . s h a p e = = ( 5 , 1 ) ) a = n p . r a n d o m . r a n d n ( 5 , 1 ) → c o l u m    v e c t o r a=np.random.randn(5) //wrong\rightarrow a=a.reshape(5,1)\\ assert(a.shape==(5,1))\\ a=np.random.randn(5,1)\rightarrow colum\;vector a=np.random.randn(5)//wronga=a.reshape(5,1)assert(a.shape==(5,1))a=np.random.randn(5,1)columvector
:

$$

$$

你可能感兴趣的:(机器学习,逻辑回归,人工智能)