TensorFlow 之 Automatic differentiation and gradient tape

Automatic differentiation and gradient tape

​ 之前我们介绍了Tensor 以及在其上的操作，下面我们介绍一下自动微分技术，—用来优化模型参数的关键。

​ tensorflow 提供了用于自动微分的API，来计算一个函数的导数。一种更接近数学的求导方法是：先写一个python函数，封装好对参数的运算。然后使用tf.contrib.eager.gradients_function 来创建一个函数计算上面封装好的函数的导函数（可指定对哪个参数求导）。同时，只要嵌套调用该函数，即可求高阶导。

Gradient tapes

什么是梯度带呢？对于每一个可导的TF函数操作，其都有对应的导函数。

对于上面提到的用户封装的一个函数，TF都会记录下这样一系列的操作，称之为tape

然后根据tape和每一个primitive操作（即上面提高的TF函数），就可以使用reverse mode differentiation

方法来计算用户封装的函数啦

tf.GradientTape（一点点臃肿，但是更详细）

有的时候，函数不是那么好封装，比如你需要对中间的某个变量进行求导。这个时候你可以使用tf.GradientTape context（python的上下文管理机制）

All computation inside the context of a tf.GradientTape is “recorded”.

Higher-order gradients

GradientTape 上下文管理器中的所有相关操作都会记录下来用于automatic differentiation.

If gradients are computed in that context, then the gradient computation is recorded as well.

As a result, the exact same API works for higher-order gradients as well.

详见代码

Code1

import tensorflow as tf 
import matplotlib.pyplot as plt
from math import pi

tf.enable_eager_execution()
tfe = tf.contrib.eager

def f(x):
    return tf.square(tf.sin(x))

print( f(pi/2).numpy() == 1.0 )
grad_f = tfe.gradients_function(f) # f的梯度函数
print(tf.abs(grad_f((pi/2,pi))).numpy())

# Higher-order gradients
def grad(f):  #相当于直接返回一个一阶梯度函数
    return lambda x : tfe.gradients_function(f)(x)[0]

x = tf.lin_space(-2*pi,2*pi,100) # 100 points between -2pi ~ 2pi

plt.plot(x, f(x), label = 'f')
plt.plot(x, grad(f)(x), label = 'first derivative')
plt.plot(x, grad(grad(f))(x), label = 'second derivative')
plt.plot(x, grad(grad(grad(f)))(x), label = 'third derivative')
plt.legend()
plt.show()

Code2

'''
Gradient tapes
'''
import tensorflow as tf 

tf.enable_eager_execution()
tfe = tf.contrib.eager # shorthand for some symbols


# x^y
def f(x,y): 
    output = 1
    # Must use range(int(y)) instead of range(y) in Python 3 when
    # using TensorFlow 1.10 and earlier. Can use range(y) in 1.11+
    for i in range(int(y)): # you can use for loop (#^.^#)
        output = tf.multiply(output, x)
    return output         


# d x^y / d x
def g(x,y):
    # Return the gradient of 'f' with respect to it's first parameter  default?
    return tfe.gradients_function(f)(x,y)[0]


print( f(3,2).numpy() )
print( g(3.0,2).numpy() )
print( f(3,3).numpy() )
print( g(3.0,3).numpy() )

Code3

'''
At times it may be inconvenient to encapsulate(封装) computation of 
interest into a function. For example, if you want the gradient
of the output with respect to intermediate(中间的) values computed in
the function. In such cases, the slightly more verbose but 
explicit(明确的)  **tf.GradientTape**  context is useful. All computation 
inside the context(上下文) of a **tf.GradientTape** is "recorded".
'''
import tensorflow as tf 

tf.enable_eager_execution()
tfe = tf.contrib.eager # shorthand for some symbols

x= tf.ones((2, 2))

with tf.GradientTape(persistent = True) as t: # persistent 持久的
    t.watch(x)
    y = tf.reduce_sum(x) # 就是求和的意思(降维)
    z = tf.multiply(y,y)


#use the same tape to compute the derivative of z with 
# respect to the intermediate value y
dz_dy = t.gradient(z, y) # 对y求导
print(dz_dy.numpy())


# Derivative of z with respect to the original input tensor x
dz_dx = t.gradient(z, x) # 对x求导
print(dz_dx.numpy())


'''
higher-order gradients

Operations inside of the GradientTape context manager
 are recorded for automatic differentiation. 

 If gradients are computed in that context, 
 then the gradient computation is recorded as well. 

  As a result, the exact same API works for higher-order 
  gradients as well. For example:
'''

x = tf.Variable(1.0)  # Convert the Python 1.0 to a Tensor object
with tf.GradientTape() as t:
    with tf.GradientTape() as t2: 
        y = x*x*x
        # Compute the gradient inside the 't' context manager
        # which means the gradient computation is differentiable as well.
    dy_dx = t2.gradient(y,x)
d2y_dx2 = t.gradient(dy_dx, x)
print(dy_dx.numpy(), d2y_dx2.numpy())