学习使用tf.contrib.learn框架开发机器学习程序

最近在学习用TensorFlow开发CNN的时候，看到官方的一篇教程Creating Estimators in tf.contrib.learn，介绍了使用封装好的一些工具开发训练程序的方法，觉得很不错，在这里记录一下学习的心得。
tf.contrib下面的东西很多，google自己没有称之为框架，但我感觉借助于tf.contrib.learn.Estimator，也等于是定义了一个框架，因为好多训练程序的流程是差不多的。

文档中建议的流程如下:

• Instantiate an Estimator 创建Estimator
• Construct a custom model function 自定义模型函数
• Configure a neural network using tf.contrib.layers 配置用到的Layers
• Choose an appropriate loss function from tf.contrib.losses 定义计算loss的方法
• Define a training op for your model 定义训练（即收敛loss）方法
• Generate and return predictions （我怎么觉得计算prediction应该放到loss前面）

创建Estimator

系统中已经有一些现成的，在learn.estimator目录下，比如：

• LinearClassifier: Constructs a linear classification model.
• LinearRegressor: Constructs a linear regression model.
• DNNClassifier: Construct a neural network classification model.
• DNNRegressor: Construct a neural network regressions model.

看样子应该是可以直接就用了，但是暂时还没有去研究。

自定义Estimator

---

nn = tf.contrib.learn.Estimator(model_fn=model_fn, model_dir=None, config=None, params=model_params, feature_engineering_fn=None)

• model_fn: 模型函数，具体后面再讨论
• model_dir: 指定log和参数保存文件的位置
• config: 待研究
• params: 传递给model_fn的自定义参数，dict形式
• * feature_engineering_fn*: 待研究

原文的例子：

# Learning rate for the model
LEARNING_RATE = 0.001

# Set model params
model_params = {"learning_rate": LEARNING_RATE}

nn = tf.contrib.learn.Estimator(
    model_fn=model_fn, params=model_params)

定义模型函数(model_fn)

---

model_fn的参数是：

def model_fn(features, targets, mode, params):
   # Logic to do the following:
   # 1. Configure the model via TensorFlow operations
   # 2. Define the loss function for training/evaluation
   # 3. Define the training operation/optimizer
   # 4. Generate predictions
   return predictions, loss, train_op

• features: 训练数据，也就是x，格式没有统一定义，取决于算法的需要；由调用fit(), evaluate(), or predict()的时候传递进来；
• targets: 训练数据的接口，也就是y；fit，evaluate的时候会传递；
• mode： 用来区分fit/evaluate/predict
  
  • tf.contrib.learn.ModeKeys.TRAIN
  • tf.contrib.learn.ModeKeys.EVAL
  • tf.contrib.learn.ModeKeys.INFER
• params: 自定义参数，没有可以不传

返回结果是一个三元组：

• predictions (required in INFER and EVAL modes)
  predictions = {"results": tensor_of_predictions}
  In INFER mode, the dict that you return from model_fn will then be returned by predict(), so you can construct it in the format in which you’d like to consume it. 也就是说这个prediction dict的格式没有要求，根据需要来定义。
  In EVAL mode, the dict is used by metric functions to compute metrics. Any MetricSpec objects passed to the metrics argument of evaluate() must have a prediction_key that matches the key name of the corresponding predictions in predictions. 看这个意思，是说evaluate的时候，要指定要用predictions中的哪个key来做比较(是不是这样，待验证)

• loss (required in EVAL and TRAIN mode). 这个就是通常意义的loss；

• train_op (required only in TRAIN mode). An Op that runs one step of training.

定义Hidden Layers

对于卷积神经网络来说，一般就是卷积层和池化层，tf.contrib.layers下有一些定义好的类：

• tf.contrib.layers.convolution2d
• tf.contrib.layers.max_pool2d
• tf.contrib.layers.avg_pool2d
• ….

定义全连接层

系统有一些预定义的类：

• relu(inputs, num_outputs). Create a layer of num_outputs nodes fully connected to the previous layer inputs with a ReLu activation function (tf.nn.relu):
  hidden_layer = tf.contrib.layers.relu(inputs=input_layer, num_outputs=10)

• relu6(inputs, num_outputs). Create a layer of num_outputs nodes fully connected to the previous layer hidden_layer with a ReLu 6 activation function (tf.nn.relu6):
  second_hidden_layer = tf.contrib.layers.relu6(inputs=hidden_layer, num_outputs=20)

• linear(inputs, num_outputs). Create a layer of num_outputs nodes fully connected to the previous layer second_hidden_layer with no activation function, just a linear transformation:
  output_layer = tf.contrib.layers.linear(inputs=second_hidden_layer, num_outputs=3)

当然还可以通过定制fully_connected的参数来定义自己需要的全连接层：

output_layer = tf.contrib.layers.fully_connected(inputs=second_hidden_layer, num_outputs=10, activation_fn=tf.sigmoid)


定义loss函数

tf.contrib.losses 下包含一些预定义的loss, 比如：

• absolute_difference(predictions, targets). Calculates loss using the absolute-difference formula (also known as L1 loss).
• log_loss(predictions, targets). Calculates loss using the logistic loss forumula (typically used in logistic regression).
• mean_squared_error(predictions, targets). Calculates loss using the mean squared error (MSE; also known as L2 loss).

定义train_op

一般按这样来定义：

train_op = tf.contrib.layers.optimize_loss(
loss=loss,
global_step=tf.contrib.framework.get_global_step(),
learning_rate=params["learning_rate"],
optimizer="SGD")


• loss: 就是前面定义的loss函数
• global_step: An integer Variable representing the step counter to increment for each model training run. Can easily be created/incremented in TensorFlow via the get_global_step() function (具体作用待研究)
• learning_rate: 应该就是类似于梯度下降的步长之类的意思，比如0.01

• optimizer: 优化方法，指定在tf.contrib.layers.optimizers中定义的方法，比如

  • SGD. Implementation of gradient descent (tf.train.GradientDescentOptimizer) 梯度下降
  • Adagrad. Implementation of the AdaGrad optimization algorithm (tf.train.AdagradOptimizer)
  • Adam. Implementation of the Adam optimization algorithm (tf.train.AdamOptimizer)
  • Ftrl. Implementation of the FTRL-Proximal (“Follow The (Proximally) Regularized Leader”) algorithm (tf.train.FtrlOptimizer)
  • Momentum. Implementation of stochastic gradient descent with momentum (tf.train.MomentumOptimizer)

  • RMSProp. Implementation of the RMSprop algorithm (tf.train.RMSPropOptimizer)

    除了名称之外，还可以有多种方法指定optimizer，参见文档。

综合以上的内容，一个完整的model_fn的例子：

def model_fn(features, targets, mode, params):
“”“Model function for Estimator.”“”

# Connect the first hidden layer to input layer
# (features) with relu activation
first_hidden_layer = tf.contrib.layers.relu(features, 10)

#Connect the second hidden layer to first hidden layer with relu
second_hidden_layer = tf.contrib.layers.relu(first_hidden_layer, 10)

# Connect the output layer to second hidden layer (no activation fn)
output_layer = tf.contrib.layers.linear(second_hidden_layer, 1)

# Reshape output layer to 1-dim Tensor to return predictions
predictions = tf.reshape(output_layer, [-1])
predictions_dict = {“ages”: predictions}

# Calculate loss using mean squared error
loss = tf.contrib.losses.mean_squared_error(predictions, targets)

train_op = tf.contrib.layers.optimize_loss(
loss=loss,
global_step=tf.contrib.framework.get_global_step(),
learning_rate=params[“learning_rate”],
optimizer=”SGD”)

return predictions_dict, loss, train_op

训练，评估和预测

# Fit
nn.fit(x=training_set.data, y=training_set.target, steps=5000)

# Score accuracy
ev = nn.evaluate(x=test_set.data, y=test_set.target, steps=1)
loss_score = ev["loss"]
print("Loss: %s" % loss_score) 

# Print out predictions
predictions = nn.predict(x=prediction_set.data,
                         as_iterable=True)
for i, p in enumerate(predictions):
  print("Prediction %s: %s" % (i + 1, p["ages"]))