Deploying Keras model on Tensorflow Serving

keras训练了个二分类的模型。需求是把keras模型跑到 tensorflow serving上 （TensorFlow Serving 系统用于在生产环境中运行模型）

keras模型转 tensorflow模型

我把 keras模型转tensorflow serving模型所使用的方法如下：

1、要拿到算法训练好的keras模型文件（一个HDF5文件）

该文件应该包含：

• 模型的结构，以便重构该模型
• 模型的权重
• 训练配置（损失函数，优化器等）
• 优化器的状态，以便于从上次训练中断的地方开始

2、编写 keras模型转tensorflow serving模型的代码

import pandas as pd
import os
import tensorflow as tf

tf.logging.set_verbosity(tf.logging.INFO)
...
def build_model():
    ############
    ...
    return model


def save_model_for_production(model, version, path='prod_models'):
    tf.keras.backend.set_learning_phase(1)
    if not os.path.exists(path):
        os.mkdir(path)
    export_path = os.path.join(
        tf.compat.as_bytes(path),
        tf.compat.as_bytes(version))
    builder = tf.saved_model.builder.SavedModelBuilder(export_path)

    model_input = tf.saved_model.utils.build_tensor_info(model.input)
    model_output = tf.saved_model.utils.build_tensor_info(model.output)

    prediction_signature = (
        tf.saved_model.signature_def_utils.build_signature_def(
            inputs={'inputs': model_input},
            outputs={'output': model_output},
            method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))

    with tf.keras.backend.get_session() as sess:
        builder.add_meta_graph_and_variables(
            sess=sess, tags=[tf.saved_model.tag_constants.SERVING],
            signature_def_map={
                'predict':
                    prediction_signature,
            })

        builder.save()


if __name__ == '__main__':
    model_file = './my_model.h5'
    if (os.path.isfile(model_file)):
        print('model file detected. Loading.')
        model = tf.keras.models.load_model(model_file)
    else:
        print('No model file detected.  Starting from scratch.')
        model = build_model()
        model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
        model.save(model_file)

    model.fit(X_train, y_train, batch_size=100, epochs=1, validation_data=(X_test, y_test))
    model.summary()

    export_path = "tf-model"
    save_model_for_production(model, "1", export_path)

上面的例子将模型保存到 tf-model目录下
tf-model目录结构如下：

tf-model/
└── 1
    ├── saved_model.pb
    └── variables
        ├── variables.data-00000-of-00001
        └── variables.index

saved_model.pb 是能在 tensorflow serving跑起来的模型。

3、跑模型

tensorflow_model_server --port=9000 --model_name="username" --model_base_path="/data/models/tf-model/"

标准输出如下（算法模型已成功跑起来了）：

Running ModelServer at 0.0.0.0:00 ...

4、客户端代码

#!/usr/bin/env python  
# encoding: utf-8  

""" 
@version: v1.0 
@author: zwqjoy 
@contact: [email protected] 
@site: https://blog.csdn.net/zwqjoy 
@file: client
@time: 2018/6/29 15:02 
"""

from __future__ import print_function
from grpc.beta import implementations
import tensorflow as tf

from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2
import numpy as np

tf.app.flags.DEFINE_string('server', 'localhost:9000',
                           'PredictionService host:port')
FLAGS = tf.app.flags.FLAGS


def main(_):
    host, port = FLAGS.server.split(':')
    channel = implementations.insecure_channel(host, int(port))
    stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
    # Send request

    # See prediction_service.proto for gRPC request/response details.
    data = np.array([4, 0, 0, 0, 1, 0, 1])
    data = data.astype(np.float32)

    request = predict_pb2.PredictRequest()
    request.model_spec.name = 'username'  # 这个name跟tensorflow_model_server  --model_name="username" 对应
    request.model_spec.signature_name = 'predict'  # 这个signature_name  跟signature_def_map 对应
    request.inputs['inputs'].CopyFrom(
        tf.contrib.util.make_tensor_proto(data, shape=(1, 7)))  # shape跟 keras的model.input类型对应
    result = stub.Predict(request, 10.0)  # 10 secs timeout
    print(result)


if __name__ == '__main__':
    tf.app.run()

 

客户端跑出的结果是：

outputs {
  key: "output"
  value {
    dtype: DT_FLOAT
    tensor_shape {
      dim {
        size: 1
      }
      dim {
        size: 1
      }
    }
    float_val: 0.976889811523
  }
}

 

float_val: 0.976889811523 就是我们需要的结果(概率)

 

keras模型转 tensorflow模型的一些说明

1、 keras 保存模型

可以使用model.save(filepath)将Keras模型和权重保存在一个HDF5文件中，该文件将包含：

• 模型的结构，以便重构该模型
• 模型的权重
• 训练配置（损失函数，优化器等）
• 优化器的状态，以便于从上次训练中断的地方开始

当然这个 HDF5 也可以是用下面的代码生成

from keras.models import load_model
model.save('my_model.h5')

2、 keras 加载模型

keras 加载模型(中间部分代码省略了)：

 

import numpy as np
from keras.datasets import mnist
from keras.utils import np_utils
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.models import load_model
# 载入数据
(x_train,y_train),(x_test,y_test) = mnist.load_data()
# (60000,28,28)
print('x_shape:',x_train.shape)
# (60000)
print('y_shape:',y_train.shape)
# (60000,28,28)->(60000,784)
x_train = x_train.reshape(x_train.shape[0],-1)/255.0
x_test = x_test.reshape(x_test.shape[0],-1)/255.0
# 换one hot格式
y_train = np_utils.to_categorical(y_train,num_classes=10)
y_test = np_utils.to_categorical(y_test,num_classes=10)

# 载入模型
model = load_model('model.h5')

# 评估模型
loss,accuracy = model.evaluate(x_test,y_test)

print('\ntest loss',loss)
print('accuracy',accuracy)

# 训练模型
model.fit(x_train,y_train,batch_size=64,epochs=2)

# 评估模型
loss,accuracy = model.evaluate(x_test,y_test)

print('\ntest loss',loss)
print('accuracy',accuracy)

# 保存参数，载入参数
model.save_weights('my_model_weights.h5')
model.load_weights('my_model_weights.h5')

 

keras 模型转tensorflow serving 模型的一些坑

希望能让新手少走一些弯路

坑1：过时的生成方法

有些方法已经过时了（例如下面这种）:

from tensorflow_serving.session_bundle import exporter

export_path = ... # where to save the exported graph
export_version = ... # version number (integer)

saver = tf.train.Saver(sharded=True)
model_exporter = exporter.Exporter(saver)
signature = exporter.classification_signature(input_tensor=model.input,
                                              scores_tensor=model.output)
model_exporter.init(sess.graph.as_graph_def(),
                    default_graph_signature=signature)
model_exporter.export(export_path, tf.constant(export_version), sess)

如果使用这种过时的方法，用tensorflow serving 跑模型的时候会提示：

WARNING:tensorflow:From test.py:107: Exporter.export (from tensorflow.contrib.session_bundle.exporter) is deprecated and will be removed after 2017-06-30.
Instructions for updating:
No longer supported. Switch to SavedModel immediately.

从warning中 显然可以知道这种方法要被抛弃了，不再支持这种方法了， 建议我们转用 SaveModel方法。

填坑大法： 使用 SaveModel

def save_model_for_production(model, version, path='prod_models'):
    tf.keras.backend.set_learning_phase(1)
    if not os.path.exists(path):
        os.mkdir(path)
    export_path = os.path.join(
        tf.compat.as_bytes(path),
        tf.compat.as_bytes(version))
    builder = tf.saved_model.builder.SavedModelBuilder(export_path)

    model_input = tf.saved_model.utils.build_tensor_info(model.input)
    model_output = tf.saved_model.utils.build_tensor_info(model.output)

    prediction_signature = (
        tf.saved_model.signature_def_utils.build_signature_def(
            inputs={'inputs': model_input},
            outputs={'output': model_output},
            method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))

    with tf.keras.backend.get_session() as sess:
        builder.add_meta_graph_and_variables(
            sess=sess, tags=[tf.saved_model.tag_constants.SERVING],
            signature_def_map={
                'predict':
                    prediction_signature,
            })

        builder.save()

参考：

https://www.jianshu.com/p/91aae37f1da6 

Deploying Keras model on Tensorflow Serving with GPU support

https://github.com/amir-abdi/keras_to_tensorflow