使用Docker和TensorFlow Serving部署服务

目录

1.启动docker

2.模型由来

3.部署模型

   1） 拉取镜像

   2）启动服务

   3）查看模型状态

   4）查看模型的输入输出

4.模型接口预测：REST

---

1.启动docker

       安装详细介绍可参考：https://blog.csdn.net/weixin_42530066/article/details/107850996

2.模型由来

      此处使用已训练好的model，需注意TensorFlow Serving是使用SavedModel格式来保存模型的。由于SavedModel包含了完整的TensorFlow程序，包括权重和计算，它不需要运行原始的模型构建代码，因此对共享或部署使用TFLite，TensorFlow.js，TensorFLow Serving或者TensorFlow Hub很有用

注：关于SavedModel可参考：https://www.tensorflow.org/guide/saved_model，此处不详细介绍

文件目录树如下：

└─exported_model
    └─saved_model
        └─1
            └─savedmodel.pb

3.部署模型

     获取官网的tfserving，参考链接，链接中有详细的docker pull命令，链接如下：https://hub.docker.com/r/tensorflow/serving/tags

1） 拉取镜像docker pull tensorflow/serving

λ docker pull tensorflow/serving
Using default tag: latest
latest: Pulling from tensorflow/serving
Digest: sha256:4a45f9bcaf96e71ef1b6d824e490562cc962ea66e1213ab06015a7da8054a53b
Status: Image is up to date for tensorflow/serving:latest
docker.io/tensorflow/serving:latest

2）启动服务

docker run -p 8500:8500 -p 8501:8501 --name tfserving --mount type=bind,source=/exported_model/saved_model,target=/models/saved_model -e MODEL_NAME=saved_model -t tensorflow/serving

     参数：

      端口8500：TensorFlow Serving提供的gRPC端口

      端口8501：REST API服务端口

      source=模型存放的绝对路径

      -e MODEL_NAME=saved_model指TensorFlow Serving需要加载的模型名称

     输出结果如下：

2020-08-06 10:25:18.022501: I tensorflow_serving/model_servers/server.cc:86] Building single TensorFlow model file config:  model_name: saved_model model_base_path: /models/saved_model
2020-08-06 10:25:18.058268: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-08-06 10:25:18.058323: I tensorflow_serving/model_servers/server_core.cc:575]  (Re-)adding model: saved_model
2020-08-06 10:25:18.594777: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: saved_model version: 1}
2020-08-06 10:25:18.594894: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: saved_model version: 1}
2020-08-06 10:25:18.594938: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: saved_model version: 1}
2020-08-06 10:25:18.596078: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/saved_model/1
2020-08-06 10:25:19.515086: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-08-06 10:25:19.515310: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:295] Reading SavedModel debug info (if present) from: /models/saved_model/1
2020-08-06 10:25:19.780411: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:234] Restoring SavedModel bundle.
2020-08-06 10:25:19.781742: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:244] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /models/saved_model/1/variables/variables.index
2020-08-06 10:25:19.781852: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:364] SavedModel load for tags { serve }; Status: success: OK. Took 1185786 microseconds.
2020-08-06 10:25:19.801197: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at /models/saved_model/1/assets.extra/tf_serving_warmup_requests
2020-08-06 10:25:19.808024: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: saved_model version: 1}
2020-08-06 10:25:19.829410: I tensorflow_serving/model_servers/server.cc:355] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2020-08-06 10:25:19.849060: I tensorflow_serving/model_servers/server.cc:375] Exporting HTTP/REST API at:localhost:8501 ...

3）查看模型状态

     使用curl命令查看启动服务状态及模型状态，TensorFlow Serving使用1作为模型的版本号

λ curl http://localhost:8501/v1/models/saved_model
{
 "model_version_status": [
  {
   "version": "1",
   "state": "AVAILABLE",
   "status": {
    "error_code": "OK",
    "error_message": ""
   }
  }
 ]
}

4）查看模型的输入输出

saved_model_cli show --dir /exported_model/saved_model/1/ --all

     注：1为version，version不同填写不一样

     输出如下：

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:                
                                                                             
signature_def['serving_default']:                                                       
  The given SavedModel SignatureDef contains the following input(s):                    
    inputs['inputs'] tensor_info:                                                       
        dtype: DT_UINT8                                                                 
        shape: (-1, -1, -1, 3)                                                          
        name: image_tensor:0                                                            
  The given SavedModel SignatureDef contains the following output(s):                   
    outputs['detection_boxes'] tensor_info:                                             
        dtype: DT_FLOAT                                                                 
        shape: (-1, 100, 4)                                                             
        name: detection_boxes:0                                                         
    outputs['detection_classes'] tensor_info:                                           
        dtype: DT_FLOAT                                                                 
        shape: (-1, 100)                                                                
        name: detection_classes:0                                                       
    outputs['detection_multiclass_scores'] tensor_info:                                 
        dtype: DT_FLOAT                                                                 
        shape: (-1, 100, 2)                                                             
        name: detection_multiclass_scores:0                                             
    outputs['detection_scores'] tensor_info:                                            
        dtype: DT_FLOAT                                                                 
        shape: (-1, 100)                                                                
        name: detection_scores:0                                                        
    outputs['num_detections'] tensor_info:                                              
        dtype: DT_FLOAT                                                                 
        shape: (-1)                                                                     
        name: num_detections:0                                                          
    outputs['raw_detection_boxes'] tensor_info:                                         
        dtype: DT_FLOAT                                                                 
        shape: (-1, -1, 4)                                                              
        name: raw_detection_boxes:0                                                     
    outputs['raw_detection_scores'] tensor_info:                                        
        dtype: DT_FLOAT                                                                 
        shape: (-1, -1, 2)                                                              
        name: raw_detection_scores:0                                                    
  Method name is: tensorflow/serving/predict              

4.模型接口预测：REST

   安装：pip install tensorflow-serving-api

   TensorFlow Serving提供REST API和gRPC两种请求方式，接下来具体介绍REST

   脚本如下：

import numpy as np
import json
from PIL import Image
import requests
import time

MODEL_NAME = "saved_model"
URL_TFSERVING = f"http://127.0.0.1:8501/v1/models/{MODEL_NAME}:predict"
PATH_IMAGE = "test.jpg"
PATH_LABEL_MAP = "./exported_model/label_map.json"
THRESHOLD = 0.3


def predict(label_map, path_image):
    img = Image.open(path_image)
    img = img.resize((320, 320))
    print(img.size)
    data_to_send = {
        "instances": np.expand_dims(img, axis=0).tolist(),
    }

    start = time.time()
    ret = requests.post(URL_TFSERVING, timeout=200, json=data_to_send)

    detect_ret = ret.json()

    width, height = img.size
    predictions = detect_ret['predictions'][0]

    cnt = 0
    for score in predictions['detection_scores']:
        if score > THRESHOLD:
            cnt += 1
        else:
            break
    detection_boxes = predictions['detection_boxes'][:cnt]
    detection_classes = predictions['detection_classes'][:cnt]

    boxes = [
        {'xmin': int(width * b[1]), 'ymin': int(height * b[0]), 'xmax': int(width * b[3]), 'ymax': int(height * b[2])}
        for b in detection_boxes
    ]
    labels = [label_map.get(str(int(i + 1e-6)), "UNKONWN") for i in detection_classes]
    print(time.time() - start)
    return boxes, labels


if __name__ == '__main__':
    with open(PATH_LABEL_MAP, 'r') as f:
        label_map = json.load(f)
    boxes, labels = predict(label_map, PATH_IMAGE)
    print(boxes)
    print(labels)

   输出如下：

λ python3 pb_tfserving_local_with_label_map.py
(320, 320)
2.8160481452941895
[{'xmin': 199, 'ymin': 69, 'xmax': 266, 'ymax': 222}]
['water_reflect']

查看进程：

λ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                              NAMES
18fd96918d0a        tensorflow/serving   "/usr/bin/tf_serving…"   30 minutes ago      Up 30 minutes       0.0.0.0:8500-8501->8500-8501/tcp   tfserving

停止服务进程：

docker stop id     #id=18fd96918d0a
docker stop 18fd96918d0a

以上仅供参考

参考：

        https://medium.com/tensorflow/serving-ml-quickly-with-tensorflow-serving-and-docker-7df7094aa008

        https://www.tensorflow.org/tfx/tutorials/serving/rest_simple