OpenVINO的部署和使用
现在几乎每家硬件或互联网公司都推出了自家的机器学习框架，小米的mace、谷歌的TensorFlow、Facebook的Torch等等。今天要介绍的是Inter公司出品的OpenVINO。OpenVINO主要分为Model Optimize和Inference两个部分。第一个就是把不同机器学习平台下训练出来的模型转换成OpenVINO可以是别的xml和bin两个组成的模型。然后执行Inference部分。
安装
OpenVINO的安装在其官网的介绍中还是比较详细的。支持Linux、Windows、Linux with FPGA和MacOS。这里就以Linux举例说明，因为之前采用docker方案安装的，配置的就是Linux环境。官网写的非常清楚，这里只是做了重复说明。
首先下载，下载前需要先注册，之后会收到一封邮件，按照邮件说明下载就可以了（有个小插曲：在Mac上用Chrome浏览器打开注册界面有点乱码，最后使用Safari打开）。接着解压tar -xvzf l_openvino_toolkit_p_
/home//inference_engine_samples /home//openvino_models
接下来，你可以选择是图形界面安装sudo ./install_GUI.sh，或者采用命令行脚本安装sudo ./install.sh。然后下载安装其依靠的框架，cd /opt/intel/openvino/install_dependencies -> sudo -E ./install_openvino_dependencies.sh.
最后我们需要设置一下环境变量，否者OpenVINO下的所需要依靠的相关文件会找不到。
source /opt/intel/openvino/bin/setupvars.sh
这种设置环境变量的方法是比较笨拙的，关机后下次开机就还需要执行一次该命令。
我们可以打开自己本地的.bashrc文件，然后在其下面加载该命令即可解决该问题：
vi
下面就开始具体的模型转换配置环节，你可以选择识别全部的模型配置，也可以到具体的么某一个框架，比如TensorFlow。
全部的配置方法是：进入cd /opt/intel/openvino/deployment_tools/model_optimizer/install_prerequisites这个文件夹下，执行：sudo ./install_prerequisites.sh。
配置单个的，如TensorFlow，可以选择执行：sudo ./install_prerequisites_tf.sh
现在就可以使用了。其他的，如安装显卡的配置或者视觉加速等还是查看官网吧。
使用
可以使用官方提供的demo来看一下你部署的成果，进入：cd /opt/intel/openvino/deployment_tools/demo后，有两个官方例子，一个是分类的一个是目标检测的。执行其中一个：./demo_squeezenet_download_convert_run.sh。
进入，/opt/intel/openvino/deployment_tool，发现model_optimizer和Inference_enginer两个文件夹。这里面的文件是如何使用OpenVINO的重要指引。
一，Model Optimize
首先进入model_optimizer文件夹，这里使用TensorFlow的pb模型转成OpenVINO的xml和bin模型。假设之前的TensorFlow转换需要的配置已经安装完成。这里有个mo_tf.py的Python文件。相应的有其他机器学习框架的转换文件，比如说caffe的mo_caffe.py。
在运行mo_tf.py文件时，需要传入几个必要的参数，如输入模型的路径和输出的路径等等，其中包括，input_model：输入pb文件的路径，output_dir:输出的xml和bin模型路径。data_type：数据类型，可以为：FP16,FP32等。具体参数在官网也可查询到。python mp_tf.py --input_model user/model_path/xxx.pb。这些参数都可以在model_optimizer文件夹下的文件中找到。可以阅读源码的，这里列举一下通用和TensorFlow相关的。
通用的参数如下：
common_group.add_argument('--input_model', '-w', '-m',
help='Tensorflow*: a file with a pre-trained model ' +
' (binary or text .pb file after freezing).\n' +
' Caffe*: a model proto file with model weights',
action=CanonicalizePathCheckExistenceAction,
type=readable_file)
common_group.add_argument('--model_name', '-n',
help='Model_name parameter passed to the final create_ir transform. ' +
'This parameter is used to name ' +
'a network in a generated IR and output .xml/.bin files.')
common_group.add_argument('--output_dir', '-o',
help='Directory that stores the generated IR. ' +
'By default, it is the directory from where the Model Optimizer is launched.',
default=get_absolute_path('.'),
action=CanonicalizePathAction,
type=writable_dir)
common_group.add_argument('--input_shape',
help='Input shape(s) that should be fed to an input node(s) of the model. '
'Shape is defined as a comma-separated list of integer numbers enclosed in '
'parentheses or square brackets, for example [1,3,227,227] or (1,227,227,3), where '
'the order of dimensions depends on the framework input layout of the model. '
'For example, [N,C,H,W] is used for Caffe* models and [N,H,W,C] for TensorFlow* '
'models. Model Optimizer performs necessary transformations to convert the shape to '
'the layout required by Inference Engine (N,C,H,W). The shape should not contain '
'undefined dimensions (? or -1) and should fit the dimensions defined in the input '
'operation of the graph. If there are multiple inputs in the model, --input_shape '
'should contain definition of shape for each input separated by a comma, for '
'example: [1,3,227,227],[2,4] for a model with two inputs with 4D and 2D shapes.')
common_group.add_argument('--scale', '-s',
type=float,
help='All input values coming from original network inputs will be ' +
'divided by this ' +
'value. When a list of inputs is overridden by the --input ' +
'parameter, this scale ' +
'is not applied for any input that does not match with ' +
'the original input of the model.')
common_group.add_argument('--reverse_input_channels',
help='Switch the input channels order from RGB to BGR (or vice versa). Applied to '
'original inputs of the model if and only if a number of channels equals 3. Applied '
'after application of --mean_values and --scale_values options, so numbers in '
'--mean_values and --scale_values go in the order of channels used in the original '
'model.',
action='store_true')
common_group.add_argument('--log_level',
help='Logger level',
choices=['CRITICAL', 'ERROR', 'WARN', 'WARNING', 'INFO',
'DEBUG', 'NOTSET'],
default='ERROR')
common_group.add_argument('--input',
help='The name of the input operation of the given model. ' +
'Usually this is a name of the ' +
'input placeholder of the model.')
common_group.add_argument('--output',
help='The name of the output operation of the model. ' +
'For TensorFlow*, do not add :0 to this name.')
common_group.add_argument('--mean_values', '-ms',
help='Mean values to be used for the input image per channel. ' +
'Values to be provided in the (R,G,B) or [R,G,B] format. ' +
'Can be defined for desired input of the model, for example: ' +
'"--mean_values data[255,255,255],info[255,255,255]". ' +
'The exact meaning and order ' +
'of channels depend on how the original model was trained.',
default=())
common_group.add_argument('--scale_values',
help='Scale values to be used for the input image per channel. ' +
'Values are provided in the (R,G,B) or [R,G,B] format. ' +
'Can be defined for desired input of the model, for example: ' +
'"--scale_values data[255,255,255],info[255,255,255]". ' +
'The exact meaning and order ' +
'of channels depend on how the original model was trained.',
default=())
# TODO: isn't it a weights precision type
common_group.add_argument('--data_type',
help='Data type for all intermediate tensors and weights. ' +
'If original model is in FP32 and --data_type=FP16 is specified, all model weights ' +
'and biases are quantized to FP16.',
choices=["FP16", "FP32", "half", "float"],
default='float')
common_group.add_argument('--disable_fusing',
help='Turn off fusing of linear operations to Convolution',
action='store_true')
common_group.add_argument('--disable_resnet_optimization',
help='Turn off resnet optimization',
action='store_true')
common_group.add_argument('--finegrain_fusing',
help='Regex for layers/operations that won\'t be fused. ' +
'Example: --finegrain_fusing Convolution1,.*Scale.*')
common_group.add_argument('--disable_gfusing',
help='Turn off fusing of grouped convolutions',
action='store_true')
common_group.add_argument('--enable_concat_optimization',
help='Turn on concat optimization',
action='store_true')
common_group.add_argument('--move_to_preprocess',
help='Move mean values to IR preprocess section',
action='store_true')
# we use CanonicalizeDirCheckExistenceAction instead of readable_dirs to handle empty strings
common_group.add_argument("--extensions",
help="Directory or a comma separated list of directories with extensions. To disable all "
"extensions including those that are placed at the default location, pass an empty "
"string.",
default=import_extensions.default_path(),
action=CanonicalizePathCheckExistenceAction,
type=readable_dirs_or_empty)
common_group.add_argument("--batch", "-b",
type=check_positive,
default=None,
help="Input batch size")
common_group.add_argument("--version",
action='store_true',
help="Version of Model Optimizer")
common_group.add_argument('--silent',
help='Prevent any output messages except those that correspond to log level equals '
'ERROR, that can be set with the following option: --log_level. '
'By default, log level is already ERROR. ',
action='store_true',
default=False)
common_group.add_argument('--freeze_placeholder_with_value', help='Replaces input layer with constant node with '
'provided value, e.g.: "node_name->True"',
default=None)
common_group.add_argument('--generate_deprecated_IR_V2',
help='Force to generate legacy/deprecated IR V2 to work with previous versions of the'
' Inference Engine. The resulting IR may or may not be correctly loaded by'
' Inference Engine API (including the most recent and old versions of Inference'
' Engine) and provided as a partially-validated backup option for specific'
' deployment scenarios. Use it at your own discretion. By default, without this'
' option, the Model Optimizer generates IR V3.',
action='store_true')
common_group.add_argument('--keep_shape_ops',
help='[ Experimental feature ] Enables `Shape` operation with all children keeping. '
'This feature makes model reshapable in Inference Engine',
action='store_true', default=False)TensorFlow相关参数的具体详情如下：
tf_group.add_argument('--input_model_is_text',
help='TensorFlow*: treat the input model file as a text protobuf format. If not specified, ' +
'the Model Optimizer treats it as a binary file by default.',
action='store_true')
tf_group.add_argument('--input_checkpoint', type=str, default=None, help="TensorFlow*: variables file to load.",
action=CanonicalizePathCheckExistenceAction)
tf_group.add_argument('--input_meta_graph',
help='Tensorflow*: a file with a meta-graph of the model before freezing',
action=CanonicalizePathCheckExistenceAction,
type=readable_file)
tf_group.add_argument('--saved_model_dir', default=None,
help="TensorFlow*: directory representing non frozen model",
action=CanonicalizePathCheckExistenceAction,
type=readable_dirs)
tf_group.add_argument('--saved_model_tags', type=str, default=None,
help="Group of tag(s) of the MetaGraphDef to load, in string format, separated by ','. "
"For tag-set contains multiple tags, all tags must be passed in.")
tf_group.add_argument('--tensorflow_subgraph_patterns',
help='TensorFlow*: a list of comma separated patterns that will be applied to ' +
'TensorFlow* node names to ' +
'infer a part of the graph using TensorFlow*.')
tf_group.add_argument('--tensorflow_operation_patterns',
help='TensorFlow*: a list of comma separated patterns that will be applied to ' +
'TensorFlow* node type (ops) ' +
'to infer these operations using TensorFlow*.')
tf_group.add_argument('--tensorflow_custom_operations_config_update',
help='TensorFlow*: update the configuration file with node name patterns with input/output '
'nodes information.',
action=CanonicalizePathCheckExistenceAction)
tf_group.add_argument('--tensorflow_use_custom_operations_config',
help='TensorFlow*: use the configuration file with custom operation description.',
action=CanonicalizePathCheckExistenceAction)
tf_group.add_argument('--tensorflow_object_detection_api_pipeline_config',
help='TensorFlow*: path to the pipeline configuration file used to generate model created '
'with help of Object Detection API.',
action=CanonicalizePathCheckExistenceAction)
tf_group.add_argument('--tensorboard_logdir',
help='TensorFlow*: dump the input graph to a given directory that should be used with TensorBoard.',
default=None,
action=CanonicalizePathCheckExistenceAction)
tf_group.add_argument('--tensorflow_custom_layer_libraries',
help='TensorFlow*: comma separated list of shared libraries with TensorFlow* custom '
'operations implementation.',
default=None,
action=CanonicalizePathCheckExistenceAction)
tf_group.add_argument('--disable_nhwc_to_nchw',
help='Disables default translation from NHWC to NCHW',
action='store_true')对TensorFlow的的OP还不能完全支持，如有不能识别的OP，对不同框架下，处理的方式有所不同，并且同一框架下有多种方法选择，官网同样给了说明自定义操作层。编码的具体方法。
对于模型转换中常见的一些问题，官网给出了一些解释：Model Optimizer Frequently Asked Questions
二，Inference
进入，/opt/intel/openvino/deployment_tools/inference_engine/samples，发现有好多文件夹，涉及到多种类型的demo，分类、目标检测、目标分割等。这里面都是C或者C++写的。如何用，每一个文件夹里面都有相应的介绍。
有一个python_samples的文件夹，进入后发现，这是Python版本的实现方法。里面也同样涉及到传入参数，打开文件一目了然：
def build_argparser():
parser = ArgumentParser(add_help=False)
args = parser.add_argument_group('Options')
args.add_argument('-h', '--help', action='help', default=SUPPRESS, help='Show this help message and exit.')
args.add_argument("-m", "--model", help="Path to an .xml file with a trained model.", required=True, type=str)
args.add_argument("-i", "--input", help="Path to a folder with images or path to an image files", required=True,
type=str, nargs="+")
args.add_argument("-l", "--cpu_extension",
help="Optional. Required for CPU custom layers. "
"Absolute MKLDNN (CPU)-targeted custom layers. Absolute path to a shared library with the "
"kernels implementations", type=str, default=None)
args.add_argument("-pp", "--plugin_dir", help="Path to a plugin folder", type=str, default=None)
args.add_argument("-d", "--device",
help="Specify the target device to infer on; CPU, GPU, FPGA, HDDL or MYRIAD is acceptable. Sample "
"will look for a suitable plugin for device specified. Default value is CPU", default="CPU",
type=str)
args.add_argument("-nt", "--number_top", help="Number of top results", default=10, type=int)
args.add_argument("-ni", "--number_iter", help="Number of inference iterations", default=1, type=int)
args.add_argument("--mean_val_r", "-mean_val_r",
help="Mean value of red chanel for mean value subtraction in postprocessing ", default=0,
type=float)
args.add_argument("--mean_val_g", "-mean_val_g",
help="Mean value of green chanel for mean value subtraction in postprocessing ", default=0,
type=float)
args.add_argument("--mean_val_b", "-mean_val_b",
help="Mean value of blue chanel for mean value subtraction in postprocessing ", default=0,
type=float)
args.add_argument("-pc", "--perf_counts", help="Report performance counters", default=False, action="store_true")
return parser其中必要的参数为-i和-m，他们分别指定了要输入的图片和模型路径。
另外，可以仔细研究这里每个任务的推理源码，包括C++或者Python的，有利于更加详细了解OpenVINO的调用接口，从而自己编写相应的文件。同样，转换成功以后，推理阶段的OP也有可能不支持，自定义层编写成库的形式可以参考官网
实战：
官网给了一个yolo-v3的例子，在上一小节的时候也说了，下载好OpenVINO后，已经给出了多种类型的神经网络模型加载的源代码，包括C++和Python。源码已经有了，这里主要是介绍一下中间可能包括的一些小坑。
首先Python：官网给出的步骤
转好pb模型后，/opt/intel/openvino/deployment_tools/inference_engine/samples/python_samples，找到yolo的Python代码。采用OpenVINO运行。
可能出现的问题是，无法加载OpenVINO框架的一些信息，这时，你需要配置一下环境变量，前面也已经介绍，source /opt/intel/openvino/bin/setupvars.sh
解决这个以后有可能发现，这里面有很多OP操作，OpenVINO不支持，比如：
[ ERROR ] Following layers are not supported by the plugin for specified device CPU:
detector/yolo-v3-tiny/Conv_9/BiasAdd/YoloRegion, detector/yolo-v3-tiny/ResizeNearestNeighbor, detector/yolo-v3-tiny/Conv_12/BiasAdd/YoloRegion
[ ERROR ] Please try to specify cpu extensions library path in sample's command line parameters using -l or --cpu_extension command line argument这里可以通过加载官网已经写好的动态库来加载，具体位置在：
{HOME}/inference_engine_samples_build/intel64/Release/lib/libcpu_extension.so进入home，命令行 cd ~
那么在执行Python文件的时候 加载 -l
可参考以下代码：
python3 object_detection_demo_yolov3_async.py -m frozen_darknet_yolov3_model.xml -i football2.mp4 -l ./root/inference_engine_samples_build/intel64/Release/lib/libcpu_extension.so --label coconames.txt以上的所有文件 我都写在了和object_detection_demo_yolov3_async.py同意目录下。
C++ 实例
官网参考
1.使用已经编译好的Makefile文件
如果Python跑通了，那么C++的难度就会降低很多。 毕竟参数都已经配置好。C++的主要在于编译这块。
我们在/opt/intel/openvino/deployment_tools/inference_engine/samples 这里看到的是源码。其编译的Makefile文件其实也已经写好了，在
{HOME}/inference_engine_samples_build目录下进入object_detection_demo_yolov3_async文件夹下，看到已经有一个Makefile文件，命令行进入该文件夹下，并在命令行键入 make
可能遇到的问题是：找不到cmake 文件路径，即 /usr/local/Cellar/cmake/3.10.2/bin/cmake, No such file or directory.
这次按照他说的路径进入，一步一步检查，这个可能是电脑上安装的cmake版本问题。找到自己的版本，打开Makefile文件打开，修改里面的内容即可。
编译成功后，可执行文件在
{HOME}/inference_engine_samples_build/intel64/Release/object_detection_demo_yolov3_async然后命令行键入：
./object_detection_demo_yolov3_async -m frozen_darknet_yolov3_model.xml -i football2.mp4 -l libcpu_extension.so同样，这次包括自定义的op操作动态库我也写在了同一文件夹下。
2. 使用cmake 自己编译
cmake文件，OpenVINO也已经帮写好了，这里重复一遍只是为了以后自己修改官网给的demo程序或者添加了部分文件后，该如何编译成可执行文件。
进入/opt/intel/openvino/deployment_tools/inference_engine/samples后，这里有个CMakeLists.txt 文件，这个就是跟目录下的cmake文件。在每个项目中，比如说object_detection_demo_yolov3_async又分别有子CMakeLists.txt。根目录下的CMakeLists.txt下建立build文件夹用来放cmake编译过程中产生的中间文件和各子工程产生的Makefile文件。cd build 进入，然后执行cmake ..。这样在build文件夹中就产生了和{HOME}/inference_engine_samples_build目录下一样的Makefile文件，通过编译make Makefile 就可以得到相应的可执行文件
卸载：
卸载的步骤也很简单，可以选择页面卸载，
首先进入有卸载页面的文件夹内
cd /opt/intel/openvino/openvino_toolkit_uninstaller卸载：sudo ./uninstall_GUI.sh
删除环境变量：vim ~/.bashrc， 键入‘i’进入编辑模式，光标移到最后一行删除‘source /opt/intel/openvino/bin/setupvars.sh’这行。
然后esc 退出编辑模式，使用:wq保存退出。
在home目录下删除openvino_models、inference_engine_demos_build等相关文件。