【Tensorflow】GPU训练ssd模型

1. 硬件环境

操作系统： Windows 10
CPU：i7 8700k 3.7GHZ
GPU：Geforce GTX 1080ti
内存： 8G

2. 文件目录

3. 安装步骤

• 安装主板集成显卡驱动

  下个驱动精灵安装即可，安装驱动完成后，再关机插入Geforce GTX 1080ti
  

• 安装Geforce GTX 1080ti驱动

  1. 可以去nvida官网下载，我下载的是388.43-desktop-win10-64bit-international-whql
  2. 如果提示“此NVIDA驱动程序不支持此操作系统类型”
      原因是，windows 10操作系统版本太老的缘故，去“更新”里面更新下操作系统（如果你的版本很老，会耗时很长）
  

• 安装Anaconda5.0.1 For Windows

  Anaconda：https://www.continuum.io/downloads
  Python 3.6 version 64BIT INSTALLER：https://repo.continuum.io/archive/Anaconda3-5.0.1-Windows-x86_64.exe
  

• 安装Tensorflow-gpu

  conda install pip
  pip install --ignore-installed --upgrade tensorflow-gpu
  

• 添加系统环境变量PYTHONPATH
  环境变量-->(系统变量下新建PYTHONPATH)加入以下内容：

  C:\suredata\0bject-Detector-App;C:\suredata\0bject-Detector-App\slim
  

• 安装vc_redist.x64.exe
  由于tensorflow只支持CUDA8.0和CUDNN v6 版本，所以要先装，要不然会有以下错误：

  ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'
  

  安装CUDA8.0 + cudnn6.0：
  注意： 有的说cudnn5.1也可以，但是我试了win10最新系统上，cudnn5.1不可以，仍然提示上面错误

  1. 安装CUDA8.0
      [下载地址](https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_win10-exe)
  2. cudnn6.0      
     [下载地址](https://developer.nvidia.com/rdp/cudnn-download#a-collapse6-8)
  3. 把解压后的文件夹复制到CUDA8.0的安装目录文件夹内。
     默认CUDA8.0的安装目录为：C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
  

3. 训练

• 管理员权限打开cmd并执行步骤2的命令开始训练

  1. cd C:\suredata\0bject-Detector-App
  2. 执行如下命令开始训练：python object_detection\train.py --logoutstderr --train_dir=sd_train\ssd_model_2017_12_21\output --pipeline_config_path=sd_train\ssd_model_2017_12_21\ssd_mobilenet_v1_pascal.config
  

训练情况：

开始训练

• 生成模型
  训练完成后用下列命令生成可用模型：

  python object_detection\export_inference_graph.py --input_type image_tensor --pipeline_config_path sd_train\ssd_model_2017_12_21\ssd_mobilenet_v1_pascal.config --trained_checkpoint_prefix sd_train\ssd_model_2017_12_21\output\model.ckpt-30000 --output_directory sd_train\ssd_model_2017_12_21\savedModel
  

  如果出现以下错误：

  C:\suredata\0bject-Detector-App>python object_detection\export_inference_graph.py --input_type image_tensor --pipeline_config_path sd_train\ssd_model_2017_12_21\ssd_mobilenet_v1_pascal.config --trained_checkpoint_prefix sd_train\ssd_model_2017_12_21\output\model.ckpt-30000 --output_directory sd_train\ssd_model_2017_12_21\savedModel                                                        
  Traceback (most recent call last):
  File "object_detection\export_inference_graph.py", line 119, in 
    tf.app.run()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "object_detection\export_inference_graph.py", line 115, in main
    FLAGS.output_directory, input_shape)
  File "C:\suredata\0bject-Detector-App\object_detection\exporter.py", line 427, in export_inference_graph
    input_shape, optimize_graph, output_collection_name)
  File "C:\suredata\0bject-Detector-App\object_detection\exporter.py", line 353, in _export_inference_graph
    postprocessed_tensors = detection_model.postprocess(output_tensors)
  File "C:\suredata\0bject-Detector-App\object_detection\meta_architectures\ssd_meta_arch.py", line 405, in postprocess
    class_predictions_without_background)
  File "C:\suredata\0bject-Detector-App\object_detection\builders\post_processing_builder.py", line 94, in score_converter_fn
    scaled_logits = tf.divide(logits, logit_scale, name='scale_logits')
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 309, in divide
    return DivideDelegateWithName(x, name) / y
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 294, in __truediv__
    return _truediv_python3(self.x, y, self.name)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 981, in _truediv_python3
    (x_dtype, y_dtype))   TypeError: x and y must have the same dtype, got tf.float32 != tf.int32 
  

  可以通过将C:\suredata\0bject-Detector-App\object_detection\builders\post_processing_builder.py中的：

  def _score_converter_fn_with_logit_scale(tf_score_converter_fn, logit_scale):
  """Create a function to scale logits then apply a Tensorflow function."""
  def score_converter_fn(logits):
    scaled_logits = tf.divide(logits, logit_scale, name='scale_logits')
    return tf_score_converter_fn(scaled_logits, name='convert_scores')
  score_converter_fn.__name__ = '%s_with_logit_scale' % (
      tf_score_converter_fn.__name__)
  return score_converter_fn
  

  修改为：

  def _score_converter_fn_with_logit_scale(tf_score_converter_fn, logit_scale):
  """Create a function to scale logits then apply a Tensorflow function."""
  def score_converter_fn(logits):
    cr = logit_scale
    cr = tf.constant([[cr]],tf.float32)
    scaled_logits = tf.divide(logits, cr, name='scale_logits')
    return tf_score_converter_fn(scaled_logits, name='convert_scores')
  score_converter_fn.__name__ = '%s_with_logit_scale' % (
      tf_score_converter_fn.__name__)
  return score_converter_fn
  

  再执行生成模型命令：

  

  生成模型

  
  

  可以看到问题解决。

几个问题：

• GPU利用率始终上不去20%左右，CPU80%
  原因是：