python+opencv+caffe+摄像头做目标检测

首先之前已经成功的使用Python做图像的目标检测，这回因为项目最终是需要用摄像头的，

所以实现摄像头获取图像，并且用Python调用CAFFE接口来实现目标识别

首先是摄像头请选择支持Linux万能驱动兼容V4L2的摄像头，

因为之前用学ARM的时候使用的Smart210，我已经确认我的摄像头是支持的，

我把摄像头插上之後自然就在 /dev 目录下看到多了一个video0的文件，

这个就是摄像头的设备文件了，所以我就没有额外处理驱动的部分

一、检测环境

再来在开始前因为之前按着国嵌的指导手册安装的opencv3.2当时没有开启V4L2及GTK_2.x的支持，

所以後面遇到了一连串的问题，请大家如下面方法检测

$ python

1.检测Python的V4L2支持及摄像头驱动是否正常

进入Python之後如下命令

Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cap = cv2.VideoCapture(0)
>>> print cap.isOpened()
True
>>>

如果 返回True就代表摄像头及你的opencv的V4L2支持就已经完全正常了

如果返回False就代表opencv或是摄像头有问题叁考後面的修改方式

2.再来因为我们要把摄像头的影像生成窗口，所以我们需要检测Python的gtk支持如下

>>> import cv2
>>> cv2.namedWindow('test',cv2.WINDOW_AUTOSIZE)
>>> 

如果没有报任何错误就代表gtk也是正常的
如如果出现下面提示

OpenCV Error: Unspecified error (The function is not implemented. Rebuild the library with Windows,
 GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, 
then re-run cmake or configure script) in cvNamedWindow,

那就代表opencv不支持gtk

如果上面两个测试都是好使的你可以跳过本步骤

首先我发现我不论如何重新编译opencv发现python一直都是有问题的，

最後发现是因为不知道什麽时候已经装过 python-opencv的包了

可以用命令

$ pip list |grep opencv

如果发现有任何跟opencv有关的包都可以利用 pip uninstall xxx 来移除

假设看到 pythom-opencv的包那就 pip uninstall opencv-python 来移除

还有检查dpkg -i |grep opencv 如果跟python的opencv有关的包也得移除

$ sudo apt-get remove python-opencv

然後到你之前安装opencv3.2的那个源码目录，

$ cd build
$ make uninstall

这样就会卸载之前安装的opencv

再来V4L的头文件已经改名了，但是opencv会默认使用linux/videodev.h所以要做个软鍊接

$ ln -s /usr/include/libv4l1-videodev.h /usr/include/linux/videodev.h

opencv安装过程中会自动的检测相关的包，以及一些依赖，

先列出我安装的包，但是因为环境多少有点不同，下面会教大家如何看缺少的包

$ sudo apt-get install  libgphoto2-dev v4l2ucp libv4l-dev dv4l libwebcam0-dev libgtkglext1-dev libunicap2-dev 

再来执行cmake

$ cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_TBB=ON -D WITH_V4L=ON -D WITH_GTK=ON -D WITH_GTK_2_X -D WITH_OPENGL=ON \
-D WITH_CUDA=ON -D WITH_CUBLAS=ON -D BUILD_PYTHON_SUPPORT=ON \
-D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib-3.2.0/modules/ ../opencv-3.2.0

其中如果V4L2使用share库也可以把-D WITH_V4L: = ON 换成 

-D WITH_LIBV4L=ON

在cmake的过程中会有类似如下的提示

Detected version of GNU GCC: 54 (504)
FP16: Feature disabled
Found OpenEXR: /usr/lib/x86_64-linux-gnu/libIlmImf.so
Checking for module 'libucil'
  No package 'libucil' found
Looking for linux/videodev.h
Looking for linux/videodev.h - found
Looking for linux/videodev2.h
Looking for linux/videodev2.h - found
Looking for sys/videoio.h
Looking for sys/videoio.h - not found
Checking for module 'libavresample'
  No package 'libavresample' found
Found TBB: build
found IPP (ICV version): 9.0.1 [9.0.1]
at: /mnt/sdb/ubuntu/install/opencv/build/3rdparty/ippicv/ippicv_lnx
CUDA detected: 8.0
CUDA NVCC target flags: -gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-D_FORCE_INLINES
LAPACK_IMPL: Atlas, LAPACK_LIBRARIES: /usr/lib/liblapack.so;/usr/lib/libcblas.so;/usr/lib/libatlas.so
Could NOT find JNI (missing:  JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH) 
Could NOT find Matlab (missing:  MATLAB_MEX_SCRIPT MATLAB_INCLUDE_DIRS MATLAB_ROOT_DIR MATLAB_LIBRARIES MATLAB_LIBRARY_DIRS MATLAB_MEXEXT MATLAB_ARCH MATLAB_BIN) 
VTK is not found. Please set -DVTK_DIR in CMake to VTK build directory, or to VTK install subdirectory with VTKConfig.cmake file

General configuration for OpenCV 3.2.0 =====================================
  Version control:               unknown

  Platform:
    Timestamp:                   2017-08-14T00:31:55Z
    Host:                        Linux 4.10.0-30-generic x86_64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               Release

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          /usr/lib/x86_64-linux-gnu/libpng.so /usr/lib/x86_64-linux-gnu/libtiff.so /usr/lib/x86_64-linux-gnu/libjasper.so /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so gtk-x11-2.0 gdk-x11-2.0 pangocairo-1.0 atk-1.0 cairo gdk_pixbuf-2.0 gio-2.0 pangoft2-1.0 pango-1.0 gobject-2.0 glib-2.0 fontconfig freetype gthread-2.0 gtkglext-x11-1.0 gdkglext-x11-1.0 GLU GL Xmu Xt SM ICE pangox-1.0 X11 gmodule-2.0 gstbase-1.0 gstreamer-1.0 gstvideo-1.0 gstapp-1.0 gstriff-1.0 gstpbutils-1.0 unicap dc1394 avcodec-ffmpeg avformat-ffmpeg avutil-ffmpeg swscale-ffmpeg gphoto2 gphoto2_port exif dl m pthread rt /usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so /usr/lib/liblapack.so /usr/lib/libcblas.so /usr/lib/libatlas.so cudart nppc nppi npps cublas cufft -L/usr/local/cuda-8.0/lib64
    3rdparty dependencies:       libwebp zlib libjpeg tbb

  OpenCV modules:
    To be built:                 cudev core cudaarithm flann imgproc ml video cudabgsegm cudafilters cudaimgproc cudawarping imgcodecs photo shape videoio cudacodec highgui objdetect ts features2d calib3d cudafeatures2d cudalegacy cudaobjdetect cudaoptflow cudastereo stitching superres videostab python2
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 java python3 viz

  GUI: 
    QT:                          NO
    GTK+ 2.x:                    YES (ver 2.24.30)
    GThread :                    YES (ver 2.48.2)
    GtkGlExt:                    YES (ver 1.2.0)
    OpenGL support:              YES (/usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so)
    VTK support:                 NO

  Media I/O: 
    ZLib:                        build (ver 1.2.8)
    JPEG:                        libjpeg (ver 90)
    WEBP:                        build (ver 0.3.1)
    PNG:                         /usr/lib/x86_64-linux-gnu/libpng.so (ver 1.2.54)
    TIFF:                        /usr/lib/x86_64-linux-gnu/libtiff.so (ver 42 - 4.0.6)
    JPEG 2000:                   /usr/lib/x86_64-linux-gnu/libjasper.so (ver 1.900.1)
    OpenEXR:                     /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so (ver 2.2.0)
    GDAL:                        NO
    GDCM:                        NO

  Video I/O:
    DC1394 1.x:                  NO
    DC1394 2.x:                  YES (ver 2.2.4)
    FFMPEG:                      YES
      avcodec:                   YES (ver 56.60.100)
      avformat:                  YES (ver 56.40.101)
      avutil:                    YES (ver 54.31.100)
      swscale:                   YES (ver 3.1.101)
      avresample:                NO
    GStreamer:                   
      base:                      YES (ver 1.8.3)
      video:                     YES (ver 1.8.3)
      app:                       YES (ver 1.8.3)
      riff:                      YES (ver 1.8.3)
      pbutils:                   YES (ver 1.8.3)
    OpenNI:                      NO
    OpenNI PrimeSensor Modules:  NO
    OpenNI2:                     NO
    PvAPI:                       NO
    GigEVisionSDK:               NO
    Aravis SDK:                  NO
    UniCap:                      NO
    UniCap ucil:                 NO
    V4L/V4L2:                    YES/YES
    XIMEA:                       NO
    Xine:                        NO
    gPhoto2:                     YES

  Parallel framework:            TBB (ver 4.4 interface 9002)

  Other third-party libraries:
    Use IPP:                     9.0.1 [9.0.1]
         at:                     /mnt/sdb/ubuntu/install/opencv/build/3rdparty/ippicv/ippicv_lnx
    Use IPP Async:               NO
    Use VA:                      NO
    Use Intel VA-API/OpenCL:     NO
    Use Lapack:                  YES (/usr/lib/liblapack.so /usr/lib/libcblas.so /usr/lib/libatlas.so)
    Use Eigen:                   NO
    Use Cuda:                    YES (ver 8.0)
    Use OpenCL:                  YES
    Use OpenVX:                  NO
    Use custom HAL:              NO

  NVIDIA CUDA
    Use CUFFT:                   YES
    Use CUBLAS:                  YES
    USE NVCUVID:                 NO
    NVIDIA GPU arch:             20 30 35 37 50 52 60 61
    NVIDIA PTX archs:
    Use fast math:               NO

  OpenCL:                        
    Include path:                /mnt/sdb/ubuntu/install/opencv/opencv-3.2.0/3rdparty/include/opencl/1.2
    Use AMDFFT:                  NO
    Use AMDBLAS:                 NO

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.1)
    packages path:               lib/python2.7/dist-packages

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.5.2)

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        Matlab not found or implicitly disabled

  Documentation:
    Doxygen:                     /usr/bin/doxygen (ver 1.8.11)

  Tests and samples:
    Tests:                       YES
    Performance tests:           YES
    C/C++ Examples:              NO

  Install path:                  /usr/local

  cvconfig.h is in:              /mnt/sdb/ubuntu/install/opencv/build
-----------------------------------------------------------------

Configuring done

开始的部份会检查一堆依赖，请仔细看如可有缺包，可以百度一下怎么安装，因为我已经装完了就不提示了，

不过有些包我也找不到我就没解决了如上面的

Checking for module 'libucil'
  No package 'libucil' found

这个我还真没找到怎么解决，不过反正问题不大

再来下面的部份一定要注意几个部分一定要有

  GUI: 
    GTK+ 2.x:                    YES (ver 2.24.30)

Video I/O:
    V4L/V4L2:                    YES/YES
    FFMPEG:                      YES
  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.1)
    packages path:               lib/python2.7/dist-packages

主要就是上面几个个非常重要，必須要装上

再来就正常 make

$ make -j8
$ make install

设置nccl的ld环境

$  vi /etc/ld.so.conf.d/nccl.conf

加上下面目录

/usr/local/nccl/lib/

然后执行

$ sudo ldconfig

安装完后回python按步骤一再次检查是否环境都好使了


二、撰写Python测试程序

# -*- coding:utf-8 -*-
# 用于模型的单张图像分类操作
import os
os.environ['GLOG_minloglevel'] = '2' # 将caffe的输出log信息不显示，必须放到import caffe前
import caffe # caffe 模块
from caffe.proto import caffe_pb2
from google.protobuf import text_format
import numpy as np
import cv2
import matplotlib.pyplot as plt
import time
import skimage.io

global num
num = 0



def detect(image1,net):
    # 传进来的image1的dtype为uint8
    # print image1.shape
    # print image1.dtype
    # print image1.size

    # image = np.array(image1, dtype=np.float32)
    # image = caffe.io.resize_image(image1, (480, 640))
    image = skimage.img_as_float(image1).astype(np.float32)
    # image = caffe.io.resize_image(image2, (300, 300))

    # skimage.io.imsave("photo.png", image)
    # cv2.imwrite("photo.png", image)
    # image = caffe.io.load_image(caffe_root + 'examples/images/bird.jpg')
    # 以下方式读取的imaged的dtype为float32
    # image = caffe.io.load_image(caffe_root + 'photo.png')
    # image = caffe.io.load_image(image1)

    # 改变dtype
    # image.dtype = 'float32'
    # print 'mode:'+image.mode
    # print image.shape
    # print image.dtype
    # print image.size

    # plt.imshow(image)

    # * Run the net and examine the top_k results
    # In[5]:
    global num
    num += 1
    print 'image num:' + str(num)

    transformed_image = transformer.preprocess('data', image)
    net.blobs['data'].data[...] = transformed_image

    time_start=time.time()
    # Forward pass.
    net.forward()
	
    time_end=time.time() 
    print 'time:' + str(time_end-time_start) + ' s'



    loc = net.blobs['bbox-list'].data[0]
    print(loc)
    #查看了结构文件发现在CAFFE一开始图像输入的时候就已经将图片缩小了，宽度1248高度384
    #然后我们在net.blobs['bbox-list'].data得到的是侦测到的目标座标，但是是相对于1248*384的
    #所以我们要把座标转换回相对原大小的位置，下面im.shape是保存在原尺寸的宽高，
    for l in range(len(loc)):
		xmin = int(loc[l][0] * image.shape[1] / 1248)
		ymin = int(loc[l][1] * image.shape[0] / 384)
		xmax = int(loc[l][2] * image.shape[1] /1248)
		ymax = int(loc[l][3] * image.shape[0] / 384)
		#在该座标位置画一个方框
		cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (55 / 255.0, 255 / 255.0, 155 / 255.0), 2)
    # 显示结果
    
    #plt.imshow(image, 'brg')
    #plt.show()
    cv2.imshow('img', image)
    


def show_info(cam):
    print 'POS_FRAMES:'+str(cam.get(1))
    print 'FRAME_COUNT:'+str(cam.get(7))
    print 'FORMAT:'+str(cam.get(8))
    print 'MODE:'+str(cam.get(9))
    print 'SATURATION:'+str(cam.get(12))
    print 'FPS:'+str(cam.get(5))

#CPU或GPU模型转换
caffe.set_mode_gpu()
#caffe.set_mode_cpu()
#caffe.set_device(0)

caffe_root = '/var/smb/work/mycode/'
# 网络参数（权重）文件
caffemodel = caffe_root + 'module/detectnet/snapshot_iter_2391.caffemodel'
# 网络实施结构配置文件
deploy = caffe_root + 'module/detectnet/deploy.prototxt'


img_root = caffe_root + 'data/'

# 网络实施分类
net = caffe.Net(deploy,  # 定义模型结构
                caffemodel,  # 包含了模型的训练权值
                caffe.TEST)  # 使用测试模式(不执行dropout)

# 加载ImageNet图像均值 (随着Caffe一起发布的)
print(os.environ['PYTHONPATH'])
#mu = np.load(os.environ['PYTHONPATH'] + '/caffe/imagenet/ilsvrc_2012_mean.npy')
#mu = mu.mean(1).mean(1)  # 对所有像素值取平均以此获取BGR的均值像素值

# 图像预处理
transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape})
transformer.set_transpose('data', (2,0,1))
#transformer.set_mean('data', mu)
transformer.set_raw_scale('data', 255)
transformer.set_channel_swap('data', (2,1,0))

# 处理图像
cam = cv2.VideoCapture(0) 
if cam.isOpened():
    cam.set(3, 400)
    cam.set(4, 300)
    cam.set(5, 3)
    time.sleep(6)
    cam.set(15, -8.0)
    size = (int(cam.get(3)), int(cam.get(4)))
    print 'size:'
    print size

cv2.namedWindow('img', cv2.WINDOW_NORMAL)

# cnt=2
# while cnt:
#     cnt -= 1
while cam.isOpened():
    ret, img = cam.read()
    if ret:
        #show_info(cam)
        detect(img,net)

    if 0xFF == ord('q') & cv2.waitKey(5) == 27:
        break
    # time.sleep(0.033)
cam.release()
cv2.destroyAllWindows()

介面上会打印bbox也就是侦测到的目标在图像的座标，另外请自行修改python代码里的相关目录，

我用自己训练的KITTI数据集，用于侦测车辆，因为拍不到车子拿手机欺骗一下，好使