ROS kinetic 机器视觉

ROS 中的图像数据
二维图像
 显示图像类型
kinetic@vm:~$ roslaunch usb_cam usb_cam-test.launch
kinetic@vm:~$ rostopic list
/image_view/output
/image_view/parameter_descriptions
/image_view/parameter_updates
/rosout
/rosout_agg
/usb_cam/camera_info
/usb_cam/image_raw
/usb_cam/image_raw/compressed
/usb_cam/image_raw/compressed/parameter_descriptions
/usb_cam/image_raw/compressed/parameter_updates
/usb_cam/image_raw/compressedDepth
/usb_cam/image_raw/compressedDepth/parameter_descriptions
/usb_cam/image_raw/compressedDepth/parameter_updates
/usb_cam/image_raw/theora
/usb_cam/image_raw/theora/parameter_descriptions
/usb_cam/image_raw/theora/parameter_updates
kinetic@vm:~$ rostopic info /usb_cam/image_raw
Type: sensor_msgs/Image

Publishers: 
 * /usb_cam (http://vm:34615/)

Subscribers: 
 * /image_view (http://vm:38655/)

kinetic@vm:~$ rosmsg show sensor_msgs/Image 
std_msgs/Header header    //消息头， 包含消息序号，时间戳和绑定坐标关系；
        uint32 seq    // 消息的序号
        time stamp    // 时间戳
        string frame_id    // 绑定坐标关系
uint32 height    // 图像的纵向分辨率；
uint32 width    // 图像的横向分辨率；
string encoding    // 图像的编码格式，包含RGB 、YUV等常用格式，不涉及图像压缩编码；
uint8 is_bigendian    // 图像数据的大小端存储模式；
uint32 step    //一行图像数据的字节数量，作为数据的步长参数；
uint8[] data // 存储图像数据的数组，大小为step * heigh 个字节;

如：1080*720分辨率的摄像头产生一帧图像的数据大小是3*1080*720

查看压缩图像
ubuntu@T430:~$ rqt_image_view
ubuntu@T430:~$ rostopic info /usb_cam/image_raw/compressed
Type: sensor_msgs/CompressedImage
Publishers:
 * /usb_cam (http://T430:40229/)
Subscribers:
 * /rqt_gui_cpp_node_5882 (http://T430:37783/)

图像压缩消息
kinetic@vm:~$ rosmsg show sensor_msgs/CompressedImage 
std_msgs/Header header
        uint32 seq
        time stamp
        string frame_id
string format        // 图像压缩编码格式(jpeg、png、bmp)

uint8[] data        // 存储图像数据的数组

三维图像
显示点云类型
kinetic@vm:~$ roslaunch freenect_launch freenect.launch
ubuntu@T430:~$ rostopic list
/camera/depth/camera_info
/camera/depth/disparity
/camera/depth/image
/camera/depth/image/compressed
/camera/depth/image/compressed/parameter_descriptions
/camera/depth/image/compressed/parameter_updates
/camera/depth/image/compressedDepth
/camera/depth/image/compressedDepth/parameter_descriptions
/camera/depth/image/compressedDepth/parameter_updates
/camera/depth/image/theora
/camera/depth/image/theora/parameter_descriptions
/camera/depth/image/theora/parameter_updates
/camera/depth/image_raw
/camera/depth/image_raw/compressed
/camera/depth/image_raw/compressed/parameter_descriptions
/camera/depth/image_raw/compressed/parameter_updates
/camera/depth/image_raw/compressedDepth
/camera/depth/image_raw/compressedDepth/parameter_descriptions
/camera/depth/image_raw/compressedDepth/parameter_updates
/camera/depth/image_raw/theora
/camera/depth/image_raw/theora/parameter_descriptions
/camera/depth/image_raw/theora/parameter_updates
/camera/depth/image_rect
/camera/depth/image_rect/compressed
/camera/depth/image_rect/compressed/parameter_descriptions
/camera/depth/image_rect/compressed/parameter_updates
/camera/depth/image_rect/compressedDepth
/camera/depth/image_rect/compressedDepth/parameter_descriptions
/camera/depth/image_rect/compressedDepth/parameter_updates
/camera/depth/image_rect/theora
/camera/depth/image_rect/theora/parameter_descriptions
/camera/depth/image_rect/theora/parameter_updates
/camera/depth/image_rect_raw
/camera/depth/image_rect_raw/compressed
/camera/depth/image_rect_raw/compressed/parameter_descriptions
/camera/depth/image_rect_raw/compressed/parameter_updates
/camera/depth/image_rect_raw/compressedDepth
/camera/depth/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth/image_rect_raw/compressedDepth/parameter_updates
/camera/depth/image_rect_raw/theora
/camera/depth/image_rect_raw/theora/parameter_descriptions
/camera/depth/image_rect_raw/theora/parameter_updates
/camera/depth/points
/camera/depth_rectify_depth/parameter_descriptions
/camera/depth_rectify_depth/parameter_updates
/camera/depth_registered/camera_info
/camera/depth_registered/disparity
/camera/depth_registered/hw_registered/image_rect
/camera/depth_registered/hw_registered/image_rect/compressed
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect/compressedDepth
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect/theora
/camera/depth_registered/hw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw
/camera/depth_registered/hw_registered/image_rect_raw/compressed
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/theora
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered/image
/camera/depth_registered/image/compressed
/camera/depth_registered/image/compressed/parameter_descriptions
/camera/depth_registered/image/compressed/parameter_updates
/camera/depth_registered/image/compressedDepth
/camera/depth_registered/image/compressedDepth/parameter_descriptions
/camera/depth_registered/image/compressedDepth/parameter_updates
/camera/depth_registered/image/theora
/camera/depth_registered/image/theora/parameter_descriptions
/camera/depth_registered/image/theora/parameter_updates
/camera/depth_registered/image_raw
/camera/depth_registered/image_raw/compressed
/camera/depth_registered/image_raw/compressed/parameter_descriptions
/camera/depth_registered/image_raw/compressed/parameter_updates
/camera/depth_registered/image_raw/compressedDepth
/camera/depth_registered/image_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/image_raw/compressedDepth/parameter_updates
/camera/depth_registered/image_raw/theora
/camera/depth_registered/image_raw/theora/parameter_descriptions
/camera/depth_registered/image_raw/theora/parameter_updates
/camera/depth_registered/points
/camera/depth_registered/sw_registered/camera_info
/camera/depth_registered/sw_registered/image_rect
/camera/depth_registered/sw_registered/image_rect/compressed
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect/compressedDepth
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect/theora
/camera/depth_registered/sw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw
/camera/depth_registered/sw_registered/image_rect_raw/compressed
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/theora
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered_rectify_depth/parameter_descriptions
/camera/depth_registered_rectify_depth/parameter_updates
/camera/driver/parameter_descriptions
/camera/driver/parameter_updates
/camera/ir/camera_info
/camera/ir/image_raw
/camera/ir/image_raw/compressed
/camera/ir/image_raw/compressed/parameter_descriptions
/camera/ir/image_raw/compressed/parameter_updates
/camera/ir/image_raw/compressedDepth
/camera/ir/image_raw/compressedDepth/parameter_descriptions
/camera/ir/image_raw/compressedDepth/parameter_updates
/camera/ir/image_raw/theora
/camera/ir/image_raw/theora/parameter_descriptions
/camera/ir/image_raw/theora/parameter_updates
/camera/ir/image_rect_ir
/camera/ir/image_rect_ir/compressed
/camera/ir/image_rect_ir/compressed/parameter_descriptions
/camera/ir/image_rect_ir/compressed/parameter_updates
/camera/ir/image_rect_ir/compressedDepth
/camera/ir/image_rect_ir/compressedDepth/parameter_descriptions
/camera/ir/image_rect_ir/compressedDepth/parameter_updates
/camera/ir/image_rect_ir/theora
/camera/ir/image_rect_ir/theora/parameter_descriptions
/camera/ir/image_rect_ir/theora/parameter_updates
/camera/ir_rectify_ir/parameter_descriptions
/camera/ir_rectify_ir/parameter_updates
/camera/projector/camera_info
/camera/rgb/camera_info
/camera/rgb/image_color
/camera/rgb/image_color/compressed
/camera/rgb/image_color/compressed/parameter_descriptions
/camera/rgb/image_color/compressed/parameter_updates
/camera/rgb/image_color/compressedDepth
/camera/rgb/image_color/compressedDepth/parameter_descriptions
/camera/rgb/image_color/compressedDepth/parameter_updates
/camera/rgb/image_color/theora
/camera/rgb/image_color/theora/parameter_descriptions
/camera/rgb/image_color/theora/parameter_updates
/camera/rgb/image_mono
/camera/rgb/image_mono/compressed
/camera/rgb/image_mono/compressed/parameter_descriptions
/camera/rgb/image_mono/compressed/parameter_updates
/camera/rgb/image_mono/compressedDepth
/camera/rgb/image_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_mono/compressedDepth/parameter_updates
/camera/rgb/image_mono/theora
/camera/rgb/image_mono/theora/parameter_descriptions
/camera/rgb/image_mono/theora/parameter_updates
/camera/rgb/image_raw
/camera/rgb/image_raw/compressed
/camera/rgb/image_raw/compressed/parameter_descriptions
/camera/rgb/image_raw/compressed/parameter_updates
/camera/rgb/image_raw/compressedDepth
/camera/rgb/image_raw/compressedDepth/parameter_descriptions
/camera/rgb/image_raw/compressedDepth/parameter_updates
/camera/rgb/image_raw/theora
/camera/rgb/image_raw/theora/parameter_descriptions
/camera/rgb/image_raw/theora/parameter_updates
/camera/rgb/image_rect_color
/camera/rgb/image_rect_color/compressed
/camera/rgb/image_rect_color/compressed/parameter_descriptions
/camera/rgb/image_rect_color/compressed/parameter_updates
/camera/rgb/image_rect_color/compressedDepth
/camera/rgb/image_rect_color/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_color/compressedDepth/parameter_updates
/camera/rgb/image_rect_color/theora
/camera/rgb/image_rect_color/theora/parameter_descriptions
/camera/rgb/image_rect_color/theora/parameter_updates
/camera/rgb/image_rect_mono
/camera/rgb/image_rect_mono/compressed
/camera/rgb/image_rect_mono/compressed/parameter_descriptions
/camera/rgb/image_rect_mono/compressed/parameter_updates
/camera/rgb/image_rect_mono/compressedDepth
/camera/rgb/image_rect_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_mono/compressedDepth/parameter_updates
/camera/rgb/image_rect_mono/theora
/camera/rgb/image_rect_mono/theora/parameter_descriptions
/camera/rgb/image_rect_mono/theora/parameter_updates
/camera/rgb_debayer/parameter_descriptions
/camera/rgb_debayer/parameter_updates
/camera/rgb_rectify_color/parameter_descriptions
/camera/rgb_rectify_color/parameter_updates
/camera/rgb_rectify_mono/parameter_descriptions
/camera/rgb_rectify_mono/parameter_updates
/diagnostics
/rosout
/rosout_agg
/tf
/tf_static
kinetic@vm:~$
Type: sensor_msgs/PointCloud2
Publishers: 
 * /camera/camera_nodelet_manager (http://vm:40763/)
Subscribers: None
kinetic@vm:~$
std_msgs/Header header
        uint32 seq
        time stamp
        string frame_id
uint32 height    // 点云图像的纵向分辨率
uint32 width       // 点云图像的横向分辨率
sensor_msgs/PointField[] fields    // 每个点的数据类型
        uint8 INT8=1        // 定义了一些常量宏
        uint8 UINT8=2
        uint8 INT16=3
        uint8 UINT16=4
        uint8 INT32=5
        uint8 UINT32=6
        uint8 FLOAT32=7
        uint8 FLOAT64=8
        string name
        uint32 offset
        uint8 datatype
        uint32 count
bool is_bigendian    // 数据的大小端存储模式
uint32 point_step    // 单点的数据字节步长
uint32 row_step    // 一列数据的字节步长
uint8[] data    // 点云数据的存储数组，总字节大小为 row_step * height   
bool is_dense    // 是否有无效点

点云单帧数量很大，如果使用分布式网络传输，需要考虑是否能满足数据的传输要求，或者针对数据进行压缩。

摄像头标定
        摄像头这种精密仪器对光学器件的要求较高，摄像头生成的物体图像往往会发生畸变，为避免数据源造成的误差，需要针对摄像头的参数进行标定。
安装标定功能包
kinetic@vm:~$ sudo apt-get install ros-kinetic-camera-calibration
使用棋盘格标定靶

摄像头标定流程
启动摄像头
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch
启动标定包
kinetic@vm:~$ rosrun camera_calibration cameracalibrator.py --size 8x6 --square 0.024 image:=/usb_cam/image_raw camera:=/usb_cam

参数说明：    
    size ： 标定棋盘格的内部角点个数，这里使用的棋盘格一共有6行，每行有8个内部角点；
    square：这个参数对应每个棋盘格的边长，单位是米；一个小正方形的边长为0.024m
    image 和 camera：设置摄像头发布的图像话题

X：标定靶在摄像头视野中左右移动；
Y：标定靶在摄像头视野中上下移动；
Size：标定靶在摄像视野中前后移动；
SKew：标定靶在摄像头视野中倾斜转动

都变为绿色，表示满足标定需求，
点击CALIBRATE ，进行计算参数

点击SAVE进行保存，路径 ('Wrote calibration data to', '/tmp/calibrationdata.tar.gz') , 解压 提取出标定文件 ost.yaml 拷贝到功能包下面。
解压之后，出现标定采集的图像和 ost.txt 、ost.yaml 文件。我们需要使用ost.yaml。
ubuntu@T430:/tmp$ cp ost.yaml ~/catkin_ws/src/robot_vision/camera_calibration.yaml

kinect标定流程
启动Kinect
kinetic@vm:~$ roslaunch robot_vision freenect.launch
启动彩色摄像头
kinetic@vm:~$ rosrun camera_calibration cameracalibrator.py image:=/camera/rgb/image_raw camera:=/camera/rgb --size 8x6 --square 0.024
标定红外摄像头
kinetic@vm:~$ rosrun camera_calibration cameracalibrator.py image:=/camera/ir/image_raw camera:=/camera/ir --size 8x6 --square 0.024

使用标定文件
摄像头

camera_calibration.yaml" />

Kinect
kinect_rgb_calibration.yaml" />

kinect_depth_calibration.yaml" />

使用标定文件可能产生的错误

原因 : 标定文件中的 camera_name 参数与实际传感器名称不匹配
比如根据上图所示的告警，分别将两个标定文件中的 camera_name 参数修改为 rgb_xxxxxxxxxxxx  depth_xxxxxxxxxxxxx
image_width: 640
image_height: 480
camera_name: head_camera

camera_matrix:

ROS+OpenCV 应用实例（人脸识别、物体跟踪）
        OpenCV ( Open Source Computer Vision Library ) 基于BSD许可发行的跨平台开源计算机视觉库，由一系列C 函数和C++类构成，同时提供C++、Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法，而且对非商业应用和商业应用都是免费的，可以直接访问硬件摄像头， 可以直接访问硬件摄像头，并且还提供了一个简单的GUI系统--highgui。
安装OpenCV
kinetic@vm:~$ sudo apt-get install ros-kinetic-vision-opencv libopencv-dev python-opencv
ROS 与 OpenCV 集成框架

运行
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch 
kinetic@vm:~$ rosrun robot_vision cv_bridge_test.py
kinetic@vm:~$ rqt_image_view

OpenCV 直接显示：

ROS 通过话题显示：

imgmsg_to_cv2()： 将ROS图像消息转换成OpenCV图像数据；
cv2_to_imgmsg()： 将OpenCV格式的图像数据转换成ROS图像消息；
输入参数：图像消息流  和 转换的图像数据格式

基于Haar特征的级联分类器对象检测算法

初始化
    完成ROS节点、图识别参数的设置

ROS 图像回调函数
    将图像转换成OpenCV的数据格式，然后预处理之后开始调用人脸识别的功能函数，最后把识别结果发布
    imgmsg_to_cv2(...)    将ROS数据格式的图像转化成OpenCV的图像格式
    cv2.cvtColor(...)    创建灰度图像
    cv2.equalizeHist(...)  创建平衡直方图，减少光线影响

人脸识别
    调用OpenCV 提供的人脸识别接口，与数据库中的人脸特征进行匹配

运行
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch
kinetic@vm:~$ roslaunch robot_vision face_detector.launch

kinetic@vm:~$ rqt_image_view 

跟踪物体的特征点

初始化
    完成ROS节点、图识别参数的设置

图像处理
    将图像转换成OpenCV格式；完成图像预处理之后开始针对两帧图像进行比较，基于图像差异识别到运行的物体，最后标识识别结果并发布。

运行
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch
kinetic@vm:~$ roslaunch robot_vision motion_detector.launch 

kinetic@vm:~$ rqt_image_view

二维码识别

参考网址：wiki.ros.org/ar_track_alvar

修改更新源，阿里云 找不到源。
修改为如下源：
deb http://ros.exbot.net/rospackage/ros/ubuntu xenial main

安装二维码识别功能包
kinetic@vm:~$ sudo apt-get install ros-kinetic-ar-track-alvar

安装路径
kinetic@vm:~$ ls /opt/ros/kinetic/share/ar_track_alvar
bundles  cmake  launch  pa

创建二维码
kinetic@vm:~$ roscore
kinetic@vm:~$ rosrun ar_track_alvar createMarker
如：kinetic@vm:~$ rosrun ar_track_alvar createMarker 0

配置：
kinetic@vm:~$ roscd robot_vision/config
kinetic@vm:~/catkin_ws/src/robot_vision/config$ ls
ar_track_camera.rviz  ar_track_kinect.rviz

创建大小为5cm的二维码，内容为 0 和 1
kinetic@vm:~$ rosrun ar_track_alvar createMarker -s 5 0
kinetic@vm:~$ rosrun ar_track_alvar createMarker -s 5 1
生成的信息保存在 home 目录下

启动摄像头二维码识别示例
摄像头校准文件启动
kinetic@vm:~$ roslaunch robot_vision usb_cam_with_calibration.launch 
kinetic@vm:~$ roslaunch robot_vision ar_track_camera.launch 

启动摄像头，需要加载标定文件，否则可能无法识别二维码。
看到是一个三维的，摄像头达到测距的效果是通过二位大小变化实现的

查看识别到的二维码位姿
kinetic@vm:~$ rostopic echo /ar_pose_marker

启动kinect二维码识别示例
kinetic@vm:~$ roslaunch robot_vision freenect.launch  
kinetic@vm:~$ roslaunch robot_vision ar_track_kinect.launch 

扩展内容：物体识别与机器学习
Object Recognition Kitchen (ORK)
TensorFlow Object Detection API

参考：
ROS cv_bridge wiki :  http://wiki.ros.org/cv_bridge
ROS opencv_apps :  http://wiki.ros.org/opencv_apps