ROS kinetic 机器视觉

ROS 中的图像数据
二维图像
 显示图像类型

kinetic@vm:~$ roslaunch usb_cam usb_cam-test.launch
kinetic@vm:~$ rostopic list

/image_view/output
/image_view/parameter_descriptions
/image_view/parameter_updates
/rosout
/rosout_agg
/usb_cam/camera_info
/usb_cam/image_raw
/usb_cam/image_raw/compressed
/usb_cam/image_raw/compressed/parameter_descriptions
/usb_cam/image_raw/compressed/parameter_updates
/usb_cam/image_raw/compressedDepth
/usb_cam/image_raw/compressedDepth/parameter_descriptions
/usb_cam/image_raw/compressedDepth/parameter_updates
/usb_cam/image_raw/theora
/usb_cam/image_raw/theora/parameter_descriptions
/usb_cam/image_raw/theora/parameter_updates
kinetic@vm:~$ rostopic info /usb_cam/image_raw
Type: sensor_msgs/Image

Publishers: 
 * /usb_cam (http://vm:34615/)

Subscribers: 
 * /image_view (http://vm:38655/)

kinetic@vm:~$ rosmsg show sensor_msgs/Image 
std_msgs/Header header    //消息头, 包含消息序号,时间戳和绑定坐标关系;
        uint32 seq    // 消息的序号
        time stamp    // 时间戳
        string frame_id    // 绑定坐标关系
uint32 height    // 图像的纵向分辨率;
uint32 width    // 图像的横向分辨率;
string encoding    // 图像的编码格式,包含RGB 、YUV等常用格式,不涉及图像压缩编码;
uint8 is_bigendian    // 图像数据的大小端存储模式;
uint32 step    //一行图像数据的字节数量,作为数据的步长参数;
uint8[] data // 存储图像数据的数组,大小为step * heigh 个字节;

如:1080*720分辨率的摄像头产生一帧图像的数据大小是3*1080*720

查看压缩图像
ubuntu@T430:~$ rqt_image_view
ubuntu@T430:~$ rostopic info /usb_cam/image_raw/compressed
Type: sensor_msgs/CompressedImage
Publishers:
 * /usb_cam (http://T430:40229/)
Subscribers:
 * /rqt_gui_cpp_node_5882 (http://T430:37783/)

图像压缩消息
kinetic@vm:~$ rosmsg show sensor_msgs/CompressedImage 
std_msgs/Header header
        uint32 seq
        time stamp
        string frame_id
string format        // 图像压缩编码格式(jpeg、png、bmp)

uint8[] data        // 存储图像数据的数组

三维图像
显示点云类型
kinetic@vm:~$ roslaunch freenect_launch freenect.launch
ubuntu@T430:~$ rostopic list
/camera/depth/camera_info
/camera/depth/disparity
/camera/depth/image
/camera/depth/image/compressed
/camera/depth/image/compressed/parameter_descriptions
/camera/depth/image/compressed/parameter_updates
/camera/depth/image/compressedDepth
/camera/depth/image/compressedDepth/parameter_descriptions
/camera/depth/image/compressedDepth/parameter_updates
/camera/depth/image/theora
/camera/depth/image/theora/parameter_descriptions
/camera/depth/image/theora/parameter_updates
/camera/depth/image_raw
/camera/depth/image_raw/compressed
/camera/depth/image_raw/compressed/parameter_descriptions
/camera/depth/image_raw/compressed/parameter_updates
/camera/depth/image_raw/compressedDepth
/camera/depth/image_raw/compressedDepth/parameter_descriptions
/camera/depth/image_raw/compressedDepth/parameter_updates
/camera/depth/image_raw/theora
/camera/depth/image_raw/theora/parameter_descriptions
/camera/depth/image_raw/theora/parameter_updates
/camera/depth/image_rect
/camera/depth/image_rect/compressed
/camera/depth/image_rect/compressed/parameter_descriptions
/camera/depth/image_rect/compressed/parameter_updates
/camera/depth/image_rect/compressedDepth
/camera/depth/image_rect/compressedDepth/parameter_descriptions
/camera/depth/image_rect/compressedDepth/parameter_updates
/camera/depth/image_rect/theora
/camera/depth/image_rect/theora/parameter_descriptions
/camera/depth/image_rect/theora/parameter_updates
/camera/depth/image_rect_raw
/camera/depth/image_rect_raw/compressed
/camera/depth/image_rect_raw/compressed/parameter_descriptions
/camera/depth/image_rect_raw/compressed/parameter_updates
/camera/depth/image_rect_raw/compressedDepth
/camera/depth/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth/image_rect_raw/compressedDepth/parameter_updates
/camera/depth/image_rect_raw/theora
/camera/depth/image_rect_raw/theora/parameter_descriptions
/camera/depth/image_rect_raw/theora/parameter_updates
/camera/depth/points
/camera/depth_rectify_depth/parameter_descriptions
/camera/depth_rectify_depth/parameter_updates
/camera/depth_registered/camera_info
/camera/depth_registered/disparity
/camera/depth_registered/hw_registered/image_rect
/camera/depth_registered/hw_registered/image_rect/compressed
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect/compressedDepth
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect/theora
/camera/depth_registered/hw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw
/camera/depth_registered/hw_registered/image_rect_raw/compressed
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/theora
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered/image
/camera/depth_registered/image/compressed
/camera/depth_registered/image/compressed/parameter_descriptions
/camera/depth_registered/image/compressed/parameter_updates
/camera/depth_registered/image/compressedDepth
/camera/depth_registered/image/compressedDepth/parameter_descriptions
/camera/depth_registered/image/compressedDepth/parameter_updates
/camera/depth_registered/image/theora
/camera/depth_registered/image/theora/parameter_descriptions
/camera/depth_registered/image/theora/parameter_updates
/camera/depth_registered/image_raw
/camera/depth_registered/image_raw/compressed
/camera/depth_registered/image_raw/compressed/parameter_descriptions
/camera/depth_registered/image_raw/compressed/parameter_updates
/camera/depth_registered/image_raw/compressedDepth
/camera/depth_registered/image_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/image_raw/compressedDepth/parameter_updates
/camera/depth_registered/image_raw/theora
/camera/depth_registered/image_raw/theora/parameter_descriptions
/camera/depth_registered/image_raw/theora/parameter_updates
/camera/depth_registered/points
/camera/depth_registered/sw_registered/camera_info
/camera/depth_registered/sw_registered/image_rect
/camera/depth_registered/sw_registered/image_rect/compressed
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect/compressedDepth
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect/theora
/camera/depth_registered/sw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw
/camera/depth_registered/sw_registered/image_rect_raw/compressed
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/theora
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered_rectify_depth/parameter_descriptions
/camera/depth_registered_rectify_depth/parameter_updates
/camera/driver/parameter_descriptions
/camera/driver/parameter_updates
/camera/ir/camera_info
/camera/ir/image_raw
/camera/ir/image_raw/compressed
/camera/ir/image_raw/compressed/parameter_descriptions
/camera/ir/image_raw/compressed/parameter_updates
/camera/ir/image_raw/compressedDepth
/camera/ir/image_raw/compressedDepth/parameter_descriptions
/camera/ir/image_raw/compressedDepth/parameter_updates
/camera/ir/image_raw/theora
/camera/ir/image_raw/theora/parameter_descriptions
/camera/ir/image_raw/theora/parameter_updates
/camera/ir/image_rect_ir
/camera/ir/image_rect_ir/compressed
/camera/ir/image_rect_ir/compressed/parameter_descriptions
/camera/ir/image_rect_ir/compressed/parameter_updates
/camera/ir/image_rect_ir/compressedDepth
/camera/ir/image_rect_ir/compressedDepth/parameter_descriptions
/camera/ir/image_rect_ir/compressedDepth/parameter_updates
/camera/ir/image_rect_ir/theora
/camera/ir/image_rect_ir/theora/parameter_descriptions
/camera/ir/image_rect_ir/theora/parameter_updates
/camera/ir_rectify_ir/parameter_descriptions
/camera/ir_rectify_ir/parameter_updates
/camera/projector/camera_info
/camera/rgb/camera_info
/camera/rgb/image_color
/camera/rgb/image_color/compressed
/camera/rgb/image_color/compressed/parameter_descriptions
/camera/rgb/image_color/compressed/parameter_updates
/camera/rgb/image_color/compressedDepth
/camera/rgb/image_color/compressedDepth/parameter_descriptions
/camera/rgb/image_color/compressedDepth/parameter_updates
/camera/rgb/image_color/theora
/camera/rgb/image_color/theora/parameter_descriptions
/camera/rgb/image_color/theora/parameter_updates
/camera/rgb/image_mono
/camera/rgb/image_mono/compressed
/camera/rgb/image_mono/compressed/parameter_descriptions
/camera/rgb/image_mono/compressed/parameter_updates
/camera/rgb/image_mono/compressedDepth
/camera/rgb/image_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_mono/compressedDepth/parameter_updates
/camera/rgb/image_mono/theora
/camera/rgb/image_mono/theora/parameter_descriptions
/camera/rgb/image_mono/theora/parameter_updates
/camera/rgb/image_raw
/camera/rgb/image_raw/compressed
/camera/rgb/image_raw/compressed/parameter_descriptions
/camera/rgb/image_raw/compressed/parameter_updates
/camera/rgb/image_raw/compressedDepth
/camera/rgb/image_raw/compressedDepth/parameter_descriptions
/camera/rgb/image_raw/compressedDepth/parameter_updates
/camera/rgb/image_raw/theora
/camera/rgb/image_raw/theora/parameter_descriptions
/camera/rgb/image_raw/theora/parameter_updates
/camera/rgb/image_rect_color
/camera/rgb/image_rect_color/compressed
/camera/rgb/image_rect_color/compressed/parameter_descriptions
/camera/rgb/image_rect_color/compressed/parameter_updates
/camera/rgb/image_rect_color/compressedDepth
/camera/rgb/image_rect_color/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_color/compressedDepth/parameter_updates
/camera/rgb/image_rect_color/theora
/camera/rgb/image_rect_color/theora/parameter_descriptions
/camera/rgb/image_rect_color/theora/parameter_updates
/camera/rgb/image_rect_mono
/camera/rgb/image_rect_mono/compressed
/camera/rgb/image_rect_mono/compressed/parameter_descriptions
/camera/rgb/image_rect_mono/compressed/parameter_updates
/camera/rgb/image_rect_mono/compressedDepth
/camera/rgb/image_rect_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_mono/compressedDepth/parameter_updates
/camera/rgb/image_rect_mono/theora
/camera/rgb/image_rect_mono/theora/parameter_descriptions
/camera/rgb/image_rect_mono/theora/parameter_updates
/camera/rgb_debayer/parameter_descriptions
/camera/rgb_debayer/parameter_updates
/camera/rgb_rectify_color/parameter_descriptions
/camera/rgb_rectify_color/parameter_updates
/camera/rgb_rectify_mono/parameter_descriptions
/camera/rgb_rectify_mono/parameter_updates
/diagnostics
/rosout
/rosout_agg
/tf
/tf_static

kinetic@vm:~$
Type: sensor_msgs/PointCloud2
Publishers: 
 * /camera/camera_nodelet_manager (http://vm:40763/)
Subscribers: None
kinetic@vm:~$
std_msgs/Header header
        uint32 seq
        time stamp
        string frame_id
uint32 height    // 点云图像的纵向分辨率
uint32 width       // 点云图像的横向分辨率
sensor_msgs/PointField[] fields    // 每个点的数据类型
        uint8 INT8=1        // 定义了一些常量宏
        uint8 UINT8=2
        uint8 INT16=3
        uint8 UINT16=4
        uint8 INT32=5
        uint8 UINT32=6
        uint8 FLOAT32=7
        uint8 FLOAT64=8
        string name
        uint32 offset
        uint8 datatype
        uint32 count
bool is_bigendian    // 数据的大小端存储模式
uint32 point_step    // 单点的数据字节步长
uint32 row_step    // 一列数据的字节步长
uint8[] data    // 点云数据的存储数组,总字节大小为 row_step * height   
bool is_dense    // 是否有无效点

点云单帧数量很大,如果使用分布式网络传输,需要考虑是否能满足数据的传输要求,或者针对数据进行压缩。

摄像头标定
       
摄像头这种精密仪器对光学器件的要求较高,摄像头生成的物体图像往往会发生畸变,为避免数据源造成的误差,需要针对摄像头的参数进行标定。
安装标定功能包
kinetic@vm:~$ sudo apt-get install ros-kinetic-camera-calibration
使用棋盘格标定靶

ROS kinetic 机器视觉_第1张图片

摄像头标定流程
启动摄像头
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch
启动标定包
kinetic@vm:~$ rosrun camera_calibration cameracalibrator.py --size 8x6 --square 0.024 image:=/usb_cam/image_raw camera:=/usb_cam

参数说明:    
    size : 标定棋盘格的内部角点个数,这里使用的棋盘格一共有6行,每行有8个内部角点;
    square:这个参数对应每个棋盘格的边长,单位是米;一个小正方形的边长为0.024m
    image 和 camera:设置摄像头发布的图像话题

ROS kinetic 机器视觉_第2张图片
X:标定靶在摄像头视野中左右移动;
Y:标定靶在摄像头视野中上下移动;
Size:标定靶在摄像视野中前后移动;
SKew:标定靶在摄像头视野中倾斜转动

都变为绿色,表示满足标定需求,
点击CALIBRATE ,进行计算参数
ROS kinetic 机器视觉_第3张图片
点击SAVE进行保存,路径 ('Wrote calibration data to', '/tmp/calibrationdata.tar.gz') , 解压 提取出标定文件 ost.yaml 拷贝到功能包下面。
解压之后,出现标定采集的图像和 ost.txt 、ost.yaml 文件。我们需要使用ost.yaml。
ubuntu@T430:/tmp$ cp ost.yaml ~/catkin_ws/src/robot_vision/camera_calibration.yaml

kinect标定流程
启动Kinect
kinetic@vm:~$ roslaunch robot_vision freenect.launch
启动彩色摄像头
kinetic@vm:~$ rosrun camera_calibration cameracalibrator.py image:=/camera/rgb/image_raw camera:=/camera/rgb --size 8x6 --square 0.024
标定红外摄像头
kinetic@vm:~$ rosrun camera_calibration cameracalibrator.py image:=/camera/ir/image_raw camera:=/camera/ir --size 8x6 --square 0.024

使用标定文件
摄像头

camera_calibration.yaml" />

Kinect
kinect_rgb_calibration.yaml" />

kinect_depth_calibration.yaml" />

使用标定文件可能产生的错误

原因 : 标定文件中的 camera_name 参数与实际传感器名称不匹配
比如根据上图所示的告警,分别将两个标定文件中的 camera_name 参数修改为 rgb_xxxxxxxxxxxx  depth_xxxxxxxxxxxxx
image_width: 640
image_height: 480
camera_name: head_camera

camera_matrix:

ROS+OpenCV 应用实例(人脸识别、物体跟踪)
        OpenCV ( Open Source Computer Vision Library ) 基于BSD许可发行的跨平台开源计算机视觉库,由一系列C 函数和C++类构成,同时提供C++、Python、Ruby、MATLAB等语言的接口,实现了图像处理和计算机视觉方面的很多通用算法,而且对非商业应用和商业应用都是免费的,可以直接访问硬件摄像头, 可以直接访问硬件摄像头,并且还提供了一个简单的GUI系统--highgui。
安装OpenCV
kinetic@vm:~$ sudo apt-get install ros-kinetic-vision-opencv libopencv-dev python-opencv
ROS 与 OpenCV 集成框架

ROS kinetic 机器视觉_第4张图片

运行
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch 
kinetic@vm:~$ rosrun robot_vision cv_bridge_test.py
kinetic@vm:~$ rqt_image_view

OpenCV 直接显示:

ROS kinetic 机器视觉_第5张图片

ROS 通过话题显示:

ROS kinetic 机器视觉_第6张图片

imgmsg_to_cv2(): 将ROS图像消息转换成OpenCV图像数据;
cv2_to_imgmsg(): 将OpenCV格式的图像数据转换成ROS图像消息;
输入参数:图像消息流  和 转换的图像数据格式

基于Haar特征的级联分类器对象检测算法

ROS kinetic 机器视觉_第7张图片

初始化
    完成ROS节点、图识别参数的设置

ROS 图像回调函数
    将图像转换成OpenCV的数据格式,然后预处理之后开始调用人脸识别的功能函数,最后把识别结果发布
    imgmsg_to_cv2(...)    将ROS数据格式的图像转化成OpenCV的图像格式
    cv2.cvtColor(...)    创建灰度图像
    cv2.equalizeHist(...)  创建平衡直方图,减少光线影响

人脸识别
    调用OpenCV 提供的人脸识别接口,与数据库中的人脸特征进行匹配

运行
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch
kinetic@vm:~$ roslaunch robot_vision face_detector.launch

kinetic@vm:~$ rqt_image_view 

跟踪物体的特征点

ROS kinetic 机器视觉_第8张图片

初始化
    完成ROS节点、图识别参数的设置

图像处理
    将图像转换成OpenCV格式;完成图像预处理之后开始针对两帧图像进行比较,基于图像差异识别到运行的物体,最后标识识别结果并发布。

运行
kinetic@vm:~$ roslaunch robot_vision usb_cam.launch
kinetic@vm:~$ roslaunch robot_vision motion_detector.launch 

kinetic@vm:~$ rqt_image_view

二维码识别

参考网址:wiki.ros.org/ar_track_alvar

修改更新源,阿里云 找不到源。
修改为如下源:
deb http://ros.exbot.net/rospackage/ros/ubuntu xenial main

ROS kinetic 机器视觉_第9张图片

ROS kinetic 机器视觉_第10张图片
安装二维码识别功能包
kinetic@vm:~$ sudo apt-get install ros-kinetic-ar-track-alvar

安装路径
kinetic@vm:~$ ls /opt/ros/kinetic/share/ar_track_alvar
bundles  cmake  launch  pa

创建二维码
kinetic@vm:~$ roscore
kinetic@vm:~$ rosrun ar_track_alvar createMarker
如:kinetic@vm:~$ rosrun ar_track_alvar createMarker 0

配置:
kinetic@vm:~$ roscd robot_vision/config
kinetic@vm:~/catkin_ws/src/robot_vision/config$ ls
ar_track_camera.rviz  ar_track_kinect.rviz

创建大小为5cm的二维码,内容为 0 和 1
kinetic@vm:~$ rosrun ar_track_alvar createMarker -s 5 0
kinetic@vm:~$ rosrun ar_track_alvar createMarker -s 5 1
生成的信息保存在 home 目录下

启动摄像头二维码识别示例
摄像头校准文件启动
kinetic@vm:~$ roslaunch robot_vision usb_cam_with_calibration.launch 
kinetic@vm:~$ roslaunch robot_vision ar_track_camera.launch 

启动摄像头,需要加载标定文件,否则可能无法识别二维码。
看到是一个三维的,摄像头达到测距的效果是通过二位大小变化实现的

查看识别到的二维码位姿
kinetic@vm:~$ rostopic echo /ar_pose_marker

启动kinect二维码识别示例
kinetic@vm:~$ roslaunch robot_vision freenect.launch  
kinetic@vm:~$ roslaunch robot_vision ar_track_kinect.launch 

扩展内容:物体识别与机器学习
Object Recognition Kitchen (ORK)
TensorFlow Object Detection API


参考:
ROS cv_bridge wiki :  http://wiki.ros.org/cv_bridge
ROS opencv_apps :  http://wiki.ros.org/opencv_apps

你可能感兴趣的:(ROS)