OpenCV(opencv_apps)在ROS中的视频图像的应用（重点讲解哈里斯角点的检测）
1、引言
通过opencv_apps，你可以在ROS中以最简单的方式运行OpenCV提供的许多功能，也就是说，运行一个与功能相对应的launch启动文件，就可以跳过为OpenCV的许多功能编写OpenCV应用程序代码，非常的方便。
对于想熟悉每个细节的伙伴们，可以去看源码，对于熟悉视觉操作很有帮助。
官方说明：http://wiki.ros.org/opencv_apps
github源码：https://github.com/ros-perception/opencv_apps
2、启动操作
2.1、opencv_apps.launch
先来启动摄像头等相关操作，需要启动opencv_apps.launch文件，我们先来了解下：
gedit /home/jetson/workspace/catkin_ws/src/jetbot_ros/launch/opencv_apps.launch
这里<include>节点包含一个jetson_csi_cam.launch启动文件以及一个名为img_transform的节点
其中jetson_csi_cam.launch我们查看下里面的内容：
cat /home/jetson/workspace/catkin_ws/src/jetson_nano_csi_cam_ros/launch/jetson_csi_cam.launch里面是一些摄像头的参数设置和启动摄像头，通过GSCAM开源包将GStreamer图像流引入到ROS中，转换成sensor_msgs/Image类型的Image话题，发布到ROS中，供其他节点使用。
2.2、img_transform.py
然后就是做这些操作的节点，我们来看下它的源码：
gedit /home/jetson/workspace/catkin_ws/src/jetbot_ros/scripts/img_transform.py#!/usr/bin/env python
# -*- coding: utf-8 -*-
import rospy
import cv2 as cv
from cv_bridge import CvBridge
from sensor_msgs.msg import Image
def topic(msg):
if not isinstance(msg, Image):
return
bridge = CvBridge()
frame = bridge.imgmsg_to_cv2(msg, "bgr8")
# Canonical input image size
frame = cv.resize(frame, (640, 480))
if img_flip == True: frame = cv.flip(frame, 1)
# opencv mat -> ros msg
msg = bridge.cv2_to_imgmsg(frame, "bgr8")
pub_img.publish(msg)
if __name__ == '__main__':
rospy.init_node("pub_img", anonymous=False)
img_topic = rospy.get_param("~img_topic", "/csi_cam_0/image_raw")
img_flip = rospy.get_param("~img_flip", False)
sub_img = rospy.Subscriber(img_topic, Image, topic)
pub_img = rospy.Publisher("/image", Image, queue_size=10)
rate = rospy.Rate(2)
rospy.spin()这里就是定义一个pub_img节点，订阅CSI摄像头相关的话题，然后通过Image话题进行发布，供其余节点使用。其中获取参数：rospy.get_param("~img_topic", "/csi_cam_0/image_raw")，如果没有在~img_topic获取到，就使用/csi_cam_0/image_raw默认值。同样的rospy.get_param("~img_flip", False)如果没有获取到~img_flip的值，就默认为False
2.3、打开相机
熟悉摄像头的相关操作之后，我们就来做一些准备工作，开启相机：
roslaunch jetbot_ros opencv_apps.launch
查看方式：rqt_image_view，如下图：
或者网页查看：rosrun web_video_server web_video_server
然后使用IP+端口就可以查看了，如下图：
然后点击里面的话题，就可以看到摄像视频了，如下图：
我们来看下开启了哪些话题：rostopic list
/csi_cam_0/camera_info
/csi_cam_0/image_raw
/csi_cam_0/image_raw/compressed
/csi_cam_0/image_raw/compressed/parameter_descriptions
/csi_cam_0/image_raw/compressed/parameter_updates
/csi_cam_0/image_raw/compressedDepth
/csi_cam_0/image_raw/compressedDepth/parameter_descriptions
/csi_cam_0/image_raw/compressedDepth/parameter_updates
/csi_cam_0/image_raw/theora
/csi_cam_0/image_raw/theora/parameter_descriptions
/csi_cam_0/image_raw/theora/parameter_updates
/image
/rosout
/rosout_agg
是一些关于csi摄像头相关的话题，接下来我们对里面大量的示例做一个演示
3、哈里斯角点
因为我们都是在opencv_apps包里面，所以先来这个所在工作区间的这个包的launch目录里面。
查看下有哪些启动文件：ls /home/jetson/workspace/catkin_ws/src/opencv_apps/launch/
adding_images.launch hough_circles.launch
camshift.launch hough_lines.launch
contour_moments.launch hsv_color_filter.launch
convex_hull.launch lk_flow.launch
corner_harris.launch people_detect.launch
discrete_fourier_transform.launch phase_corr.launch
edge_detection.launch pyramids.launch
face_detection.launch rgb_color_filter.launch
face_recognition.launch segment_objects.launch
fback_flow.launch simple_flow.launch
find_contours.launch smoothing.launch
general_contours.launch threshold.launch
goodfeature_track.launch watershed_segmentation.launch
hls_color_filter.launch
可以看到对图像操作的功能还是挺多的，有霍夫圆、霍夫直线、轮廓矩、LK光流、哈里斯角点、边缘、人物、面部识别等等
3.1、corner_harris.launch
判断某个点是图像中的角点，这里的对角点的检测，我们花多点时间重点来说明下，后面的节点基本就是展示为主。
roslaunch opencv_apps corner_harris.launch
启动如下，使用一张棋盘格式的图片让它识别：
可以看到图片中的角点，有很多的小圆圈给标注着，这里的角点检测的多少和准确度，取决于上面的threshold阈值大小，可以自行调节，调小了，角点数量就会多，相对而言准确度也在下降，调大阈值，角点数量就相应减少，准确度要高。
我们来看下这个启动文件里面的内容：
cat /home/jetson/workspace/catkin_ws/src/opencv_apps/launch/corner_harris.launch
3.2、corner_harris.cpp
前面是一些参数设置，后面是一个node节点部分，其中type="corner_harris"可以知道使用的是C++写的，因为如果是Python写的，就是type="corner_harris.py"这样的形式，当然那里也注释说明了是corner_harris.cpp
其中C++代码文件的地址：ls /home/jetson/workspace/catkin_ws/build/opencv_apps
我们来看下这个哈里斯角点的实际代码：
gedit /home/jetson/workspace/catkin_ws/build/opencv_apps/corner_harris.cpp
/*********************************************************************
* Software License Agreement (BSD License)
*
* Copyright (c) 2016, Kentaro Wada.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials provided
* with the distribution.
* * Neither the name of the Kentaro Wada nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*********************************************************************/
#include
#include
int main(int argc, char **argv)
{
ros::init(argc, argv, "corner_harris", ros::init_options::AnonymousName);
if (ros::names::remap("image") == "image") {
ROS_WARN("Topic 'image' has not been remapped! Typical command-line usage:\n"
"\t$ rosrun image_rotate image_rotate image:= [transport]");
}
// need to use 1 worker thread to prevent freezing in imshow, calling imshow from mutiple threads
//nodelet::Loader manager(false);
ros::param::set("~num_worker_threads", 1); // need to call Loader(bool provide_ros_api = true);
nodelet::Loader manager(true);
nodelet::M_string remappings;
nodelet::V_string my_argv(argv + 1, argv + argc);
my_argv.push_back("--shutdown-on-close"); // Internal
manager.load(ros::this_node::getName(), "opencv_apps/corner_harris", remappings, my_argv);
ros::spin();
return 0;
}
这段代码就是初始化corner_harris节点，image话题的重映射等，然后用圆圈来标注角点，这个操作会用到接下来要讲的一个插件。
3.3、corner_harris_nodelet.cpp
上面的核心代码，会用到Nodelet，关于这个Nodelet插件的解释：
允许用户在ROS节点中添加自定义功能，Nodelet使得开发人员能够将复杂的代码封装到可重用的插件中，这些插件可以像其他ROS节点一样进行部署和通信，开发人员可以编写更加模块化和可维护的代码，提高ROS系统的可扩展性和可重用性。
更关键的是效率问题，Nodelet提供一种方法，可以让多个算法程序在一个进程中，使用共享指针shared_ptr来实现零拷贝通信，以降低因为拷贝传输大数据（比如图像流，点云等）而延迟的问题，换句话说就是将多个node捆绑在一起管理，使得同一个manager里面的话题的数据传输更快，因为不会在进程内传递消息时进行复制而产生的效率下降。
查看插件文件：ls /home/jetson/workspace/catkin_ws/src/opencv_apps/src/nodelet
我们打开哈里斯角点的插件代码来看下：
gedit /home/jetson/workspace/catkin_ws/src/opencv_apps/src/nodelet/corner_harris_nodelet.cpp
// -*- coding:utf-8-unix; mode: c++; indent-tabs-mode: nil; c-basic-offset: 2; -*-
/*********************************************************************
* Software License Agreement (BSD License)
*
* Copyright (c) 2016, JSK Lab.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials provided
* with the distribution.
* * Neither the name of the Kei Okada nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*********************************************************************/
#include
#include "opencv_apps/nodelet.h"
#include
#include
#include
#include
#include
#include
#include "opencv_apps/CornerHarrisConfig.h"
// https://github.com/opencv/opencv/blob/master/samples/cpp/tutorial_code/TrackingMotion/cornerHarris_Demo.cpp
/**
* @function cornerHarris_Demo.cpp
* @brief Demo code for detecting corners using Harris-Stephens method
* @author OpenCV team
*/
namespace opencv_apps
{
class CornerHarrisNodelet : public opencv_apps::Nodelet
{
image_transport::Publisher img_pub_;
image_transport::Subscriber img_sub_;
image_transport::CameraSubscriber cam_sub_;
ros::Publisher msg_pub_;
boost::shared_ptr it_;
typedef opencv_apps::CornerHarrisConfig Config;
typedef dynamic_reconfigure::Server ReconfigureServer;
Config config_;
boost::shared_ptr reconfigure_server_;
int queue_size_;
bool debug_view_;
std::string window_name_;
static bool need_config_update_;
int threshold_;
void reconfigureCallback(Config& new_config, uint32_t level)
{
config_ = new_config;
threshold_ = config_.threshold;
}
const std::string& frameWithDefault(const std::string& frame, const std::string& image_frame)
{
if (frame.empty())
return image_frame;
return frame;
}
void imageCallbackWithInfo(const sensor_msgs::ImageConstPtr& msg, const sensor_msgs::CameraInfoConstPtr& cam_info)
{
doWork(msg, cam_info->header.frame_id);
}
void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
doWork(msg, msg->header.frame_id);
}
static void trackbarCallback(int /*unused*/, void* /*unused*/)
{
need_config_update_ = true;
}
void doWork(const sensor_msgs::ImageConstPtr& image_msg, const std::string& input_frame_from_msg)
{
// Work on the image.
try
{
// Convert the image into something opencv can handle.
cv::Mat frame = cv_bridge::toCvShare(image_msg, sensor_msgs::image_encodings::BGR8)->image;
// Do the work
cv::Mat dst, dst_norm, dst_norm_scaled;
dst = cv::Mat::zeros(frame.size(), CV_32FC1);
cv::Mat src_gray;
if (frame.channels() > 1)
{
cv::cvtColor(frame, src_gray, cv::COLOR_BGR2GRAY);
}
else
{
src_gray = frame;
cv::cvtColor(src_gray, frame, cv::COLOR_GRAY2BGR);
}
/// Detector parameters
int block_size = 2;
int aperture_size = 3;
double k = 0.04;
/// Detecting corners
cv::cornerHarris(src_gray, dst, block_size, aperture_size, k, cv::BORDER_DEFAULT);
/// Normalizing
cv::normalize(dst, dst_norm, 0, 255, cv::NORM_MINMAX, CV_32FC1, cv::Mat());
cv::convertScaleAbs(dst_norm, dst_norm_scaled);
/// Drawing a circle around corners
for (int j = 0; j < dst_norm.rows; j++)
{
for (int i = 0; i < dst_norm.cols; i++)
{
if ((int)dst_norm.at(j, i) > threshold_)
{
cv::circle(frame, cv::Point(i, j), 5, cv::Scalar(0), 2, 8, 0);
}
}
}
/// Create window
if (debug_view_)
{
cv::namedWindow(window_name_, cv::WINDOW_AUTOSIZE);
const int max_threshold = 255;
if (need_config_update_)
{
config_.threshold = threshold_;
reconfigure_server_->updateConfig(config_);
need_config_update_ = false;
}
cv::createTrackbar("Threshold:", window_name_, &threshold_, max_threshold, trackbarCallback);
}
if (debug_view_)
{
cv::imshow(window_name_, frame);
int c = cv::waitKey(1);
}
// Publish the image.
sensor_msgs::Image::Ptr out_img =
cv_bridge::CvImage(image_msg->header, sensor_msgs::image_encodings::BGR8, frame).toImageMsg();
img_pub_.publish(out_img);
}
catch (cv::Exception& e)
{
NODELET_ERROR("Image processing error: %s %s %s %i", e.err.c_str(), e.func.c_str(), e.file.c_str(), e.line);
}
}
void subscribe() // NOLINT(modernize-use-override)
{
NODELET_DEBUG("Subscribing to image topic.");
if (config_.use_camera_info)
cam_sub_ = it_->subscribeCamera("image", queue_size_, &CornerHarrisNodelet::imageCallbackWithInfo, this);
else
img_sub_ = it_->subscribe("image", queue_size_, &CornerHarrisNodelet::imageCallback, this);
}
void unsubscribe() // NOLINT(modernize-use-override)
{
NODELET_DEBUG("Unsubscribing from image topic.");
img_sub_.shutdown();
cam_sub_.shutdown();
}
public:
virtual void onInit() // NOLINT(modernize-use-override)
{
Nodelet::onInit();
it_ = boost::shared_ptr(new image_transport::ImageTransport(*nh_));
pnh_->param("queue_size", queue_size_, 3);
pnh_->param("debug_view", debug_view_, false);
if (debug_view_)
{
always_subscribe_ = true;
}
window_name_ = "CornerHarris Demo";
reconfigure_server_ = boost::make_shared >(*pnh_);
dynamic_reconfigure::Server::CallbackType f =
boost::bind(&CornerHarrisNodelet::reconfigureCallback, this, _1, _2);
reconfigure_server_->setCallback(f);
img_pub_ = advertiseImage(*pnh_, "image", 1);
onInitPostProcess();
}
};
bool CornerHarrisNodelet::need_config_update_ = false;
} // namespace opencv_apps
namespace corner_harris
{
class CornerHarrisNodelet : public opencv_apps::CornerHarrisNodelet
{
public:
virtual void onInit() // NOLINT(modernize-use-override)
{
ROS_WARN("DeprecationWarning: Nodelet corner_harris/corner_harris is deprecated, "
"and renamed to opencv_apps/corner_harris.");
opencv_apps::CornerHarrisNodelet::onInit();
}
};
} // namespace corner_harris
#include
PLUGINLIB_EXPORT_CLASS(opencv_apps::CornerHarrisNodelet, nodelet::Nodelet);
PLUGINLIB_EXPORT_CLASS(corner_harris::CornerHarrisNodelet, nodelet::Nodelet); 主要关注其中检测角点的方法：
cv::cornerHarris(src_gray, dst, block_size, aperture_size, k, cv::BORDER_DEFAULT);参数说明如下：
src_gray：输入的灰度Mat矩阵或浮点图像
dst：存储着哈里斯角点检测的结果，跟源图的尺寸和类型一样
block_size：邻域的大小
aperture_size：Sobel边缘检测滤波器大小
k：Harris中间参数，经验值0.04~0.06
cv::BORDER_DEFAULT：插值类型
我们也可以看到在发布和订阅时，用到的就是指针。
发布时：
sensor_msgs::Image::Ptr out_img =cv_bridge::CvImage(image_msg->header, sensor_msgs::image_encodings::BGR8, frame).toImageMsg();
img_pub_.publish(out_img);订阅时：
Nodelet::onInit();
it_ = boost::shared_ptr(new image_transport::ImageTransport(*nh_));
if (config_.use_camera_info)
cam_sub_ = it_->subscribeCamera("image", queue_size_, &CornerHarrisNodelet::imageCallbackWithInfo, this);
else
img_sub_ = it_->subscribe("image", queue_size_, &CornerHarrisNodelet::imageCallback, this); 这样我们在消息传递时，就只需要传指针了，当然这里是针对同一台设备的进程间通信，如果是不同设备那还是需要解引用传输实际内容，因为我们知道ROS的分布节点进行通信的协议是XML-RPC，本质也是HTTP协议，只不过编码格式是XML类型，它们之间的传输还得是拷贝内容进行通信。
查看节点关系：rqt_graph，我这里将调试节点隐藏，显得更清晰点，如下图：
可以看到CSI摄像头图像获取需要经过转换之后最终通过/Image话题发布给哈里斯角点算法处理。
识别角点的原理，简单来说就是在特征窗口里面，如果灰度发生了较大的变化，就认为这里是一个角点，有兴趣的可以查阅：harris.cpp
4、霍夫直线检测
接下来的部分，就没有上述那么去分析了，只是熟悉下常用的几个功能。
霍夫直线的检测在计算机视觉和图像处理中用途广泛，可以用于边缘检测、直线检测等。
实际场景中，可以通过从图像中检测出边缘，然后通过识别直线或曲线，将这些边缘连接起来，形成完整的物体。
另外无人驾驶的发展，对于自动化检测道路、车道线等应用也有着广泛的应用。
启动launch文件：roslaunch opencv_apps hough_lines.launch
很好的检测到了我身上的衣服以及上面的“中国”文字，如下图：
5、图像轮廓矩
contour_moments.launch是启动识别图像中轮廓的矩函数，这里的轮廓矩也可以理解成轮廓的特征，它也有着很广泛的应用：
目标识别：提取图像中物体的轮廓特征，可以对目标进行识别和分类。
目标检测：通过检测物体的形状和轮廓，来确定目标的位置。
图像分割：因为可以对不同的区域进行目标的识别，所以也可以帮助其进行图像的分割。
医学领域：可以用来识别图像中的组织器官和患病部位，从而提取特征，进行医学的诊断。
启动：roslaunch opencv_apps contour_moments.launch，如下图：
6、LK光流
LK光流是描述目标运动的方法，利用LK光流可以实现对目标的追踪，从而知道目标的位姿。
LK光流法的前提条件如下：
亮度恒定：一个像素点随着时间的变化，其亮度值（像素灰度值）是不能变化的。
小运动：时间的变化不会引起位置的剧烈变化。这样才能利用相邻帧之间的位置变化引起的灰度值变化，去求取灰度对位置的偏导数。
空间一致：前一帧的相邻像素点在后一帧也是相邻的，因为为了求解x，y方向的速度，需要建立方程组，而空间一致假设就可以利用邻域n个像素点来建立n个方程。
启动：roslaunch opencv_apps lk_flow.launch，如下图：
7、相机相位位移
检测相机移动的快慢，或者里面目标的运动快慢
启动：roslaunch opencv_apps phase_corr.launch，如下图：
移动越快，圆就越大
大概的介绍就先到这儿吧，另外一些关于OpenCV的文章，有兴趣的可以查阅：
OpenCV自带的HAAR级联分类器对脸部(人脸、猫脸等)的检测识别
OpenCV的HSV颜色空间在无人车中颜色识别的应用