Ubuntu20.04LTS下安装Intel Realsense D435i驱动与ROS包

目标

在Ubuntu20.04LTS系统下安装D435i的驱动SDK2和ROS包，实现在ROS环境中使用D435i。

一、D435i简介

D435i是Inter公司RealSense系列摄像头的一种，采用英特尔最新的深度感应硬件和软件，封装成易于集成的产品。相比D435，D435i添加了IMU（惯性测量单元），不仅为 SLAM 和跟踪应用打开了大门，还使手持扫描系统的注册和校准更容易，在虚拟/增强现实和无人机等领域很重要。
D435i是一个RGB-D相机，也就是我们常说的深度相机，目前深度相机的实现原理主要分为三种：分别是结构光，Tof，双目成像。REALSENSE使用的是结构光的方案。正面的四个摄像头，从左向右以次是左红外相机，红外点阵投射仪，右红外相机，和RGB相机。

尽管具有红外投射器，但并不是采用红外反射测距。它的作用仅仅是投射不可见的固定的红外纹理样式，提高在纹理不明显的环境中（例如白墙）的深度计算精度，辅助双目视觉测距。左右两个相机将图像数据送入内置的深度处理器，在其中基于双目测距的原理计算每个像素的深度值。
最高 1280×720 双目深度分辨率;
最高 1920×1080 RGB 分辨率;
最高 90 FPS 深度视频流。
深度流与普通 RGB 视频流类似，只不过每个像素点的值不再是 RGB 或灰度值，而是物体相对于相机的距离。
只能对相同帧率的 RGB 与深度视频流做同步设置。
双目 baseline 为 50 mm;
深度探测范围 0.2 m ~ 10 m;
深度坐标系以左侧相机为中心（下图中 centerline of 1/4-20 是指三脚架螺丝空的中心）

二、环境配置

系统：Ubuntu20.04LTS
ROS：1.15.14
摄像机：D435i

三、RealSense的SDK2.0安装

方式一：(优先)
1.注册服务器的公钥

sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE

2.将服务器添加到存储库列表中

sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u

3.安装SDK2
方式一（apt）：
基础安装

sudo apt-get install librealsense2-dkms
sudo apt-get install librealsense2-utils

选装

sudo apt-get install librealsense2-dev
sudo apt-get install librealsense2-dbg

Tip:安装dev包后，可使用g++ -std=c++11 filename.cpp -lrealsense2 或 别的IDE来编译具有librealsense的应用。
测试安装结果

realsense-viewer 

方式二（源代码）：
下载librealsense，然后进入该目录，运行下列指令安装和编译依赖项：

sudo apt-get install libudev-dev pkg-config libgtk-3-dev
sudo apt-get install libusb-1.0-0-dev pkg-config
sudo apt-get install libglfw3-dev
sudo apt-get install libssl-dev

sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && udevadm trigger 
mkdir build
cd build
cmake ../ -DBUILD_EXAMPLES=true
make
sudo make install

测试安装结果

realsense-viewer 

四、ROS包安装

方式一（apt）：
安装ROS版本的realsense2_camera

sudo apt-get install ros-$ROS_DISTRO-realsense2-camera
sudo apt-get install ros-$ROS_DISTRO-realsense2-description

方式二（源代码）：
进入catkin_ws工作空间的src目录，下载realsense和ddynamic_reconfigure，然后用ROS的catkin_make指令编译。

cd ~/catkin_ws/src
git clone https://github.com/IntelRealSense/realsense-ros.git
git clone https://github.com/pal-robotics/ddynamic_reconfigure.git
cd ~/catkin_ws && catkin_make

安装rgbd-launch
rgbd_launch是一组打开RGBD设备，并load 所有nodelets转化 raw depth/RGB/IR 流到深度图(depth image), 视差图(disparity image)和点云(point clouds)的launch文件集。

sudo apt-get install ros-noetic-rgbd-launch

测试编译结果：

roslaunch realsense2_camera demo_pointcloud.launch 

五、摄像机CV的ROS包节点

在catkin_ws工作空间中创建一个usb_camera包，含有一个图像消息发布节点img_publisher与图像消息接收显示节点img_viewer。
package.xml：

<buildtool_depend>catkin</buildtool_depend>
  <build_depend>cv_bridge</build_depend>
  <build_depend>image_transport</build_depend>
  <build_depend>roscpp</build_depend>
  <build_depend>sensor_msgs</build_depend>
  <build_depend>std_msgs</build_depend>
  <build_depend>opencv2</build_depend>

  <build_export_depend>cv_bridge</build_export_depend>
  <build_export_depend>image_transport</build_export_depend>
  <build_export_depend>roscpp</build_export_depend>
  <build_export_depend>sensor_msgs</build_export_depend>
  <build_export_depend>std_msgs</build_export_depend>
  <build_export_depend>opencv2</build_export_depend>

  <exec_depend>cv_bridge</exec_depend>
  <exec_depend>image_transport</exec_depend>
  <exec_depend>roscpp</exec_depend>
  <exec_depend>sensor_msgs</exec_depend>
  <exec_depend>std_msgs</exec_depend>
 <exec_depend>opencv2</exec_depend>

  <!-- The export tag contains other, unspecified, tags -->
  <export>
    <!-- Other tools can request additional information be placed here -->

img_publisher.cpp节点：

#include 
#include 
#include 
#include 
#include 
#include 
 
int main(int argc, char** argv)
{
  ros::init(argc, argv, "img_publisher");
  ros::NodeHandle nh;
  image_transport::ImageTransport it(nh);
  image_transport::Publisher pub = it.advertise("camera/image", 1);
 
  cv::VideoCapture cap;
  cv::Mat frame;
    
  int deviceID=0;
  
  if(argc>1)
	deviceID=argv[1][0]-'0';
  int apiID=cv::CAP_ANY;
  cap.open(deviceID+apiID);
  
  if(!cap.isOpened()){
	std::cerr<<"ERROR! Unable to open camera"<<std::endl;
	return -1;
  }
 
  ros::Rate loop_rate(30);
  while (nh.ok()) {
	cap.read(frame);
	if(!frame.empty()){
		sensor_msgs::ImagePtr msg = cv_bridge::CvImage(std_msgs::Header(), "bgr8", frame).toImageMsg();
		pub.publish(msg);
	}
    	ros::spinOnce();
    	loop_rate.sleep();
  }
  return 0;
}

img_viewer：

//接收并查看图像消息的节点img_viewer.cpp
#include 
#include 
#include 
#include 
 
 //依赖包有std_msgs（消息传递），roscpp（c++），cv_bridge（ros和opencv图像消息转换），sensor_msgs（传感器消息），image_transport（图像编码传输）
void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
  try
  {
    cv::imshow("view", cv_bridge::toCvShare(msg, "bgr8")->image);
  }
  catch (cv_bridge::Exception& e)
  {
    ROS_ERROR("Could not convert from '%s' to 'bgr8'.", msg->encoding.c_str());
  }
}
 
int main(int argc, char **argv)
{
  ros::init(argc, argv, "img_viewer");
  ros::NodeHandle nh;
  cv::namedWindow("view");
  cv::startWindowThread();
  image_transport::ImageTransport it(nh);
  image_transport::Subscriber sub = it.subscribe("camera/image", 1, imageCallback);
  ros::spin();
  cv::destroyWindow("view");
  return 0;
}

CMakeLists.txt：

cmake_minimum_required(VERSION 3.0.2)
project(usb_camera)

## Compile as C++11, supported in ROS Kinetic and newer
# add_compile_options(-std=c++11)

## Find catkin macros and libraries
## if COMPONENTS list like find_package(catkin REQUIRED COMPONENTS xyz)
## is used, also find other catkin packages

find_package(catkin REQUIRED COMPONENTS
  cv_bridge
  image_transport
  roscpp
  sensor_msgs
  std_msgs
)

## System dependencies are found with CMake's conventions
# find_package(Boost REQUIRED COMPONENTS system)

find_package(OpenCV REQUIRED)
## Uncomment this if the package has a setup.py. This macro ensures
## modules and global scripts declared therein get installed
## See http://ros.org/doc/api/catkin/html/user_guide/setup_dot_py.html
# catkin_python_setup()

################################################
## Declare ROS messages, services and actions ##
################################################

## To declare and build messages, services or actions from within this
## package, follow these steps:
## * Let MSG_DEP_SET be the set of packages whose message types you use in
##   your messages/services/actions (e.g. std_msgs, actionlib_msgs, ...).
## * In the file package.xml:
##   * add a build_depend tag for "message_generation"
##   * add a build_depend and a exec_depend tag for each package in MSG_DEP_SET
##   * If MSG_DEP_SET isn't empty the following dependency has been pulled in
##     but can be declared for certainty nonetheless:
##     * add a exec_depend tag for "message_runtime"
## * In this file (CMakeLists.txt):
##   * add "message_generation" and every package in MSG_DEP_SET to
##     find_package(catkin REQUIRED COMPONENTS ...)
##   * add "message_runtime" and every package in MSG_DEP_SET to
##     catkin_package(CATKIN_DEPENDS ...)
##   * uncomment the add_*_files sections below as needed
##     and list every .msg/.srv/.action file to be processed
##   * uncomment the generate_messages entry below
##   * add every package in MSG_DEP_SET to generate_messages(DEPENDENCIES ...)

## Generate messages in the 'msg' folder
# add_message_files(
#   FILES
#   Message1.msg
#   Message2.msg
# )

## Generate services in the 'srv' folder
# add_service_files(
#   FILES
#   Service1.srv
#   Service2.srv
# )

## Generate actions in the 'action' folder
# add_action_files(
#   FILES
#   Action1.action
#   Action2.action
# )

## Generate added messages and services with any dependencies listed here
# generate_messages(
#   DEPENDENCIES
#   sensor_msgs#   std_msgs
# )

################################################
## Declare ROS dynamic reconfigure parameters ##
################################################

## To declare and build dynamic reconfigure parameters within this
## package, follow these steps:
## * In the file package.xml:
##   * add a build_depend and a exec_depend tag for "dynamic_reconfigure"
## * In this file (CMakeLists.txt):
##   * add "dynamic_reconfigure" to
##     find_package(catkin REQUIRED COMPONENTS ...)
##   * uncomment the "generate_dynamic_reconfigure_options" section below
##     and list every .cfg file to be processed

## Generate dynamic reconfigure parameters in the 'cfg' folder
# generate_dynamic_reconfigure_options(
#   cfg/DynReconf1.cfg
#   cfg/DynReconf2.cfg
# )

###################################
## catkin specific configuration ##
###################################
## The catkin_package macro generates cmake config files for your package
## Declare things to be passed to dependent projects
## INCLUDE_DIRS: uncomment this if your package contains header files
## LIBRARIES: libraries you create in this project that dependent projects also need
## CATKIN_DEPENDS: catkin_packages dependent projects also need
## DEPENDS: system dependencies of this project that dependent projects also need
catkin_package(
#  INCLUDE_DIRS include
#  LIBRARIES usb_camera
#  CATKIN_DEPENDS cv_bridge image_transport roscpp sensor_msgs std_msgs
#  DEPENDS system_lib
)

###########
## Build ##
###########

## Specify additional locations of header files
## Your package locations should be listed before other locations
include_directories(
# include
  ${catkin_INCLUDE_DIRS}
  ${OpenCV_INCLUDE_DIRS}
)

## Declare a C++ library
# add_library(${PROJECT_NAME}
#   src/${PROJECT_NAME}/usb_camera.cpp
# )

## Add cmake target dependencies of the library
## as an example, code may need to be generated before libraries
## either from message generation or dynamic reconfigure
# add_dependencies(${PROJECT_NAME} ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

## Declare a C++ executable
## With catkin_make all packages are built within a single CMake context
## The recommended prefix ensures that target names across packages don't collide
# add_executable(${PROJECT_NAME}_node src/usb_camera_node.cpp)

## Rename C++ executable without prefix
## The above recommended prefix causes long target names, the following renames the
## target back to the shorter version for ease of user use
## e.g. "rosrun someones_pkg node" instead of "rosrun someones_pkg someones_pkg_node"
# set_target_properties(${PROJECT_NAME}_node PROPERTIES OUTPUT_NAME node PREFIX "")

## Add cmake target dependencies of the executable
## same as for the library above
# add_dependencies(${PROJECT_NAME}_node ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

## Specify libraries to link a library or executable target against
# target_link_libraries(${PROJECT_NAME}_node
#   ${catkin_LIBRARIES}
# )
add_executable(img_publisher src/img_publisher.cpp)
target_link_libraries(img_publisher ${catkin_LIBRARIES} ${OpenCV_LIBS})

add_executable(img_viewer src/img_viewer.cpp)
target_link_libraries(img_viewer ${catkin_LIBRARIES} ${OpenCV_LIBS})

#############
## Install ##
#############

# all install targets should use catkin DESTINATION variables
# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html

## Mark executable scripts (Python etc.) for installation
## in contrast to setup.py, you can choose the destination
# catkin_install_python(PROGRAMS
#   scripts/my_python_script
#   DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark executables for installation
## See http://docs.ros.org/melodic/api/catkin/html/howto/format1/building_executables.html
# install(TARGETS ${PROJECT_NAME}_node
#   RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark libraries for installation
## See http://docs.ros.org/melodic/api/catkin/html/howto/format1/building_libraries.html
# install(TARGETS ${PROJECT_NAME}
#   ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
#   LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
#   RUNTIME DESTINATION ${CATKIN_GLOBAL_BIN_DESTINATION}
# )

## Mark cpp header files for installation
# install(DIRECTORY include/${PROJECT_NAME}/
#   DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
#   FILES_MATCHING PATTERN "*.h"
#   PATTERN ".svn" EXCLUDE
# )

## Mark other files for installation (e.g. launch and bag files, etc.)
# install(FILES
#   # myfile1
#   # myfile2
#   DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
# )

#############
## Testing ##
#############

## Add gtest based cpp test target and link libraries
# catkin_add_gtest(${PROJECT_NAME}-test test/test_usb_camera.cpp)
# if(TARGET ${PROJECT_NAME}-test)
#   target_link_libraries(${PROJECT_NAME}-test ${PROJECT_NAME})
# endif()

## Add folders to be run by python nosetests
# catkin_add_nosetests(test)

六、问题排查

1. 查找usb摄像头
在最初的节点运行时，提示找不到usb摄像头，改变摄像机的索引号也不行。查摄像头占用情况：

ls  /dev/video*
/dev/video0  /dev/video1  /dev/video2  /dev/video3  /dev/video4  /dev/video5

摄像机索引没问题。
根据输出信息[ WARN:0] OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Cannot identify device '/dev/video0'. [ WARN:0] OpenCV | GStreamer warning: unable to start pipeline [ WARN:0] OpenCV | GStreamer warning: GStreamer: pipeline have not been created
查找相关文献，试探性的调试如下：

find /lib/modules/ -name "*v4l2*.ko"

显示如下信息：

/lib/modules/5.13.0-30-generic/kernel/drivers/media/v4l2-core/v4l2-flash-led-class.ko
/lib/modules/5.13.0-30-generic/kernel/drivers/media/v4l2-core/v4l2-dv-timings.ko
/lib/modules/5.13.0-30-generic/kernel/drivers/media/v4l2-core/v4l2-fwnode.ko
/lib/modules/5.13.0-30-generic/kernel/drivers/media/v4l2-core/v4l2-mem2mem.ko
/lib/modules/5.13.0-30-generic/kernel/drivers/media/i2c/adv7511-v4l2.ko
/lib/modules/5.13.0-30-generic/kernel/drivers/media/common/videobuf2/videobuf2-v4l2.ko
/lib/modules/5.13.0-30-generic/kernel/drivers/media/common/v4l2-tpg/v4l2-tpg.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/v4l2-core/v4l2-flash-led-class.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/v4l2-core/v4l2-dv-timings.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/v4l2-core/v4l2-fwnode.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/v4l2-core/v4l2-mem2mem.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/i2c/adv7511-v4l2.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/common/videobuf2/videobuf2-v4l2.ko
/lib/modules/5.13.0-27-generic/kernel/drivers/media/common/v4l2-tpg/v4l2-tpg.ko

发现相关的ko，其中，adv7511-v4l2.ko这个是与usb摄像机相关的文件。运行：

sudo modprobe adv7511-v4l2

然后就可以打开USB摄像机了。modprobe命令用于自动处理可载入模块，设置摄像头设备后相当于默认使用该设备。
另外，用v4l2-ctl --list-devices 命令可以查看摄像头对应的设备号，如果没有v4l2-ctl，可先安装：

sudo apt install v4l-utils

v4l2-ctl --list-devices
Video Capture 9 (usb-0000:00:14.0-3):
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/video4
	/dev/video5
	/dev/media0
	/dev/media1

经过上述操作后并不一定完全解决问题。
2. 了解Gstreamer框架
gstreamer是一个开发流式多媒体应用的开源框架，是为更好的处理音视频而设计开发的，但又不限于处理音视频，只要是有明显的数据流特征的应用。
gstreamer的功能核心是pipeline框架以及用于扩展功能的插件。通过gstreamer多媒体数据协商机制和同步机制，pipeline能够很好的地处理流式数据。开发者可以灵活利用已有插件，而且还可以自定义特定功能插件。

[1] Ubantu20.04安装libRealsense SDK 2.0 和ROS Wrapper for Intel RealSense Devices
[2] Ubuntu20.04+RealSense D435i
[3] ubuntu opencv 打开摄像头失败
[4] Gstreamer基础知识介绍