ZED2相机+NVIDIA NX使用及检测目标功能1

ZED相机使用

zed ros wrapper分为三部分：

1. zed-ros-wrapper : 提供 ZED ROS Wrapper 节点的主包
2. zed-ros-interfaces：声明自定义主题、服务和操作的包
3. zed-ros-examples：一个支持包，包含有关如何使用 ZED ROS Wrapper 的示例和教程

启动ZED节点：

1. ZED相机： roslaunch zed_wrapper zed.launch
2. ZED 迷你相机： roslaunch zed_wrapper zedm.launch
3. ZED 2相机： roslaunch zed_wrapper zed2.launch
4. ZED 2i相机： roslaunch zed_wrapper zed2i.launch

设置相机参数：修改文件param/common.yaml（所有相机通用参数）和param/zed2.yaml
参数调整：https://www.stereolabs.com/docs/ros/zed-node/

RVIZ可视化ZED2：roslaunch zed_display_rviz display_zed2.launch
配置RVIZ：https://www.stereolabs.com/docs/ros/rviz/
显示其余ZED数据：https://www.stereolabs.com/docs/ros/
注：

1. rgb/image_rect_color : 颜色校正图像（默认为左传感器）      
2. zed/zed_node/point_cloud/cloud_registered：3D彩色点云
3. mapping/fused_cloud：融合颜色点云，仅在启用映时发布

检测目标功能

1.对象检测参数：https://www.stereolabs.com/docs/ros/zed-node/ 对象检测类别及其他API参考：https://www.stereolabs.com/docs/api/python/classpyzed_1_1sl_1_1OBJECT__SUBCLASS.html

（人、车辆、包、动物、电子设备、水果和蔬菜）
model： 要使用的对象检测模块‘0’: MULTI_CLASS_BOX - ‘1’: MULTI_CLASS_BOX_ACCURATE - ‘2’: HUMAN_BODY_FAST - ‘3’: HUMAN_BODY_ACCURATE - ‘4’: MULTI_CLASS_BOX_MEDIUM - ‘5’: HUMAN_BODY_MEDIUM
object_tracking_enabled：启用/禁用对检测到的对象的跟踪 TRUE/FALSE

2.检测到的对象msg：https://www.stereolabs.com/docs/ros/object-detection/

zed_interfaces/ObjectsStamped消息定义为：

#Standard Header
std_msgs/Header header

#Array of `object_stamped` topics
zed_interfaces/Object[] objects

zed_interfaces/Object定义为：

# Object label
string label

# Object label ID
int16 label_id

# Object sub
string sublabel

# Object confidence level (1-99)
float32 confidence

# Object centroid position
float32[3] position

# Position covariance
float32[6] position_covariance

# Object velocity
float32[3] velocity

# Tracking available
bool tracking_available

# Tracking state
# 0 -> OFF (object not valid)
# 1 -> OK
# 2 -> SEARCHING (occlusion occurred, trajectory is estimated)
int8 tracking_state

# Action state
# 0 -> IDLE
# 2 -> MOVING
int8 action_state

# 2D Bounding box projected to Camera image
zed_interfaces/BoundingBox2Di bounding_box_2d

# 3D Bounding box in world frame
zed_interfaces/BoundingBox3D bounding_box_3d

# 3D dimensions (width, height, lenght)
float32[3] dimensions_3d

# Is skeleton available?
bool skeleton_available

# 2D Bounding box projected to Camera image of the person head
zed_interfaces/BoundingBox2Df head_bounding_box_2d

# 3D Bounding box in world frame of the person head
zed_interfaces/BoundingBox3D head_bounding_box_3d

# 3D position of the centroid of the person head
float32[3] head_position

# 2D Person skeleton projected to Camera image
zed_interfaces/Skeleton2D skeleton_2d

# 3D Person skeleton in world frame
zed_interfaces/Skeleton3D skeleton_3d

子消息定义：

zed_interfaces/BoundingBox2Df：

#      0 ------- 1
#      |         |
#      |         |
#      |         |
#      3 ------- 2
zed_interfaces/Keypoint2Df[4] corners
zed_interfaces/BoundingBox2Df：

#      0 ------- 1
#      |         |
#      |         |
#      |         |
#      3 ------- 2
zed_interfaces/Keypoint2Di[4] corners
zed_interfaces/BoundingBox3D：

#      1 ------- 2
#     /.        /|
#    0 ------- 3 |
#    | .       | |
#    | 5.......| 6
#    |.        |/
#    4 ------- 7
zed_interfaces/Keypoint3D[8] corners
zed_interfaces/Keypoint2Df：

float32[2] kp
zed_interfaces/Keypoint2Di：

uint32[2] kp
zed_interfaces/Keypoint3D：

float32[3] kp
zed_interfaces/Skeleton2D：

# Skeleton joints indices
#        16-14   15-17
#             \ /
#              0
#              |
#       2------1------5
#       |    |   |    |
#       |    |   |    |
#       3    |   |    6
#       |    |   |    |
#       |    |   |    |
#       4    8   11   7
#            |   |
#            |   |
#            |   |
#            9   12
#            |   |
#            |   |
#            |   |
#           10   13
zed_interfaces/Keypoint2Df[18] keypoints
zed_interfaces/Skeleton3D：

# Skeleton joints indices
#        16-14   15-17
#             \ /
#              0
#              |
#       2------1------5
#       |    |   |    |
#       |    |   |    |
#       3    |   |    6
#       |    |   |    |
#       |    |   |    |
#       4    8   11   7
#            |   |
#            |   |
#            |   |
#            9   12
#            |   |
#            |   |
#            |   |
#           10   13
zed_interfaces/Keypoint3D[18] keypoints

3.nodelets的使用：https://www.stereolabs.com/docs/ros/zed-nodelets/

4.读取IMU、设备位置路径等信息：https://www.stereolabs.com/docs/ros/sensor-data/以及https://www.stereolabs.com/docs/ros/zed-node/

5.检测人、其他物品

1. 参数修改:common.yaml，zed2i.yaml，在zed2i.yaml中model内修改检测目标等
2. ZED SDK 检测图像中存在的所有对象并计算它们的 3D 位置和速度。 对象与相机的距离以公制单位（例如米）表示，并从相机左眼后部到场景对象计算。（https://www.stereolabs.com/docs/object-detection/）

6.rosbag存信息及plotjuggler展示

plotjuggler下载时使用：sudo apt install ros-melodic-plotjuggler-ros而不是sudo apt install ros-melodic-plotjuggler，才能实时订阅信息显示，否则只能读取bag。
补充信息

https://facontidavide.github.io/PlotJuggler/visualization_howto/index.html

rosbag：

http://wiki.ros.org/rosbag/Commandline
https://guyuehome.com/35777
https://guyuehome.com/34588