（6）点云数据处理学习——RGBD图
1、主要参考
（1）官网的地址
RGBD images — Open3D 0.16.0 documentation
2、实现和测试
2.1 功能说明
Open3D有一个用于图像的数据结构。它支持各种函数，如read_image、write_image、filter_image和draw_geometries。一个Open3D图像可以直接转换为numpy数组。
一个Open3D RGBDImage由两个图像rgbimage .depth和rgbimage .color组成。我们要求两个图像注册到相同的相机帧和具有相同的分辨率。下面的教程展示了如何从许多著名的RGBD数据集中读取和使用RGBD图像。
2.2Redwood 数据集（Redwood dataset）
（1）数据集说明
在本节中，我们将展示如何从Redwood数据集[Choi2015]_读取和可视化RGBDImage。
Redwood格式在16位单通道图像中存储深度。整数值表示以毫米为单位的深度测量。它是Open3D解析深度图像的默认格式。
（2）使用的函数
create_from_color_and_depth( color_raw, depth_raw)
功能说明：默认的转换函数create_rgbd_image_from_color_and_depth从一对彩色和深度图像创建一个RGBDImage。彩色图像被转换为灰度图像，存储在浮动范围[0,1]中。深度图像存储在float中，以米为单位表示深度值。
（3）测试代码
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
print("Read Redwood dataset")
redwood_rgbd = o3d.data.SampleRedwoodRGBDImages()
color_raw = o3d.io.read_image(redwood_rgbd.color_paths[0])
depth_raw = o3d.io.read_image(redwood_rgbd.depth_paths[0])
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
color_raw, depth_raw)
print(rgbd_image)注意open3d自动下载解压后在下面位置
几张椅子的图挺漂亮的
2.2.1显示一下RGB图和深度图
（1）可以通过np.asarray直接对深度图和彩色图进行处理
color_img = np.asarray(rgbd_image.color)
depth_img = np.asarray(rgbd_image.depth)
print(np.size(color_img))
print(color_img[0][0])
print(depth_img[0][0])（2）测试代码
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
print("Read Redwood dataset")
redwood_rgbd = o3d.data.SampleRedwoodRGBDImages()
color_raw = o3d.io.read_image(redwood_rgbd.color_paths[0])
depth_raw = o3d.io.read_image(redwood_rgbd.depth_paths[0])
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
color_raw, depth_raw)
print(rgbd_image)
#640*480的图
color_img = np.asarray(rgbd_image.color)
depth_img = np.asarray(rgbd_image.depth)
print(np.size(color_img))
print(color_img[0][0])
print(depth_img[0][0])
plt.subplot(1, 2, 1)
plt.title('Redwood grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('Redwood depth image')
plt.imshow(rgbd_image.depth)
plt.show()（3）测试的结果
2.2.2RGBD图像可以转换为点云图
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
print("Read Redwood dataset")
redwood_rgbd = o3d.data.SampleRedwoodRGBDImages()
color_raw = o3d.io.read_image(redwood_rgbd.color_paths[0])
depth_raw = o3d.io.read_image(redwood_rgbd.depth_paths[0])
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
color_raw, depth_raw)
print(rgbd_image)
#640*480的图
# color_img = np.asarray(rgbd_image.color)
# depth_img = np.asarray(rgbd_image.depth)
# print(np.size(color_img))
# print(color_img[0][0])
# print(depth_img[0][0])
# plt.subplot(1, 2, 1)
# plt.title('Redwood grayscale image')
# plt.imshow(rgbd_image.color)
# plt.subplot(1, 2, 2)
# plt.title('Redwood depth image')
# plt.imshow(rgbd_image.depth)
# plt.show()
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
rgbd_image,
o3d.camera.PinholeCameraIntrinsic(
o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.5)
o3d.visualization.draw_geometries([pcd])
这里使用了PinholeCameraIntrinsicParameters。PrimeSenseDefault作为默认相机参数。它的图像分辨率为640x480，焦距(fx, fy) =(525.0, 525.0)，光心(cx, cy) =(319.5, 239.5)。单位矩阵被用作默认的外部参数。pcd.transform在点云上应用上下翻转转换，以达到更好的可视化目的。
2.3 SUN数据集(SUN dataset)
（1）说明
在本节中，我们将展示如何读取和可视化SUN数据集[Song2015]_的RGBDImage。
本教程与上面处理Redwood数据集的教程几乎相同。唯一的区别是我们使用了转换函数create_rgbd_image_from_sun_format来解析SUN数据集中的深度图像。
（2）测试
1）下载后的数据集如下图所示
2）测试代码
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
print("Read SUN dataset")
sun_rgbd = o3d.data.SampleSUNRGBDImage()
color_raw = o3d.io.read_image(sun_rgbd.color_path)
depth_raw = o3d.io.read_image(sun_rgbd.depth_path)
rgbd_image = o3d.geometry.RGBDImage.create_from_sun_format(color_raw, depth_raw)
print(rgbd_image)
plt.subplot(1, 2, 1)
plt.title('SUN grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('SUN depth image')
plt.imshow(rgbd_image.depth)
plt.show()
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
rgbd_image,
o3d.camera.PinholeCameraIntrinsic(
o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.5)
o3d.visualization.draw_geometries([pcd])
转变成的点云图如下图所示
2.4 NYU数据集(NYU dataset)
本部分展示如何从NYU数据集[Silberman2012]_读取和可视化RGBDImage。
（1）说明
本部分教程与上面处理Redwood数据集的教程几乎相同，但有两点不同。首先，纽约大学的图片不是标准的jpg或png格式。因此，我们使用mpimg。imread将彩色图像作为numpy数组读取，并将其转换为Open3D图像。调用另一个辅助函数read_nyu_pgm从NYU数据集中使用的特殊大端pgm格式中读取深度图像。其次，我们使用一个不同的转换函数create_rgbd_image_from_nyu_format来解析SUN数据集中的深度图像
（2）测试代码
import open3d as o3d
import numpy as np
import matplotlib.image as mpimg
import re
# This is special function used for reading NYU pgm format
# as it is written in big endian byte order.
def read_nyu_pgm(filename, byteorder='>'):
with open(filename, 'rb') as f:
buffer = f.read()
try:
header, width, height, maxval = re.search(
b"(^P5\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
except AttributeError:
raise ValueError("Not a raw PGM file: '%s'" % filename)
img = np.frombuffer(buffer,
dtype=byteorder + 'u2',
count=int(width) * int(height),
offset=len(header)).reshape((int(height), int(width)))
img_out = img.astype('u2')
return img_out
print("Read NYU dataset")
# Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
# MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
nyu_rgbd = o3d.data.SampleNYURGBDImage()
color_raw = mpimg.imread(nyu_rgbd.color_path)
depth_raw = read_nyu_pgm(nyu_rgbd.depth_path)
color = o3d.geometry.Image(color_raw)
depth = o3d.geometry.Image(depth_raw)
rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
print(rgbd_image)下载解压后的图片如下图所示
（3）显示一下
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
import matplotlib.image as mpimg
import re
# This is special function used for reading NYU pgm format
# as it is written in big endian byte order.
def read_nyu_pgm(filename, byteorder='>'):
with open(filename, 'rb') as f:
buffer = f.read()
try:
header, width, height, maxval = re.search(
b"(^P5\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
except AttributeError:
raise ValueError("Not a raw PGM file: '%s'" % filename)
img = np.frombuffer(buffer,
dtype=byteorder + 'u2',
count=int(width) * int(height),
offset=len(header)).reshape((int(height), int(width)))
img_out = img.astype('u2')
return img_out
print("Read NYU dataset")
# Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
# MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
nyu_rgbd = o3d.data.SampleNYURGBDImage()
color_raw = mpimg.imread(nyu_rgbd.color_path)
depth_raw = read_nyu_pgm(nyu_rgbd.depth_path)
color = o3d.geometry.Image(color_raw)
depth = o3d.geometry.Image(depth_raw)
rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
print(rgbd_image)
plt.subplot(1, 2, 1)
plt.title('NYU grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('NYU depth image')
plt.imshow(rgbd_image.depth)
plt.show()
（4）转换为点云图像显示
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
import matplotlib.image as mpimg
import re
# This is special function used for reading NYU pgm format
# as it is written in big endian byte order.
def read_nyu_pgm(filename, byteorder='>'):
with open(filename, 'rb') as f:
buffer = f.read()
try:
header, width, height, maxval = re.search(
b"(^P5\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n])*"
b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
except AttributeError:
raise ValueError("Not a raw PGM file: '%s'" % filename)
img = np.frombuffer(buffer,
dtype=byteorder + 'u2',
count=int(width) * int(height),
offset=len(header)).reshape((int(height), int(width)))
img_out = img.astype('u2')
return img_out
print("Read NYU dataset")
# Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
# MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
nyu_rgbd = o3d.data.SampleNYURGBDImage()
color_raw = mpimg.imread(nyu_rgbd.color_path)
depth_raw = read_nyu_pgm(nyu_rgbd.depth_path)
color = o3d.geometry.Image(color_raw)
depth = o3d.geometry.Image(depth_raw)
rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
print(rgbd_image)
plt.subplot(1, 2, 1)
plt.title('NYU grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('NYU depth image')
plt.imshow(rgbd_image.depth)
plt.show()
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
rgbd_image,
o3d.camera.PinholeCameraIntrinsic(
o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.5)
o3d.visualization.draw_geometries([pcd])
2.5 TUM数据集(TUM dataset)
（1）说明
本节展示如何从TUM数据集[Strum2012]_读取和可视化RGBDImage。
本教程与上面处理Redwood数据集的教程几乎相同。唯一的区别是我们使用转换函数create_rgbd_image_from_tum_format来解析TUM数据集中的深度图像。
（2）测试
注意：官网教程有误
# tum_rgbd = o3d.data.SampleSUNRGBDImage()
tum_rgbd = o3d.data.SampleTUMRGBDImage()
或者手动下载
https://github.com/isl-org/open3d_downloads/releases/download/20220201-data/SampleTUMRGBDImage.zip
import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
print("Read TUM dataset")
# 方法一：官网下载
#tum_rgbd = o3d.data.SampleTUMRGBDImage()
# color_raw = o3d.io.read_image(tum_rgbd.color_path)
# depth_raw = o3d.io.read_image(tum_rgbd.depth_path)
## 方法二 本地下载后使用 https://github.com/isl-org/open3d_downloads/releases/download/20220201-data/SampleTUMRGBDImage.zip
tum_rgbd_color_path = "D:/RGBD_CAMERA/python_3d_process/SampleTUMRGBDImage/TUM_color.png"
tum_rgbd_depth_path = "D:/RGBD_CAMERA/python_3d_process/SampleTUMRGBDImage/TUM_depth.png"
color_raw = o3d.io.read_image(tum_rgbd_color_path)
depth_raw = o3d.io.read_image(tum_rgbd_depth_path)
rgbd_image = o3d.geometry.RGBDImage.create_from_tum_format(color_raw, depth_raw)
print(rgbd_image)
plt.subplot(1, 2, 1)
plt.title('TUM grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('TUM depth image')
plt.imshow(rgbd_image.depth)
plt.show()
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
rgbd_image,
o3d.camera.PinholeCameraIntrinsic(
o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# # Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.35)
o3d.visualization.draw_geometries([pcd])
转换后的点云图像如下
