HoloLens2 Visible-Light Tracking Camera (VLC) 内参获取

问题描述

    OpenCV中的许多功能都需要输入相机内参(Camera Intrinsics)。然而，HoloLens2并没有提供直接获取相机内参的API，what is provided is a function (MapImagePointToCameraUnitPlane()) from pixel to unit ray in the camera space。详见HoloLens2的官方论文HoloLens 2 Research Mode as a Tool for Computer Vision Research。
    本文介绍获取HoloLens2 Visible-Light Tracking 相机内参的另一种方法——从Lookup Table (LuT)中求解相机内参。

求解方法

1. 获取LuT
LuT通过微软提供的SteamRecorder工程得到，工程链接。
将工程中需要记录的相机类型修改为VLC：

将工程部署到HL并运行，LuT将被离线保存：

2.从LuT中求解相机内参

import numpy as np

def ReadFile(filepath):
    with open(filepath, mode='rb') as depth_file:
        lut = np.frombuffer(depth_file.read(), dtype="f")
        lut = np.reshape(lut, (-1, 3))
    return lut

def Fit_Linearity(x_array, y_array):
    # x_array自变量 y_array因变量
    # 方程个数
    m = len(x_array)
    # 计算过程
    sum_x = np.sum(x_array)
    sum_y = np.sum(y_array)
    sum_xy = np.sum(x_array * y_array)
    sum_xx = np.sum(x_array ** 2)
    b = (sum_y * sum_xx - sum_x * sum_xy) / (m * sum_xx - (sum_x) ** 2)
    k = (m * sum_xy - sum_x * sum_y) / (m * sum_xx - (sum_x) ** 2)
    return k, b

def Fit_CameraIntrinsics(lut, h, w):
    """
    u = fx*xc+x0
    v = fy*yc+y0
    """
    where = np.where(lut[:, 2] != 0)
    lut = lut[where]
    xc = lut[:, 0] / lut[:, 2]
    yc = lut[:, 1] / lut[:, 2]
    u = np.arange(0.5, w, 1, float)
    u = np.tile(u, h)
    u = u[where]
    v = np.arange(0.5, h, 1, float)
    v = v.repeat(w)
    v = v[where]
    fx, x0 = Fit_Linearity(xc, u)
    fy, y0 = Fit_Linearity(yc, v)
    intrinsics = np.array([[fx, 0, x0], [0, fy, y0], [0, 0, 1]])
    return intrinsics

if __name__ == '__main__':
    # 设置图像分辨率
    height = 480
    width = 640
    filepath = 'D:\\Paper\\HoloLens\\Record\\2022-12-27-053238\\VLC RR_lut.bin' # the file path of LuT that you saved
    lut = ReadFile(filepath)
    camera_intrinsics = Fit_CameraIntrinsics(lut, height,width)
    print(camera_intrinsics)

最终输出相机内参如下：

总结

参考链接

https://blog.csdn.net/scy261983626/article/details/117224024
https://github.com/microsoft/HoloLens2ForCV/issues/37