使用opencv实现camera calibration

1. 使用opencv实现camera calibration(摄像机校准/标定)

使用opencv提供的demo(源码为calibration.cpp)实现单目摄像头的image的camera calibration的

1. 1. 基本处理过程

一般步骤分为两个部分，如下：

1. 获取camera的内外参和畸变相关系数

1. 从图片列表中读取图像，或从camera中获取图像。
2. cvtColor，把图像转换成灰度图；
3. 基于BGR图像，使用findChessboardCorners, 获取corners
4. 基于灰度图，使用cornerSubPix，获取更精确的corners
5. 把每张图的Corner points 放入images points，二维向量，第一维对应于图像，第二维对应于图像内的corners points；
6. 判断当前images points存放的图像的数量是否超过了10张，如果超过了，则

• 构造obj points ；
• calibrateCamera来获取camera的内外参和畸变相关系数等；
• 通过projectPoints和L2 norm来计算误差；
• 保存camera的内外参和使用的image points到输出文件中；
• 结束
•  

基于camera的内参和畸变相关系数对图像进行畸变校正

1. 从图片列表中读取图像，或从camera中获取图像。
2. initUndistortRectifyMap，计算畸变和修正转换的映射；
3. Remap对图像进行修正。
4. 结束

 

1. 1. Demo的入参

demo的输入参数：

1. -w=6，board_width，棋盘的x-axis有6个Corner；
2. -h=9，board_height，棋盘的y-axis有9个Corner；
3. Inputdata=cameraChessboard/path.yml，它的组织形式按seq方式组织如下：

 

%YAML:1.0

---

image:

   - "/data_1/image/cameraChessboard/IR_gain0x0_exp0x0_0.ppm.jpg"

   - "/data_1/image/cameraChessboard/IR_gain0x0_exp0x0_7.ppm.jpg"

   - "/data_1/image/cameraChessboard/IR_gain0x0_exp0x0_8.ppm.jpg"

   - "/data_1/image/cameraChessboard/IR_gain0x0_exp0x0_9.ppm.jpg"

1. Pattern，默认的CHESSBOARD。
2. square size，默认为1；
3. -op:把检测的图像的points写入output文件；
4. -o=:采用默认的，生成out_camera_data.yml文件
5. -oe:把camera的外参写入output文件；
6. -su:显示经过camera标定后的进行畸变校正后的图像；

 

1. 1. 结果分析

结果分析：

%YAML:1.0

---

calibration_time: "2018年08月17日 星期五 11时23分31秒"

nframes: 44                                       进行计算的图像数量

image_width: 640                              图像的像素

image_height: 480

board_width: 6                                    水平方向corners数量

board_height: 9                                   垂直方向corners数量

square_size: 1.                                     默认方格的大小

aspectRatio: 1.                                    

flags: 2                                                  calibrateCamera函数进行标定时调整的方式

camera_matrix: !!opencv-matrix          内参矩阵3x3见上面的描述

   rows: 3

   cols: 3

   dt: d

   data: [ 4.6958590239705154e+02, 0., 3.0591894265453857e+02, 0.,

       4.6958590239705154e+02, 2.4178386307453937e+02, 0., 0., 1. ]

distortion_coefficients: !!opencv-matrix      

（畸变相关系数，见畸变相关的描述，排列的顺序是k_1,k_2,p_1,p_2,k_3,[k_4,...]

   rows: 5

   cols: 1

   dt: d

   data: [ -2.4717467623780354e-02, 7.0579448978055140e-02,

       -6.3739666531135988e-03, -2.2351753848386508e-03,

       -4.5992594230796928e-02 ]

avg_reprojection_error: 3.1941260400210775e-01            平均误差

per_view_reprojection_errors: !!opencv-matrix               每个图像的误差统计

   rows: 44

   cols: 1

   dt: f

   data: [ 2.38993719e-01, 2.23887086e-01, 2.77384073e-01,

       3.50579470e-01, 3.56113821e-01, 3.43528956e-01, 3.36770862e-01,

       3.70998591e-01, 2.50331372e-01, 3.63689899e-01, 3.79765749e-01,

       3.50891143e-01, 3.61140341e-01, 4.00476992e-01, 4.04239237e-01,

       4.00084198e-01, 4.07668471e-01, 4.06589746e-01, 4.13059950e-01,

       2.50337362e-01, 4.17318434e-01, 2.57471532e-01, 3.22800189e-01,

       3.06288958e-01, 2.91867703e-01, 3.00722539e-01, 3.05220723e-01,

       2.90379196e-01, 2.92704344e-01, 2.83548862e-01, 2.55934417e-01,

       2.87461996e-01, 2.81719893e-01, 2.88423806e-01, 2.91990936e-01,

       2.76637942e-01, 2.92850018e-01, 2.87989706e-01, 2.85537958e-01,

       2.95785844e-01, 2.91026205e-01, 2.55670249e-01, 2.49187574e-01,

       2.54454195e-01 ]

extrinsic_parameters: !!opencv-matrix            每个图像的外参，三个旋转参数，三个平移参数

   rows: 44

   cols: 6

   dt: d

   data:[ -1.0252698275569812e-03, 1.0134541447830549e-01,

       3.1146873534273984e+00, 7.2093171284807358e+00,

       8.0120685080172516e-01, 2.1125111161235626e+01,

       -3.1164670130659883e-03, 1.0227472942994664e-01,

       3.1148242287664636e+00, 7.2148129695101835e+00,

       8.1320260621666129e-01, 2.1130417808260610e+01,

......................................................

 

 

1. 1. 函数介绍
      1. findChessboardCorners函数---找到corners
2. 函数原型：bool findChessboardCorners(InputArray image, Size patternSize, OutputArray corners, int flags=CALIB_CB_ADAPTIVE_THRESH+CALIB_CB_NORMALIZE_IMAGE )
3. 输入：

• image – Source chessboard view. It must be an 8-bit grayscale or color image.
• patternSize – Number of inner corners per a chessboard row and column ( patternSize = cvSize(points_per_row,points_per_colum) = cvSize(columns,rows) ).
• flags –Various operation flags that can be zero or a combination of the following values:

• CALIB_CB_ADAPTIVE_THRESH Use adaptive thresholding to convert the image to black and white, rather than a fixed threshold level (computed from the average image brightness).
• CALIB_CB_NORMALIZE_IMAGE Normalize the image gamma with equalizeHist() before applying fixed or adaptive thresholding.
• CALIB_CB_FILTER_QUADS Use additional criteria (like contour area, perimeter, square-like shape) to filter out false quads extracted at the contour retrieval stage.
• CALIB_CB_FAST_CHECK Run a fast check on the image that looks for chessboard corners, and shortcut the call if none is found. This can drastically speed up the call in the degenerate condition when no chessboard is observed.

1. 输出：

• corners – Output array of detected corners.

1. 返回值：如果找到corners，就返回1，否则返回0；

 

1. 1. 1. cornerSubPix函数---获取更加精确的corners的位置
2. 函数原型：void cornerSubPix(InputArray image, InputOutputArray corners, Size winSize, Size zeroZone, TermCriteria criteria)
3. 输入：

• image – Input image.
• winSize – Half of the side length of the search window. For example, if winSize=Size(5,5) , then a 5*2+1 \times 5*2+1 = 11 \times 11 search window is used.
• zeroZone – Half of the size of the dead region in the middle of the search zone over which the summation in the formula below is not done. It is used sometimes to avoid possible singularities of the autocorrelation matrix. The value of (-1,-1) indicates that there is no such a size.我们一般采用默认值(-1,-1) 。
• criteria – Criteria for termination of the iterative process of corner refinement. That is, the process of corner position refinement stops either after criteria.maxCount iterations or when the corner position moves by less than criteria.epsilon on some iteration.

1. 输入&&输出：

• corners –  Initial coordinates of the input corners && Output array of  refined coordinates corners.

1. 返回值：无；
2. 调用举例：

cornerSubPix( viewGray, pointbuf, Size(11,11), Size(-1,-1), TermCriteria( TermCriteria::EPS+TermCriteria::COUNT, 30, 0.1 ));

 

1. 1. 1. drawChessboardCorners---用直线把一个个corners连接起来
2. 函数原型：void drawChessboardCorners(InputOutputArray image, Size patternSize, InputArray corners, bool patternWasFound)
3. 输入：

• patternSize – Number of inner corners per a chessboard row and column (patternSize = cv::Size(points_per_row,points_per_column)).
• corners – Array of detected corners, the output of findChessboardCorners.
• patternWasFound – Parameter indicating whether the complete board was found or not. The return value of findChessboardCorners() should be passed here.

1. 输入&&输出：

• image– Destination image. It must be an 8-bit color image.

1. 返回值：无；
2. 调用举例：

drawChessboardCorners( view, boardSize, Mat(pointbuf), found );

 

1. 1. 1. calibrateCamera函数---进行camera参数标定
2. 函数原型：double calibrateCamera(InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints, Size imageSize, InputOutputArray cameraMatrix, InputOutputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, int flags=0, TermCriteria criteria=TermCriteria( TermCriteria::COUNT+TermCriteria::EPS, 30, DBL_EPSILON) )
3. 输入：

• objectPoints– In the new interface it is a vector of vectors of calibration pattern points in the calibration pattern coordinate space (e.g. std::vector>). The outer vector contains as many elements as the number of the pattern views. If the same calibration pattern is shown in each view and it is fully visible, all the vectors will be the same. Although, it is possible to use partially occluded patterns, or even different patterns in different views. Then, the vectors will be different. The points are 3D, but since they are in a pattern coordinate system, then, if the rig is planar, it may make sense to put the model to a XY coordinate plane so that Z-coordinate of each input object point is 0.在实际使用上把Z轴坐标设置为0；对于chessboard方式做标定时，因为棋盘是正方形的固定大小，因此，可以基于宽和高的corners的个数直接设置object Points的坐标。
• imagePoints –In the new interface it is a vector of vectors of the projections of calibration pattern points (e.g. std::vector>). imagePoints.size() and objectPoints.size() and imagePoints[i].size() must be equal to objectPoints[i].size() for each i.
• point_counts – 二维向量，标识所有图像的所有corners。第二维是一张图像的所有corners的坐标点；第一维是所有的图像。
• imageSize –图像的大小。 Size of the image used only to initialize the intrinsic camera matrix.
•   A = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1} . If CV_CALIB_USE_INTRINSIC_GUESS and/or CV_CALIB_FIX_ASPECT_RATIO are specified, some or all of fx, fy, cx, cy must be initialized before calling the function.
• flags –Different flags that may be zero or a combination of the following values:

• CV_CALIB_USE_INTRINSIC_GUESS cameraMatrix contains valid initial values of fx, fy, cx, cy that are optimized further. Otherwise, (cx, cy) is initially set to the image center ( imageSize is used), and focal distances are computed in a least-squares fashion. Note, that if intrinsic parameters are known, there is no need to use this function just to estimate extrinsic parameters. Use solvePnP() instead.
• CV_CALIB_FIX_PRINCIPAL_POINT The principal point is not changed during the global optimization. It stays at the center or at a different location specified when CV_CALIB_USE_INTRINSIC_GUESS is set too.
• CV_CALIB_FIX_ASPECT_RATIO The functions considers only fy as a free parameter. The ratio fx/fy stays the same as in the input cameraMatrix . When CV_CALIB_USE_INTRINSIC_GUESS is not set, the actual input values of fx and fy are ignored, only their ratio is computed and used further.
• CV_CALIB_ZERO_TANGENT_DIST Tangential distortion coefficients  (p_1, p_2) are set to zeros and stay zero.
• CV_CALIB_FIX_K1,...,CV_CALIB_FIX_K6 The corresponding radial distortion coefficient is not changed during the optimization. If CV_CALIB_USE_INTRINSIC_GUESS is set, the coefficient from the supplied distCoeffs matrix is used. Otherwise, it is set to 0.
• CV_CALIB_RATIONAL_MODEL Coefficients k4, k5, and k6 are enabled. To provide the backward compatibility, this extra flag should be explicitly specified to make the calibration function use the rational model and return 8 coefficients. If the flag is not set, the function computes and returns only 5 distortion coefficients.

• criteria – Termination criteria for the iterative optimization algorithm.

1. 输出：

• cameraMatrix – 3x3的camera内参矩阵。Output 3x3 floating-point camera matrix。.
• distCoeffs – 畸变相关系数，输出的排列方式是k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]])。Output vector of distortion coefficients  (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]]) of 4, 5, or 8 elements.
• Rvecs：所有图像的旋转向量。 Output vector of rotation vectors (see Rodrigues() ) estimated for each pattern view (e.g. std::vector>). That is, each k-th rotation vector together with the corresponding k-th translation vector (see the next output parameter description) brings the calibration pattern from the model coordinate space (in which object points are specified) to the world coordinate space, that is, a real position of the calibration pattern in the k-th pattern view (k=0.. M -1).
• tvecs –所有图像的平移向量。 Output vector of translation vectors estimated for each pattern view.

1. 返回值：；
2. 调用举例：

 

1. 1. 1. projectPoints---把3D的object Points投影到camera的image上
2. 函数原型：void projectPoints(InputArray objectPoints, InputArray rvec, InputArray tvec, InputArray cameraMatrix, InputArray distCoeffs, OutputArray imagePoints, OutputArray jacobian=noArray(), double aspectRatio=0 )
3. 输入：

• objectPoints – Array of object points, 3xN/Nx3 1-channel or 1xN/Nx1 3-channel (or vector ), where N is the number of points in the view.
• rvec – Rotation vector. See Rodrigues() for details.
• tvec – Translation vector.
• cameraMatrix – Camera matrix  A。
• distCoeffs – Input vector of distortion coefficients  (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]]) of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.
• aspectRatio –(可选的参数) Optional “fixed aspect ratio” parameter. If the parameter is not 0, the function assumes that the aspect ratio (fx/fy) is fixed and correspondingly adjusts the jacobian matrix.

1. 输出：

• imagePoints – Output array of image points, 2xN/Nx2 1-channel or 1xN/Nx1 2-channel, or vector .
• jacobian –(可选的参数) Optional output 2Nx(10+) jacobian matrix of derivatives of image points with respect to components of the rotation vector, translation vector, focal lengths, coordinates of the principal point and the distortion coefficients. In the old interface different components of the jacobian are returned via different output parameters.

 

1. 调用举例：

projectPoints(Mat(objectPoints[i]), rvecs[i], tvecs[i],cameraMatrix, distCoeffs, imagePoints2);

 

1. 1. 1. Norm函数---计算范数
2. 函数原型：double norm(InputArray src1, int normType=NORM_L2, InputArray mask=noArray())
3. 输入：

• src1 – first input array.
• src2 – second input array of the same size and the same type as src1.
• normType – type of the norm (see the details below).
• mask – optional operation mask; it must have the same size as src1 and CV_8UC1 type.

1. 输出：
2. 返回值：计算结果；
3. 调用举例：

err = norm(Mat(imagePoints[i]), Mat(imagePoints2), NORM_L2);

计算这两个vector Points的L2范数，用于误差测量。

 

1. 1. 1. initUndistortRectifyMap函数---计算出校正畸变和修正变换的映射map
2. 函数原型：void initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2)
3. 输入：

• cameraMatrix – camera的内参矩阵。Input camera matrix  。
• distCoeffs – 畸变相关系数。Input vector of distortion coefficients  (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]]) of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.
• R – Optional rectification transformation in the object space (3x3 matrix). R1 or R2 , computed by stereoRectify() can be passed here. If the matrix is empty, the identity transformation is assumed. In cvInitUndistortMap R assumed to be an identity matrix.。这个主要通过stereoRectify函数获取的R1或R2.主要用于立体视觉处理。
• newCameraMatrix –新的camera内参矩阵。 New camera matrix  。当前的实例是通过getOptimalNewCameraMatrix函数获取的。
• size – Undistorted image size.一般同要做畸变校正的图像相同。
• m1type – Type of the first output map that can be CV_32FC1 or CV_16SC2 . See convertMaps() for details.我们当前使用的是是 CV_16SC2

1. 输出：

• map1 – The first output map.
• map2 – The second output map.

 

1. 返回值： ；
2. 调用举例：

initUndistortRectifyMap(cameraMatrix, distCoeffs, Mat(),getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0),imageSize, CV_16SC2, map1, map2);

 

1. 1. 1. Remap函数---通用的几何变换
2. 函数原型：void remap(InputArray src, OutputArray dst, InputArray map1, InputArray map2, int interpolation, int borderMode=BORDER_CONSTANT, const Scalar& borderValue=Scalar())
3. 输入：

• src – Source image.
• map1 – The first map of either (x,y) points or just x values having the type CV_16SC2 , CV_32FC1 , or CV_32FC2 . See convertMaps() for details on converting a floating point representation to fixed-point for speed.
• map2 – The second map of y values having the type CV_16UC1 , CV_32FC1 , or none (empty map if map1 is (x,y) points), respectively.
• interpolation – Interpolation method (see resize() ). The method INTER_AREA is not supported by this function.
• borderMode – Pixel extrapolation method (see borderInterpolate() ). When borderMode=BORDER_TRANSPARENT , it means that the pixels in the destination image that corresponds to the “outliers” in the source image are not modified by the function.
• borderValue – Value used in case of a constant border. By default, it is 0.

 

1. 输出：

• dst – Destination image. It has the same size as map1 and the same type as src .

1. 返回值： ；
2. 调用举例：

remap(view, rview, map1, map2, INTER_LINEAR);

 

1. 1. 1. getOptimalNewCameraMatrix函数---返回新的基于随机缩放参数的camera 内参矩阵

 

1. 函数原型：Mat getOptimalNewCameraMatrix(InputArray cameraMatrix, InputArray distCoeffs, Size imageSize, double alpha, Size newImgSize=Size(), Rect* validPixROI=0, bool centerPrincipalPoint=false )
2. 输入：

• cameraMatrix –camera的内参。 Input camera matrix.
• distCoeffs – 畸变相关系数。Input vector of distortion coefficients  (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]]) of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.
• imageSize – 计算camera内参所用的图像的大小。Original image size.
• alpha – Free scaling parameter between 0 (when all the pixels in the undistorted image are valid) and 1 (when all the source image pixels are retained in the undistorted image). See stereoRectify() for details.我们使用时，把它设置为1.
• new_imag_size – 修正后的图像大小，默认是同原始图像大小相同。Image size after rectification. By default,it is set to imageSize .
• validPixROI –(可选参数) Optional output rectangle that outlines all-good-pixels region in the undistorted image. See roi1, roi2 description in stereoRectify() .
• centerPrincipalPoint –(可选参数) Optional flag that indicates whether in the new camera matrix the principal point should be at the image center or not. By default, the principal point is chosen to best fit a subset of the source image (determined by alpha) to the corrected image.

 

1. 输出：
2. 返回值： Output new camera matrix.
3. 举例：

getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0)

接下来学习立体视觉部分内容；