基于AR的实时算法在虚拟屏幕上播放视频及实现

        AR（Augmented Reality）是计算机视觉方面一个重要的研究方向，原来听说过，感觉很有意思，故究其原理，用一定的视觉算法加以实现。

这里主要用到相机标定和一些图像的投影透视变换的技术，标定的目的是获取相机的内外参，从而获得投影矩阵P，然后进行下面的点映射之间的关系。一般情况下的相机标定是非常成熟不过的东西了，技术文章也是铺天盖地，目前绝大数标定方法都属于张正友的方法，OpenCV和Matlab官方都是采用张正友的，在此不具体阐述，如有需要，请参考官方文档。但在现实应用中，张正友方法并非总是合适，要适合自己的场合使用，就需要掌握额外的高等数学知识，尤其是对相机矩阵P的理解。本篇博客不深入透析标定原理，坐标关系，只做简单说明。透视变换可参考我这篇博客原理。

一、基于AR的Demo

直接上动态gif图，把一部电影片段加进去了：），（限于CSDN只能上传小于5M的图片，没办法长传完整的( ▼-▼ )）

                             

二、代码实现

运行以下代码前，需要实现对相机进行标定，简单标定可参考这里。“mydata.mat”为标定后存储的参数。为了提高速度，程序基本上没有循环，尽量采用矩阵向量运算。

function main()
% AR demo
% author:cuixingxing
% email: [email protected]
% 2018.8.4
%

close all;
clear;clc;

%% 导入各类参数,AR增强现实，加入图片
load('mydata.mat')% 相机标定的数据，保存相机cameraParameters对象,图像点和世界坐标点集
logo = imread('demo.png');% 读入自己的图片，图片任意
  
%% 
movingPts = [1,1;
    size(logo,2),1;
    1,size(logo,1);
    size(logo,2),size(logo,1)];
cap1 = webcam(3);
squareSize = 38.72;% 标定版方格宽度 mm

v = VideoWriter('my_AR.avi');% save
open(v);
colors = randi(255,12,3);% 12 条棱线颜色
player1 = vision.DeployableVideoPlayer('Location', [20, 400]);
frame1 = snapshot(cap1);
step(player1, frame1);

while player1.isOpen()
    logo = imresize(logo,[200,320]);% 电影图像
    frame1 = snapshot(cap1);
    drawImg = frame1;
    [imagePoints,boardSize,isused]=detectCheckerboardPoints(frame1);
    if all(isused)&& size(imagePoints,1)==size(worldPoints1,1)
        rows = boardSize(1)-1;
        cols = boardSize(2) - 1;
        % 世界坐标中4个角点坐标,这个定义好
        point3D_1 = [0,0,0];
        point3D_2 = [cols*squareSize,point3D_1(2),0];
        point3D_3 = [point3D_1(1),rows*squareSize,0];
        point3D_4 = [point3D_1(1),point3D_1(2),-220];% 单位毫米
        
        % 抠图到长方体顶面
        [rotMat,transVec] = extrinsics(imagePoints,worldPoints1,cameraParams1);
        projectMat = cameraMatrix(cameraParams1,rotMat,transVec);
        [lines,points2D] = GetSquareEdge(point3D_1,point3D_2,point3D_3,point3D_4,projectMat);
        tform = fitgeotrans(movingPts,points2D(4:7,:),'projective');
        logo_tf = imwarp(logo,tform);
        [x,y,Height,Width] = GetHW(points2D(4:7,:),frame1);
        if isempty(x)
            continue;
        end
        logo_tf = imresize(logo_tf,[Height,Width]);
        ROI = frame1(y:y+Height-1,x:x+Width-1,:);
        ROI(logo_tf~=0) = logo_tf(logo_tf~=0);
        frame1(y:y+Height-1,x:x+Width-1,:) = ROI;
        
        % 在图像上画轴
        drawImg = insertShape(frame1,'Line',lines,'LineWidth',8,'Color',colors);
        step(player1, drawImg);
    else
        step(player1, drawImg);
    end 
    writeVideo(v,drawImg);
end
close(v);
release(player1);

 一些子函数：

注意点顺序关系图为：

function [lines,points2D] = GetSquareEdge(point3D_1,point3D_2,point3D_3,point3D_4,projectMat)
% 功能：根据相邻的4个角点，获取正方形的棱边
% 输入：point3D_1，1*3 double ，[x,y,z]，坐标原点世界坐标
%      point3D_2，1*3 double ，[x,y,z]，相邻角点世界坐标
%      point3D_3，1*3 double ，[x,y,z]，相邻角点世界坐标
%      point3D_4，1*3 double ，[x,y,z]，相邻角点世界坐标
%      projectMat，4*3 double,相机投影矩阵,注意有的地方是3*4，有个转置关系
% 输出： lines, 12*4 double,形如[x1,y1,x2,y2; x3,y3,x4,y4,...]每行为每条线的起始点和终点坐标
%       points2D, 8*2 double，图像上顶点坐标
%
% author:cuixingxing
% email: [email protected]
% 2018.8.4
%

point3D_5 = [point3D_2(1),point3D_1(2),point3D_4(3)];
        point3D_6 = [point3D_1(1),point3D_3(2),point3D_4(3)];
        point3D_7 = [point3D_2(1),point3D_3(2),point3D_4(3)];
        point3D_8 = [point3D_2(1),point3D_3(2),point3D_1(3)];
        points3D = [point3D_1;
            point3D_2;
            point3D_3;
            point3D_4;
            point3D_5;
            point3D_6;
            point3D_7;
            point3D_8];
        
%% points2D
 points2D = [points3D,ones(8,1)]*projectMat;
        points2D = points2D./points2D(:,3);
        points2D = points2D(:,1:2);
        lines = [points2D(1,:),points2D(2,:);
            points2D(1,:),points2D(3,:);
            points2D(2,:),points2D(8,:);
            points2D(3,:),points2D(8,:);
            points2D(1,:),points2D(4,:);
            points2D(3,:),points2D(6,:);
            points2D(2,:),points2D(5,:);
            points2D(7,:),points2D(8,:);
            points2D(4,:),points2D(6,:);
            points2D(4,:),points2D(5,:);
            points2D(5,:),points2D(7,:);
            points2D(6,:),points2D(7,:)];

 

function [minx,miny,Height,Width] = GetHW(points,frame)
% 功能：获取二维图像上n个平面点集points的最小平行于轴的矩形
% 输入：points, n*2 double ,[x,y]点集坐标，像素坐标
%      frame,原图像
%输出： minWidth,矩形宽度，1*1 double
%      minHeight,矩形高度，1*1 double
%
% author:cuixingxing
% email: [email protected]
% 2018.8.4
%

minx = round(min(points(:,1)));
maxx = round(max(points(:,1)));
miny = round(min(points(:,2)));
maxy = round(max(points(:,2)));

Width = maxx-minx;
Height = maxy-miny;

if minx<=0||miny<=0||minx+Width>=size(frame,2)||miny+Height>=size(frame,1)
    minx =[];
    miny = [];
    Width = [];
    Height = [];
end