The Documentatiion of Kinect Face Tracking
Face Tracking
微软是在其1.5版本以后的SDK中加入这个功能的。在结合微软提供的Kinect SDK 和 Face Tracking SDK 能够在你开发的应用程序中轻松的实现实时的人脸追踪。整体来讲，Face Tracking SDK 通过人脸追踪引擎对Kinect相机的捕获数据进行分析，进而推断出头部姿态和面部表情，并且进一步将这些信息实时的应用到应用中。
在使用Face Tracking SDK 首先必须安装Kinect SDK，并且还 要满足一些运行Face Tracking SDK的系统硬件要求。
System Requirments(系统硬件要求)
最小的硬件需求
- 双核CPU,主频在2.66GHZ或者更快
- Wdiows 7 或者Window 8 操作系统，并且支持Microsoft® DirectX® 9.0c
- 2GB内存空间
- Kinect相机
微软的Visual Studio Express 提供了一个良好的开发、调试、测试的集成开发环境，如果使用的是Visual Studio C++编译器，那么需要做如下两件事：
- 在你的源文件中包含 FaceTrackLib.h 头文件
- 指定要包含头文件的路径
#ifdef _WINDOWS
#include
#include
#endif
And comment it out:
//#ifdef _WINDOWS
#include
#include
//#endif Coordinate System(坐标系统)
Face Tracking SDK 使用和kinect一样的坐标系统，因此三维追踪结果也是按Kinect的坐标输出的。在这个坐标系统中，坐标的原点是在相机的镜头处，Z轴指向人，Y轴垂直向上，如图所示：
API Description(函数调用接口描述)
Face Tracking SDK 是以COM组件的DLL的形式给用户提供脸部追踪功能，用户通过调用接口实现想要的功能。下面先介绍四个主要的COM接口
| 接口 | 接口描述 |
|---|---|
| IFTFaceTracker | The main interface for face tracking. |
| IFTResult | A result of a face tracking operation |
| IFTImage | A helper interface that wraps various image buffers. |
| IFTModel | A fitted 3D face model. |
对于以上接口你可以使用FTCreateFaceTracker()函数创建一个IFTFaceTracker对象，可以使用FTCreateImage()函数创建一个IFImage对象，对于IFTResult和IFTModel来说是被IFTFaceTracker创建的。
同事Face Tracking SDK 可以使用如下的数据结构：
| Structure | Description |
|---|---|
| FT_SENSOR_DATA | Contains all input data for a face tracking operation. |
| FT_CAMERA_CONFIG | Contains the configuration of the video or depth sensor which frames are being tracked. |
| FT_VECTOR2D | Contains the points of a 2D vector. |
| FT_VECTOR3D | Contains the points of a 3D vector. |
| FT_TRIANGLE | Contains a 3D face model triangle. |
| FT_WEIGHTED_RECT | Contains a weighted rectangle returned by the Face Tracking SDK. |
// This example assumes that the application provides
// void* cameraFrameBuffer, a buffer for an image, and that there is a method
// to fill the buffer with data from a camera, for example
// cameraObj.ProcessIO(cameraFrameBuffer)
// Create an instance of face tracker
IFTFaceTracker* pFT = FTCreateFaceTracker();
if(!pFT)
{
// Handle errors
}
FT_CAMERA_CONFIG myColorCameraConfig = {640, 480, 1.0}; // width, height, focal length
FT_CAMERA_CONFIG myDepthCameraConfig = {640, 480}; // width, height
HRESULT hr = pFT->Initialize(&myColorCameraConfig, &myDepthCameraConfig, NULL, NULL);
if( FAILED(hr) )
{
// Handle errors
}
// Create IFTResult to hold a face tracking result
IFTResult* pFTResult = NULL;
hr = pFT->CreateFTResult(&pFTResult);
if(FAILED(hr))
{
// Handle errors
}
// prepare Image and SensorData for 640x480 RGB images
IFTImage* pColorFrame = FTCreateImage();
if(!pColorFrame)
{
// Handle errors
}
// Attach assumes that the camera code provided by the application
// is filling the buffer cameraFrameBuffer
pColorFrame->Attach(640, 480, cameraFrameBuffer, FORMAT_UINT8_R8G8B8, 640*3);
FT_SENSOR_DATA sensorData;
sensorData.pVideoFrame = &colorFrame;
sensorData.ZoomFactor = 1.0f;
sensorData.ViewOffset = POINT(0,0);
bool isTracked = false;
// Track a face
while ( true )
{
// Call your camera method to process IO and fill the camera buffer
cameraObj.ProcessIO(cameraFrameBuffer); // replace with your method
// Check if we are already tracking a face
if(!isTracked)
{
// Initiate face tracking. This call is more expensive and
// searches the input image for a face.
hr = pFT->StartTracking(&sensorData, NULL, NULL, pFTResult);
if(SUCCEEDED(hr) && SUCCEEDED(pFTResult->Status))
{
isTracked = true;
}
else
{
// Handle errors
isTracked = false;
}
}
else
{
// Continue tracking. It uses a previously known face position,
// so it is an inexpensive call.
hr = pFT->ContinueTracking(&sensorData, NULL, pFTResult);
if(FAILED(hr) || FAILED (pFTResult->Status))
{
// Handle errors
isTracked = false;
}
}
// Do something with pFTResult.
// Terminate on some criteria.
}
// Clean up.
pFTResult->Release();
pColorFrame->Release();
pFT->Release();Face Tracking Interfaces(接口详细讲解)
- IFTFaceTracker
IFTFaceTracker 提供了一个CreateFTResult方法，用于创建一个用来保存face tracking 结果的对象IFTResult，在开始进行秒不追踪的时候，TFTResult是必须要被创建的。在应用程序中，TFTFaceTracker方法在FT_SENSOR_DATA这个图像数据中画出一些潜在的头部区域，方法调用者根据实际情况决定要对哪个头进行跟踪。
在开始面部跟踪前，要先调用StartTracking方法。StartTracking方法是一个开销非常大的方法，要实现脸部的定位，决定脸部的方位，并且开始跟踪。可以通过传入一个hint去跟踪指定的脸也可以通过传入NULL去遍历整幅图像——第一个被找到的脸将会被跟踪。当返回一个非空的pFTResult指针时，表示StartTracking被成功的调用。
当应用程序需要长时间的跟踪face时，需要调用ContinueTracking函数。保持调用ContinueTracking函数直到你想终止face tracking 或者face tracking 失败，例如被追踪的人走出了摄像头的外部。如果发生face tracking失败，失败的信息将会在pFTResult 相应的状态位中显示出来。当要重新开始face tracking时，应用程序就要重新调用StartTracking 和ContinueTracking。
ContinueTracking相对StartTracking来说调用的开销要小的多，因此对于StartTracking来说只需要在开始face Tracking的时候调用，之后调用ContinueTracking即可。
- IFTResult
- GetFaceRect —— 一个将被追踪face框起来的矩形
- Get2DShapePoints ——2D(X,Y)坐标。并且一共有87个这样的2D坐标
- IFTModel
IFTModel提供了一个完成将追踪信息向3D空间转化的一个功能接口。他通过IFTFaceTracker::GetFaceModel()方法创建。这个接口通过不同的方法获取众多不同的模型：
GetSUCount, GetAUCount – returns number of shape units (SU) or animation units (AU) used in the 3D linear model
GetTriangles – returns 3D model mesh triangles (indexes of vertices). Each triangle has 3 vertex indexes listed in the clockwise fashion.
GetVertexCount – returns number of vertices in the 3D model mesh
Also, IFTModel provides two methods to get a 3D face model in either the video camera space or projected onto the video camera image plane. These methods are:
Get3DShape - returns the 3D face model vertices transformed by the passed Shape Units, Animation Units, scale stretch, rotation and translation
GetProjectedShape - Returns the 3D face model vertices transformed by the passed Shape Units, Animation Units, scale stretch, rotation and translation and projected to the video frame
下面的代码演示了完成转换：
HRESULT VisualizeFaceModel(
IFTImage* pColorImg,
IFTModel* pModel,
FT_CAMERA_CONFIG const* pCameraConfig,
FLOAT const* pSUCoef,
FLOAT zoomFactor,
POINT viewOffset,
IFTResult* pAAMRlt,
UINT32 color
)
{
if (!pColorImg || !pModel || !pCameraConfig || !pSUCoef || !pAAMRlt)
{
return E_POINTER;
}
HRESULT hr = S_OK;
UINT vertexCount = pModel->GetVertexCount();
FT_VECTOR2D* pPts2D = reinterpret_cast
(_malloca(sizeof(FT_VECTOR2D) * vertexCount));
if (pPts2D)
{
FLOAT *pAUs;
UINT auCount;
hr = pAAMRlt->GetAUCoefficients(&pAUs, &auCount);
if (SUCCEEDED(hr))
{
FLOAT scale, rotationXYZ[3], translationXYZ[3];
hr = pAAMRlt->Get3DPose(&scale, rotationXYZ, translationXYZ);
if (SUCCEEDED(hr))
{
hr = pModel->GetProjectedShape(pCameraConfig, zoomFactor, viewOffset,
pSUCoef, pModel->GetSUCount(), pAUs, auCount,
scale, rotationXYZ, translationXYZ, pPts2D, vertexCount);
if (SUCCEEDED(hr))
{
POINT* p3DMdl = reinterpret_cast
(_malloca(sizeof(POINT) * vertexCount));
if (p3DMdl)
{
for (UINT i = 0; iGetTriangles(&pTriangles, &triangleCount);
if (SUCCEEDED(hr))
{
struct EdgeHashTable
{
UINT32* pEdges;
UINT edgesAlloc;
void Insert(int a, int b)
{
UINT32 v = (min(a, b) << 16) | max(a, b);
UINT32 index = (v + (v << 8)) * 49157, i;
for ( i = 0; i
(_malloca(sizeof(UINT32) * eht.edgesAlloc));
if (eht.pEdges)
{
ZeroMemory(eht.pEdges,
sizeof(UINT32) * eht.edgesAlloc);
for (UINT i = 0; i < triangleCount; ++i)
{
eht.Insert(pTriangles[i].i, pTriangles[i].j);
eht.Insert(pTriangles[i].j, pTriangles[i].k);
eht.Insert(pTriangles[i].k, pTriangles[i].i);
}
for (UINT i = 0; i < eht.edgesAlloc; ++i)
{
eht.pEdges[i] & pColorImg->DrawLine(
p3DMdl[eht.pEdges[i] >> 16],
p3DMdl[eht.pEdges[i] & 0xFFFF],
color, 1 );
}
_freea(eht.pEdges);
}
// Render the face rect in magenta
RECT rectFace;
hr = pAAMRlt->GetFaceRect(&rectFace);
if (SUCCEEDED(hr))
{
POINT leftTop = {rectFace.left, rectFace.top};
POINT rightTop = {rectFace.right - 1, rectFace.top};
POINT leftBottom = {rectFace.left,
rectFace.bottom - 1};
POINT rightBottom = {rectFace.right - 1,
rectFace.bottom - 1};
UINT32 nColor = 0xff00ff;
SUCCEEDED(hr = pColorImg->DrawLine(leftTop, rightTop, nColor, 1)) &
SUCCEEDED(hr = pColorImg->DrawLine(rightTop, rightBottom, nColor, 1)) &
SUCCEEDED(hr = pColorImg->DrawLine(rightBottom, leftBottom, nColor, 1)) &
SUCCEEDED(hr = pColorImg->DrawLine(leftBottom, leftTop, nColor, 1));
}
}
_freea(p3DMdl);
}
else
{
hr = E_OUTOFMEMORY;
}
}
}
}
_freea(pPts2D);
}
else
{
hr = E_OUTOFMEMORY;
}
return hr;
}