FFMPEG录屏（12）---- 捕获桌面(Duplication)

最近增加了对Duplication API捕获桌面的支持，记录一下过程和其中遇到的问题。

参考资料

Desktop Duplication Api

AccquireNextFrame

DXGI_OUTDUPL_POINTER_SHAPE_TYPE

官方Demo

吐槽

DX这套接口是真的烦，真的烦，为了获取到duplication接口，你得初始化一堆东西啊啊啊啊啊啊啊！初始化的你眼花缭乱啊啊啊啊啊啊啊啊！

准备工作

避免系统中没有d3d依赖，所有d3d接口通过动态加载方式引入程序。准备一个简单的函数用来加载动态库。

1. 获取系统路径

static char system_path[260] = { 0 };

static bool get_system_path() {

	if (strlen(system_path) == 0) {
		UINT ret = GetSystemDirectoryA(system_path, MAX_PATH);
		if (!ret) {
			al_fatal("failed to get system directory :%lu", GetLastError());
			return false;
		}
	}

	return true;
}

2. 加载动态库

HMODULE load_system_library(const char * name)
{
	if (get_system_path() == false) return NULL;

	char base_path[MAX_PATH] = { 0 };
	strcpy(base_path, system_path);
	strcat(base_path, "\\");
	strcat(base_path, name);

	HMODULE module = GetModuleHandleA(base_path);
	if (module)
		return module;

	module = LoadLibraryA(base_path);
	if (!module) {
		al_error("failed load system library :%lu", GetLastError());
	}

	return module;
}

3. 释放动态库

void free_system_library(HMODULE handle)
{
	FreeModule(handle);
}

这里讲一下GetModuleHandle和LoadLibrary，前者会首先看当前进程空间是否已经引入了模块，如果有则计数器加一，并返回模块句柄。

干点正事

1. 引入d3d11.dll和dxgi.dll

HMODULE _d3d11 = load_system_library("d3d11.dll");
HMODULE _dxgi = load_system_library("dxgi.dll");

2. 获取D3D11CreateDevice函数指针

PFN_D3D11_CREATE_DEVICE create_device = 
				(PFN_D3D11_CREATE_DEVICE)GetProcAddress(_d3d11, "D3D11CreateDevice");

3. 创建一个d3d设备

ID3D11Device* _d3d_device;
HRESULT hr = S_OK;

// Driver types supported
D3D_DRIVER_TYPE driver_types[] =
{
	D3D_DRIVER_TYPE_HARDWARE,
	D3D_DRIVER_TYPE_WARP,
	D3D_DRIVER_TYPE_REFERENCE,
};
UINT n_driver_types = ARRAYSIZE(driver_types);

// Feature levels supported
D3D_FEATURE_LEVEL feature_levels[] =
{
	D3D_FEATURE_LEVEL_11_0,
	D3D_FEATURE_LEVEL_10_1,
	D3D_FEATURE_LEVEL_10_0,
	D3D_FEATURE_LEVEL_9_1
};
UINT n_feature_levels = ARRAYSIZE(feature_levels);

D3D_FEATURE_LEVEL feature_level;

// Create device
for (UINT driver_index = 0; driver_index < n_driver_types; ++driver_index)
{
	hr = create_device(nullptr, driver_types[driver_index], nullptr, 0, feature_levels, n_feature_levels,
		D3D11_SDK_VERSION, &_d3d_device, &feature_level, &_d3d_ctx);
	if (SUCCEEDED(hr)) break;
}

需要注意的一点是D3D11CreateDevice函数的第一个参数*IDXGIAdapter pAdapter，引入微软的说明：

A pointer to the video adapter to use when creating a device. Pass NULL to use the default adapter, which is the first adapter that is enumerated by IDXGIFactory1::EnumAdapters.
Note Do not mix the use of DXGI 1.0 (IDXGIFactory) and DXGI 1.1 (IDXGIFactory1) in an application. Use IDXGIFactory or IDXGIFactory1, but not both in an application.

什么意思呢，就是这里会指定你要基于哪个显示适配器创建你的d3d device，再换句话，就是你要捕获哪个屏幕的画面，如果你有多屏幕的话。这里我们传入nullptr，捕获默认的主显示器。

4. 根据d3d设备获取对应的dxgi设备接口

IDXGIDevice* dxgi_device = nullptr;
HRESULT hr = _d3d_device->QueryInterface(__uuidof(IDXGIDevice), reinterpret_cast<void**>(&dxgi_device));

5. 获取dxgi设备适配器

IDXGIAdapter* dxgi_adapter = nullptr;
hr = dxgi_device->GetParent(__uuidof(IDXGIAdapter), reinterpret_cast<void**>(&dxgi_adapter));
dxgi_device->Release();
dxgi_device = nullptr;

6. 获取设备输出接口

IDXGIOutput* dxgi_output = nullptr;
hr = dxgi_adapter->EnumOutputs(_output_index, &dxgi_output);
dxgi_adapter->Release();
dxgi_adapter = nullptr;

7. 获取设备输出描述

DXGI_OUTPUT_DESC _output_des;
dxgi_output->GetDesc(&_output_des);

8. 根据设备输出接口创建一个duplication接口

IDXGIOutput1* dxgi_output1 = nullptr;
hr = dxgi_output->QueryInterface(__uuidof(dxgi_output1), reinterpret_cast<void**>(&dxgi_output1));
dxgi_output->Release();
dxgi_output = nullptr;

IDXGIOutputDuplication *_duplication;
hr = dxgi_output1->DuplicateOutput(_d3d_device, &_duplication);
dxgi_output1->Release();
dxgi_output1 = nullptr;

至此，我们的初始化工作告一段落了。。。其中有一些名词如Output、Adaptor、Interface等翻译可能有些不准确导致看起来有些怪，所以还是建议每个函数去看一下官方文档为准，以防理解有误。

捕获图像

1. 获取一帧DXGI资源

IDXGIResource* dxgi_res = nullptr;

// Get new frame
DXGI_OUTDUPL_FRAME_INFO frame_info;
HRESULT hr = _duplication->AcquireNextFrame(500, frame_info, &dxgi_res);

// Timeout will return when desktop has no chane
if (hr == DXGI_ERROR_WAIT_TIMEOUT) return AE_TIMEOUT;

if (FAILED(hr)) 
	return AE_DUP_ACQUIRE_FRAME_FAILED;

// If still holding old frame, destroy it
if (_image)
{
	_image->Release();
	_image = nullptr;
}

这里要注意的是AcquireNextFrame的返回值，当桌面画面没有变化或没有新图像到来时会返回DXGI_ERROR_WAIT_TIMEOUT，此时无需做图像更新操作，直接返回循环等待下一帧图像。

Return value
AcquireNextFrame returns:

• S_OK if it successfully received the next desktop image.
• DXGI_ERROR_ACCESS_LOST if the desktop duplication interface is invalid. The desktop duplication interface typically becomes invalid when a different type of image is displayed on the desktop. Examples of this situation are:
  – Desktop switch
  – Mode change
  – Switch from DWM on, DWM off, or other full-screen application
  In this situation, the application must release the IDXGIOutputDuplication interface and create a new IDXGIOutputDuplication for the new content.
• DXGI_ERROR_WAIT_TIMEOUT if the time-out interval elapsed before the next desktop frame was available.
• DXGI_ERROR_INVALID_CALL if the application called AcquireNextFrame without releasing the previous frame.
• E_INVALIDARG if one of the parameters to AcquireNextFrame is incorrect; for example, if pFrameInfo is NULL.
• Possibly other error codes that are described in the DXGI_ERROR topic.

其他错误代码则需要释放释放duplication接口并重新创建，最常见的是系统session切换的时候，比如运行需要管理员权限的程序弹出权限请求画面时，不仅捕获不到桌面画面，还需要重新初始化duplication接口。在之前的项目中，有以系统服务运行的屏幕捕获程序，需要在session变化时切换不同的捕获方式，比如切换到GDI捕获。

2. 获取一帧图像纹理

ID3D11Texture2D *_image;
hr = dxgi_res->QueryInterface(__uuidof(ID3D11Texture2D), reinterpret_cast<void **>(&_image));
dxgi_res->Release();
dxgi_res = nullptr;

3. 创建新的纹理并拷贝图像

// Copy old description
D3D11_TEXTURE2D_DESC frame_desc;
_image->GetDesc(&frame_desc);

// Create a new staging buffer for fill frame image
ID3D11Texture2D *new_image = NULL;
frame_desc.Usage = D3D11_USAGE_STAGING;
frame_desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
frame_desc.BindFlags = 0;
frame_desc.MiscFlags = 0;
frame_desc.MipLevels = 1;
frame_desc.ArraySize = 1;
frame_desc.SampleDesc.Count = 1;
hr = _d3d_device->CreateTexture2D(&frame_desc, NULL, &new_image);

// Copy next staging buffer to new staging buffer
_d3d_ctx->CopyResource(new_image, _image);

4. 将图像从GPU映射到内存中

// Create staging buffer for map bits
IDXGISurface *dxgi_surface = NULL;
hr = new_image->QueryInterface(__uuidof(IDXGISurface), (void **)(&dxgi_surface));
new_image->Release();

// Map buff to mapped rect structure
DXGI_MAPPED_RECT mapped_rect;
hr = dxgi_surface->Map(&mapped_rect, DXGI_MAP_READ);

4. 拷贝图像数据到缓冲区内 （此处有错误，更正见文章尾部）

memcpy(_buffer, mapped_rect.pBits, _buffer_size);
dxgi_surface->Unmap();

这里需要注意的是，Duplication API捕获到的桌面数据格式总是BGRA，所以我们的缓冲区初始化大小就是width * height * 4

捕获鼠标

1. 定义鼠标数据结构体

typedef struct _PTR_INFO
{
	_Field_size_bytes_(BufferSize) BYTE* buff;
	DXGI_OUTDUPL_POINTER_SHAPE_INFO shape;
	POINT position;
	bool visible;
	UINT size;
	UINT output_index;
	LARGE_INTEGER pre_timestamp;
} DUPLICATION_CURSOR_INFO;

DUPLICATION_CURSOR_INFO _cursor_info;

2. 判断鼠标形状、坐标是否发生变化

// A non-zero mouse update timestamp indicates that there is a mouse position update and optionally a shape change
if (frame_info->LastMouseUpdateTime.QuadPart == 0)
	return AE_NO;

bool b_updated = true;

// Make sure we don't update pointer position wrongly
// If pointer is invisible, make sure we did not get an update from another output that the last time that said pointer
// was visible, if so, don't set it to invisible or update.
if (!frame_info->PointerPosition.Visible && (_cursor_info.output_index != _output_index))
	b_updated = false;

// If two outputs both say they have a visible, only update if new update has newer timestamp
if (frame_info->PointerPosition.Visible && _cursor_info.visible && (_cursor_info.output_index != _output_index) && (_cursor_info.pre_timestamp.QuadPart > frame_info->LastMouseUpdateTime.QuadPart))
	b_updated = false;

其中frame_info为AcquireNextFrame时获取的图像信息

3. 更新鼠标坐标、时间戳、visible等信息

// Update position
if (b_updated)
{
	_cursor_info.position.x = frame_info->PointerPosition.Position.x + _output_des.DesktopCoordinates.left;
	_cursor_info.position.y = frame_info->PointerPosition.Position.y + _output_des.DesktopCoordinates.top;
	_cursor_info.output_index = _output_index;
	_cursor_info.pre_timestamp = frame_info->LastMouseUpdateTime;
	_cursor_info.visible = frame_info->PointerPosition.Visible != 0;
}

// No new shape only update cursor positions & visible state
if (frame_info->PointerShapeBufferSize == 0)
{
	return AE_NO;
}

这里注意的是当发现PointerShapeBufferSize 为0时，表示鼠标没有更新形状，可能只是更新了坐标和visible属性。

4. 拷贝鼠标形状数据

// Old buffer too small
if (frame_info->PointerShapeBufferSize > _cursor_info.size)
{
	if (_cursor_info.buff)
	{
		delete[] _cursor_info.buff;
		_cursor_info.buff = nullptr;
	}
	_cursor_info.buff = new (std::nothrow) BYTE[frame_info->PointerShapeBufferSize];
	if (!_cursor_info.buff)
	{
		_cursor_info.size = 0;
		return AE_ALLOCATE_FAILED;
	}

	// Update buffer size
	_cursor_info.size = frame_info->PointerShapeBufferSize;
}

// Get shape
UINT BufferSizeRequired;
HRESULT hr = _duplication->GetFramePointerShape(frame_info->PointerShapeBufferSize, reinterpret_cast<VOID*>(_cursor_info.buff), &BufferSizeRequired, &(_cursor_info.shape));
if (FAILED(hr))
{
	delete[] _cursor_info.buff;
	_cursor_info.buff = nullptr;
	_cursor_info.size = 0;
	return AE_DUP_GET_CURSORSHAPE_FAILED;
}

这里需要注意一点，因为鼠标除了系统鼠标外、各种软件是有可能绘制自己的鼠标形状的，比如PS等绘图软件，所以鼠标形状的大小随时会变，因此要动态的扩容鼠标形状缓冲区。

重头戏来了，绘制鼠标

这个方法搜遍全网，没有找到任何参考资料，只有微软例子中给出了对鼠标形状的处理以及绘制，但是实时绘制到窗体上输出，而我们需要和桌面图像合并用来压缩，结合微软对几种鼠标类型的解释和绘制例子实现了绘制鼠标到桌面图形中。

1. BGRA数据存储方式

原谅我无耻的盗图，而且格式是RGBA，但凑合着看吧。
首先要明白一个概念是，BGRA数据的存储格式，是按照行扫描的方式，什么意思呢，就是假设屏幕19201080，那么就把屏幕分成1080行，1920列，把整个屏幕分成19201080个像素点，每个像素点有R、G、B、A三个颜色和一个透明度表示。
也就是说，你的屏幕数据映射到内存中后就是存了这么一个数据数组，里面包含了每个像素点的数据，一个像素点占用4个字节。

2. 为什么要自己画鼠标

You need to use the desktop duplication API to determine if your client app must draw the mouse pointer shape onto the desktop image. Either the mouse pointer is already drawn onto the desktop image that IDXGIOutputDuplication::AcquireNextFrame provides or the mouse pointer is separate from the desktop image. If the mouse pointer is drawn onto the desktop image, the pointer position data that is reported by AcquireNextFrame (in the PointerPosition member of DXGI_OUTDUPL_FRAME_INFO that the pFrameInfo parameter points to) indicates that a separate pointer isn’t visible. If the graphics adapter overlays the mouse pointer on top of the desktop image, AcquireNextFrame reports that a separate pointer is visible. So, your client app must draw the mouse pointer shape onto the desktop image to accurately represent what the current user will see on their monitor.

To draw the desktop’s mouse pointer, use the PointerPosition member of DXGI_OUTDUPL_FRAME_INFO from the pFrameInfo parameter of AcquireNextFrame to determine where to locate the top left hand corner of the mouse pointer on the desktop image. When you draw the first frame, you must use the IDXGIOutputDuplication::GetFramePointerShape method to obtain info about the shape of the mouse pointer. Each call to AcquireNextFrame to get the next frame also provides the current pointer position for that frame. On the other hand, you need to use GetFramePointerShape again only if the shape changes. So, keep a copy of the last pointer image and use it to draw on the desktop unless the shape of the mouse pointer changes.

大致意思嘛就是，这个视情况而定，有的时候或者说有的设备会把鼠标绘制在图像数据中，有的呢则是分开绘制，你要处理的就是当发现有鼠标信息更新刚好又有鼠标形状，那就自己合并吧。

3. 三种鼠标类型

• DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MONOCHROME
  The pointer type is a monochrome mouse pointer, which is a monochrome bitmap. The bitmap’s size is specified by width and height in a 1 bits per pixel (bpp) device independent bitmap (DIB) format AND mask that is followed by another 1 bpp DIB format XOR mask of the same size.
• DXGI_OUTDUPL_POINTER_SHAPE_TYPE_COLOR
  The pointer type is a color mouse pointer, which is a color bitmap. The bitmap’s size is specified by width and height in a 32 bpp ARGB DIB format.
• DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MASKED_COLOR
  The pointer type is a masked color mouse pointer. A masked color mouse pointer is a 32 bpp ARGB format bitmap with the mask value in the alpha bits. The only allowed mask values are 0 and 0xFF. When the mask value is 0, the RGB value should replace the screen pixel. When the mask value is 0xFF, an XOR operation is performed on the RGB value and the screen pixel; the result replaces the screen pixel.

说实在的，文档就给了这么几句说明让我有点抓狂，还不如给三个处理的例子来的实在。
后两个我看懂了，第一个嘛，恕我直言还没完全弄明白。

• DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MONOCHROME
  大体意思是，这种类型是个黑白的鼠标图形，但是有掩码什么的blablablabla…
• DXGI_OUTDUPL_POINTER_SHAPE_TYPE_COLOR
  大体意思呢，是个彩色位图，并且是按照ARGB来存储的，直接贴图就是了
• DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MASKED_COLOR
  是说是带有掩码的彩色位图，然后掩码只有0x00和0xFF，掩码是0x00的时候呢，要把鼠标像素点完全替换掉对应的屏幕像素点，如果掩码是0xFF呢要和屏幕像素数据进行异或运算

4. 干货有点长

void record_desktop_duplication::draw_cursor()
{
	if (_cursor_info.visible == false) return;

	int cursor_width = 0, cursor_height = 0, left = 0, top = 0;

	cursor_width = _cursor_info.shape.Width;
	cursor_height = _cursor_info.shape.Height;

	// In case that,the value of position is negative value
	left = abs(_cursor_info.position.x - _rect.left);
	top = abs(_cursor_info.position.y - _rect.top);

	// Notice here
	if (_cursor_info.shape.Type == DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MONOCHROME)
		cursor_height = cursor_height / 2;

	//Skip invisible pixel
	cursor_width = min(_width - (left + cursor_width), cursor_width);
	cursor_height = min(_height - (top + cursor_height), cursor_height);

	//al_debug("left:%d top:%d width:%d height:%d type:%d", left, top, cursor_width, height, _cursor_info.shape.Type);

	switch (_cursor_info.shape.Type)
	{

		// The pointer type is a color mouse pointer, 
		// which is a color bitmap. The bitmap's size 
		// is specified by width and height in a 32 bpp 
		// ARGB DIB format.
		// should trans cursor to BGRA?
		case DXGI_OUTDUPL_POINTER_SHAPE_TYPE_COLOR:
		{
			unsigned int *cursor_32 = reinterpret_cast<unsigned int*>(_cursor_info.buff);
			unsigned int *screen_32 = reinterpret_cast<unsigned int*>(_buffer);

			for (int row = 0; row < cursor_height; row++) {
				for (int col = 0; col < cursor_width; col++) {
					unsigned int cur_cursor_val = cursor_32[col + (row * (_cursor_info.shape.Pitch / sizeof(UINT)))];
						
					//Skip black or empty value
					if (cur_cursor_val == 0x00000000)
						continue;
					else
						screen_32[(abs(top) + row) *_width + abs(left) + col] = cur_cursor_val;//bit_reverse(cur_cursor_val);
				}
			}
			break;
		}

		// The pointer type is a monochrome mouse pointer, 
		// which is a monochrome bitmap. The bitmap's size 
		// is specified by width and height in a 1 bits per 
		// pixel (bpp) device independent bitmap (DIB) format 
		// AND mask that is followed by another 1 bpp DIB format 
		// XOR mask of the same size.
		case DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MONOCHROME:
		{
			unsigned int *cursor_32 = reinterpret_cast<unsigned int*>(_cursor_info.buff);
			unsigned int *screen_32 = reinterpret_cast<unsigned int*>(_buffer);

			for (int row = 0; row < cursor_height; row++) {
				BYTE MASK = 0x80;
				for (int col = 0; col < cursor_width; col++) {
					// Get masks using appropriate offsets
					BYTE AndMask = _cursor_info.buff[(col / 8) + (row  * (_cursor_info.shape.Pitch))] & MASK;
					BYTE XorMask = _cursor_info.buff[(col / 8) + ((row + cursor_height) * (_cursor_info.shape.Pitch))] & MASK;
					UINT AndMask32 = (AndMask) ? 0xFFFFFFFF : 0xFF000000;
					UINT XorMask32 = (XorMask) ? 0x00FFFFFF : 0x00000000;

					// Set new pixel
					screen_32[(abs(top) + row) *_width + abs(left) + col] = (screen_32[(abs(top) + row) *_width + abs(left) + col] & AndMask32) ^ XorMask32;

					// Adjust mask
					if (MASK == 0x01)
					{
						MASK = 0x80;
					}
					else
					{
						MASK = MASK >> 1;
					}
				}
			}
			break;
		}
		// The pointer type is a masked color mouse pointer. 
		// A masked color mouse pointer is a 32 bpp ARGB format 
		// bitmap with the mask value in the alpha bits. The only 
		// allowed mask values are 0 and 0xFF. When the mask value
		// is 0, the RGB value should replace the screen pixel. 
		// When the mask value is 0xFF, an XOR operation is performed 
		// on the RGB value and the screen pixel; the result replaces the screen pixel.
		case DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MASKED_COLOR:
		{
			unsigned int *cursor_32 = reinterpret_cast<unsigned int*>(_cursor_info.buff);
			unsigned int *screen_32 = reinterpret_cast<unsigned int*>(_buffer);

			for (int row = 0; row < cursor_height; row++) {
				for (int col = 0; col < cursor_width; col++) {
					unsigned int cur_cursor_val = cursor_32[col + (row * (_cursor_info.shape.Pitch / sizeof(UINT)))];
					unsigned int cur_screen_val = screen_32[(abs(top) + row) *_width + abs(left) + col];
					unsigned int mask_val = 0xFF000000 & cur_cursor_val;

					if (mask_val) {
						//0xFF: XOR operation is performed on the RGB value and the screen pixel
						cur_screen_val = (cur_screen_val ^ cur_cursor_val) | 0xFF000000;
					}
					else {
						//0x00: the RGB value should replace the screen pixel
						cur_screen_val = cur_cursor_val | 0xFF000000;
					}
				}
			}
			break;
		}
		default:
			break;
	}
}

需要注意的是

//Notice here
if (_cursor_info.shape.Type == DXGI_OUTDUPL_POINTER_SHAPE_TYPE_MONOCHROME)
height = height / 2;

我不知道为什么这样做，文档没有说明，我也是看了微软的例子这么做才知道，而且这个类型的鼠标height确实是64，宽度是32
还有要注意的是在根据坐标计算像素点偏移的时候，最好能自己画一张点图，这样不容易出错。

至此，我们的duplication 捕获桌面告一段落，需要优化的是鼠标绘制的时候，ARGB要转BGRA，我做了转换但是效果不好。

其他诸如帧率控制、重新初始化等请点击末尾的源码传送门。加星是对我最大的支持。

拜了个拜！

GitHub传送门

screen-recorder

---

2020.6.20更正

4. 拷贝图像数据到缓冲区内 （此处有错误，更正见文章尾部）

memcpy(_buffer, mapped_rect.pBits, _buffer_size);
dxgi_surface->Unmap();

这里需要注意的是，Duplication API捕获到的桌面数据格式总是BGRA，所以我们的缓冲区初始化大小就是width * height * 4

这里在产品新版本发布后，发现很多分辨率下无法正常捕获到图像，把一帧图像保存成位图发现也是错误的。
最终锁定到这个MAP函数。

typedef struct DXGI_MAPPED_RECT {
INT Pitch;
BYTE *pBits;
} DXGI_MAPPED_RECT;

Members
Pitch
Type: INT
A value that describes the width, in bytes, of the surface.
pBits
Type: BYTE*
A pointer to the image buffer of the surface.

其中的Pitch参数表示图像的宽，这里我也不知道怎么翻译比较准确，前文讲到过屏幕像素的行列扫描方式，这里描述的就是在pBits中，一行像素数据的大小由Pitch决定，后来Debug发现，果然它并不总是等于with*4,那么就好办了，把拷贝函数改成如下：

int dst_rowpitch = frame_desc.Width * 4;
		for (int h = 0; h < frame_desc.Height; h++) {
			memcpy_s(_buffer + h*dst_rowpitch, dst_rowpitch, (BYTE*)mapped_rect.pBits + h*mapped_rect.Pitch, min(mapped_rect.Pitch, dst_rowpitch));
		}

问题解决！

参考资料：Desktop Screen Capture