承接上一篇【渲染逆向】Hook D3D API.
DX11的Hook流程
程序的入口点(DllMain)在win32_libentry.cpp中
BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
{
if(ul_reason_for_call == DLL_PROCESS_ATTACH)
{
BOOL ret = add_hooks();
SetLastError(0);
return ret;
}
return TRUE;
}
在函数add_hooks的结尾会调用LibraryHooks::RegisterHooks()
void LibraryHooks::RegisterHooks()
{
BeginHookRegistration();
for(LibraryHook *lib : LibList())
lib->RegisterHooks();
EndHookRegistration();
}
LibList函数返回LibraryHook对象的列表，而向列表中加入对象，是LibraryHook调用构造函数的时候。
LibraryHook::LibraryHook()
{
LibList().push_back(this);
}
显然LibraryHook是一个基类，会有超级多需要Hook的库来继承这一类，而我们现在关心的是DX11的库。
class D3D11Hook : LibraryHook
{
public:
void RegisterHooks();
private:
static D3D11Hook d3d11hooks;
//something....
}
可以发现D3D11Hooks有一个静态成员，当这个类型初始化时，会自动生成一个D3D11Hook对象，并调用父类LibraryHook的构造函数，将自身加入Lib列表中。
当函数add_hooks遍历列表，调用RegisterHooks时，会调用D3D11Hook::RegisterHooks。
void RegisterHooks()
{
//something....
CreateDevice.Register("d3d11.dll", "D3D11CreateDevice", D3D11CreateDevice_hook);
CreateDeviceAndSwapChain.Register("d3d11.dll", "D3D11CreateDeviceAndSwapChain",
D3D11CreateDeviceAndSwapChain_hook);
}
这里会调用HookedFunction::Register，然后调用LibraryHooks::RegisterFunctionHook，将Hook所需的数据（dll库名称、函数名称、原始函数地址、hook函数地址）记录到数据结构中，之后统一用IAT hook进程函数hook，如果有需要，可以在这里修改为其他hook类型，例如：
void LibraryHooks::RegisterFunctionHook(const char *libraryName, const FunctionHook &hook)
{
//something...
if(!_stricmp(libraryName, "d3d11.dll") || !_stricmp(libraryName, "dxgi.dll"))
{
std::cout << "inline hook " << libraryName
<< "\t-\t" << hook.function.c_str() << std::endl;
*hook.orig = GetProcAddress(GetModuleHandleA(libraryName), hook.function.c_str());
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach((PVOID *)hook.orig, hook.hook);
DetourTransactionCommit();
}
else
{
s_HookData->DllHooks[strlower(rdcstr(libraryName))].FunctionHooks.push_back(hook);
}
}
这里我就修改了一下，假如是d3d11.dll或dxgi.dll的函数，则进行inline hook，而其他库函数则保持原样。
dxgi.dll的Hook同理:
class DXGIHook : LibraryHook
{
public:
void RegisterHooks()
{
LibraryHooks::RegisterLibraryHook("dxgi.dll", NULL);
CreateDXGIFactory.Register("dxgi.dll", "CreateDXGIFactory", CreateDXGIFactory_hook);
CreateDXGIFactory1.Register("dxgi.dll", "CreateDXGIFactory1", CreateDXGIFactory1_hook);
CreateDXGIFactory2.Register("dxgi.dll", "CreateDXGIFactory2", CreateDXGIFactory2_hook);
GetDebugInterface.Register("dxgi.dll", "DXGIGetDebugInterface", DXGIGetDebugInterface_hook);
GetDebugInterface1.Register("dxgi.dll", "DXGIGetDebugInterface1", DXGIGetDebugInterface1_hook);
}
这些都是Hook需要查询地址的函数，而类似SetRenderTargets之类的虚表函数的Hook方式，放到下一节来找。
DX11截帧流程
RenderDoc按键或调用提供的api截帧都是调用core.h中的RenderDoc::TriggerCapture(uint32_t numFrames)函数记录需要截几帧。
void TriggerCapture(uint32_t numFrames) { m_Cap = numFrames; }
同文件下又提供了另一个方法bool RenderDoc::ShouldTriggerCapture(uint32_t frameNumber)。
bool RenderDoc::ShouldTriggerCapture(uint32_t frameNumber)
{
bool ret = m_Cap > 0;
if(m_Cap > 0)
m_Cap--;
//something....
return ret;
继续查找引用可以找到一个相对有用的函数——WrappedID3D11Device::Present，这个函数在结尾调用的ShouldTriggerCapture，并且这个函数中还可以调整OverlayText(RenderDoc左上角的覆盖层)。
//在WrappedID3D11Device::Present中
if(IsBackgroundCapturing(m_State) && RenderDoc::Inst().ShouldTriggerCapture(m_FrameCounter))
{
RenderDoc::Inst().StartFrameCapture((ID3D11Device *)this, swapper->GetHWND());
m_AppControlledCapture = false;
m_CapturedFrames.back().frameNumber = m_FrameCounter;
}
void RenderDoc::StartFrameCapture(void *dev, void *wnd)
{
IFrameCapturer *frameCap = MatchFrameCapturer(dev, wnd);
if(frameCap)
{
frameCap->StartFrameCapture(dev, wnd);
m_CapturesActive++;
}
}
frameCap是IFrameCapturer的指针，IFrameCapturer是个接口，而实现这个接口的是WrappedID3D11Device。
void WrappedID3D11Device::StartFrameCapture(void *dev, void *wnd)
{
//something...
m_State = CaptureState::ActiveCapturing;
//something...
}
在这里激活了当前抓取状态，core.h中封装了一层函数：
constexpr inline bool IsActiveCapturing(CaptureState state)
{
return state == CaptureState::ActiveCapturing;
}
当程序调用渲染api时，会进入到RenderDoc定义的Hook函数中，在调用原有函数后，会检查截帧状态，假如处于截帧状态，则记录API，这就是抓帧的实现，贴一个例子，下面是一个Wrap(包装)好的api：
void WrappedID3D11DeviceContext::IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY Topology)
{
SCOPED_LOCK_OPTIONAL(m_pDevice->D3DLock(), m_pDevice->D3DThreadSafe());
DrainAnnotationQueue();
m_EmptyCommandList = false;
//调用原有函数
SERIALISE_TIME_CALL(m_pRealContext->IASetPrimitiveTopology(Topology));
//如果处于截帧状态，则序列化API和参数
if(IsActiveCapturing(m_State))
{
USE_SCRATCH_SERIALISER();
SCOPED_SERIALISE_CHUNK(D3D11Chunk::IASetPrimitiveTopology);
SERIALISE_ELEMENT(m_ResourceID).Named("Context"_lit).TypedAs("ID3D11DeviceContext *"_lit);
Serialise_IASetPrimitiveTopology(GET_SERIALISER, Topology);
m_ContextRecord->AddChunk(scope.Get());
}
m_CurrentPipelineState->Change(m_CurrentPipelineState->IA.Topo, Topology);
VerifyState();
}
借此我们对这些Wrap的函数产生兴趣，例如WrappedID3D11Device，查看声明：
class WrappedID3D11Device : public IFrameCapturer, public ID3DDevice, public ID3D11Device5
这个类竟然实现了ID3DDevice这个DX接口。
断点跟踪下这个类的构造函数调用堆栈
-D3D11Hook::D3D11CreateDevice_hook
-D3D11Hook::D3D11CreateDeviceAndSwapChain_hook
-D3D11Hook::Create_Internal
-WrappedID3D11Device::WrappedID3D11Device
第一个函数D3D11Hook::D3D11CreateDevice_hook是RenderDoc的hook函数，本文的第一节已经提到了；在D3D11Hook::Create_Internal中，调用了原始的D3D11CreateDeviceAndSwapChain方法，然后用返回的device构造wrap device，最后将wrap返回，由于wrap device同样实现了ID3D11Device类的接口，因此在使用上和DirectX的Device没有区别：
HRESULT Create_Internal(RealD3D11CreateFunction real, __in_opt IDXGIAdapter *pAdapter,
D3D_DRIVER_TYPE DriverType, HMODULE Software, UINT Flags,
__in_ecount_opt(FeatureLevels) CONST D3D_FEATURE_LEVEL *pFeatureLevels,
UINT FeatureLevels, UINT SDKVersion,
__in_opt CONST DXGI_SWAP_CHAIN_DESC *pSwapChainDesc,
__out_opt IDXGISwapChain **ppSwapChain, __out_opt ID3D11Device **ppDevice,
__out_opt D3D_FEATURE_LEVEL *pFeatureLevel,
__out_opt ID3D11DeviceContext **ppImmediateContext)
{
//...
//调用原有D3D11CreateDeviceAndSwapChain方法
HRESULT ret = real(pAdapter, DriverType, Software, Flags, pFeatureLevels, FeatureLevels,
SDKVersion, pUsedSwapDesc, ppSwapChain, ppDevice, pFeatureLevel, NULL);
//构造wrap device
WrappedID3D11Device *wrap = new WrappedID3D11Device(*ppDevice, params);
//将wrap device的地址通过指针返回
*ppDevice = wrap;
//...
}
其他DX组件，例如IDXGIFactory、ID3D11DeviceContext的hook方式与此类似。
坑
报个坑，Nvidia 驱动安装程序Geforce Experience的游戏内覆盖如果开启，有几率会弹NVAPI_NO_IMPLEMENT错误。