PyEmu源码阅读笔记
PyEmu是一个用python编写的x86指令仿真器，所谓仿真器就是个类似沙盒或虚拟机的东西，在仿真器下运行的程序将被完全掌控，所有代码由仿真器模拟执行，具有极强的可控性并且恶意软件一般也无法对机器造成伤害
PyEmu由于作者的精心编写以及python语言本身的优点使得PyEmu源码具有极强的可读性
This is a short readme describing the layout of PyEmu.
PyContext.py: A module containing a class for defining a context to pass between modules in the emulator
PyCPU.py: The CPU class implements each instruction and is responsible for executing and maintaining state
PyDebug.py: A simple class to ease some debugging tasks
PyEmu.py: The user facing class that implements the public methods available for use. Also is responsible for initiating the memory and cpu classes
PyInstruction.py: A helper class for providing abstracted access to the pydasm instruction structures
PyMemory.py: A module containing the memory managers responsible for fetching and storing memory
PyOS.py: A rough implementation of needed OS specific structures for process creation and control.
lib/
pefile.py: Ero Carrera's pefile implementation
pydasm.pyd: Ero Carrera's libdasm python wrapper
ctypes/_ctypes.pyd: Ctypes library needed for PyOS.py
PyContext.py: A module containing a class for defining a context to pass between modules in the emulator
PyCPU.py: The CPU class implements each instruction and is responsible for executing and maintaining state
PyDebug.py: A simple class to ease some debugging tasks
PyEmu.py: The user facing class that implements the public methods available for use. Also is responsible for initiating the memory and cpu classes
PyInstruction.py: A helper class for providing abstracted access to the pydasm instruction structures
PyMemory.py: A module containing the memory managers responsible for fetching and storing memory
PyOS.py: A rough implementation of needed OS specific structures for process creation and control.
lib/
pefile.py: Ero Carrera's pefile implementation
pydasm.pyd: Ero Carrera's libdasm python wrapper
ctypes/_ctypes.pyd: Ctypes library needed for PyOS.py
这是PyEmu的README文件，简短的描述了代码的文件组织、结构，其中lib目录中是3个库文件，一个是python标准库ctypes，用来让python拥有像c语言一样的底层控制能力，pefile文件则包含一些对于pe格式文件的结构定义、操作。pydasm则是反汇编引擎libdasm库，作为一个仿真器在仿真之前首先要做的就是理解x86机器码的意图，所以一个反汇编引擎是必须的。
再看PyEmu仿真器主要由3个子系统构成：PyCPU、PyMemory、PyEmu，也是3个主要类的名字。其中PyCPU是最为核心的一个类，负责大量复杂的工作，从上W行的代码量就可以看出，所以暂时不去管它
PyEmu就是一个仿真器的抽象，负责处理任何来自用户的意图，并负担起指挥PyCPU,PyMemory运行的重则，而在其之上继承出了3个子类PyDbgPyEmu、IDAPyEmu、PEPyEmu用以支持PyDbg,IDA,以及独立使用的仿真器，整体结构如下：
PyCPU PyMemory
| |
| |
|________ ________|
|
PyEmu
其实如果去看源码的话会发现，暂时不去考虑PyCPU和PyMemory的话PyEmu这个类是非常好理解的，它利用PyCPU和PyMemory的强大能力，提供了一系列的字典用来存储各种调试器中的各种内存，堆栈，代码，异常断点处理例程功能给用户使用，达到和调试器一样的效果，然后一些信息的获取就是简单地封装了底层PyCPU和PyMemory类的方法
下面贴出PyEmu源代码的缩略版，以便更清楚的观察PyEmu的实现，根据名字就可以简单地判断函数的功能：
#!/usr/bin/env python
########################################################################
#
# PyEmu: scriptable x86 emulator
#
#
# License: None
#
########################################################################
sys.path.append(r'C:\Program Files\IDA\python')
from PyCPU import PyCPU
from PyContext import PyContext
from PyMemory import *
from PyOS import *
'''
PyEmu:
The main emulator class. This class implements the public methods
for controlling the emulator. This includes handlers, and initialization.
'''
class PyEmu:
DEBUG = 0
def __init__(self):
#
# raise_exception: This method gets called when an exception happens
#
def raise_exception(self, exception, address):
#
# debug: A public method for setting global debug levels
#
def debug(self, level):
#
# execute: A public method for executing instructions
#
def execute(self, steps=1, start=0x0, end=0x0):
#
# get_register: A public method to retrieve a register for the user
def get_register(self, register):
#
# set_register: A public method for setting a registers value
#
def set_register(self, register, value, name=""):
#
# get_stack_variable: A public method for setting stack local variables
#
def get_stack_variable(self, offset, size=0):
#
# set_stack_variable: A public method for setting a local stack variable
#
def set_stack_variable(self, offset, value, size=0, name=""):
#
# get_stack_argument: A public method to get a functions stack argument
#
def get_stack_argument(self, offset, size=0):
#
# set_stack_argument: A public method to set up a stack argument
#
def set_stack_argument(self, offset, value, name=""):
#
# get_memory: A public method for fetching arbitrary memory
#
def get_memory(self, address, size=0):
#
# get_memory_string: A public method to fetch a string from memory
#
def get_memory_string(self, address):
#
# set_memory: A public method for setting arbitrary memory
#
def set_memory(self, address, value, size=0):
#
# get_selector: A public method for fetching a selector from the LDT
#
def get_selector(self, selector):
#
# set_register_handler: A public method for setting a custom register
#
def set_register_handler(self, register, handler):
#
# set_mnemonic_handler: A public method for setting a custom mnemonic
#
def set_mnemonic_handler(self, mnemonic, handler):
#
# set_opcode_handler: A public method for setting a custom handler
#
def set_opcode_handler(self, opcode, handler):
#
# set_pc_handler: A public method for setting a custom handler on
#
def set_pc_handler(self, address, handler):
#
# set_exception_handler: A public method for setting a custom
#
def set_exception_handler(self, exception, handler):
#
# set_library_handler: A public method for setting a custom
#
def set_library_handler(self, function, handler):
#
# set_interrupt_handler: A public method for setting a custom
#
def set_interrupt_handler(self, interrupt, handler):
#
# set_memory_handler: A public method for setting a custom handler
#
def set_memory_handler(self, address, handler):
#
# set_memory_read_handler: A public memory for setting a custom handler
def set_memory_read_handler(self, handler):
#
# set_memory_write_handler: A public memory for setting a custom handler
def set_memory_write_handler(self, handler):
#
# set_memory_access_handler: A public memory for setting a custom handler
def set_memory_access_handler(self, handler):
#
# set_stack_read_handler: A public memory for setting a custom handler
#
def set_stack_read_handler(self, handler):
#
# set_stack_write_handler: A public memory for setting a custom handler
def set_stack_write_handler(self, handler):
#
# set_stack_access_handler: A public memory for setting a custom handler
#
def set_stack_access_handler(self, handler):
#
# set_heap_read_handler: A public memory for setting a custom handler
#
def set_heap_read_handler(self, handler):
#
# set_heap_write_handler: A public memory for setting a custom handler
def set_heap_write_handler(self, handler):
#
# set_heap_access_handler: A public memory for setting a custom handler
#
def set_heap_access_handler(self, handler):
#
# dump_regs: A public method to dump the regs from the CPU
#
def dump_regs(self):
#
# dump_stack: A public method to dump the stack from ESP and EBP
#
def dump_stack(self, count=64):
#
# get_disasm: A public method to get a pretty dump of the current
def get_disasm(self):
'''
PyDbgPyEmu:
The ugliest class name ever. Really the PyEmu class for handling
PyDbg operation. It is responsible for talking between the
emulator and the real process. This is what the user would instantiate.
'''
class PyDbgPyEmu(PyEmu):
def __init__(self, dbg):
def setup_context(self):
'''
IDAPyEmu:
The purposed class for emulating in IDA Pro. This has to set up
some basic operating environments for the executable. This is what
the user will be instantiating.
'''
class IDAPyEmu(PyEmu):
def __init__(self, stack_base=0x0095f000, stack_size=0x1000, heap_base=0x000a0000, heap_size=0x2000, frame_pointer=True):
#
# setup_os: Adds a new thread based on which OS you are using
#
def setup_os(self):
#
# setup_context: Sets the needed stack pointers so we can execute
#
def setup_context(self):
'''
PEPyEmu:
The purposed class for emulating from a raw PE executable. This has
This is what the user will be instantiating.
'''
class PEPyEmu(PyEmu):
def __init__(self, stack_base=0x0095f000, stack_size=0x1000, heap_base=0x000a0000, heap_size=0x2000, frame_pointer=True):
#
# setup_os: Adds a new thread based on which OS you are using
def setup_os(self):
#
# setup_context: Sets the needed stack pointers so we can execute
def setup_context(self):
#
# load: Loads the sections of a binary into the emulator memory
#
def load(self, exename):
接着是PyMemory类，这个文件的源码首先将系统中一个页面的内存抽象出一个页面类PyMemoryPage，并仿真一个页面所具有的权限、数据等属性。
然后才是PyMemory类，PyMemory仿真了内存中的各种情况，包括跨页读写内存等内存管理功能，做出一个真实内存该有的行为。紧接着就是继承出PyDbgMemory,IDAMemory,PEMemory分别为3个仿真器提供内存管理接口，下面贴出实现缩略版：
#!/usr/bin/env python
########################################################################
#
# PyEmu: scriptable x86 emulator
#
#
# License: None
#
########################################################################
'''
PyMemoryPage:
A class that allows us to define some properties and methods for each
page of memory in our cache. This could be used to further define
permissions and attributes as needed
'''
class PyMemoryPage:
DEBUG = 0
PAGESIZE = 4096
READ = 0x1
WRITE = 0x2
EXECUTE = 0x4
def __init__(self, address, data="", permissions=0x0):
def get_data(self):
def get_permissions(self):
def set_data(self, data):
def set_permissions(self, permissions):
def set_debug(self, level):
def set_r(self):
def set_w(self):
def set_x(self):
def set_rw(self):
def set_rx(self):
def set_rwx(self):
def is_r(self):
def is_w(self):
def is_x(self):
def is_rx(self):
def is_rwx(self):
'''
PyMemory:
The base class for handling memory requests from the PyCPU and PyEmu.
This class should be extended by any custom memory managers.
'''
class PyMemory:
DEBUG = 0
PAGESIZE = 4096
def __init__(self, emu):
#
# get_memory: Fetches memory first checking local cache, then
#
def get_memory(self, address, size):
#
# set_memory: Set an address to a specific value. This can be a
#
def set_memory(self, address, value, size):
#
# get_available_page: Will return the next available page starting from address
#
def get_available_page(self, address):
#
# is_valid: A helper function to check for a address in our cache
#
def is_valid(self, address):
def get_page(self, page):
def set_debug(self, level):
#
# dump_memory: This dumps the data from memory optionally writing
#
def dump_memory(self, filename=None):
#
# dump_pages: This will dump all the currently cached memory pages.
def dump_pages(self, data=False):
'''
PyDbgMemory:
This is the pydbg memory manager. It extends the base PyMemory class
This is responsible for nothing more than handling requests for
memory if needed. In this case a fetch of unknown memory will make a
call to ReadProcessMemory via the dbg instance.
'''
class PyDbgMemory(PyMemory):
def __init__(self, emu, dbg):
#
# allocate_page: Allocates a page for addition into the cache
#
def allocate_page(self, page):
#
# get_page: This fetches the page from pydbg
#
def get_page(self, page):
'''
IDAMemory:
This is the ida memory manager. It extends the base PyMemory class
and is responsible for handling any unknown memory requests. In IDA
this is a tricky call cause we can either throw an exception on invalid
memory accesses or go ahead and fulfill them in case the user did not
set everything up properly. Its really a personal choice.
'''
class IDAMemory(PyMemory):
def __init__(self, emu):
#
# allocate_page: Allocates a page for addition into the cache
#
def allocate_page(self, page):
#
# get_page: Handles unknown memory requests from the base class.
#
def get_page(self, page):
'''
PEMemory:
This is the raw PE file memory handler that is responsible for handling
requests from the base class. Like the others it requests memory when
needed.
'''
class PEMemory(PyMemory):
def __init__(self, emu):
#
# allocate_page: Allocates a page for addition into the cache
#
def allocate_page(self, page):
#
# allocate: Allocates a block of memory
#
def allocate(self, size):
#
# get_page: Stores a page in the base class cache
#
def get_page(self, page):
另外还有几个辅助的类如PyContext:模拟寄存器环境的结构
PyFlags:专门用于模拟x86的标志寄存器操作
PyInstruction:实现各种x86指令结构、信息的存储、操作
PyOS:实现操作系统结构如TEB、PEB、LDT等结构的模拟
接下来就是仿真器中最为核心也最为复杂的系统PyCPU类了
源码中PyCPU首先定义了一个python字典的标志寄存器位图，PyCPU自身则定义了所有的寄存器，包括标志寄存器的每一个标志位作为数据成员，并且定义了一个已执行指令的字典缓存，大大提升PyCPU在仿真过程中的性能，减少重复指令的解析(解析需要调用反汇编引擎)，之后定义了一个仿真器支持的指令表字典，每个指令对应一个指令仿真处理例程。
然后用get_register和set_register例程分别仿真了寄存器的读写操作，与此相对应的既然是寄存器的读写，自然包括了寄存器读写所可能需要拦截调用的用户定义的处理函数。
同理，get_memory和set_memory也是对内存读写的仿真，既然是内存读写，不管是堆栈还是别的位置，一律需要处理。cpu中存在的符号扩展同样有一个专门函数来处理，然后是一些有用的环境获取、设置，标志寄存器获取、设置等函数。
接下来就是一个指令仿真函数execute,用于执行一条当前PyCPU状态下的指令，函数需要处理库的调用，库的具体调用需要用户提供，因为仿真器实际只能提供到指令的仿真，那些API调用还是需要我们来提供，之后读取一定字节的数据传给反汇编引擎进行反汇编，存储反汇编后的结果，然后调用相关指令助记符对应的指令仿真函数进行真正的指令级别的仿真。
再下面是一个get_memory_address函数，用来获取指令中指定操作数的内存地址，所以这段代码比较像反汇编引擎中的指令解析，需要解析指令得出操作数所指的地址，暂时还不知道这个函数是用来干嘛的。再下面就提供了一些有用的dump操作以便随时查看整个仿真器内部的寄存器、堆栈等状态；
这里贴出仿真器使用中最常用的函数，用来执行指令的函数，是PyCPU类中的成员函数：
这里贴出仿真器使用中最常用的函数，用来执行指令的函数，是PyCPU类中的成员函数：
def execute(self):
# Check our program counter handlers
if self.EIP in self.emu.pc_handlers:
if not self.emu.pc_handlers[self.EIP](self.emu, self.EIP):
return False
if self.EIP in self.emu.os.libraries:
library = self.emu.os.libraries[self.EIP]
if self.DEBUG > 1:
print "[*] Calling 0x%08x:%s" % (self.EIP, library['name'])
if library['name'] in self.emu.library_handlers:
result = self.emu.library_handlers[library['name']](library['name'], library['address'])
if not result:
return False
else:
return result
else:
print "[*] Need a handler for [%s]" % (library)
return False
oldeip = self.EIP
#
# We track instructions executed so we can greatly increase performance
#
if self.EIP not in self.executed_instructions:
# Fetch raw instruction from memory, 13 bytes seems to be the largest possible instruction
rawinstruction = self.get_memory(self.EIP, 13)
if not rawinstruction:
print "[!] Problem fetching raw bytes from 0x%08x" % (self.EIP)
return False
# Decode instruction from raw returning a pydasm.instruction
instruction = pydasm.get_instruction(rawinstruction, pydasm.MODE_32)
if not instruction:
print "[!] Problem decoding instruction"
return False
# Create our python class for instruction, we do this in case we ever leave pydasm
pyinstruction = PyInstruction(instruction)
self.executed_instructions[self.EIP] = pyinstruction
else:
pyinstruction = self.executed_instructions[self.EIP]
if self.DEBUG > 0:
print "[*] Executing [0x%x][%x] %s" % (self.EIP, pyinstruction.opcode, self.get_disasm())
# We must split any prefix sense we use flags
pyinstruction.mnemonic = pyinstruction.mnemonic.split()
if pyinstruction.mnemonic[0] in ["rep", "repe", "repne", "lock"]:
pyinstruction.mnemonic = pyinstruction.mnemonic[1]
else:
pyinstruction.mnemonic = pyinstruction.mnemonic[0]
# Check if we support this instruction
if pyinstruction.mnemonic in self.supported_instructions:
# Execute!
if not self.supported_instructions[pyinstruction.mnemonic](pyinstruction):
return False
else:
print "[!] Unsupported instruction %s" % pyinstruction.mnemonic
return False
# If EIP has not changed we advance to the next instruction in code
if self.EIP == oldeip:
self.EIP += pyinstruction.length
# Everything checked out
return True
剩下的代码就是各种指令仿真的具体实现，这是仿真器最关键的部分，直接决定仿真器能否正确运行，同样，开发一个虚拟机也免不了进行一套自己的仿真操作。
这里贴上一个movsb指令以实例：
这里贴上一个movsb指令以实例：
def MOVSB(self, instruction):
op1 = instruction.op1
op2 = instruction.op2
oo = instruction.operand_so()
ao = instruction.address_so()
if oo:
osize = 2
else:
osize = 4
if ao:
asize = 2
else:
asize = 4
op1value = ""
op2value = ""
op3value = ""
op1valuederef = None
op2valuederef = None
#A4 MOVSB
if instruction.opcode == 0xa4:
osize = 1
if ao:
if instruction.rep():
repcount = self.get_register16("CX")
while repcount > 0:
op1value = self.ES + self.get_register16("DI")
op2value = self.DS + self.get_register16("SI")
op2valuederef = self.get_memory(op2value, osize)
self.set_memory(op1value, op2valuederef, osize)
if not self.DF:
self.set_register16("DI", op1value + osize)
self.set_register16("SI", op2value + osize)
else:
self.set_register16("DI", op1value - osize)
self.set_register16("SI", op2value - osize)
repcount -= 1
self.set_register16("CX", repcount)
else:
op1value = self.ES + self.get_register16("DI")
op2value = self.DS + self.get_register16("SI")
op2valuederef = self.get_memory(op2value, osize)
self.set_memory(op1value, op2valuederef, osize)
if not self.DF:
self.set_register16("DI", op1value + osize)
self.set_register16("SI", op2value + osize)
else:
self.set_register16("DI", op1value - osize)
self.set_register16("SI", op2value - osize)
else:
if instruction.rep():
repcount = self.get_register32("ECX")
while repcount > 0:
op1value = self.get_register32("EDI")
op2value = self.get_register32("ESI")
op2valuederef = self.get_memory(op2value, osize)
self.set_memory(op1value, op2valuederef, osize)
if not self.DF:
self.set_register32("EDI", op1value + osize)
self.set_register32("ESI", op2value + osize)
else:
self.set_register32("EDI", op1value - osize)
self.set_register32("ESI", op2value - osize)
repcount -= 1
self.set_register32("ECX", repcount)
else:
op1value = self.get_register32("EDI")
op2value = self.get_register32("ESI")
op2valuederef = self.get_memory(op2value, osize)
self.set_memory(op1value, op2valuederef, osize)
if not self.DF:
self.set_register32("EDI", op1value + osize)
self.set_register32("ESI", op2value + osize)
else:
self.set_register32("EDI", op1value - osize)
self.set_register32("ESI", op2value - osize)
opcode = instruction.opcode
if opcode in self.emu.opcode_handlers:
if op1valuederef != None and op2valuederef == None:
self.emu.opcode_handlers[opcode](self.emu, opcode, self.get_register32("EIP"), op1valuederef, op2value, op3value)
elif op2valuederef != None and op1valuederef == None:
self.emu.opcode_handlers[opcode](self.emu, opcode, self.get_register32("EIP"), op1value, op2valuederef, op3value)
else:
self.emu.opcode_handlers[opcode](self.emu, opcode, self.get_register32("EIP"), op1value, op2value, op3value)
else:
return False
mnemonic = instruction.mnemonic.upper()
if mnemonic in self.emu.mnemonic_handlers:
if op1valuederef != None and op2valuederef == None:
self.emu.mnemonic_handlers[mnemonic](self.emu, mnemonic, self.get_register32("EIP"), op1valuederef, op2value, op3value)
elif op2valuederef != None and op1valuederef == None:
self.emu.mnemonic_handlers[mnemonic](self.emu, mnemonic, self.get_register32("EIP"), op1value, op2valuederef, op3value)
else:
self.emu.mnemonic_handlers[mnemonic](self.emu, mnemonic, self.get_register32("EIP"), op1value, op2value, op3value)
return True
经过阅读，这上万行代码全是仿真各种指令的实现，而且几乎都是上面例子这样的模式实现，真正阅读起来很多代码是可以跳过的，同时发现这些仿真实现几乎都带有反汇编引擎的影子，嵌入的反汇编引擎只起到了得出指令助记符的功能（个人认为这里有很大的优化空间即将仿真过程嵌入反汇编引擎中，当然只是个设想，可能作者也是认为用这么点性能去换开发效率得不偿失），就是说要编写出这些仿真代码可能需要你同时非常了解x86汇编指令的格式并一定程度上的解析他。