Windows GUI自动化控制工具之python uiAutomation

对 Windows GUI进行自动化控制的工具有很多，比如pywinauto、pyautogui、pywin32、Autoit、airtest、UIAutomation等，UI Automation API是微软提供的自动化框架，可在支持 Windows Presentation Foundation (WPF) 的所有操作系统上使用，支持的应用类型更多。本文介绍封装了UI Automation API的Python uiautomation 模块的使用方法。

Python uiautomation 模块由yinkaisheng 开发，封装了微软 UI Automation API，支持自动化Win32，MFC，WPF，Modern UI(Metro UI), Qt, IE, Firefox, Chrome和基于Electron开发的应用程序。

环境准备

uiautomation安装

最新版uiautomation2.0只支持Python 3版本，但不要使用3.7.6和3.8.1这两个版本，因为comtypes包在这两个版本中不能正常工作。

pip安装uiautomation：

$ pip install uiautomation

检查是否安装成功：

$ pip list | findstr uiautomation
uiautomation       2.0.18

安装完成后，在Python的Scripts(我的路径为C:\Program Files\Python37\Scripts)目录中会有一个文件automation.py，是用来枚举控件树结构的一个脚本。

可运行 automation.py -h查看命令帮助:

$ python automation.py -h
UIAutomation 2.0.18 (Python 3.7.2, 64 bit)
usage
-h      show command help
-t      delay time, default 3 seconds, begin to enumerate after Value seconds, this must be an integer
        you can delay a few seconds and make a window active so automation can enumerate the active window
-d      enumerate tree depth, this must be an integer, if it is null, enumerate the whole tree
-r      enumerate from root:Desktop window, if it is null, enumerate from foreground window
-f      enumerate from focused control, if it is null, enumerate from foreground window
-c      enumerate the control under cursor, if depth is < 0, enumerate from its ancestor up to depth
-a      show ancestors of the control under cursor
-n      show control full name, if it is null, show first 30 characters of control's name in console,
        always show full name in log file @AutomationLog.txt
-p      show process id of controls

if UnicodeError or LookupError occurred when printing,
try to change the active code page of console window by using chcp or see the log file @AutomationLog.txt
chcp, get current active code page
chcp 936, set active code page to gbk
chcp 65001, set active code page to utf-8

examples:
automation.py -t3
automation.py -t3 -r -d1 -m -n
automation.py -c -t3

进程查看器

对 Windows GUI进行自动化控制需要使用进程查看器工具对GUI界面元素进行定位，定位工具有很多，这里推荐使用微软提供的inspect.exe 或者 Accessibility Insights 这两款工具。

inspect.exe

inspect.exe 是 Windows SDK 自带的一个进程查看器，可以用来查看系统正在运行的进程信息、模块、线程、堆栈跟踪等详细数据。

Windows SDK下载地址为：https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/

建议直接到这里下载inspect.exe：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/tree/master/inspect

64位系统版本的inspect.exe也可以点击这里下载。

Accessibility Insights

Accessibility Insights 是微软开发的一款辅助功能测试工具。它可以帮助开发者测试 web 应用、Windows 桌面应用和 Android 应用的可访问性，确保这些应用程序符合无障碍标准。

Accessibility Insights获取的控件属性信息没有inspect.exe全面，使用起来更加流畅。下载为：https://accessibilityinsights.io/downloads/

控件对象模型

微软 UIAutomation API定义了支持的控件类型和对应的模型（Pattern），所有支持的控件类型可参考：https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpatternmapping

控件类型	必须支持的模型	可选模型	Does not support
Button	None	ExpandCollapse, Invoke, Toggle, Value	None
Calendar	Grid, Table	Scroll, Selection	Value
CheckBox	Toggle	None	None
Edit	None	RangeValue, Text, Value	None
List	None	Grid, MultipleView, Scroll, Selection	Table
ListItem	SelectionItem	CustomNavigation, ExpandCollapse, GridItem, Invoke, ScrollItem, Toggle, Value	None
Menu	None	None	None
MenuBar	None	Dock, ExpandCollapse, Transform	None
MenuItem	None	ExpandCollapse, Invoke, SelectionItem, Toggle	None
RadioButton	SelectionItem	None	Toggle
SplitButton	ExpandCollapse, Invoke	None	None
Tab	Selection	Scroll	None
TabItem	SelectionItem	None	Invoke
Table	Grid, GridItem, Table, TableItem	None	None
Text	None	GridItem, TableItem, Text	Value
TitleBar	None	None	None
ToolBar	None	Dock, ExpandCollapse, Transform	None

python uiautomation库对UIAutomation API定义的各个Control和Pattern进行了封装。

下面来看使用python uiautomation操作Windows自带计算器的例子。

uiautomation库示例

控制计算器

可以使用inspect.exe来定位计算器元素：

示例脚本如下：

import os
import uiautomation as auto
import subprocess

class uiautoCalc(Loggers):
    """uiautomation控制计算器
    """
    def __init__(self):
        super().__init__()
        self.logger = Loggers().myLogger()
        auto.uiautomation.DEBUG_SEARCH_TIME =True 
        auto.uiautomation.SetGlobalSearchTimeout(2) # 设置全局搜索超时时间
        self.calcWindow = auto.WindowControl(searchDepth=1, Name='计算器', desc='计算器窗口') # 计算器窗口
        if not self.calcWindow.Exists(0,0):
            subprocess.Popen('calc')# 设置窗口前置
            self.calcWindow = auto.WindowControl(
            searchDepth=1, Name='计算器', desc='计算器窗口')
        self.calcWindow.SetActive() # 激活窗口
        self.calcWindow.SetTopmost(True) # 设置为顶层

    def gotoScientific(self):
        self.calcWindow.ButtonControl(AutomationId='TogglePaneButton', desc='打开导航').Click(waitTime=0.01)        
        self.calcWindow.ListItemControl(AutomationId='Scientific', desc='选择科学计算器').Click(waitTime=0.01)
        clearButton = self.calcWindow.ButtonControl(AutomationId='clearEntryButton', desc='点击CE清空输入')
        if clearButton.Exists(0,0):
            clearButton.Click(waitTime=0)
        else:
            self.calcWindow.ButtonControl(AutomationId='clearButton', desc='点击C清空输入').Click(waitTime=0.01)

    def getKeyControl(self):
        automationId2key ={'num0Button':'0','num1Button':'1','num2Button':'2','num3Button':'3','num4Button':'4','num5Button':'5','num6Button':'6','num7Button':'7','num8Button':'8','num9Button':'9','decimalSeparatorButton':'.','plusButton':'+','minusButton':'-','multiplyButton':'*','divideButton':'/','equalButton':'=','openParenthesisButton':'(','closeParenthesisButton':')'}        
        calckeys = self.calcWindow.GroupControl(ClassName='LandmarkTarget')
        keyControl ={}
        for control, depth in auto.WalkControl(calckeys, maxDepth=3):
            if control.AutomationId in automationId2key:
                self.logger.info(control.AutomationId)
                keyControl[automationId2key[control.AutomationId]]= control
        return keyControl

    def calculate(self, expression, keyControl):
        expression =''.join(expression.split())
        if not expression.endswith('='):
            expression +='='
            for char in expression:
                keyControl[char].Click(waitTime=0)
        self.calcWindow.SendKeys('{Ctrl}c', waitTime=0.1)
        return auto.GetClipboardText()

    def calc_demo(self):
        """计算器示例
        :return : 
        """        
        self.gotoScientific() # 选择科学计算器        
        keyControl = self.getKeyControl() # 获取按键控件
        result     = self.calculate('(1 + 2 - 3) * 4 / 5.6 - 7', keyControl)
        print('(1 + 2 - 3) * 4 / 5.6 - 7 =', result)
        self.calcWindow.CaptureToImage('calc.png', x=7, y=0, width=-14, height=-7) # 截图
        self.calcWindow.GetWindowPattern().Close() # 关闭计算机

if __name__ == "__main__":
    ui = uiautoCalc()
    ui.calc_demo()

脚本执行动图：

参考文档

1. https://github.com/pywinauto/pywinauto

2. https://cloud.tencent.com/developer/article/2213048

3. https://github.com/yinkaisheng/Python-UIAutomation-for-Windows

4. Python UIAutomation文档：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/blob/master/readme_cn.md

5. https://www.cnblogs.com/Yinkaisheng/p/3444132.html

6. GitHub - jacexh/pyautoit: Python binding for AutoItX3.dll

7. GitHub - mhammond/pywin32: Python for Windows (pywin32) Extensions

8. Accessibility tools - Inspect - Win32 apps | Microsoft Learn

9. Accessibility Insights

--THE END--