2019 年 12 个最好的桌面软件自动化测试工具（注：原文未翻译）

12 Best Automation Tools for Testing Desktop Apps in 2019

This post is frequently updated. Last updated: 14 Jun, 2019

Although the demand for desktop app testing is not growing as fast as mobile and web app testing, it’s still a crucial day-to-day duty of many testers, especially those who work on sophisticated legacy systems in complex enterprise environments. Ignoring desktop app automation is simply a luxury that those testers can’t afford.

Desktop applications are computer programs that must be installed, configured and executed on a desktop operating system such as Windows, Linux and macOS. End-users need a desktop computer or laptop with sufficient hardware resources to access the functionalities of these apps.

Common technologies to build desktop apps include Java (Swing, AWT, SWT, JavaFX), .NET (UWP, WPF, WinForms, VB.NET), Windows API (win32), GDI, MFC, Visual Basic, Visual C++, PowerBuilder, SAPGUI, Oracle Forms, Telerik, DevExpress, Infragistics, Xlib, Cocoa, etc.

This article lists out the best automation tools to automate functional testing for desktop apps. If you’re looking for a tool that fits your specific needs, this list is definitely helpful for you to explore, evaluate, compare and eventually adopt the best tool for your automation needs.

---

NOTE:
• The tools are not ranked.
• You may notice that many of the tools on the list are commercial. This is
due to the complex nature of automating desktop apps. Most tools that
are able to automate desktop apps are usually proprietary, and there are
limited open-source options.

Standard Criteria

When assessing the pros and cons of a specific tool, I used the following criteria.

#	Criteria	Description
1	Cross operating systems	Can the tool run on multiple OS (Windows, Linux, macOS)?
2	Cross automation platforms	Can the tool automate desktop apps built by various technologies such as Java, WPF, UWP, SAPGUI, Oracle Forms, etc.?
3	Programming languages support	Does the tool support writing automation tests in various programming languages such as Java, C#, Python, Ruby, JavaScript, etc.?
4	Test runner support	Does the tool support various test runners or unit test frameworks such as Junit, xUnit, Pytest, TestNG, Mocha, etc.?
5	Automation technology	What kind of automation technologies does the tool support? Does it provide its own proprietary technology or build on top of open-source solutions such as Selenium WebDriver?
6	Element locating technique	What types of element locators does the tool support? Does it allow testers to inspect the internal properties of UI objects? Does it support its proprietary technique?
7	Built-in UI inspection tool	Does the tool provide a built-in tool to inspect the internals of UI objects, or does it rely on a third-party tool?
8	Page Object Model (POM) support	Does the tool separate the automation tests from the UI element locators as in the form of the Page Object Model (POM)?
9	Coding skills required	How much coding proficiency does the tool require?
10	Record & playback tool	Does the tool provide built-in record & playback capabilities?
11	Built-in ALM integrations	Does the tool integrate with ALM tools such as Jenkins, TFS, Azure, AWS, CircleCI, etc.?
12	Price	Is the tool free and open-source? If it’s commercial, does it provide a freemium option?

1. WinAppDriver

Figure 1 – WinAppDriver architecture

WinAppDriver is built on top of the WebDriver protocol – the gold standard for web & mobile app testing. You can use WinAppDriver to drive any Windows apps such as classic Windows controls (Win32), WPF, WinForms, and UWP using a wide variety of language bindings (C#, Java, Python, Ruby), and test runners (MSTest, JUnit, Nunit, etc.).

Pros

Cons

• You can use any language and test runner you prefer to test your Windows apps thanks to WebDriver/Appium • It’s WebDriver-based so Selenium/Appium testers can use it with ease • You can effortlessly integrate with Selenium and Appium projects in the same language • You can perform advanced gestures such as pressure, tilt, barrel button (Surface pen), etc. • You can use XPath and other properties to locate Windows UI elements • You can use the industry standard POM • It’s FREE, open-source and backed by Microsoft

• Only runs on Windows 10, not macOS or Linux • Cannot automate apps written by other tech stacks such as: Java, DevExpress, Infragistics, SAPGUI and Oracle Forms, etc. • No built-in spy tool. You must use inspect.exe (Windows SDK) and mentally map inspect.exe properties to WinAppDriver properties • You must know how to code & architect a maintainable test automation framework • No record & playback capabilities • No built-in support for ALM integrations

If you’re only interested in automating Windows apps with WebDriver, and you know how to code, you should go with WinAppDriver. If you’re not confident with coding, adopting WinAppDriver may not be the best option for you.

2. Winium

Figure 2 – Winium test script

Winium.Desktop is an open-source test automation framework for WinForms and WPF app testing. Just like WinAppDriver, it’s based on WebDriver protocol. However, the tool is considerably weaker than WinAppDriver in several areas.

Pros

Cons

• You can use any language such as Java, Objective-C, Javascript/NodeJS, PHP, Python, Ruby, C#, etc. • You can use any test runner (JUnit, TestNG, NUnit, PyUnit, etc.) • You can automate WPF and WinForms • It’s also WebDriver-based so using the tools is easier if you’re familiar with Selenium/Appium • You can use XPath to locate Windows UI elements • You can use the industry standard Page Object Model (POM) • It’s FREE and open-source

• The code base is not actively maintained (last commit was in 2016) • Only runs on Windows 10, not macOS or LinuxCannot automate apps written by other tech stacks such as Java, DevExpress, Infragistics, SAPGUI, Oracle Forms, etc. • The API is not comprehensive enough • No built-in spy tool. You must use inspect.exe or any other third-party tools • You must know how to code & architect a maintainable test automation frameworks • No record & playback capabilities • No built-in support for ALM integrations

For open-source projects, active maintenance is key. Winium has not been actively maintained since 2016 so if you go with it, you have to fix bugs or add new improvements on your own. Your team needs extensive technical knowledge and skills. Ultimately Winium is a less desirable choice than WinApp driver.

3. TestArchitect

Figure 3 – TestArchitect IDE showing a keyword-driven test

TestArchitect is a Keyword-Driven Testing tool. Testers with modest programming skills can learn quickly thanks to an extensive keyword library. TestArchitect also features a modern Keyword-Driven Testing method called Action-Based Testing. TestArchitect appears in Forrester Research reports in recent years.

Pros

Cons

• TestArchitect runs on Windows, Linux and macOS • Complete feature set for test development, test execution and test management • You’re able to write cross-platform tests jumping from desktop to web, mobile, API, database and more platforms. • You can automate apps written by different tech stacks such as .NET, Java, classic Windows controls, Oracle Forms, DevExpress, etc. • You can locate UI elements using various techniques including XPath, CSS selector, ID, name, class, etc.Built-in object spy tool • Built-in record-playback • You don’t have to learn coding since tests are keyword-driven • You can easily integrate TestArchitect with many ALM tools like Team Foundation Server, Azure DevOps, Zephyr, Jenkins, HP QC, etc. • Action-Based Testing is a relatively more modern Keyword-Driven Testing method than other keyword-driven approaches • TestArchitect has its own standalone IDE

• Besides ABT, you can only use 3 languages: Java, Python and C# to write actions. No other languages supported. • Only one test runner provided: TestArchitect Controller. • Automation technologies are proprietary. • You cannot push TestArchitect test assets into Git for version control because TestArchitect has its own built-in version control • Commercial product so you have to pay (flexible pricing). However, you can use TestArchitect Team for free with a limited number of test cases. • You cannot run tests in parallel on one machine but you can distribute tests to a network of machines/VMs/Docker

TestArchitect is relatively stronger than open-source solutions in terms of usability, width and depth of support. If you don’t want to build a framework from scratch, you should go with this battle-tested framework. Besides, the tool is well maintained by a professional engineering team, unlike Winium. The caveat? You need a budget for the upfront cost of purchasing it. Luckily, the tool does offer a Freemium version with limited number of test cases.

注：下面视频为 YouTube(播放源) 嵌入；如果不显示说明本博客平台不支持。

Quick Introduction to TestArchitect

4. Ranorex

Figure 4 – Ranorex Studio showing a keyword-driven test

Ranorex is a test automation tool that has been in the market for a long time. They offer proprietary automation technologies to automate a wide range of desktop, web and mobile apps. Ranorex also appears in Forrester Research reports in recent years.

Pros

Cons

• Ranorex runs on Windows, macOS and Linux • You can compose tests in their standalone Ranorex Studio instead of using a third party IDE such as Visual Studio Code • Tests are keyword-driven so it’s easier to maintain in the long run. • Keyword-driven testing eliminates most of the coding needs so you could develop tests faster without coding skills • Keywords can be scripted using VB.NET and C#. You could mix record-and-playback keywords with scripted keywords in one test • Built-in object spy tool • Built-in record-playback • Built-in ALM integrations such as Bamboo, HP Quality Center, TFS/MTM, Jenkins, TeamCity, etc.

• Only one proprietary test runner supported • Automation technologies are proprietary. • The UI element locating technique called RanorexPath is their own invention. Nobody else adopts this technique so you can’t use any third-party tools besides the built-in Spy tool. • In the Omni-channel Functional Test Automation Tools 2018 report conducted by Forrester, Ranorex was ranked as having the weakest current offering • Commercial product so you have to pay (~$5,800 per floating license)

Ranorex’s presence in the Japan market is strong thanks to their Japanese localization and reliable Record & Playback. For other markets, Ranorex is not that strong. Invest in Ranorex if you prefer Record & Playback as the primary method of test development.

5. Tricentis Tosca

Figure 5 – Tricentis Tosca UI

Tricentis provides a wide support for virtually all test activities. They offer customers a Continuous Testing platform that covers test design, UI testing, API testing, service virtualization and test data management. With the recent acquisition of QASymphony (qTest), they further expand their scope to test management.

Pros

Cons

• Tosca is easier to adopt by non-technical testers thanks to its model-based method • Tosca can automate a wide range of platforms like .NET, Java, browsers, mobile apps, etc. • Cross-platform end to end tests jumping from desktop, to mobile, web, SAP and API in one flow. • Tosca supports C# and JavaLocating elements using various properties including ID, text, and anchor. • Built-in object spy tool called XScan • Built-in record-playbackBuilt-in ALM integrations and test management tools, especially QASymphony qTest • Automatically generate test data (the feature is actually called “generate instances for a test case”). • Gartner ranked the tool as leader in their recent automation testing reports

• Cannot run on macOS and Linux • Element locators are stored in “modules” which are similar to page objects but it’s harder to use than page objects • Proprietary test runnerModel-based approach (noun-first keywords) create unnecessary friction in test creating and maintenance problems • Too much clicking when writing tests • Cannot integrate with Git for version control of test assets since Tosca has its own repository. However, you can link GitHub revisions to Tosca revisions. • Commercial options only, no freemium

Tricentis Tosca is very friendly to manual testers who want to transition to test automation engineers. The company claims that given 3 weeks, they can train 85 out of 100 manual testers to become automation engineers using Tosca. On the contrary, only 15 out of 100 manual testers became automation engineers given 3 months of training (yes it’s months, not weeks). However, albeit easier to adopt at first, Tricentis Tosca will gradually become more complex and difficult to use in the long run.

6. TestComplete

Figure 6 – TestComplete BDD-style test cases

SmartBear TestComplete has been on the automation market since 1999 so it’s obviously one of the veterans. TestComplete (TC) offers a comprehensive solution that covers desktop, web, and mobile. They’ve recently added a hybrid UI matching method combining visual recognition and object recognition.

Pros

Cons

• TC is easier to adopt by non-technical testers thanks to its keyword-driven method • TC can automate a wide range of platforms like .NET, Java, browsers, mobile apps, etc. • Cross-platform end to end tests jumping from API, to desktop, mobile, and web in one flow. • TC supports JavaScript, Python, VBScript, Jscript, C++, Delphi, and C# • Locating elements using various properties including ID, text, XPath. • Built-in object spy tool • Built-in record-playback that runs reliably • Built-in ALM integrations such as Jenkins, HP Quality Center, etc. • Gartner ranked the tool as leader in their 2018 report

• Cannot run on macOS and Linux • Proprietary test runner • Noun-first keyword approach creates unnecessary friction in test creating • Too much clicking when writing tests • Cannot integrate with Git for version control of test assets since TC has its own repository. • Commercial options only, no freemium.

TestComplete is indeed one of the best tools on this list. Consider checking it out.

7. Micro Focus UFT(QTP)

Figure 7 – UFT’s GUI test

UFT/QTP is one of the oldest tools that still survives in the test automation market until today. UFT is a powerful UI testing tool. It allows you to test a wide range of platforms including desktop, web, web services, SAP, and mobile. It also integrates seamlessly with HP Quality Center.

Pros

Cons

• UFT is easier to adopt by non-technical testers thanks to its keyword-driven method • UFT can automate a wide range of platforms like .NET, Java, browsers, mobile apps, web services, SAP, ComponentOne, DevExpress, UIA, databases, etc. • Cross-platform end to end tests jumping from API, to desktop, mobile, and web in one flow. • Locating elements using various object properties and XPath. • Ability to integrate with Git, Subversion and TortoiseSVN for version control of test assets • Built-in object spy tool • Built-in record-playback that runs reliably • Built-in ALM integrations, especially with HP Quality Center/ HP ALM, etc.

• Cannot run on macOS and Linux • Proprietary test runner • UFT supports only VBScript • Noun-first keyword approach creates unnecessary friction in test creating • Too much clicking when writing tests • Commercial so you must pay

If you already have HP QC as your test management system, UFT is your natural choice for test automation.

8. Sikulix

Figure 8 – SikuliX IDE showing an image-based test

SikuliX is an open-source tool that automates anything you see on screen using image recognition and OCR. The name actually means “God’s Eye” in Huichol Indian culture. Check it out here.

Pros

Cons

• Java-based so it can run on Windows, Linux and macOS • Since SikuliX is image-based you can automate any platforms (Java, .NET, SAP, etc.) • You don’t have to know the internals of UI objects on screen since those UI controls are recognized by images • You can write keyword-driven tests that are relatively more readable using Robot Framework. Not much coding required. • You can automate mobile apps via mobile emulators and web apps on desktop browsers • You can use Python, Ruby and Javascript to write test scenarios • No need for object spy tool • FREE and open source

• SikuliX needs a real screen connected. That means you cannot run tests in headless mode. • Image-based automation is slower and more fragile than the object-based technique • You cannot run tests in parallel on one machine since SikuliX takes the entire control of mouse and keyword interactions • No built-in record-playback capabilities • No built-in ALM integrations • Poor documentation, user guides and community support

SikuliX is suitable for small projects since it was designed to automate various mundane tasks on Windows. However, once you scale up, you’ll find unsolvable problems due to the nature of the image recognition technique. Additionally, image recognition is way slower than object recognition.

9. Robot Framework

Figure 9 – A test case in Robot Framework (credit: robotframework.org)

Robot Framework is another open-source tool that is keyword-driven. Robot is flexible since you could use it with many desktop automation libraries such as SwingLibrary (Java Swing GUI), SikuliLibrary (SikuliX), SapGuiLibrary, JavaFXLibrary (JavaFX apps), EclipseLibrary (Eclipse RCP/SWT), and AutoItLibrary (AutoIt).

Pros	Cons
• Can run on multiple OS: Windows, Linux, macOS • Write easy to understand tests in keyword-driven format so not much coding required • Support many platforms such as desktop, web, and mobile • Support many test runners • Multiple element locator types • Can use page objects • Can run multi tests in parallel using Pabot • Strong community support • It’s FREE	• You to use Eclipse or RIDE to write tests • Tests are tab-delimited so it’s hard to view • No built-in object spy tool • No built-in ALM integrations • No built-in record-playback • Out-of-date documentation & user guides

Robot Framework is an excellent alternative to expensive commercial tools. If your budget is tight, consider adopting Robot FW. However, beware of the not-so-obvious investment in learning and maintaining the framework in the long run.

10. AutoIt

Figure 10 – AutoIt script to automate the Windows Calculator app

AutoIt is a freeware designed to automate Windows GUI. Although the initial purpose of AutoIt was to “roll out” and configure thousands of PCs, you can use AutoIt for test automation too. Fun fact: AutoIt became popular in part thanks to Selenium’s inability to interact with Windows dialogs that you manually use to upload files to a web page.

Pros

Cons

• AutoIt scripts can be compiled into standalone executables and executed from a Selenium-Java program • Provides its own IDE to write scripts based on the free SciTE source code editor • Built-in tool to inspect UI elements • Can use many object properties to identify UI controls • The project is actively maintained • It’s FREE but closed source

• Compatible with many Windows versions from XP to Windows 10 but cannot run on Linux and macOS • Only support its own BASIC-like scripting language • Coding required • You cannot run tests in parallel • Unique test runner • You cannot automate newer apps written by UWP, WPF or other platforms such as Java, DevExpress, etc. • You must handle timing on your own through hardcoded sleeps • No built-in record-playback • No built-in ALM integrations • Not popular so it’s harder to get community support

AutoIt works for some simple automation scenarios. If you want a free and lightweight solution to get through simple automation challenges, use AutoIt. Also note that if you like Selenium, you ought to use AutoIt to work around pop-up dialogs.

11. White Framework

Figure 11 – Sample test case written in White

White framework helps you automate rich client applications written by Win32, WinForms, WPF, Silverlight and JavaSWT. It’s based on .NET and depends on Microsoft’s UIAutomation (UIA) to drive desktop apps.

Pros	Cons
• Support multiple .NET language • Built on top of UI • Automation and Windows API so it’s powerful for each niche • FREE and open source	• Cannot run on Linux and macOS • No built-in IDE • No built-in object spy tool, must use Windows SDK Inspect, UI Automation Verify or Snoop • Not actively maintained by the community (last commit was 2017) • Poor documentation and sample code

White is not a strong contender on this list due to its lack of capabilities and narrow target automation platforms. Try to avoid using White if you want a maintainable tool that can help scale up your test project. That said, small quick and dirty projects can still leverage White thanks to its free cost.

12. Conformiq

Figure 12 – Conformiq’s test case design

Conformiq is not strictly an automation tool. Its core offerings (Creator and Transformer) are tools that help you model the business flows of the app under test, generate tests and transform those tests to executable scripts. Conformiq totally relies on 3rd party automation libraries such as QTP/UFT for desktop testing and Selenium for web testing.

Pros

Cons

• Provides its own Eclipse-like IDE • Supports desktop, web and mobile apps depending the automation tool • Model-based testing (MBT) so you can visualize your test flows using a model • Auto generate test cases from a model • Keyword-driven / BDD test scripts so it’s more friendly to non-developers • Can work with Selenium, UFT, Tricentis Tosca, Parasoft SOAtest, Eggplant, Appium, Cucumber, etc. • Built-in tool to inspect UI elements • Built-in ALM integrations with HP QC

• Java-based so Conformiq runs across Windows and Linux but not macOS • No built-in record playback • More expensive since you have to purchase UFT on top of Conformiq’s cost

Conformiq works really well for test teams with more business expertise and less technical. It helps you visualize the test flows as a whole through a model so you don’t miss test cases and bugs. However, having to purchase additional UFT licenses is a big entry barrier.

Conclusion

This list was composed as of early 2019. The tool market for automation testing is very competitive. So you can expect radical changes and innovative features being introduced every day. If you spot some outdated information, please leave a comment. I’ll fix ASAP.

Have you seen better desktop app testing tools not listed above? Comment the tool names and why you think they are the best below. I’ll add them as soon as possible.

Disclaimer: This article is not endorsed by, directly affiliated with, maintained, authorized, or sponsored by any of the companies mentioned in this article (Microsoft, Ranorex, Tricentis, SmartBear LLC, HP, Micro Focus, and the listed open-source project teams). All product and company names are the registered trademarks of their original owners. The use of any trade name or trademark is for identification and reference purposes only and does not imply any association with the trademark holder or their product brand.