UIPath进阶教程-2-UI Automation

Sometimes the usual manual routine is not the optimal way for automation. Carefully explore the application’s behavior and UiPath’s integration/features before committing to a certain approach.

2.1 Desktop Automation

UI automation goes at its best when Robots and applications run on the same machine because UiPath can integrate directly with the technology behind the application to identify elements, trigger events and get data behind the scenes.
2.1.1. Input Methods
There are three methods UiPath uses for triggering a Click or a Type Into an application. These are displayed as properties in all activities that deal with UI automation.

If SimulateType or SimulateClick are selected, Studio hooks into the application and triggers the event handler of an indicated UI element (button, edit box)
● If SendWindowMessages is selected, Studio posts the event details to the application message loop and the application’s window procedure dispatches it to the target UI
element internally
● Studio signals system drivers with hardware events if none of the above option are selected and lets the operating system dispatch the details towards the target element

These methods should be tried in the order presented, as Simulate and WindowMessages are faster and also work in the background, but they depend mostly on the technology behind the application.
Hardware events work 100% as Studio performs actions just like a human operator (e.g. moving the mouse pointer and clicking at a particular location), but in this case, the application being automated needs to be visible on the screen. This can be seen as a drawback, since there is the risk that the user can interfere with the automation.
2.1.2. Selectors
Sometimes the automatically generated selectors propose volatile attribute values to identify elements and manual intervention is required to calibrate the selectors. A reliable selector should successfully identify the same element every time in all conditions, in dev, test and production environments and no matter the usernames logged on to the applications.
Here are some tips of how to improve a selector in Selector Editor or UiExplorer:
 Replace attributes with volatile values with attributes that look steady and meaningful
 Replace variable parts of an attribute value with wildcards (*)
 If an attribute’s value is all wildcard (e.g. name=’*’) then attribute should be removed
 If editing attributes doesn’t help, try adding more intermediary containers
 Avoid using idx attribute unless it is a very small number like 1 or 2

In the selector above, we notice the page title has a reference to the time when selector was recorded and also that some attributes have randomly looking IDs. Tweaking the attributes, we can come up with a better selector than UiPath recorder proposed.

2.1.3. Containers
Similar to file paths, selectors can be full or partial (relative). Full selectors start with a window or html identifier and have all necessary information to find an element on the whole desktop, while partial selectors work only inside an attach/container that specifies the top-level window where elements belong:
 OpenBrowser
 OpenApplication
 AttachBrowser
 AttachWindow
There are several advantages to using containers with partial selectors instead of full selectors:
 Visually groups activities that work on the same application
 Is slightly faster, not seeking for the top window every time
 Makes it easier to manage top level selectors in case manual updates are necessary
 Essential when working on two instances of the same application

2.2 Image Automation

Image recognition is the last approach to automating applications if nothing else works to identify UI elements on the screen (like selectors or keyboard shortcuts). Because image matching requires elements to be fully visible on the screen and that all visible details are the same at runtime as during development, when resorting to image automation extra care should be taken to ensure the reliability of the process. Selecting more/less of an image than needed might lead to an image not found or a false positive match.
2.2.1. Resolution Considerations
Image matching is sensitive to environment variations like desktop theme or screen resolution. When the application runs in Citrix, the resolution should be kept greater or equal than when recording the workflows. Otherwise, small image distortions can be compensated by slightly lowering the captured image Accuracy factor.
Check how the application layout adjusts itself to different resolutions to ensure visual elements proximity, especially in the case of coordinate based techniques like relative click and relative scrape.
If the automation supports different resolutions, parallel recordings can be placed inside a PickBranch activity and Robot will use either match.
2.2.2. OCR Engines
If OCR returns good results for the application, text automation is a good alternative to minimize environment influence. Google Tesseract engine works better for smaller areas and Microsoft MODI for larger ones.
Using the MODI engine in loop automations can sometimes create memory leaks. This is why it is recommended that scraping done with MODI be invoked via a separate workflow, using the Isolated property.

2.3 UI Synchronization

Unexpected behavior is likely to occur when the application is not in the state the workflow assumes it to be. The first thing to watch for is the time the application takes to respond to Robot interactions.
The DelayMS property of input enables you to wait a while for the application to respond. However, there are situations when an application’s state must be validated before proceeding with certain steps in a process. Measures may include using extra activities that wait for the desired application state before other interactions. Activities that might help include:
 ElementExists
 FindImage
 FindText

WaitElementVanish
 WaitImageVanish
 WaitScreenText (in terminals)

2.4 Background Automation

If an automation is intended to share the desktop with a human user, all UI interaction must be implemented in the background. This means that the automation has to work with UI element objects directly, thus allowing the application window to be hidden or minimized during the process.
● Use the SimulateType, SimulateClick and SendWindowMessages options for navigation and data entry via the Click and TypeInto activities
● Use the SetText, Check and SelectItem activities for background data entry
● GetText, GetFullText and WebScraping are the output activities that run in the background
● Use ElementExists to verify application state