Android抓取文字、文字位置的实现

引文：

背景在前文说了，里面实现的细节也清楚了。所以接下来，要扩展下控制框架。这是实践的记录。

使用

先放实现效果吧

import atx
d=atx.connect()
d.tap_text(u'动态')        # 点击第一个有动态的控件
print d.find_text(u'动态') # 找出全部包含"动态"的字符串

这两个函数tap_text、find_text 是这样写的

hirerachy的获取

从前文分析，用到的指令有这几个

adb shell uiautomator dump /sdcard/out.xml
adb pull /sdcard/out.xml

可以这样包装：

def __find_text_uiautomator(self, text="", full_match=True):
    remote_target = self.__remote_tmp_path(complexity=True, suffix="xml")
    local_tmp_file = self.__local_tmp_path(suffix="xml")
    try:
        adb.dump_uiautomator(serial=self.serial, port=self.port, remote_path=remote_target)
        adb.pull(serial=self.serial, port=self.port, remote_path=remote_target, local_path=local_tmp_file)
        return xmls.find(path=local_tmp_file, text=text, full_match=full_match)
    except (IOError, Exception):
        return []
    finally:
        adb.rm(serial=self.serial, port=self.port, remote_path=remote_target)
        self.__remove_local_file(local_tmp_file)

@hook_wrap(consts.EVENT_CLICK_TEXT)
def _tap_text(self, text="", full_match=True, target_index=0, try_times=1, frequency=0.3, safe=True):
    try_times = try_times if try_times > 0 else 1
    x = -1
    y = -1
    for i in range(try_times):
        result = self.__find_text_uiautomator(text=text, full_match=full_match)
        if len(result) > 0 and len(result) > target_index:
            x, y = result[target_index]["point"]
            self._tap(x, y)
            break
        else:
            time.sleep(frequency)
    if safe or x != -1 or y != -1:
        return x, y
    else:
        raise TextNotFoundError(
            '%s,%d:Not found text %s ,try %d' %
            (self.serial, self.instance, text, try_times))

 def find_text(self, text="", timeout=15, frequency=0.3, delay=0, full_match=False):
     if delay > 0:
         time.sleep(delay)
     return self._find_text(text=text,
                            full_match=full_match,
                            try_times=int(timeout / frequency),
                            frequency=0.3,
                            safe=True)

def tap_text(self, text="", timeout=15, frequency=0.3, delay=0, full_match=True):
    if delay > 0:
        time.sleep(delay)
    self._tap_text(text=text,
                   full_match=full_match,
                   try_times=int(timeout / frequency),
                   safe=False)

hierrachy的解析

python这边做一点xml的解析,xml解析有几种方式，常见的是DOM、SAX，这里简单用SAX处理一下，主要就是在startElement找下key

要解析的内容大概是这样的

<hierarchy rotation="0">
<node 
index="1" text="所有联系人" 
resource-id="" class="android.widget.TextView" 
package="com.android.contacts" content-desc="" 
checkable="false" checked="false" 
clickable="true" enabled="true" 
focusable="false" focused="false" 
scrollable="false" long-clickable="true" 
password="false" selected="true" 
bounds="[477,240][1080,408]" />node>
hierarchy>

里面是有挺多信息的，但是目前我们需要关注的就是 text bounds，可以这样处理xml

#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = 'Yeshen'

import xml.sax
from xml.sax import ContentHandler

PATTON_PARK_MATCH = 0
PATTON_FULL_MATCH = 1


# simple self test
# python -c "import xmls as test;print test.find('../../out/window_dump.xml','收藏')"
def find(path="", text="", full_match=True):
    parser = xml.sax.make_parser()
    parser.setFeature(xml.sax.handler.feature_namespaces, 0)
    handler = FindNodeHandle(text.encode('utf-8').strip(), full_match)
    parser.setContentHandler(handler=handler)
    parser.parse(path)
    return handler.getResult()


class FindNodeHandle(ContentHandler):
    def __init__(self, target, full_match):
        ContentHandler.__init__(self)
        self._target = target.strip()
        self._isFullMatch = full_match
        self._finds = []

    def doFind(self, text, bounds):
        if self._isFullMatch:
            match = text == self._target
        else:
            match = text.find(self._target) != -1
        if match and len(bounds) > 6:  # bounds u'[0,240][477,408]'
            points = bounds[1:len(bounds) - 1].split("][")
            x = 0
            y = 0
            for point in points:
                p = point.split(",")
                if len(p) == 2:
                    x += int(p[0]) / 2
                    y += int(p[1]) / 2
            self._finds.append({'text': text, 'point': (x, y)})

    def getResult(self):
        return self._finds

    def startElement(self, tag, attributes):
        if "text" in attributes \
                and "bounds" in attributes \
                and len(attributes["text"]) > 0:
            self.doFind(
                text=attributes["text"].encode('utf-8').strip(),
                bounds=attributes["bounds"])
            # if "content-desc" in attributes:

    def endElement(self, tag):
        pass

优化

自测的时候，发现uiautomator dump令人难以置信的慢。

>> d.tap_text(u"动态")

dump_uiautomator__16:56:17.532321
__done_dump_uiautomator__16:56:19.591696
pull__16:56:19.591899
__done_pull__16:56:19.620207
tap__16:56:19.706735
__done_tap__16:56:20.573592

uiautomator dump花了2s

这个和实现有关，在前一篇博客介绍的,uiautomator用了Accessibility的功能，Accessibility是在View中缓存了一份数据，准备这些数据需要时间。在DumpCommand.java中也看到有等待的代码

//com.android.commands.uiautomator.DumpCommand
@Override
public void run(String[] args) {
    ...
    uiAutomation.waitForIdle(1000, 1000 * 10);
    ...
}

/**
 * Waits for the accessibility event stream to become idle, which is not to
 * have received an accessibility event within idleTimeoutMillis.
 * The total time spent to wait for an idle accessibility event stream is bounded
 * by the globalTimeoutMillis.
 *
 * @param idleTimeoutMillis The timeout in milliseconds between two events
 *            to consider the device idle.
 * @param globalTimeoutMillis The maximal global timeout in milliseconds in
 *            which to wait for an idle state.
 *
 * @throws TimeoutException If no idle state was detected within
 *            globalTimeoutMillis.
 */
public void waitForIdle(long idleTimeoutMillis, long globalTimeoutMillis)
        throws TimeoutException {
    synchronized (mLock) {
        throwIfNotConnectedLocked();

        final long startTimeMillis = SystemClock.uptimeMillis();
        if (mLastEventTimeMillis <= 0) {
            mLastEventTimeMillis = startTimeMillis;
        }

        while (true) {
            final long currentTimeMillis = SystemClock.uptimeMillis();
            // Did we get idle state within the global timeout?
            final long elapsedGlobalTimeMillis = currentTimeMillis - startTimeMillis;
            final long remainingGlobalTimeMillis =
                    globalTimeoutMillis - elapsedGlobalTimeMillis;
            if (remainingGlobalTimeMillis <= 0) {
                throw new TimeoutException("No idle state with idle timeout: "
                        + idleTimeoutMillis + " within global timeout: "
                        + globalTimeoutMillis);
            }
            // Did we get an idle state within the idle timeout?
            final long elapsedIdleTimeMillis = currentTimeMillis - mLastEventTimeMillis;
            final long remainingIdleTimeMillis = idleTimeoutMillis - elapsedIdleTimeMillis;
            if (remainingIdleTimeMillis <= 0) {
                return;
            }
            try {
                 mLock.wait(remainingIdleTimeMillis);
            } catch (InterruptedException ie) {
                 /* ignore */
            }
        }
    }
}

waitForIdle(1000, 1000 * 10) 这段代码什么意思呢，就是最多等10s，每次等1秒后再检查一次。这个编译出来就是uiautomator，uiautomator在android机器上的（即修改下ROM）。所以可以修改第一个参数，idleTimeoutMillis，把检查的频率提高一点。

优化之后估计速度可以在1s左右，希望更快的话，只能使用ViewServer了 :(

github

代码迟点(我勤快的话)会同步更新到github，即

https://github.com/wuyisheng/ATX