pyautogui脱离屏幕基于图片的图像定位

用过pyautogui的同学应该都知道，locateOnScreen可以传入一张图片与当前屏幕（截屏）进行匹配，但是我的需求是能基于已经截屏的图片来进行图像定位，看了pyautogui的源码没有相关的接口，突发奇想自己定义一个。

首先，参考locateOnScreen的源码

/usr/local/lib/python3.9/site-packages/pyautogui/__init__.py

    @raisePyAutoGUIImageNotFoundException
    def locateOnScreen(*args, **kwargs):
        return pyscreeze.locateOnScreen(*args, **kwargs)

/usr/local/lib/python3.9/site-packages/pyscreeze/__init__.py 

def locateOnScreen(image, minSearchTime=0, **kwargs):
    """TODO - rewrite this
    minSearchTime - amount of time in seconds to repeat taking
    screenshots and trying to locate a match.  The default of 0 performs
    a single search.
    """
    start = time.time()
    while True:
        try:
            screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here.
            retVal = locate(image, screenshotIm, **kwargs)
            try:
                screenshotIm.fp.close()
            except AttributeError:
                # Screenshots on Windows won't have an fp since they came from
                # ImageGrab, not a file. Screenshots on Linux will have fp set
                # to None since the file has been unlinked
                pass
            if retVal or time.time() - start > minSearchTime:
                return retVal
        except ImageNotFoundException:
            if time.time() - start > minSearchTime:
                if USE_IMAGE_NOT_FOUND_EXCEPTION:
                    raise
                else:
                    return None

从上面代码可以看出，截屏的图像对象是通过screenshot得到的

screenshotIm = screenshot(region=None)

# set the screenshot() function based on the platform running this module
if sys.platform.startswith('java'):
    raise NotImplementedError('Jython is not yet supported by PyScreeze.')
elif sys.platform == 'darwin':
    screenshot = _screenshot_osx
elif sys.platform == 'win32':
    screenshot = _screenshot_win32
else: # TODO - Make this more specific. "Anything else" does not necessarily mean "Linux".
    screenshot = _screenshot_linux

从上面代码可以看出screenshot是平台相关的接口，因为我是mac平台，所以参考了_screenshot_osx的实现

def _screenshot_osx(imageFilename=None, region=None):
    """
    TODO
    """
    # TODO - use tmp name for this file.
    if imageFilename is None:
        tmpFilename = 'screenshot%s.png' % (datetime.datetime.now().strftime('%Y-%m%d_%H-%M-%S-%f'))
    else:
        tmpFilename = imageFilename
    subprocess.call(['screencapture', '-x', tmpFilename])
    im = Image.open(tmpFilename)

    if region is not None:
        assert len(region) == 4, 'region argument must be a tuple of four ints'
        region = [int(x) for x in region]
        im = im.crop((region[0], region[1], region[2] + region[0], region[3] + region[1]))
        os.unlink(tmpFilename) # delete image of entire screen to save cropped version
        im.save(tmpFilename)
    else:
        # force loading before unlinking, Image.open() is lazy
        im.load()

    if imageFilename is None:
        os.unlink(tmpFilename)
    return im

从上面代码可以看出，实际上截屏的接口的实现也很简单，先使用命令截屏保存图片，再用Image.open()打开图片创建一个对象im，然后在im.load()即可，接口返回的是对象im。

由此，可以看出，只要外部传入一个图片文件路径，然后使用Image.open()得到im就可以进行图像定位了。

---

新增代码如下：

/usr/local/lib/python3.9/site-packages/pyautogui/__init__.py

    @raisePyAutoGUIImageNotFoundException
    def locateOnImage(*args, **kwargs):
        return pyscreeze.locateOnImage(*args, **kwargs)

 /usr/local/lib/python3.9/site-packages/pyscreeze/__init__.py

def locateOnImage(scr_image, target_image, minSearchTime=0, **kwargs):
    """TODO - rewrite this
    minSearchTime - amount of time in seconds to repeat taking
    screenshots and trying to locate a match.  The default of 0 performs
    a single search.
    """
    start = time.time()
    while True:
        try:
            im = Image.open(target_image)
            im.load()
            #screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here.
            retVal = locate(scr_image, im, **kwargs)
            try:
                im.fp.close()
            except AttributeError:
                # Screenshots on Windows won't have an fp since they came from
                # ImageGrab, not a file. Screenshots on Linux will have fp set
                # to None since the file has been unlinked
                pass
            if retVal or time.time() - start > minSearchTime:
                return retVal
        except ImageNotFoundException:
            if time.time() - start > minSearchTime:
                if USE_IMAGE_NOT_FOUND_EXCEPTION:
                    raise
                else:
                    return None

src_image就相当于原本locateOnScreen的image，target_image就是要基于此定位的图片，然后把原本的

screenshotIm = screenshot(region=None)

换成

im = Image.open(target_image)
im.load()

 现在就可以使用自定义接口来进行图片和图片的匹配定位了

x, y = locate_on_image('chrome.png','screen.png', confidence=0.9)
print(x, y)

输出结果：

679 1619

chrome.png

 

screen.png