Appium-Image Comparison(图像对比)

This article describes the set of image comparison features available in Appium. These features are available in all drivers and require OpenCV 3 native libs. Also, each feature is able to visualize the comparison result, so you can always track what is going on under the hood to select optimal matching parameters to achieve the best comparison results.

本文介绍了Appium中可用的一组图像比较功能.
这些特征是可用的在驱动程序中并且需要OpenCV 3本地库.
另外，每个特征都能够可视化比较结果，
因此你可以随时跟踪引擎盖下发生的情况，
以选择最佳匹配参数，
从而获得最佳的比较结果.

Prerequisites

• OpenCV 3+ native libraries

OpenCV 3+本地库

• opencv4nodejs npm module: npm i -g opencv4nodejs. By default the preinstall script of this module also downloads and makes all the required OpenCV libs from source, but this requires developer tools to be available on the host system.

opencv4nodejs模块:`npm i -g opencv4nodejs`.
默认预安装脚本也是从这个模块下载并从源代码生成所有必须的OpenCV库.
但是这样要求从主机上提供开发人员工具

• Appium Server 1.8.0+

Appium服务版本1.8.0+

Purpose

Image comparison might be handy for many automation tasks. For example: - It is necessary to figure out whether the given picture example is present on the screen - It is necessary to calculate coordinates of some predefined on-screen object - It is necessary to verify whether the current on-screen object state is similar to the expected state

图像对比或许更多的提供给自动化工作.
例如：
-它必须要弄清楚屏幕上是否有给出的图片
-它必须要去计算预定义屏幕对象的坐标
-它必须要验证屏幕上现在的对象是否与预期状态相似

Feature-based Comparison

Performs images matching by template to find possible occurrence of the partial image in the full image. Read https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_feature2d/py_matcher/py_matcher.html for more details on this topic. Such comparison is useful in case the resulting image is rotated/scaled in comparison to the original one.

执行图片的匹配通过模板
在完整图像中去查找可能存在的部分图像.
阅读：https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_feature2d/py_matcher/py_matcher
html有关更多的细节在这个主题里.
例如：
比较是有用的假如产生的图片是旋转/缩放
与原来的那个相比.

Examples

// java

byte[] screenshot = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES));
FeaturesMatchingResult result = driver
        .matchImagesFeatures(screenshot, originalImg, new FeaturesMatchingOptions()
                .withDetectorName(FeatureDetector.ORB)
                .withGoodMatchesFactor(40)
                .withMatchFunc(MatchingFunction.BRUTE_FORCE_HAMMING)
                .withEnabledVisualization());
assertThat(result.getVisualization().length, is(greaterThan(0)));
assertThat(result.getCount(), is(greaterThan(0)));
assertThat(result.getTotalCount(), is(greaterThan(0)));
assertFalse(result.getPoints1().isEmpty());
assertNotNull(result.getRect1());
assertFalse(result.getPoints2().isEmpty());
assertNotNull(result.getRect2());

All the `FeaturesMatchingOptions` builder methods above contain detailed descriptions in their docstrings.

所有的`FeaturesMatchingOptions`开发者措施上面包含详细的描写在这个文档字符串中.

# Ruby
image1 = File.read 'first/image/path.png'
image2 = File.read 'second/image/path.png'

match_result = @driver.match_images_features first_image: image1, second_image: image2
assert_equal %w(points1 rect1 points2 rect2 totalCount count), match_result.keys

match_result_visual = @driver.match_images_features first_image: image1, second_image: image2, visualize: true
assert_equal %w(points1 rect1 points2 rect2 totalCount count visualization), match_result_visual.keys
File.open('match_result_visual.png', 'wb') {
      |f| f<< Base64.decode64(match_result_visual['visualization']) }
assert File.size? 'match_result_visual.png'

Visualization Example

可视化示例

Occurrences Lookup

引用查找

Performs images matching by template to find possible occurrence of the partial image in the full image. Read https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html for more details on this topic. Such comparison is useful in case the full image is a superset of the partial image.

执行图片的匹配通过模块去查找可能出现的部分图像在整个图像中.
阅读：https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.
html有关更多的描述在这个主题里.
这样的比较是有用假如完整图像是相关的部分图像的一个父集.

There is a subtle difference between occurrence comparison and feature comparison. The former is to be used when the image to be found is a subset of the target/screenshot. The latter is to be used when the image to be found is basically the same as the target but rotated and/or scaled.

产状比较和特征比较有细微的区别.
使用前者是当图像查找是目标/屏幕截图的分组时.
使用后者是当查找的图像与目标相同但旋转的并/或缩放.

Examples

// java

byte[] screenshot = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES));
OccurrenceMatchingResult result = driver
        .findImageOccurrence(screenshot, partialImage, new OccurrenceMatchingOptions()
                .withEnabledVisualization());
assertThat(result.getVisualization().length, is(greaterThan(0)));
assertNotNull(result.getRect());

All the OccurrenceMatchingOptions builder methods above contain detailed descriptions in their docstrings.

所有关于`OccurrenceMatchingOptions`开发者的方法在上面包含详细的描述在文档字符串中.

// Typescript / Javascript
  /*
     Typescsript code for occurrence comparison using the template matching algorithm.
     It detects if an image is contained in another image (called the template).
     The image must have the same scale and look the same. However, you can add a scaling transformation beforehand.

     official doc:
     https://github.com/appium/appium/blob/master/docs/en/writing-running-appium/image-comparison.md
     OpenCV algorithm doc:
     https://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html
     official sample code:
     https://github.com/justadudewhohacks/opencv4nodejs/blob/master/examples/templateMatching.js

     You must install opencv4nodejs using the -g option.

     The Javascript client driver webdriverio does not support (in January 2020) the "-image" strategy implemented in the Appium server. You will have more power and understanding while using openCV directly. Since the appium server is in Javascript, you can do all it does with opencv in your test suite.

     The testing framework mocha can be run with typescript to have async/await.
     You need to run mocha with those options in the right order and with the associated packages installed:
     NODE_PATH=/path/to/nodejs/lig/node_modules TS_NODE_PROJECT=config/tsconfig_test.json --require ts-node/register --require tsconfig-paths/register
     You will also need to make a basic config/tsconfig_test.json
     Note that paths in tsconfig.json does not support absolute paths. Hence, you cannot move the NODE_PATH there.
  */
  import * as path from 'path';
  const cv = require(path.join(process.env.NODE_PATH, 'opencv4nodejs'));
  const isImagePresent = async () => {
     
    /// Take screenshot and read the image
    const screenImagePath = './appium_screenshot1.png';
    await driver.saveScreenshot(screenImagePath)
    const likedImagePath = './occurrence1.png';

    // Load images
    const originalMatPromise = cv.imreadAsync(screenImagePath);
    const waldoMatPromise = cv.imreadAsync(likedImagePath);
    const [originalMat, waldoMat] = await Promise.all([originalMatPromise, waldoMatPromise]);

    // Match template (the brightest locations indicate the highest match)
    // In the OpenCV doc, the option 5 refers to the algorithm called CV_TM_CCOEFF_NORMED
    const matched = originalMat.matchTemplate(waldoMat, 5);

    // Use minMaxLoc to locate the highest value (or lower, depending of the type of matching method)
    const minMax = matched.minMaxLoc();
    const {
      maxLoc: {
      x, y } } = minMax;

    // Draw bounding rectangle
    originalMat.drawRectangle(
      new cv.Rect(x, y, waldoMat.cols, waldoMat.rows),
      new cv.Vec(0, 255, 0),
      2,
      cv.LINE_8
    );

    // Open result in new window
    // If the image is too big for your screen, you need to write to a file instead.
    // Check the source of opencv4nodejs for writing an image to a file.
    cv.imshow('We\'ve found Waldo!', originalMat);
    await cv.waitKey();

    // then you know if the image was found by comparing the rectangle with a reference rectangle.
    // the structure minMax contains the property maxVal that gives the quality of the match
    // 1 is prefect match, but you may get .999. If you extract an image from the screenshot manually,
    // you will get an image that matches.
  };

image1 = File.read 'first/image/path.png'
image2 = File.read 'partial/image/path.png'

find_result = @driver.find_image_occurrence full_image: image1, partial_image: image2
assert_equal({
      'rect' => {
      'x' => 0, 'y' => 0, 'width' => 750, 'height' => 1334 } }, find_result)

find_result_visual = @driver.find_image_occurrence full_image: image1, partial_image: image2, visualize: true
assert_equal %w(rect visualization), find_result_visual.keys
File.open('find_result_visual.png', 'wb') {
      |f| f<< Base64.decode64(find_result_visual['visualization']) }
assert File.size? 'find_result_visual.png'

Visualization Example

The highlighted picture at the left bottom corner is the resulting match of

比较突出的图片在左下角

lookup.

Similarity Calculation

相似性计算

Performs images matching to calculate the similarity score between them. The flow there is similar to the one used in findImageOccurrence, but it is mandatory that both images are of equal size. Such comparison is useful in case the original image is a copy of the original one, but with changed content.

执行的图像匹配以计算他们之间的相似性分数.
这里的流程与使用`findImageOccurrence`是相同的，
但是它是强制两张图片的尺寸是相同的.
比较是有用的假如原始图像是原图像的副本，
但是内容改变了.

Examples

// java

byte[] screenshot1 = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES));
byte[] screenshot2 = Base64.encodeBase64(driver.getScreenshotAs(OutputType.BYTES));
SimilarityMatchingResult result = driver
        .getImagesSimilarity(screenshot1, screenshot2, new SimilarityMatchingOptions()
                .withEnabledVisualization());
assertThat(result.getVisualization().length, is(greaterThan(0)));
assertThat(result.getScore(), is(greaterThan(0.0)));

All the SimilarityMatchingOptions builder methods above contain detailed descriptions in their docstrings.

所有的`SimilarityMatchingOptions`开发者的方法包含在上面详细的描述.

# Ruby
image1 = File.read 'first/image/path.png'
image2 = File.read 'second/image/path.png'

get_images_result = @driver.get_images_similarity first_image: image1, second_image: image2
assert_equal({
      'score' => 0.891606867313385 }, get_images_result)

get_images_result_visual = @driver.get_images_similarity first_image: image1, second_image: image2, visualize: true
assert_equal %w(score visualization), get_images_result_visual.keys
File.open('get_images_result_visual.png', 'wb') {
      |f| f<< Base64.decode64(get_images_result_visual['visualization']) }
assert File.size? 'get_images_result_visual.png'

Visualization Example

The similarity score for two pictures above is ~0.98.

两张图片的相似度约为0.98.