Using Android's Compatibility Test Suite

From：http://bitbar.com/blog/44/using-androids-compatibility-test-suite

 

Using Android's Compatibility Test Suite

Submitted by Cezary Statkiewicz on Thu, 05/20/2010 - 18:04

• Android Compatibility Test Suite
• Android CTS
• Google Android

Hi

I'm Cezary Statkiewicz and I'm Senior Software Engineer at Bitbar.com. One of our offerings is Android CTS integration with CI environment, so today I'd like to make a short introduction to CTS usage.

One of the first things that one can see when starting using CTS is a complete lack of documentation. There is no mention in official on-line documentation about it, and if one would google a bit, he'd find only a couple of blog posts and some questions on android mailing lists. This is also one of motivators for me to write about it.

After writing this entry I've found that actually there IS quite good documentation (very broad introduction, compatibility definition document and CTS docs) on http://source.android.com/compatibility/index.html , which probably was released few days ago.

First of all, CTS is not a different story than normal testing in Android, which is based on JUnit and Instrumentation testing. It is just an extension of this process, which automates test execution, test results gathering and aggregation. CTS is made of two main elements: cts cli command and tests, that are mostly unit tests and reference applications from rest of Android platform code. Some tests are provided expilictly by this test suite. What CTS does is grouping test packages (which are actually Android apps) into test plans (there are several test plans pre-created and you can define your own plans), executes those plans on device, gets the results and creates test result file (aggregated from junit test results). It uses adb command internally.

Preparation

Since Android 2.0 CTS is available in public source repository, so anyone can come and play. I assume you are:

• using Android platform from git repository,
• using unix-like host system,
• operating in in directory with android,
• have built platform (make ) and sdk (make sdk ).

If not, see instructions on how to get the code . To build CTS you have to execute:

. build/evnsetup.sh

make cts

This will build CTS and install it in out/host/$host_type/cts/ , where $host_type is your host operating system, i.e. linux-x86 . Under this directory you can find directories and files:

• android-cts/repository/plans/ with test plans defined as xml files,
• android-cts/repository/results/ with test results,
• android-cts/repository/testcases/ with test cases for CTS.
• android-cts/repository/host_config.xml with configuration of CTS.

For convenience add to your $PATH path to out/host/linux-x86/bin/ . To run cts, first you will have to connect a device to your host machine or run emulator with some AVD (see http://developer.android.com/guide/developing/tools/avd.html for more details on creating Android Virtual Device and http://developer.android.com/guide/developing/tools/emulator.html for information on using Android's emulator). Then run adb start-server to be able to connect debugging tools with the device (ADB is used by CTS internally, so if you're interested in more details about it, read http://developer.android.com/guide/developing/tools/adb.html ). We'll be running AVD named "test_case" in console on screen without video nor audio output (since this is all about automated testing, we want to run it on remote, headless machine):

$ screen emulator @test_case -no-window -noaudio &

$ adb start-server

$ cts

Working with CTS

Now you should have working cts environment. To get help, type 'help'. You will get:

cts_host > help

Usage: command options

Avaiable commands and options:

Host:

help: show this message

exit: exit cts command line

Plan:

ls --plan: list available plans

ls --plan plan_name: list contents of the plan with specified name

add --plan plan_name: add a new plan with specified name

add --derivedplan plan_name -s/--session session_id -r/--result result_type: derive a plan from the given session

rm --plan plan_name/all: remove a plan or all plans from repository

start --plan test_plan_name: run a test plan

start --plan test_plan_name -d/--device device_ID: run a test plan using the specified device

start --plan test_plan_name -t/--test test_name: run a specific test

start --plan test_plan_name -p/--package java_package_name: run a specific java package

start --plan test_plan_name -t/--test test_name -d/--device device_ID: run a specific test using the specified device

start --plan test_plan_name -p/--package java_package_name -d/--device device_ID: run a specific java package using the specified device

Package:

ls -p/--package: list available packages

ls -p/--package package_name: list contents of the package with specified name

add -p/--package root: add packages from root to repository

rm -p/--package package_name/all: remove a package or all packages from repository

Result:

ls -r/--result: list all result of sessions

ls -r/--result -s/--session session_id: list detail case result of a specified session

ls -r/--result [pass/fail/notExecuted/timeout] -s/--session session_id: list detail cases of a specified session by the specified result.

History:

history/h: list all commands in command history

history/h count: list the latest count records in command history

history/h -e num: run the command designated by 'num' in command history

Device:

ls -d/--device: list available devices

cts_host >

As you can see, you can list test plans, list test packages (and add/remove those), execute test plans (or just some packages from them) on specified device and see some short information about test results (sessions). One note here: test plans are executed in sessions, which in general means that until test plan is not finished (i.e. interrupted with Ctrl+C), it is considered as an 'open session'. If you would run into a situation when you have many open sessions, just stop cts, remove results from out/host/linux-x86/cts/android-cts/repository/results/ and start cts again.

To see a list of sessions, execute ls -r in cts console:

$ cts

Android CTS version 2.1_r1

Device(emulator-5554) connected

cts_host > ls -r

cts_host > List of all results:

Session Test result Start time End time Test plan name

Pass Fail Timeout NotExecuted

1 5 0 0 0 2010.05.20 10:15:19 2010.05.20 10:18:39 AppSecurity

3 1 0 0 0 2010.05.20 10:51:22 2010.05.20 10:54:44 Signature

5 34 0 0 14728 2010.05.20 10:10:03 2010.05.20 10:09:50 Java

cts_host >

Standard CTS installation has 8 plans available:

cts_host > ls --plan

List of plans (8 in total):

AppSecurity

Java

Android

RefApp

Signature

Performance

VM

CTS

cts_host >

and have over 60 test packages to use:

cts_host > ls --package

Available packages (55 in total):

android.util

android.hardware

android.core.tests.luni.lang

android.media

android.performance2

android.core.tests.luni.util

android.tests.sigtest

android.database

android.example

android.core.tests.nio

android.core.tests.dom

android.core.tests.annotation

android.widget

android.core.tests.archive

android.performance3

android.core.tests.text

android.core.tests.xml

android.apidemos.cts

android.graphics

android.core.tests.crypto

android.view

android.performance5

android.core.tests.sql

android.speech

android.gesture

android.performance

android.core.tests.security

android.performance4

android.tests.appsecurity

android.permission

android.dpi2

android.core.tests.concurrent

android.location

android.core.tests.logging

android.text

android.app

android.core.tests.regex

android.bluetooth

android.core.tests.math

android.provider

android.webkit

android.content

android.core.tests.nio_char

android.dpi

android.core.tests.xnet

android.net

android.accounts

android.permission2

android.os

android.core.tests.prefs

android.core.tests.luni.io

android.telephony

android.core.tests.luni.net

android.jni

android.core.vm-tests

cts_host >

Note that some packages used in CTS are not visible in this list. Total count of test cases is over 23000, so full run of all tests takes several hours to complete.

You can run test plan by running cts shell, or non-interactively:

$ cts start --plan $PLAN_NAME

To be more exact:

$ cts start --plan Signature

Android CTS version 2.1_r1

Device(emulator-5554) connected

cts_host > start test plan Signature

CTS_INFO >>> Checking API...

CTS_INFO >>> This might take several minutes, please be patient...

API Check PASS.

(pass)

Test summary: pass=1 fail=0 timeOut=0 notExecuted=0 Total=1

Time: 202.363s

$

This will produce results in out/host/linux-x86/cts/android-cts/repository/results/$session_date , where $session_date can be 2010.05.20_10.42.58 . Contents of this directory:

cts_result.css

cts_result.xsl

logo.gif

newrule-green.png

testResult.xml

Most importaint file is testResult.xml. It contains information on host machine, test device, test plan and tests execution status:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="cts_result.xsl"?>

<TestResult endtime="Thu May 20 10:46:21 CDT 2010" starttime="Thu May 20 10:42:58 CDT 2010" testPlan="Signature" version="1.2">

<DeviceInfo>

<Screen resolution="480x320"/>

<PhoneSubInfo subscriberId="15555218135"/>

<BuildInfo Xdpi="160.0" Ydpi="160.0" androidPlatformVersion="7" buildID="ECLAIR" buildName="generic" buildVersion="2.1-update1" build_board="unknown" build_brand="generic" build_device="generic" build_fingerprint="generic/generic/generic/:2.1-update1/ECLAIR/eng.shaker.20100519.082949:eng/test-keys" build_model="generic" build_type="eng" deviceID="emulator-5554" imei="000000000000000" imsi="310260000000000" keypad="qwerty" locales="en_US" navigation="trackball" network="Android" touch="finger"/>

</DeviceInfo>

<HostInfo name="android.shaker.bitbar.com">

<Os arch="amd64" name="Linux" version="2.6.26-2-amd64"/>

<Java name="Sun Microsystems Inc." version="1.5.0_17"/>

<Cts version="2.1_r1">

<IntValue name="maxTestCount" value="600"/>

<IntValue name="maxTestsInBatchMode" value="100"/>

<IntValue name="testStatusTimeoutMs" value="300000"/>

<IntValue name="batchStartTimeoutMs" value="1800000"/>

<IntValue name="individualStartTimeoutMs" value="300000"/>

<IntValue name="signatureTestTimeoutMs" value="600000"/>

<IntValue name="packageInstallTimeoutMs" value="120000"/>

<IntValue name="postInstallWaitMs" value="10000"/>

</Cts>

</HostInfo>

<Summary failed="0" notExecuted="0" pass="1" timeout="0"/>

<TestPackage appPackageName="android.tests.sigtest" digest="" name="SignatureTest" signatureCheck="true">

<TestSuite name="android">

<TestSuite name="tests">

<TestSuite name="sigtest">

<TestCase name="SignatureTest" priority="">

<Test endtime="Thu May 20 10:46:21 CDT 2010" name="signatureTest" result="pass" starttime="Wed Dec 31 18:00:00 CST 1969"/>

</TestCase>

</TestSuite>

</TestSuite>

</TestSuite>

</TestPackage>

</TestResult>

This file is viewable under some web browsers (Firefox is recommended) - it will use appended css and display transfored XML. Note, that those files also available in out/host/linux-x86/cts/android-cts/repository/results/2010.05.20_10.42.58.zip .

There are some notes that should be made about CTS:

• CTS can be run only in one instance on one machine. Executing many instances will result in CTS_ERROR >>> Error: CTS is being used at the moment. No more than one CTS instance is allowed simultaneously message.
• Tests are executed in context of device, so their execution speed depends on device.
• Test results are aggregated from JUnit data, and during conversion from JUnit there is some information loss.