Android Compatibility Test Suite

Android Compatibility Test Suite

因为工作的需要，了解了一下Android Compatibility Test Suite(CTS),将其概要和架构做一简单总结。部分内容从PPT中摘录出来，涉及CTS实现的内容有所增加。

What is CTS?

The CTS is a tool used by device manufacturers to help ensure their devices are compatible, and to report test results for validations. The CTS is intended to be run frequently by OEMs throughout the engineering process to catch compatibility issues early.

The CTS is a free, commercial-grade test suite, available for download. The CTS represents the "mechanism" of compatibility.

The CTS is an automated testing harness that includes two major software components: the CTS test harness runs on the desktop machine and individual test cases pushed and executed on target device.

Why be compatible?

Give your users the best possible experience with the applications they run.

Make it easy for developers to write top-quality applications for your device.

Take advantage of the Android Market.

How to become compatible?

The definition enumerates the software and hardware features in a compatible Android device.

CTS is a downloadable open-source testing harness that can be used in any way you like as you develop your handset.

Submit CTS-generated report to [email protected] to claim compatibility for the device.

How the CTS works?

The CTS test harness runs on the desktop machine and manages test execution.

The test harness pushes the test case *.apk file to attached device, executes the test through instrumentation, and records the test results.

Give tests result in .xml file once all tests are executed.

How to get CTS

A. Download the latest CTS version via the linkhttp://source.android.com/compatibility/downloads.html.  The page contents(Android 4.0.3 for example)

 

Android 4.0.3 is the release of the development milestone code-named Ice Cream Sandwich. Android 4.0.3 is the current version of Android. Source code for Android 4.0.3 is found in the 'android-4.0.3_r1' branch in the open-source tree.

    Android 4.0 Compatibility Definition Document (CDD)

    Android 4.0.3 R3 Compatibility Test Suite (CTS)

    Android 4.0.3 R2 CTS Verifier (CTS verifier is intended to be run by a human operator to test functionality that cannot be tested by an automated system, such as correct functioning of a camera and sensors).

 

B. Alternatively, compile the CTS code yourself.

The CTS source code in the dir $ANDROID/cts/;

$ ./envsetup.sh

$ make cts

The build output is in $ANDROID/out/host/linux-x86/cts/, the dir tree as follows.

$ tree

.

├── docs

├── repository

│   ├── logs

│   │  └── 2012.08.17_09.13.09

│   │      ├── device_logcat_645902302998382377.zip

│   │      └── host_log_436286063990992724.zip

│   ├── plans

│   │  ├── Android.xml

│   │  ├── AppSecurity.xml

│   │  ├── CTS-TF.xml

│   │  ├── CTS.xml

│   │  ├── Java.xml

│   │  ├── RefApp.xml

│   │  ├── Signature.xml

│   │  ├── VM-TF.xml

│   │  └── VoiceDialerTestPlan.xml

│   ├── results

│   │  ├── 2012.08.17_09.13.09

│   │  │  ├── cts_result.css

│   │  │  ├── cts_result.xsl

│   │  │  ├── logo.gif

│   │  │  ├── newrule-green.png

│   │  │  └── testResult.xml

│   │  └── 2012.08.17_09.13.09.zip

│   └── testcases

│       ├── android.core.tests.libcore.package.com.apk

│       ├── android.core.tests.libcore.package.com.xml

│       ├── android.core.tests.libcore.package.dalvik.apk

│       ├── android.core.tests.libcore.package.dalvik.xml

………………

│       ├── android.core.tests.runner.apk

│       ├── android.core.vm-tests.jar

│       ├── android.core.vm-tests-tf.jar

│       ├── android.core.vm-tests-tf.xml

│       ├── android.core.vm-tests.xml

│       ├── android.tests.appsecurity.xml

│       ├── ApiDemos.apk

│       ├── ApiDemosReferenceTest.apk

│       ├── ApiDemosReferenceTest.xml

│       ├── BrowserTestCases.apk

│       ├── BrowserTestCases.xml

…….

│       ├── ExpriAnimationTestCases.apk

│       ├── ExpriAnimationTestCases.xml

│       ├── SignatureTest.apk

│       ├── SignatureTest.xml

│       ├── TestDeviceSetup.apk

│       ├── VoiceDialerTestCases.apk

│       └── VoiceDialerTest.xml

└── tools

    ├── cts-tradefed

    ├── cts-tradefed.jar

    ├── ddmlib-prebuilt.jar

    ├── hosttestlib.jar

    ├── junit.jar

    ├── README

    └── tradefed-prebuilt.jar

 

Note:

The CTS tool can be copied to and runs in any other directory.

Set up the target device

•       Your phone should be running a user build (Android 4.0 and later)

•       Ensure the Text To Speech files are installed on the device(via settings).

•       Make sure the device has a SD card plugged in and the card is empty(sdcard may be modified/erased).

•       Do a factory data reset on the device.

•       Make sure no lock pattern is set on the device, the "USB Debugging" & Stay Awake & Allow mock locations are checked.

•       Make sure device is connected to a functioning Wi-Fi network.

•       Make sure the device is at the home screen at the start of CTS, must not be used for any other tasks, do not press any keys on the device while CTS is running.

•       Set up accessibility tests and device administration tests, enable the corresponding check settings(refer to CTS UM for detail).

       adb install ./repository/testcases/CtsDelegatingAccessibilityService.apk

       adb install ./repository/testcases/CtsDeviceAdmin.apk

CTS commands

$ ./android-cts/tools/cts-tradefed

cts-tf> will be displayed, and the connected device’s information will be poped.

cts-cf> help to list all commands

The mainly used CTS command including list series and run cts series.

Running the CTS

 cts-cf> run cts --plan CTS for CTS plan, or

 cts-cf>run cts --plan <plan_name> for the selected plan.

The test progress will scroll on the PC screen and the phone device is operated by the CTS.

…It will take several hours to finish the CTS plan’s all test cases……

When finished,

1) The test-results are placed in the file: $CTS_ROOT/repository/results/<start time>.zip/testResult.xml

2) The logs(including the host log and device logcat) are under the dir $CTS_ROOT/repository/logs/

CTS test plans

•       Signature – check the API signatures described in the Android release XML files against the APIs.

•       RefApp – test reference application hehaviour.

•       AppSecurity – support the Android permissions model as defined in the Android developer documentation [Resources, 54]

•       VM-tf – The tests focus on testing the Dalvik VM.

•       Java – test the core java libraries.

•       Android – test most of the system functions.

•       CTS – including all of above test plans.

Note:

CTS vs. CTS-TF. There shouldn't be any difference between CTS and CTS-TF plans. The test runner will actually run CTS-TF if you enter CTS.

Android Test plan packages source codes are under the directory $ANDROID/cts/tests/tests/

For test plan details, view the test plan file $CTS_ROOT/repository/plans/$PLAN_NAME.xml

Test result summery

1) Device Information

Build info, Test result summery, Phone info, Features supported, Root Processes, Partitions, System Libraries.

2) Test Summary by Package Table

Test Package | Passed | Failed | Timed Out | Not Executed | Total Tests

3) Test Failures

4) Detailed Test Report.

 Table Format same as above.

Analyze test Failure 1

1) android.app.cts.DownloadManagerTest – testDownloadManager

junit.framework.AssertionFailedError: Make sure you have WiFi or some other connectivity for this test. At android.app.cts.DownloadManagerTest$DownloadCompleteReceiver.waitForDownloadComplete(DownloadManagerTest.java:247)

cause: WiFi is not enabled, no other data link.

2) libcore.java.net.InetAddressTest-- test_isReachable

java.net.UnknownHostException: Unable to resolve host "www.google.com": No address associated with hostname at java.net.InetAddress.lookupHostByName(InetAddress.java:467)

 cause: WiFi is not enabled, no other data link.

3) android.telephony.cts.SmsManagerTest – testSendMessages

java.lang.IllegalArgumentException: Invalid destinationAddress at android.telephony.SmsManager.sendTextMessage(SmsManager.java:83)

                public void sendTextMessage(String destinationAddress, String scAddress, String text, ………) {

                        if (TextUtils.isEmpty(destinationAddress)) {

                       throw new IllegalArgumentException("Invalid destinationAddress");               //line 83

                        }

                        ……

                In SmsManagerTest.java

                protected void setUp() throws Exception {

                        mDestAddr = mTelephonyManager.getLine1Number();

                        ……

                public void testSendMessages() throws InterruptedException {

                        sendTextMessage(mDestAddr, mDestAddr, mSentIntent, mDeliveredIntent);

                        ……

                protected void sendTextMessage(String destAddr, String text, PendingIntent sentIntent, PendingIntent deliveredIntent) {

                        getSmsManager().sendTextMessage(destAddr, null, text, sentIntent, deliveredIntent);

                 }

                cause: No SIM.

Add new test case to Android Test plan

1. test cases source code dir

$MYDROID/cts/tests/tests/example

2. Android.mk content

LOCAL_PATH:= $(call my-dir)

 

include $(CLEAR_VARS)

 

# Replace "Example" with your name.

LOCAL_PACKAGE_NAME := CtsExampleTestCases           ----- Android package name

 

# Don't include this package in any target.

LOCAL_MODULE_TAGS := optional

 

# When built, explicitly put it in the data partition.

LOCAL_MODULE_PATH := $(TARGET_OUT_DATA_APPS)

 

# All tests should include android.test.runner.

LOCAL_JAVA_LIBRARIES := android.test.runner

 

LOCAL_SRC_FILES := $(call all-java-files-under, src)

 

LOCAL_SDK_VERSION := current

 

include $(BUILD_PACKAGE)

 

3. AndroidManifest.xml content

<manifest xmlns:android="http://schemas.android.com/apk/res/android"

    package="com.android.cts.example">                              ------ Java package namespace

    <application>

<uses-library android:name="android.test.runner" />

    </application>

    <instrumentation android:name="android.test.InstrumentationTestRunner"

                     android:targetPackage="com.android.cts.example“     ------ Target package load and run by testrunner.

                     android:label="CTS tests of example component"/>

 

</manifest>

The java namespace package and the android:targetPackage通常是相同的，但是也有不同的情况。不同的时候，后者通常是被testrunner加载和运行的测试用例实现，而前者可能是定制的testrunner。需要注意的是，这两个要版本相同，否则在push apk时，会出现找不到targetPackage的错误。因为两个都会push到手机，所以targetPackage一般不能是Browser.apk、Phone.apk等，因为这会替换手机上的同名包，测试的将不再是手机的包，而是新push上去的包。

加上testcase的PackageFileName.apk，一共有三个package标识:

PackageFileName

namespace package

android:targetPackage

 

4. Add test package to cts/CtsTestCaseList.mk

                CTS_COVERAGE_TEST_CASE_LIST := \

                                CtsAccelerationTestCases \

                                ……

                                CtsExampleTestCases \

                                ………

 

                CTS_COVERAGE_TEST_CASE_LIST will be included in CTS_TEST_CASE_LIST, and included in CTS_CASE_LIST in build/core/tasks/cts.mk instead.

5. Re-compile cts

buildCts.py will add the test cases to Android test plan, and can find the entry in $CTS_OUT/repository/plans/Android.xml

                <?xml version="1.0" encoding="UTF-8"?>

                <TestPlan version="1.0">

                                <Entry uri="android.drm"/>

                                <Entry uri="android.example"/>                              ----- url format buildCts.py makes.

                                <Entry uri="android.gesture"/>

                </TestPlan>

 

CtsExampleTestCases.apk and its  description file CtsExampleTestCases.xml will be found in $CTS_OUT/repository/testcases/

<?xml version="1.0" encoding="UTF-8"?>

<TestPackage AndroidFramework="Android 1.0" appNameSpace="com.android.cts.example" appPackageName="android.example" name="CtsExampleTestCases" runner="android.test.InstrumentationTestRunner" version="1.0">   

        <TestSuite name="android">       

                <TestSuite name="example">           

                        <TestSuite name="cts">               

                                <TestCase name="ExampleSecondaryTest">                   

                                        <Test name="testZorch">                       

                                                <Description>              Test {@link Example#zorch}.        </Description>                       

                                        </Test>                    

                                </TestCase>               

                                <TestCase name="ExampleTest">                   

                                        <Test name="testBlort">                       

                                                <Description>               Test {@link Example#blort}.        </Description>                       

                                        </Test>                   

                                </TestCase>               

                        </TestSuite>           

                </TestSuite>       

        </TestSuite>   

</TestPackage>

 

Add a test plan ExampleTestPlan and test packageExampleTestCases in CTS.

1. Add test package source code to cts folder

                $MYDROID/cts/tests/ExampleTest

2. Verify Android.mk & AndroidManifest.xml

                Same to add test cases, but make sure don’t use the duplicated package name!!

3. Add item in CtsTestCaseList.mk

This will control the compile of the test cases

4. Add test plan ‘ExampleTestPlan' in cts/tools/utils/buildCts.py

   A. define the GenerateExampleCheckDescription function

def GenerateExampleCheckDescription(test_repository):

  """Generate the test description for the example check."""

  LogGenerateDescription('android.tests.exampletest')

  package = tools.TestPackage('ExampleTestCases', 'android.tests.exampletest')

  package.AddAttribute('appNameSpace', 'android.tests.exampletest')

  package.AddAttribute ('targetNameSpace', 'com.android.example')

  package.AddAttribute ('targetBinaryName', 'Example')

  package.AddAttribute(‘exampleCheck', 'true')

  package.AddAttribute('runner', 'android.tests.exampletest.ExampleTest')

  package.AddTest('android.tests.exampletest.ExampleTestName')

  description = open(os.path.join(test_repository, ‘ExampleTest.xml'), 'w')

  package.WriteDescription(description)

  description.close()

 

  B. and called it in GenerateTestDescriptions()

  def GenerateTestDescriptions(self):

    """Generate test descriptions for all packages."""

    pool = Pool(processes=16)

 

    # individually generate descriptions not following conventions

    pool.apply_async(GenerateSignatureCheckDescription, [self.test_repository])

    pool.apply_async(GenerateReferenceAppDescription, [self.test_repository])

    pool.apply_async(GenerateAppSecurityDescription, [self.temp_dir,

        self.test_repository, self.android_root, self.doclet_path])

 

    # generate test descriptions for android tests

    android_packages = GetSubDirectories(self.test_root)

    for package in android_packages:

      pool.apply_async(GenerateTestDescription, [self.test_root, self.temp_dir,

          self.test_repository, self.android_root, self.doclet_path, package])

 

    # generate test description for example tests

    pool.apply_async(GenerateExampleCheckDescription, [self.test_repository])

 

    pool.close()

    pool.join()

 

  C. Modify GenerateTestPlans to organize the plans

                def GenerateTestPlans(self):

                  """Generate default test plans."""

                # TODO: Instead of hard-coding the plans here, use a configuration file,

                 # such as test_defs.xml

                 packages = []

                descriptions = sorted(glob.glob(os.path.join(self.test_repository, '*.xml')))

                for description in descriptions:

                  doc = tools.XmlFile(description)

                    packages.append(doc.GetAttr('TestPackage', 'appPackageName'))

 

                plan = tools.TestPlan(packages)

                plan.Exclude('android\.core\.vm-tests')

                plan.Exclude('android\.performance.*')

                plan.Include(r'android\.core\.vm-tests-tf')

                self.__WritePlan(plan, 'CTS-TF')

                self.__WritePlan(plan, 'CTS')

 

                plan.Exclude(r'android\.tests\.sigtest')

                plan.Exclude(r'android\.core.*')

                plan.Exclude(r'android\.tests\.exampletest')     // exclude the test cases from Android Test Plan

                self.__WritePlan(plan, 'Android')

 

                plan = tools.TestPlan(packages)

                plan.Include(r'android\.core\.tests.*')

                self.__WritePlan(plan, 'Java')

 

                plan = tools.TestPlan(packages)

                plan.Include(r'android\.core\.vm-tests-tf')

                self.__WritePlan(plan, 'VM-TF')

 

                plan = tools.TestPlan(packages)

                plan.Include(r'android\.tests\.sigtest')

                self.__WritePlan(plan, 'Signature')

 

                plan = tools.TestPlan(packages)

                plan.Include(r'android\.apidemos\.cts')

                self.__WritePlan(plan, 'RefApp')

 

                plan = tools.TestPlan(packages)

                plan.Include(r'android\.tests\.appsecurity')

                self.__WritePlan(plan, 'AppSecurity')

 

                plan = tools.TestPlan(packages)

                plan.Include(r'android\.tests\.exampletest')

                self.__WritePlan(plan, 'ExampleTestPlan')

               

从buildCts的代码可以看出，Test Plan是buildCts对编译出来的Test cases进行组织形成TestPlan；虽然逻辑上TestPlan具有包含关系，如CTS包含Android、sigtest、appsecurity、RefApp、vm-tf等Test Plan，但在buildCts代码里面，它们是同level的组织关系。

 

   D. buildCts.py main entry

if __name__ == '__main__':

                builder = CtsBuilder(sys.argv)

                builder.GenerateTestDescriptions()

                builder.GenerateTestPlans()

 

  E. Compile cts, and get

out/host/linux-x86/cts/android-cts/repository/plans/ExampleTestPlan.xml , and

Exampletestcases.apk and Exampletestcases.xml in cts/android-cts/repository/testcases/

 

CTS and JUnit

CTS is an upper wrapper build on JUnit, so the test cases uses the Junit test cases rules, using android testrunner and Instrument.

The test package can use adb shell am to push and run.