android CTS 兼容测试
CTS介绍-----------------------------------------------3
什么是CTS------------------------------------------3
测试环境搭建-------------------------------------------3
测试环境--------------------------------------------3
测试前准备------------------------------------------3
测试手机配置----------------------------------------4
执行测试------------------------------------------------4
整体测试6.0以下------------------------------------4
测试相关的指令--------------------------------------5
主要针对6.0及其以下版本测试使用------------------7
针对Android7.0以及以后版本测试------------------8
android 8.1CTS 测试
测试结果输出------------------------------------------10
FAQ---------------------------------------------------10
CTS介绍
什么是CTS
CTS简介：Compatibility Test suite系列兼容测试
google定义了一个兼容性规范(Compatible Definition),而CTS就是为了确保某个测试符合该规范。从而基于Android的应用程序能在基于同一个api版本的设备上面运行。通过CTS测试的device可以获得android的商标，并且享受android market的权限
一般CTS测试有三项：Cts,Gts,Ctsverify，CTS测试工具可以在官方网上下载（http://source.android.com/compatibility/downloads.html），选择对应Android版本的CDD文档（介绍CTS测试），CTS测试包下载（测试包会不断更新），将下载的CTS测试包解压到相应文件夹里面应包含：android-cts文件夹里面包含Docs、Repository、Tools三个文件夹，
Android6.0的测试包和Android7.0的CTS测试包已经有些不同，而Media Files最新为CTS Media 1.2.，
测试环境搭建
测试环境
· linux系统（最好是64位ubuntu10.04）
· 需正确配置JavaJDK（1.7版本以上）和Android SDK 和ADB，确保系统和手机能够正常的连接
下載java jdk 1.8 地址 :
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
下載android sdk
android官网下载sdk,注意现在官网不推荐单独安装sdk，都是和Android Studio绑定在一起，单独的sdk比较难找，放在网页的最下面。
下載完成後 ,需要在下載相應的api
配置环境变量 ,這裏配置全局的环境变量
$sudo vim /etc/profile
#set java environment
JAVA_HOME=/usr/java/jdk1.8.0_181
JRE_HOME=/usr/java/jdk1.8.0_181/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export ANDROID_HOME=/home/ctstest/androidsdk/android-sdk-linux
export PATH=$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$PATH
更新下环境变量
source /etc/profile
查看配置是否正確
java
adb devices
测试前准备
· 下载需要进行测试的软件版本 -------用戶版本
· 连接IPV6网络
· 插入可用的SD卡，容量最好大于1G
· 插入可用的Sim卡，如果为双卡版本，需确保卡一的卡槽有可用的Sim卡.
· 写入合法的IMEI号，需要写入号码 ----- 卡要寫號碼
· 检查手机是否设置了CTS标志位 -------- cts標示位置
测试手机配置
· 手机恢复出厂设置（Settings→Backup andreset→Factory data reset）
· 确保SD中有足够的空间（大于1G），可将SD格式化清空。
· 确保手机里没有Google账号或者其他账号存在
· 确保日期和时间的正确(开了Wifi或者数据连接以后会自动更新时间，时区选默认的)，设置太平洋时间gtm：-8
· 设置屏幕为常亮(Settings→Display→Sleep→Never)
· 设置无锁屏(Settings→Security→Set up screenlock→None)
· 手机恢复出厂设置（Settings→Backup andreset→Factory data reset）
· 2.1_R2到4.2_R4之前，安装CtsDelegatingAccessibilityService.apk,
(在android-cts/repository/testcases下，可用adb install安装)，并进入Settings→Accessibility开启Delegating Accessibility Service.
·
· 安装CtsDeviceAdmin.apk,(在android-cts/repository/testcases下可用adbinstall CtsDeviceAdmin.apk),并进入Settings→Security→Deviceadministrators开启里面的两个包
·
· 将媒体文件(android-cts-media-1.2.zip解压出来的文件),名称为test的文件，拷贝到手机外部SD卡上，可使用脚本拷贝(在android-cts-media-1.0目录下执行bash copy_media.shall)
·
· 确保开发者选项中的USB debugging,Stay awake,Allowmock locations三个选项已勾选。
· 设置系统语言为美式英语(Settings→Language&input→Language)
· 设置系统输入法为谷歌默认输入法
(Settings→Language&input→Default→English(US))
· 打开Wifi，打开数据连接，打开GPS
· 确保USB连接稳定
执行测试
整体测试
测试之前手机需要安装CtsDeviceAdmin.apk在终端中输入：
cts/android-cts/repository/testcases$adbinstall CtsDeviceAdmin.apk
在Settings-Security-Deviceadministrators下勾选
Android Devices Manage
android.deviceadmin.cts.CtsDevicesAdminReceiver
android.deviceadmin.cts.CtsDevicesAdminReceiver2
l 用adb devices查看手机是否连上
n 进入测试环境
l 进入CTS测试
测试相关的指令
xxx@CTS-ceshi:~/cts/6.0_r14/android-cts$cd tools/
xxx@CTS-ceshi:~/cts/6.0_r14/android-cts/tools$./cts-tradefed
Android CTS 6.0_r14 build:3623550
03-17 16:46:58 I/DeviceManager: Detectednew device c23392e
l cts-tf > runcts --plan CTS --disable-reboot //开始执CTS整个测试包
l cts-tf> help 使用帮助文档
Run:
runcts --plan test_plan_name: run a test plan
runcts --package/-p : run a CTS test package
runcts --class/-c [--method/-m] : run a specific test class and/ormethod
run cts --continue-session session_ID: run allnot executed tests from a previous CTS session
runcts [options] --serial/-s device_ID: run CTS on specified device
runcts [options] --shards number_of_shards: shard a CTS run into given number ofindependent chunks, to run on multiple devices inparallel
runcts --help/--help-all: get more help on running CTS
List:
l/list d/devices: list connected devices and their state
l/list packages: list CTS test packages
l/list p/plans: list CTS test plans
l/list i/invocations: list invocations aka CTStest runs currentlyin progress
l/list c/commands: list commands: aka CTS test run commands currently inthe queue waiting to be allocated devices
l/list r/results: list CTS results currently present in the repository
Add:
addderivedplan --plan plane_name --session/-s session_id -r[pass/fail/notExecuted/timeout]: derive a plan from the given session
Dump:
d/dump l/logs: dump the tradefed logs for all running invocations
Options:
--disable-reboot: Do not reboot device after running some amount of tests.
l cts-tf > l r 查看失败项目 成功项目 ，未执行项目
Session Pass Fail Not Executed Start time Plan name Device serial(s)
0 2743 130 3555 2017.01.18_16.46.14 CTS1 24ade972
l cts-tf> run cts --continue-session 0 --disable-reboot
继续执行未执行的项目
cts-tf > l r
Session Pass Fail Not Executed Start time Plan name Device serial(s)
0 2743 130 3555 2017.01.18_16.46.14 CTS1 24ade972
l cts-tf> add derivedplan --plan testname -s 0 -r fail
l cts-tf> run cts --plan testname 执行失败项目
l 多台机器同时执行
Android6.0测试需要IPV6环境，可以用命令run cts –plan CTS –skip-preconditions
Android6.0测试需要的时间比较长，google提供了多台手机一起测试的功能：
r un cts –plan CTS –shards 手机数量 –skip-preconditions
比如我需要3台机器一起测试就可以输入命令：run cts –plan CTS –shards 3–skip-preconditions
另外：在终端中输入help可以参看cts的相关命令
run cts –plan test_plan_name 测试一个plan
run cts –packages/-p packages_name 单独测试一个package
run cts –class/-c class_name–method/-m method_name 测试一个类和方法
如：run cts -candroid.hardware.camera2.cts.BurstCaptureTest -m testYuvBurst
l i 命令查看运行时间 cts-tf > l i
l r 查看测试结果 cts-tf > l r
主要针对6.0及其以下版本测试使用
CTS v1 console command reference
Table1. This table summarizes the CTS v1 console commands for varioususes.
| Host |
Description |
| help |
Display a summary of the most commonly used commands |
| help all |
Display the complete list of available commands |
| exit |
Gracefully exit the CTS console. Console will close when all currently running tests are finished |
| Run |
Description |
| run cts |
Run the specified tests and displays progress information. One of --plan, --package, --class or --continue-session needs to be specified The CTS console can accept other commands while tests are in progress If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests If more than one device is connected, the CTS host will choose a device automatically |
| --plan |
Run the specified test plan |
| --package/-p |
Run the specified test packages |
| --class/-c |
Run the specified test class and/or method |
| --continue-session |
Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results |
| --shards |
Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel |
| --serial/-s |
Run CTS on the specific device |
| -t |
Run a specific test method |
| --force-abi 32|64 |
On 64-bit devices, run the test against only the 32-bit or 64-bit ABI |
| List |
Description |
| list packages |
List all available test packages in the repository |
| list plans |
List all available test plans in the repository |
| list invocations |
List 'run' commands currently being executed on devices |
| list commands |
List all 'run' commands currently in the queue waiting to be assigned to devices |
| list results |
List CTS results currently stored in repository |
| list devices |
List currently connected devices and their state 'Available' devices are functioning, idle devices, available for running tests 'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests 'Allocated' devices are devices currently running tests |
| Add |
Description |
| add derivedplan --plan |
Create a plan derived from given result session; use this option to rerun reports and validate test issues |
针对Android7.0 以及以后版本测试
For Android 7.0 or later, you'll use CTS v2.
CTS v2 console command reference
Table1. This table summarizes the CTS V2 console commands for varioususes.
| Host |
Description |
| help |
Display a summary of the most commonly used commands |
| help all |
Display the complete list of available commands |
| version |
Show the version. |
| exit |
Gracefully exit the CTS console. Console will close when all currently running tests are finished. |
| Run |
Description |
| run cts |
Run the default CTS plan (that is, the full CTS invocation). The CTS console can accept other commands while tests are in progress. If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests. If more than one device is connected, the CTS host will choose a device automatically. |
| --plan |
Run the specified test plan. |
| --module/-m |
Run the specified test module or modules. For example, run cts --module CtsGestureTestCases executes the gesture test module (this can be shortened to run cts -m Gesture). |
| --module/-m |
Run the specified module and test. For example, run cts -m Gesture --test android.gesture.cts. GestureTest#testGetStrokes runs the specific package, class, or test. |
| --retry |
Retry all tests that failed or were not executed from the previous sessions. Use list results to get the session id. |
| --shards |
Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel. |
| --serial/-s |
Run CTS on the specific device. |
| --include-filter |
Run only with the specified modules. |
| --exclude-filter |
Exclude the specified modules from the run. |
| --log-level-display/-l |
Run with the minimum specified log level displayed to STDOUT. Valid values: [VERBOSE, DEBUG, INFO, WARN, ERROR, ASSERT]. |
| --abi |
Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports. |
| --logcat, --bugreport, and --screenshoot-on-failure |
Give more visibility into failures and can help with diagnostics. |
| --device-token |
Specifies a given device has the given token eg. --device-token 1a2b3c4d:sim-card.. |
| --skip-device-info |
Skips collection of information about the device. Note: do not use this option when running CTS for approval. |
| --skip-preconditions |
Bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection. |
| List |
Description |
| list modules |
List all available test modules in the repository. |
| list plansor list configs |
List all available test plans (configs) in the repository. |
| list invocations |
List 'run' commands currently being executed on devices. |
| list commands |
List all 'run' commands currently in the queue waiting to be assigned to devices. |
| list results |
List CTS results currently stored in repository. |
| list devices |
List currently connected devices and their state. 'Available' devices are functioning, idle devices, available for running tests. 'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests. 'Allocated' devices are devices currently running tests. |
| Dump |
Description |
| dump logs |
Dump the tradefed logs for all running invocations. |
| ADD |
Description ps: |
| a/add s/subplan |
Create a subplan used to create a subplan |
| --session |
The session used to create a subplan |
| --result-type |
Which result to include in the subplan |
| --name/-n |
The name of the new subplan |
| Cts-tf> run cts --retry sessionId --retry-type FAILED/NOT_EXECUTED 执行未完成的任务 ，输出结果
|
|
测试模块下的单个模块
测试模块
比如我们要测试 CtsMediaTestCases ，我们知道这个模块下面有连个模块 ：
arm64-v8a CtsMediaTestCases
armeabi-v7a CtsMediaTestCases
如果要整体这是这个模块 ， 可以使用
cts-tf > run cts -m CtsMediaTestCases
要是测试模块下的其中一个小模块 ，我们就需要添加一个
| --abi |
Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports. |
cts-tf > run cts -m CtsMediaTestCases -a armeabi-v7a
android 8.1 CTS 测试
- 首先当然还是要下载测试套件 https://source.android.com/compatibility/cts/downloads Android 8.1 is the release of the development milestone code-named Oreo-MR1.The source code for the following tests can be synced with the'android-cts-8.1_r5' tag in the open-source tree.
- 在liunux上执行 ，就下载 Android8.1 R5 Compatibility Test Suite (CTS) - ARM ，如果要测试cts verifier 就顺便把Android8.1 R5 CTS Verifier - ARM下载下来
- 当然还是少不了CTS Media Files 的 CTS Media 1.4 （有4G大小了）
- 后面的手机配置基本上都是大同小异 ，这里就不在重复
- 这里按照CTS v2 console command reference 相关命令来执行
- 执行命令 run cts 默认是执行 Run the default CTS plan (that is, the full CTS invocation).
- run cts
--shards: 表示在几个设备上执行 run cts --shards 2 ,代表在2个设备上执行 - 最后执行完成后就会给你有提示 ，坐等测试结果就好了 （记得不要打开整个测试的结果 ，有时间会很大，导致浏览器卡死，所以还是打开失败的html）
测试结果输出
l 测试结束后 /repository/results/2016.05.27_21.29.17/testResult.xml使用浏览器打开，查看测试的结果,一般都只是查看错误的结果 ,整个结果都查看的话 ,太卡了, 容易浏览器卡死 ,所以一般都是查看的失败的结果
可以进入android-cts/repository/results/2016.05.27_21.29.17输入命令： sudo gedit testResult.xml 在这里必须使用命令打开，否则不能保存，可以看到测试结果：
最近因为测试 的需要,接触到了Android cts测试 ,以上是最近整理的一点东西 ,如果有不对的地方,也请看到的大神指点一二,谢谢 !
在CTS测试的过程中 ，有时间会出现download cts media的情况 ，这个时间 需要把下载的media文件存放到/tmp下 ，但是文件夹一定要要是 android-cts-media/android-cts-media-1.4这样存放的方式
补充:
VTS 测试环境搭建
首先需要搭建测试环境，我们需要以下这些组件：
+ 64-bit Ubuntu Linux
+ Java 8
+ Python 2.7
+ ADB 1.0.39
具体的搭建步骤是：
1） 安装 Python 开发包
$ sudo apt-get install python-dev2） 安装 Protocol Buffer 工具
-
$ sudo apt-get install python-protobuf -
$ sudo apt-get install protobuf-compiler
3） 安装 Python 虚拟环境相关工具
-
$ sudo apt-get install python-virtualenv -
$ sudo apt-get install python-pip
4） 在设备上启用开发者模式并打开 USB 调试功能
5） 检查设备是否能被 ADB 探测到
$ adb devices
编译完成后，我们可以在out/host/linux-x86/vts/android-vts.zip目录下找到 VTS 测试包，解压之后，进入android-vts/tools/目录，执行以下命令即可进行默认的全局 VTS 测试：
-
$ vts-tradefed -
> run vts
也可以只对某个模块进行测试：
-
$ vts-tradefed -
> run vts -m VtsHalAudioV2_0Target
还可以只对某个模块中的某一项用例进行测试：
-
$ vts-tradefed -
> run vts -m VtsHalAudioV2_0Target -t RecommendedOutputStreamConfigSupport
剩下的就是耐心等待。测试完成后我们可以在android-vts/results/目录下找到测试报告，可以在android-vts/logs/目录下看到测试日志。
VTS测试的设备 的设置 :-------刷新设备
- Flashing device
For fastboot follow the steps below
-
- Enable OEM Unlocking in Developer settings
- Disable the verified boot
$ fastboot flashing unlock
-
- Wipe user data from the device
$fastboot –w
-
- Flash correct GSI, check the security patch level of device, then choose the corresponding GSI
ex. Device security patch 1027-09-05, need to use system_aosp_arm64_ab_20170905.img
$fastboot flash system system_aosp_arm64_ab_20170905.img
重新设备 ,确认设备可以正常启动
FAQ
l 修改了如下内容 ，正常测试没有问题，可不需要修改
增加了环境变量 ，使用的是6.0的aapt ，这是要测试的相应的版本的aapt
1 。root@CTS-ceshi:~# gedit .bashrc
exportANDROID_HOME=/home/liangcheng/android-sdk-linux #/*SDK所在的路径*/
exportPATH=$PATH:/home/liangcheng/android-sdk-linux/build-tools/23.0.3
l 手机在linux上可以通过adb devices 看到 ，但是不弹出连接时间的对话框
需要查看 develop options 中 Select usb Configuration 是否是在MTP模式上