Android I/O Prefetch 学习
Android Iop 学习
思路整理：明确方向和目的
这个季度（剩余60天），我有一个重要的任务：完成Iop（I/O Prefetcher）的原理学习。目前我已经看Iop快一周了，但是还是不怎么懂，感觉距离理解还得一个多月，最主要的原因是其所涉及到的内容很广，从Android 的 FrameWork层延申到了hal层，自顶向下的链路大概是：
FrameWork{AMS，BoostFrameWork(通过反射调用Performance)}Java语言实现 ----> 中间转发层 Preformance 通过 jni ----> 调用Hal 层的 c++ 代码，c++ 代码完成预取操作。
这个学习过程有一个难点的地方是，AMS，这个地方所涉及的知识很多，又比较杂，BoostFrameWork穿插在AMS这个框架里的不少地方，来实现Iop。而AMS(ActivityManagerService)这个是Android 四大组件中最重要的一个，其功能主要是用来管理进程的启动调度，及内存管理。
在这个季度尾，我们部门有一个一小时的学习内容讲解，需要有深度的讲解一下技术，我师傅对我的安排是把Iop输出文档，并对Iop做一个有深度的讲解，目前看来，我的心里是没什么把握的，毕竟网上几乎搜不到相关的内容，唯一一篇文章，android IO Prefetch源码分析，很多地方讲的比较粗，看起来很费劲，对于我一个Android系统 新手来说，在FrameWork层的理解还是新手，不过这篇文章也很有参考意义。现在来罗列一下我目前所拥有的工具与资料：
- android IO Prefetch 源码分析
- Android 11 Iop HAL层 + BoostFrameWork源码框架 BoostFramework未穿插在 FrameWork 中
- Android 10 Iop 在FrameWork 层的源码，BoostFramework 穿插在 FrameWork中
- 随处可见的 AMS 讲解资料
- 调试的板子和环境可用于验证结果
我的任务：
- 完成 Iop 在 Android 11 的成功启动
- 测试 Iop 在 App 冷启动方面的优化效果
- 确认是否集成到新平台
标题学习计划制定
第一阶段
以目前的状态，自顶向下的学习，是效率最高的一种方案。故采取自顶向下的学习方式，来进行学习。虽说自顶向下，但是并非需要将FrameWork全学了，这样时间肯定不足，也大大脱离了目标。对于上层的学习目前只需要学习AMS，但是AMS依然很大，由我最近这几天的学习，发现AMS依然可以拆分，大致分为：
ActivityManagerService.java
ActivityStack.java
ProcessRecord.java
ActivityStackSupervisor.java
ActivityStarter.java
ActivityRecord.java
以上这几个模块需要重点关注，其余知识大致理解流程即可，以上这几个模块的代码重点阅读与理解，毕竟要在Android 11 上对这几块代码进行修改，现在拥有Android 10 编写的Iop的框架，可以进行模仿，但是前提条件是需要理解这几部分是做什么的，不然所做的代码移植就是不经思考的模仿。
第一阶段结果展望：
| 预期日期 | 截止日期 | 结果 |
|---|---|---|
| 2022/11/4 | 2022/11/10 | 完成Iop的成功启动，且输出一篇BoostFrameWork Iop部分和AMS部分互相作用机制的文档。 |
Activity 启动流程梳理
下图是Activity启动的框架流程图，包含了activity 整体流程的框架
较老Android 版本 Activity启动流程图
手动绘制的代码流程图，Activity的启动流程
-
学习Iop中遇到的一个人让我疑惑不解问题，I/O Prefetch 在AMS中是在哪些节点上生效会导致App启动被加速？
根据上图可知道Activity的启动会涉及两种情况，第一种是已经被启动过了一次，可以直接调用realStartActivityLocked()来启动，还有一种是未曾被启动过，需要调用startProcessAsync通过层层调用使用socket通知Zygote去fork一个新的进程，但最后通过层层调用还是会回到realStartActivityLocked来启动，而realStartActivityLocked 调用了findTaskToMoveToFront，而其包含了acquireAppLaunchPerfLock，acquireAppLaunchPerfLock 就是我们执行I/O Prefetch的部分，在此过程中发生Iop预读来加速App启动时间。android/frameworks/base/services/core/java/com/android/server/wm/ActivityStackSupervisor.java
void startSpecificActivity(ActivityRecord r, boolean andResume, boolean checkConfig) {
...........
boolean knownToBeDead = false;
if (wpc != null && wpc.hasThread()) {
try {
realStartActivityLocked(r, wpc, andResume, checkConfig);
return;
} catch (RemoteException e) {
Slog.w(TAG, "Exception when starting activity "
+ r.intent.getComponent().flattenToShortString(), e);
}
.......
final boolean isTop = andResume && r.isTopRunningActivity();
mService.startProcessAsync(r, knownToBeDead, isTop, isTop ? "top-activity" : "activity");
}
void findTaskToMoveToFront(Task task, int flags, ActivityOptions options, String reason,
boolean forceNonResizeable) {
.....
//top_activity = task.stack.topRunningActivityLocked();
/* App is launching from recent apps and it's a new process */
ActivityRecord top_activity = task.topRunningActivityLocked();
if(top_activity != null && top_activity.isState(DESTROYED)) {
acquireAppLaunchPerfLock(top_activity.packageName);
}
....
这里使用了BoostFramework 里面的调用，开启I/O预取，但并非第一次启动App
public void acquireAppLaunchPerfLock(String packageName) {
/* Acquire perf lock during new app launch */
if (mPerfBoost == null) {
mPerfBoost = new BoostFramework();
}
if (mPerfBoost != null) {
mPerfBoost.perfHint(BoostFramework.VENDOR_HINT_FIRST_LAUNCH_BOOST, packageName, -1, BoostFramework.Launch.BOOST_V1);
mPerfSendTapHint = true;
}
if (mPerfPack == null) {
mPerfPack = new BoostFramework();
}
if (mPerfPack != null) {
mPerfPack.perfHint(BoostFramework.VENDOR_HINT_FIRST_LAUNCH_BOOST, packageName, -1, BoostFramework.Launch.BOOST_V2);
}
// Start IOP
if (mPerfIop == null) {
mPerfIop = new BoostFramework();
}
if (mPerfIop != null) {
mPerfIop.perfIOPrefetchStart(-1,packageName,"");
}
}
public void acquireUxPerfLock(int opcode, String packageName) {
mUxPerf = new BoostFramework();
if (mUxPerf != null) {
mUxPerf.perfUXEngine_events(opcode, 0, packageName, 0);
}
}
- 第二个问题，为什么在次节点上会产生加速效果？
原因：许多应用程序在启动时需要访问I/O.很多时间会因为阻塞I / O而导致应用程序启动慢。预取数据之后，应用程序几乎可以从pagecache 中立即访问该数据，从而大大减少了应用程序启动延迟。
接下来要聊的是Activity 的状态转换关系。
基于上面的结论，我们启动一个新的App会调用到realStartActivityLocked这个函数，而此时Activity的状态正是处于OnCreate开始前，处于INITIALIZING 状态，在此处我们插入Iop的预取可实现优化。Activity 的所有状态如下：
enum ActivityState {
INITIALIZING,
RESUMED,
PAUSING,
PAUSED,
STOPPING,
STOPPED,
FINISHING,
DESTROYING,
DESTROYED,
RESTARTING_PROCESS
}
整体流程节点如下：
| 完成日期 | 完成结果 |
|---|---|
| 2022/10/30 | 梳理了Activity的启动及IO Prefetch的执行节点 |
第二阶段
第二阶段相对第一阶段较容易，因为这一周的时间已经用来看了Iop的hal层的实现机制了，且代码上的分布很紧凑，代码量小，涉及的架构不怎么庞大，正常进行学习即可，顺便学习一下JNI机制，衍射。
下图是关于Iop调用的整体流程图如下：
Iop初始化流程
首先当系统启动时，会启动 [email protected] 进程，该进程相当于一个Server端，他在最开始运行时会进行初始化，首先调用dlopen加载libqti-iopd.so动态库，初始化函数指针，加载libqti-iopd.so库中的函数，以便之后用到的时候，对函数进行调用。接下来看看他是如何做的：
void Iop::LoadIopLib() {
const char *rc = NULL;
char buf[PROPERTY_VALUE_MAX];
if (!mHandle.is_opened) {
mHandle.dlhandle = dlopen("libqti-iopd.so", RTLD_NOW | RTLD_LOCAL);
if (mHandle.dlhandle == NULL) {
ALOGE("IOP-HAL %s Failed to (dl)open iopd\n", __func__);
return;
}
dlerror();
*(void **) (&mHandle.iop_server_init) = dlsym(mHandle.dlhandle, "iop_server_init");
if ((rc = dlerror()) != NULL) {
ALOGE("IOP-HAL %s Failed to get iop_server_init\n", rc);
dlclose(mHandle.dlhandle);
mHandle.dlhandle = NULL;
return;
}
*(void **) (&mHandle.iop_server_exit) = dlsym(mHandle.dlhandle, "iop_server_exit");
if ((rc = dlerror()) != NULL) {
ALOGE("IOP-HAL %s Failed to get iop_server_exit\n", rc);
dlclose(mHandle.dlhandle);
mHandle.dlhandle = NULL;
return;
}
*(void **) (&mHandle.iop_server_submit_request) = dlsym(mHandle.dlhandle, "iop_server_submit_request");
if ((rc = dlerror()) != NULL) {
ALOGE("IOP-HAL %s Failed to get iop_server_submit_request\n", rc);
dlclose(mHandle.dlhandle);
mHandle.dlhandle = NULL;
return;
}
*(void **) (&mHandle.uxe_server_submit_request) = dlsym(mHandle.dlhandle, "uxe_server_submit_request");
if ((rc = dlerror()) != NULL) {
ALOGE("IOP-HAL %s Failed to get uxe_server_submit_request\n", __func__);
dlclose(mHandle.dlhandle);
mHandle.dlhandle = NULL;
return;
}
mHandle.is_opened = true;
}
return;
}
在加载了libqti-iopd.so动态库的函数之后，该进程便会调用iop_server_init的初始化操作，在这里调用了iop_server_init等待Framework层传下来的信息进行处理。
IIop* HIDL_FETCH_IIop(const char* /* name */) {
ALOGE("IOP-HAL: inside HIDL_FETCH_IIop");
Iop *iopObj = new (std::nothrow) Iop();
ALOGE("IOP-HAL: boot Address of iop object");
if (iopObj != NULL) {
iopObj->LoadIopLib();
ALOGE("IOP-HAL: loading library is done");
if (iopObj->mHandle.iop_server_init != NULL ) {
(*(iopObj->mHandle.iop_server_init))();
}
}
return iopObj;
}
Iop_server_init这个函数会创建了一个死循环的iop_server线程，用来处理从Framework层传下来的msg信息。
int iop_server_init() {
int rc1 = 0, stage = 0, rc2 = 0;
int uba_rc = 0;
char property[PROPERTY_VALUE_MAX];
char trace_prop[PROPERTY_VALUE_MAX];
QLOGI("IOP server starting");
/* Enable traces by adding vendor.debug.trace.perf=1 into build.prop */
if (property_get(PROP_NAME, trace_prop, NULL) > 0) {
if (trace_prop[0] == '1') {
perf_debug_output = PERF_SYSTRACE = atoi(trace_prop);
}
}
preferred_apps = (char *) malloc (strcat_sz * sizeof(char));
if (preferred_apps == NULL) {
goto error;
}
memset(preferred_apps, 0, strcat_sz);
preferred_apps[0] = '\0';
IOPevqueue.GetDataPool().SetCBs(Alloccb, Dealloccb);
rc1 = pthread_create(&iop_server_thread, NULL, iop_server, NULL);
if (rc1 != 0) {
stage = 3;
goto error;
}
iop_init = true;
if (uxe_disabled()) {
uxe_prop_disable = 1;
} else {
uxe_prop_disable = 0;
strlcpy(property,perf_get_prop("vendor.iop.uxe_trigger_freq" , "1").value, PROPERTY_VALUE_MAX);
property[PROPERTY_VALUE_MAX-1]='\0';
uxe_trigger_freq = atoi(property);
UXEevqueue.GetDataPool().SetCBs(Alloccb, Dealloccb);
rc2 = pthread_create(&uxe_server_thread, NULL, uxe_server, NULL);
uba_rc = init_uba();
if (rc2 != 0) {
goto error;
}
if (uba_rc == 0) {
QLOGE("Fail to init UBA");
goto error;
}
uxe_init = true;
}
return 1;
error:
QLOGE("Unable to create control service (stage=%d, rc1=%d rc2=%d uba_rc=%d)", stage, rc1, rc2, uba_rc);
return 0;
}
在 iop_server中，首先初始化一些参数，然后会向IOPevqueue队列申请一个msg，若队列中没有请求消息则阻塞在这里，等待队列中出现msg，若有msg，则取出来，走接下来的流程，然后创建数据库，若数据库没有被创建则创建数据库。因为我们提交的消息一般是IopStart 的过程，所以消息类型是case IOP_CMD_PERFLOCK_IOPREFETCH_START，所以我们走这条分支。接下来他会获取配置文件的配置参数，是否开启某些功能。配置文件路径 /vendor/etc/perf/perfconfigstore.xml ，配置文件基本内容如下图
可根据vendor.enable.prefetch是否为1是否开启iop预读。最开始执行信息采集的是pid > 0，所以先分析pid > 0的情况。
static void *iop_server(void *data)
{
int rc, cmd;
iop_msg_t *msg = NULL;
(void)data;
bool is_db_init = false;
char tmp_pkg_name[PKG_NAME_LEN];
int bindApp_dur = 0, disAct_dur = 0, avg_bindApp = 0, avg_disAct = 0;
int pkg_count = 0, bindApp_count = 0, pkg_mismatch = 0, da_reset = 0;
bool launching = false;
memset(tmp_pkg_name, 0, PKG_NAME_LEN);
/* Main loop */
for (;;) {
//wait for perflock commands
EventData *evData = IOPevqueue.Wait();
if (!evData || !evData->mEvData) {
continue;
}
if(!is_boot_complete())
{
QLOGE("io prefetch is disabled waiting for boot_completed");
continue;
}
if(is_db_init == false)
{
if(create_database() == 0)
{
//Success
is_db_init = true;
}
}
case IOP_CMD_PERFLOCK_IOPREFETCH_START:
{
static bool is_in_recent_list = false;
char enable_prefetch_property[PROPERTY_VALUE_MAX];
char enable_prefetch_ofr_property[PROPERTY_VALUE_MAX];
int enable_prefetcher = 0;
int enable_prefetcher_ofr = 0;
strlcpy(enable_prefetch_property,perf_get_prop("vendor.enable.prefetch" , "0").value, PROPERTY_VALUE_MAX);
enable_prefetch_property[PROPERTY_VALUE_MAX-1]='\0';
enable_prefetcher = strtod(enable_prefetch_property, NULL);
strlcpy(enable_prefetch_ofr_property,perf_get_prop("vendor.iop.enable_prefetch_ofr" , "0").value, PROPERTY_VALUE_MAX);
enable_prefetch_ofr_property[PROPERTY_VALUE_MAX-1]='\0';
enable_prefetcher_ofr = strtod(enable_prefetch_ofr_property, NULL);
// if PID < 0 consider it as playback operation
if(msg->pid < 0)
{
int ittr = 0;
char *week_day = NULL;
week_day = (char *) malloc(6*sizeof(char));
// Insert package into the table
if (week_day == NULL) {
//Malloc failed. Most-probably low on memory.
break;
}
strlcpy(pkg_info.pkg_name,msg->pkg_name,PKG_NAME_LEN);
strlcpy(tmp_pkg_name,pkg_info.pkg_name,PKG_NAME_LEN);
bindApp_dur = 0;
disAct_dur = 0;
launching = true;
time(&pkg_info.last_time_launched);
compute_time_day_slot(week_day, &time_slot);
QLOGI("UXEngine Updating Details: pkg_name: %s, week_day: %s, time_slot: %d %s\n", pkg_info.pkg_name, week_day, time_slot, tmp_pkg_name);
update_ux_pkg_details(pkg_info, week_day, time_slot, 0);
update_palm_table(msg->pkg_name, 0, 1);
QLOGI("UXEngine finished ux_pkg_details update \n");
free(week_day);
is_in_recent_list = false;
if(!enable_prefetcher)
{
QLOGE("io prefetch is disabled");
break;
}
//Check app is in recent list
for(ittr = 0; ittr < IOP_NO_RECENT_APP; ittr++)
{
if(0 == strcmp(msg->pkg_name,recent_list[ittr]))
{
is_in_recent_list = true;
QLOGE("is_in_recent_list is TRUE");
break;
}
}
// IF Application is in recent list, return
if(true == is_in_recent_list)
{
QLOGE("io prefetch is deactivate");
break;
}
if(recent_list_cnt == IOP_NO_RECENT_APP)
recent_list_cnt = 0;
//Copy the package name to recent list
strlcpy(recent_list[recent_list_cnt],msg->pkg_name,PKG_LEN);
recent_list_cnt++;
stop_capture();
stop_playback();
start_playback(msg->pkg_name);
}
// if PID > 0 then consider as capture operation
if(msg->pid > 0)
{
if(!enable_prefetcher)
{
QLOGE("io prefetch is disabled");
break;
}
if(true == is_in_recent_list)
{
QLOGE("io prefetch Capture is deactivated ");
break;
}
stop_capture();
start_capture(msg->pid,msg->pkg_name,msg->code_path,enable_prefetcher_ofr);
}
break;
}
}
pid > 0时，会先检测是否开启了iop 预取，且查看 预取的信息是否在最近的数组中，有则停止采集。接下来回收可能存在的线程，防止内存泄露，且进入start_capture中，创建一个新的线程capture_thread 进行信息采集更新至数据库中。
int start_capture(pid_t pid, char *pkg_name, char *code_path, bool ofr) {
char property[PROPERTY_VALUE_MAX];
capture_thread_arg *arg_bundle = NULL;
int len = 0;
int len_code_path = 0;
if(NULL == pkg_name)
return -1;
len = strlen(pkg_name);
if ( len <= 0)
{
QLOGW("incorrect length for pkg_name");
return -1;
}
arg_bundle = (capture_thread_arg *)malloc(sizeof(capture_thread_arg));
if (NULL == arg_bundle)
return -1;
total_files = 0;
QLOGE(" start_capture ENTER ");
arg_bundle->pid = pid;
arg_bundle->pkg_name = (char *) malloc(len+1);
arg_bundle->ofr = ofr;
len_code_path = strlen(code_path);
if (NULL == arg_bundle->pkg_name) {
free(arg_bundle);
return -1;
}
QLOGI("%d %s %zd %d\n", pid, arg_bundle->pkg_name,strlen(pkg_name),PKG_NAME_LEN);
strlcpy(arg_bundle->pkg_name, pkg_name, len+1);
strlcpy(list_code_path, code_path, len_code_path+1);
QLOGI("%d %s %zd %s %zd\n", pid, arg_bundle->pkg_name,strlen(pkg_name),
list_code_path, strlen(code_path));
halt_thread = 0;
start_capture_time = (long) now_ms();
time(&start_capture_time_s);
property_get("vendor.iop.read_fd_interval_ms", property, "50");
read_fd_interval_ms = atoi(property);
if(read_fd_interval_ms <= 0) {
QLOGI("read_fd_interval_ms is set to a non-positive value %d. Resetting to default value 50\n", read_fd_interval_ms);
property_set("read_fd_interval_ms", "50");
read_fd_interval_ms = 50;
}
QLOGI("property = %d\n", read_fd_interval_ms);
//TBD must remove comments
if(pthread_create(&capture_pthread_id, NULL, capture_thread, arg_bundle)) {
return -1;
}
QLOGI("start_capture EXIT");
return 0;
}
在capture_thread中继续读取配置参数，根据参数是否开启某些功能，根据测试红色部分不建议开，会反响优化，使冷启动时间变慢。调用get_priv_code_files和get_priv_files获取提供的有关activity的相关文件放入数组file_list中，然后调用update_pkg_details和get_priv_files更新至数据库中。
void * capture_thread(void * arg) {
int i = 0;
int pkg_len = 0;
int duration_counter = 0;
pkg_details pkg_info;
int index = 0;
int halt_counter = CAPTURE_MAX_DURATION/read_fd_interval_ms;
int polling_enable = 0;
char property[PROPERTY_VALUE_MAX];
property_get("vendor.polling_enable", property, "0");
polling_enable = atoi(property);
ATRACE_BEGIN("capture_thread");
capture_thread_arg * arg_bundle = (capture_thread_arg *)arg;
//Copy the package to same string to remove ':'
pkg_len = strlen(arg_bundle->pkg_name);
i = 0;
while(i < pkg_len && arg_bundle->pkg_name[i] != ':')
{
i++;
}
arg_bundle->pkg_name[i] = '\0';
strlcpy(list_pkg_name,arg_bundle->pkg_name,PKG_NAME_LEN);
QLOGI("pkg_name = %s",arg_bundle->pkg_name);
if(polling_enable)
{
while (duration_counter < halt_counter) {
if (halt_thread) goto cleanup;
QLOGI("Getting snapshot %d\n", arg_bundle->pid);
get_snapshot(list_pkg_name,arg_bundle->pid);
duration_counter++;
usleep(read_fd_interval_ms * 1000);
}
}
get_priv_code_files(arg_bundle->pkg_name);
get_priv_files(arg_bundle);
QLOGI("pkg_name = %s total_files = %d ",arg_bundle->pkg_name,total_files);
// Insert package into the table
strlcpy(pkg_info.pkg_name,arg_bundle->pkg_name,PKG_NAME_LEN);
time(&pkg_info.last_time_launched);
// Update Mincore data
if(arg_bundle->ofr)
{
for(index = 0; index < total_files;index++)
{
if(update_mincore_data(file_list[index]) != 0)
{
file_list[index]->mincore_array = NULL;
}
}
}
update_pkg_details(pkg_info);
update_file_details(arg_bundle->pkg_name, file_list, total_files);
delete_mark_files();
cleanup:
//log a report about how many files need insert or update this time
QLOGE("# Final entry : pkg_name file_name file_time_stamp filesize file_iop_size");
for(i = 0; i < total_files; i++) {
QLOGE("%d. Final entry : %s %s %d %d %d\n",i
,arg_bundle->pkg_name,file_list[i]->file_name
, file_list[i]->file_time_stamp, file_list[i]->filesize, file_list[i]->file_iop_size);
if (file_list[i]->mincore_array) {
free(file_list[i]->mincore_array);
file_list[i]->mincore_array = NULL;
}
free(file_list[i]);
}
ATRACE_END();
free(arg_bundle->pkg_name);
free(arg_bundle);
QLOGI("Exit capture_thread");
return NULL;
}
接下来看看get_priv_code_files 和 get_priv_files 具体获取了哪些数据？先来看看get_priv_code_files获取了哪些文件。在代码中主要获取了oat中的文件。举个例子看下：
例如.video/IqiyiVideo 目录下的 IqiyiVideo.apk lib oat等内容。capture_pkg_file是一个递归函数，主要获取文件夹里的内容，如*.odex vdex *.apk等文件
static void get_priv_code_files(char *pkg_name) {
char f_name[FILE_NAME_LEN];
DIR *dp;
//oat first
snprintf(f_name,FILE_NAME_LEN,"%s/oat",list_code_path);
QLOGI("Dir = %s",f_name);
dp = opendir(f_name);
QLOGI("First dp = %p",dp);
if (dp != NULL) {
capture_pkg_file(pkg_name, dp, f_name, -1);
closedir(dp);
} else {
QLOGI("dp is NULL");
}
//then scan apk file
dp = opendir(list_code_path);
QLOGI("First dp = %p",dp);
if (dp != NULL) {
capture_pkg_file(pkg_name, dp, list_code_path, 1);
closedir(dp);
} else {
QLOGI("dp is NULL");
}
}
接下来看看get_priv_files获取了什么文件？
在代码中主要获取 /data/user/0/pkg_name 以及 /data/data/pkg_name 文件夹下的文件，找个文件看看如下：
第二阶段结果展望：
void get_priv_files(capture_thread_arg * arg_bundle)
{
char f_name[FILE_NAME_LEN];
DIR *dp;
char trace_buf[1024];
snprintf(trace_buf, sizeof(trace_buf), "Get_prive_file");
ATRACE_BEGIN(trace_buf);
snprintf(f_name,FILE_NAME_LEN,"/data/user/0/%s",arg_bundle->pkg_name);
QLOGI("Dir = %s",f_name);
dp = opendir(f_name);
QLOGI("First dp = %p",dp);
if (dp != NULL) {
capture_pkg_file(arg_bundle->pkg_name, dp,f_name,2);
closedir(dp);
}
else
{
QLOGI("dp is NULL");
}
snprintf(f_name,FILE_NAME_LEN,"/data/data/%s",arg_bundle->pkg_name);
QLOGI("Dir = %s",f_name);
dp = opendir(f_name);
QLOGI("First dp = %p",dp);
if (dp != NULL) {
capture_pkg_file(arg_bundle->pkg_name, dp,f_name, 4);
closedir(dp);
}
else
{
QLOGI("dp is NULL");
}
ATRACE_END();
}
接下来看看如何更新到数据库里？
先调用get_package是否已经在数据库里了，如果在给他的调用次数+1，且更新上次启动时间。如果不在则调用IO_PREFETCHER_QUERY_INSERT_PKG sql语句插入数据库中。
//Update package deails with provided attributes
int update_pkg_details(pkg_details pkg_info)
{
char query_str[2048] = IO_PREFETCHER_QUERY_UPDATE_PKG_DETAILS;
/* "UPDATE io_pkg_tbl SET pkg_last_use = %lu ,pkg_use_count = pkg_use_count+1 WHERE pkg_name = '%s'"*/
pkg_details temp_pkg_info;
temp_pkg_info.pkg_name[0] = 0;
int is_file_present = get_package(pkg_info.pkg_name, &temp_pkg_info);
QLOGI("update_pkg_details");
if(NULL == db_conn)
open_db();
if(is_file_present)
{
//Update
snprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_UPDATE_PKG_DETAILS
,pkg_info.last_time_launched
,pkg_info.pkg_name);
QLOGI("\nQuery = %s \n",query_str);
iop_query_exec(query_str);
}
else
{
//Insert
snprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_INSERT_PKG
,pkg_info.pkg_name
,pkg_info.last_time_launched
);
QLOGI("\nQuery = %s \n",query_str);
iop_query_exec(query_str);
}
return 0;
}
这里通过get_file 获取io_pkg_file_tbl表的信息,更新到io_pkg_file_tbl表中。这里数据的更新使用了sql的事务来更新到数据库中。
// update detail for file with provided attibutes
int update_file_details(char * pkg_name,file_details *file_info[], int size)
{
int i = 0;
char *error=NULL;
sqlite3_stmt *stmt=NULL;
char trace_buf[1024];
char query_str[2048] = IO_PREFETCHER_QUERY_UPDATE_FILE_DETAILS;
iop_query_exec("BEGIN TRANSACTION;");
for(i=0;i<size;i++) {
file_details temp_file_info;
temp_file_info.file_name[0]=0;
int is_file_present = get_file(pkg_name, file_info[i]->file_name, &temp_file_info);
if(is_file_present) {
//Update
QLOGI("MINCORE update file %s %s study_finish %d, cache_dropped %d",
pkg_name, temp_file_info.file_name,
temp_file_info.study_finish, temp_file_info.cache_dropped);
/*
* update mincore array data if this file cache is dropped and study not finished
*/
if (temp_file_info.cache_dropped == 1 && temp_file_info.study_finish <= 2) {
snprintf(trace_buf, sizeof(trace_buf), "update file %s", file_info[i]->file_name);
ATRACE_BEGIN(trace_buf);
snprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_UPDATE_FILE_DETAILS
,file_info[i]->file_time_stamp
,file_info[i]->filesize
,file_info[i]->file_iop_size
,pkg_name
,file_info[i]->file_name);
QLOGI("\nQuery = %s \n",query_str);
int pages_in_range = (file_info[i]->filesize + PAGE_SIZE - 1)/PAGE_SIZE;
sem_wait(&mutex);
_SQLITE_CHECK_OK(sqlite3_prepare_v2(db_conn, query_str,
strlen(query_str), &stmt, (const char **)&error));
int index = sqlite3_bind_parameter_index(stmt, ":mincore_array");
if (file_info[i]->mincore_array) {
_SQLITE_CHECK_OK(sqlite3_bind_blob(stmt, index,
file_info[i]->mincore_array, pages_in_range, SQLITE_STATIC));
}
else
{
_SQLITE_CHECK_OK(sqlite3_bind_blob(stmt, index, NULL, 0, SQLITE_STATIC));
}
_SQLITE_CHECK_DONE(sqlite3_step(stmt));
sqlite3_reset(stmt);
sqlite3_finalize(stmt);
sem_post(&mutex);
ATRACE_END();
}
else
{
if (file_info[i]->cache_dropped) {
mark_cache_dropped(pkg_name, file_info[i]->file_name, 0);
}
}
}
else
{
//insert new file to db
int index;
int pagesize = sysconf(_SC_PAGESIZE);
QLOGI("MINCORE insert new file %s %s iopsize %d filesize %d",
pkg_name, file_info[i]->file_name,
file_info[i]->file_iop_size, file_info[i]->filesize);
snprintf(trace_buf, sizeof(trace_buf), "insert new file %s", file_info[i]->file_name);
ATRACE_BEGIN(trace_buf);
snprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_INSER_FILE
,pkg_name,file_info[i]->file_name
,file_info[i]->file_time_stamp
,file_info[i]->filesize
,file_info[i]->file_modify_time
,file_info[i]->file_iop_size
);
QLOGI("\nQuery = %s \n",query_str);
sem_wait(&mutex);
_SQLITE_CHECK_OK(sqlite3_prepare_v2(db_conn, query_str,
strlen(query_str), &stmt, (const char**)&error));
if (file_info[i]->mincore_array != NULL && file_info[i]->file_iop_size >pagesize) {
index = sqlite3_bind_parameter_index(stmt, ":mincore_array");
int pages_in_range = (file_info[i]->filesize + PAGE_SIZE - 1)/PAGE_SIZE;
_SQLITE_CHECK_OK(sqlite3_bind_blob(stmt, index,
file_info[i]->mincore_array, pages_in_range, SQLITE_STATIC));
}
_SQLITE_CHECK_DONE(sqlite3_step(stmt));
sqlite3_reset(stmt);
sqlite3_finalize(stmt);
sem_post(&mutex);
ATRACE_END();
}
}
iop_query_exec("COMMIT TRANSACTION;");
return 0;
}
此时采集信息的过程已经完成了，接下来应该是执行预取优化的过程了，可以回到iop_server 流程中的pid < 0的步骤，继续向下分析。黄色得部分主要是收集activity启动得时间信息，主要包括信息是工作日，上一次启动时间等信息，用于预测，如果预测成功权重+weight，预测失败，失败的权重+weight等信息，目前没有用到预测功能，先忽略。绿色的部分主要是看新启动的app是否是在最近启动的列表里，如果存在则不执行预取。如果都没问题则执行浅红色部分，执行获取数据库中的信息提前执行预取操作。ng)
if(msg->pid < 0)
{
int ittr = 0;
char *week_day = NULL;
week_day = (char *) malloc(6*sizeof(char));
// Insert package into the table
if (week_day == NULL) {
//Malloc failed. Most-probably low on memory.
break;
}
strlcpy(pkg_info.pkg_name,msg->pkg_name,PKG_NAME_LEN);
strlcpy(tmp_pkg_name,pkg_info.pkg_name,PKG_NAME_LEN);
bindApp_dur = 0;
disAct_dur = 0;
launching = true;
time(&pkg_info.last_time_launched);
compute_time_day_slot(week_day, &time_slot);
QLOGI("UXEngine Updating Details: pkg_name: %s, week_day: %s, time_slot: %d %s\n", pkg_info.pkg_name, week_day, time_slot, tmp_pkg_name);
update_ux_pkg_details(pkg_info, week_day, time_slot, 0);
update_palm_table(msg->pkg_name, 0, 1);
QLOGI("UXEngine finished ux_pkg_details update \n");
free(week_day);
is_in_recent_list = false;
if(!enable_prefetcher)
{
QLOGE("io prefetch is disabled");
break;
}
//Check app is in recent list
for(ittr = 0; ittr < IOP_NO_RECENT_APP; ittr++)
{
if(0 == strcmp(msg->pkg_name,recent_list[ittr]))
{
is_in_recent_list = true;
QLOGE("is_in_recent_list is TRUE");
break;
}
}
// IF Application is in recent list, return
if(true == is_in_recent_list)
{
QLOGE("io prefetch is deactivate");
break;
}
if(recent_list_cnt == IOP_NO_RECENT_APP)
recent_list_cnt = 0;
//Copy the package name to recent list
strlcpy(recent_list[recent_list_cnt],msg->pkg_name,PKG_LEN);
recent_list_cnt++;
stop_capture();
stop_playback();
start_playback(msg->pkg_name);
}
Iop预取优化流程
接下来我们具体看看他是如何从数据库中获取信息，然后执行iop预取的操作的。在start_playback中开一个线程去执行。
void start_playback(char *pkg_name_arg) {
char *pkg_name = NULL;
int len = 0;
QLOGE(" start_playback-1 ");
if(NULL == pkg_name_arg)
return;
len = strlen(pkg_name_arg);
if(len <= 0)
{
QLOGW("Incorrect length for pkg_name_arg");
return;
}
pkg_name = (char *) malloc(len + 1);
if (NULL == pkg_name)
return;
strlcpy(pkg_name, pkg_name_arg,len+1);
QLOGI(" %s %zd\n", pkg_name,strlen(pkg_name));
if(pthread_create(&start_playback_pthread_id, NULL, start_playback_thread, pkg_name)) {
return ;
}
QLOGI("Exit playback");
return ;
}
接下来是Iop的核心操作。get_total_file(pkg_name)这个操作是检查数据库中是否有数据，如果没有，则没有预取操作了。如果num_file>0，则表示有数据，则调用get_file_list，获取pkg_name 对应的文件信息存储到file_detail_ptr中，接下来的一对操作主要是进行数据的标记，是否要清理缓存，根据数据标签把一些不确定的因素去掉。此时已经获取到了预取信息到file_detail_ptr，调用iop_fileplayback_run(file_detail_ptr)进行预取，最后调用 commit_filelist_info （）将更新的信息更新到数据库里。
static void* start_playback_thread(void *pkg_name_arg)
{
file_details *file_detail_ptr;
int num_file = 0;
int num_of_files = 0;
char *pkg_name = (char *)pkg_name_arg;
int i = 0;
total_data = 0;
struct stat file_stat;
int pagesize = sysconf(_SC_PAGESIZE);
ATRACE_BEGIN("start_playback_thread: Enter");
QLOGI("pkg_name = %s",pkg_name);
num_file = get_total_file(pkg_name);
if( num_file <= 0)
{
QLOGI("no file to read get_total_file %d", num_file);
remove_pkg(pkg_name);
free(pkg_name);
ATRACE_END();
return NULL;
}
file_detail_ptr = (file_details *) malloc(sizeof(file_details) * num_file);
if (NULL == file_detail_ptr)
{
QLOGI("Fail to allocate memory");
remove_pkg(pkg_name);
free(pkg_name);
ATRACE_END();
return NULL;
}
num_of_files = get_file_list(pkg_name,file_detail_ptr,num_file);
QLOGI("num_of_files = %d",num_of_files);
if (num_of_files <= 0)
{
QLOGI("no file to read get_file_list");
remove_pkg(pkg_name);
free(file_detail_ptr);
free(pkg_name);
ATRACE_END();
return NULL;
}
if( num_file > MAX_NUM_FILE)
{
//free mincore array that exceed MAX_NUM_FILE
for (i=MAX_NUM_FILE; i<num_of_files; i++) {
if (file_detail_ptr[i].mincore_array)
{
free(file_detail_ptr[i].mincore_array);
file_detail_ptr[i].mincore_array=NULL;
}
}
num_file = MAX_NUM_FILE;
}
for(i = 0; i < num_file;i++)
{
int fd = -1;
char trace_buf[4096];
if (file_detail_ptr[i].disabled) {
QLOGE("skip preload disabled file %d %s", i, file_detail_ptr[i].file_name);
continue;
}
if (file_detail_ptr[i].study_finish && file_detail_ptr[i].file_iop_size < pagesize) {
file_detail_ptr[i].cache_dropped = NEED_MARK_CACHE_NOT_DROPPED;
QLOGE("skip preload ZERO needed file %d %s", i, file_detail_ptr[i].file_name);
continue;
}
fd = open(file_detail_ptr[i].file_name, O_RDONLY);
if(fd == -1)
{
mark_for_delete(pkg_name,file_detail_ptr[i].file_name);
continue;
}
if ( fstat( fd, &file_stat ) < 0 )
{
QLOGI("fail to get file stat");
mark_for_delete(pkg_name,file_detail_ptr[i].file_name);
close(fd);
continue;// goto next file
}
if ( file_stat.st_size == 0 )
{
// File zie is zero
mark_for_delete(pkg_name,file_detail_ptr[i].file_name);
close(fd);
continue;// goto next file
}
snprintf(trace_buf,4096,"Opening file %s size = %d",file_detail_ptr[i].file_name
, (int)file_stat.st_size);
ATRACE_BEGIN(trace_buf);
if (file_stat.st_mtime != file_detail_ptr[i].file_modify_time) {
if (strstr(file_detail_ptr[i].file_name,".apk")!=NULL
&& (strstr(file_detail_ptr[i].file_name,"/data/app/")!=NULL ||
strstr(file_detail_ptr[i].file_name, "/system/app/")!=NULL ||
strstr(file_detail_ptr[i].file_name, "/system/priv-app/")!=NULL))
{
//readonly /data/app/base.apk
QLOGI("apk file %s modify time changed %d/%ld, \
remove files belong to pkg, and restudy",
file_detail_ptr[i].file_name,
file_detail_ptr[i].file_modify_time, file_stat.st_mtime);
snprintf(trace_buf, sizeof(trace_buf), "apk file %s upgrade, remove pkg"
, file_detail_ptr[i].file_name);
ATRACE_BEGIN(trace_buf);
file_detail_ptr[i].mark_for_delete = NEED_MARK_FOR_DELETE_PKG;
close(fd);
ATRACE_END();
ATRACE_END(); // end for opening file
ATRACE_END(); //end for start_playback_thread enter
return NULL;
} else {
QLOGI("file %s modify time changed update db", file_detail_ptr[i].file_name);
//mark study_finish=0 and cache_dropped=1
snprintf(trace_buf, sizeof(trace_buf), "file %s modifie mark update"
, file_detail_ptr[i].file_name);
ATRACE_BEGIN(trace_buf);
//restudy if file modify time changed for .odex and .dex file
if (strstr(file_detail_ptr[i].file_name, ".odex") != NULL
|| strstr(file_detail_ptr[i].file_name, ".dex") != NULL
|| strstr(file_detail_ptr[i].file_name, ".vdex") != NULL)
{
file_detail_ptr[i].mark_for_delete = NEED_MARK_FOR_UPDATE;
file_detail_ptr[i].file_modify_time = file_stat.st_mtime;
file_detail_ptr[i].filesize = file_stat.st_size;
drop_file_cache(fd, &(file_detail_ptr[i]));
} else {//disable unstable file
file_detail_ptr[i].mark_for_delete = NEED_MARK_FOR_DISABLE;
file_detail_ptr[i].disabled = 1;
}
ATRACE_END();
}
close(fd);
continue;
}
//try if need drop cache or playback
if (file_detail_ptr[i].study_finish == 0) {
QLOGI("Drop cache of file %s", file_detail_ptr[i].file_name);
snprintf(trace_buf, sizeof(trace_buf), "Drop cache of file %s",
file_detail_ptr[i].file_name);
ATRACE_BEGIN(trace_buf);
drop_file_cache(fd, &(file_detail_ptr[i]));
file_detail_ptr[i].cache_dropped = NEED_MARK_CACHE_DROPPED;
ATRACE_END();
} else {
QLOGI("io preload of file %s", file_detail_ptr[i].file_name);
snprintf(trace_buf, sizeof(trace_buf), "add file %s %.2f / %.2f MB to task",
file_detail_ptr[i].file_name,
file_detail_ptr[i].file_iop_size/1024.0/1024,
file_detail_ptr[i].filesize/1024.0/1024);
ATRACE_BEGIN(trace_buf);
iop_fileplayback_run(file_detail_ptr[i],file_stat,fd);
ATRACE_END();
if (file_detail_ptr[i].cache_dropped) {
file_detail_ptr[i].cache_dropped = NEED_MARK_CACHE_NOT_DROPPED;
}
}
close(fd);
if(total_data > MAX_TOTAL_DATA)
{
QLOGI("Max total size limit reached Total = %d",total_data);
//STOP reading and max total reached
break;
}
ATRACE_END(); // end for opening file
}
commit_filelist_info(file_detail_ptr, num_file);
delete_mark_files();
ATRACE_END(); //end for start_playback_thread enter
QLOGI(" Total Data = %.2f MB", total_data/1024.0/1024);
for (i=0; i<num_of_files; i++) {
if (file_detail_ptr[i].mincore_array) {
free(file_detail_ptr[i].mincore_array);
file_detail_ptr[i].mincore_array=NULL;
}
}
free(pkg_name);
free(file_detail_ptr);
return NULL;
}
下面来看看iop_fileplayback_run具体执行了什么操作？
在这里主要是通过posix_fadvise 系统调用 + POSIX_FADV_WILLNEED 参数执行了预读操作，将需要加载的文件提前读到了pagecache里，使启动时读取该部分文件能直接命中缓存加速启动速度。具体可以参照posix_fadvise的作用 linux系统函数posix_fadvise_九品神元师的博客-CSDN博客。
/* Run playback operation */
static void *iop_fileplayback_run(file_details info, struct stat file_stat, int fd)
{
int page_itrr;
int total_mincore_pages = (info.file_iop_size+PAGE_SIZE-1)/PAGE_SIZE;
int total_pages = (file_stat.st_size+PAGE_SIZE-1)/PAGE_SIZE;
char *mincore_array = info.mincore_array;
char trace_buf[1024];
if (total_mincore_pages > 0) {
snprintf(trace_buf, sizeof(trace_buf), "%s read size:%.4fMB",
info.file_name,
total_mincore_pages*4/1024.0);
}
else
{
snprintf(trace_buf, sizeof(trace_buf), "%s read size:%.4fMB",
info.file_name,file_stat.st_size/1024.0/1024);
}
ATRACE_BEGIN(trace_buf);
int count_pages=0;
int read_data = 0;
int ret = 0;
QLOGI("MINCORE, PLAYBACK %s START, mincore_array: %p",
info.file_name, mincore_array);
//load each page by posix_fadvise here to avoid read_ahead
//and decrease CPU usage of copy io content result to local viriable
if (mincore_array == NULL) {
//if all file need read to cache
int len_size = file_stat.st_size;
ret = posix_fadvise(fd, 0, len_size, POSIX_FADV_WILLNEED);
read_data += len_size;
if (ret !=0) {
QLOGE("posix_fadvise failed: %s", strerror(errno));
}
} else {//load file content based on mincore result
count_pages=0;
for(page_itrr = 0; page_itrr < total_pages; page_itrr++) {
if (mincore_array[page_itrr] & 0x1) {
if (count_pages>=total_mincore_pages)
break;
ret = posix_fadvise(fd, page_itrr*PAGE_SIZE, PAGE_SIZE, POSIX_FADV_WILLNEED);
if (ret !=0) {
QLOGE("posix_fadvise failed: %s", strerror(errno));
}
read_data += PAGE_SIZE;
count_pages++;
}
}
}
//after finish preload, mark file cache advise as normal
ret = posix_fadvise(fd, 0, file_stat.st_size, POSIX_FADV_NORMAL);
if (ret !=0) {
QLOGE("posix_fadvise failed: %s", strerror(errno));
}
close(fd);
total_data += read_data;
QLOGI("MINCORE, PLAYBACK %s END real read file size = %.4fMB",
info.file_name, read_data/1024.0/1024);
ATRACE_END();
return NULL;
}
学前疑问解答
- Iop是如何怎么预取的？
主要通过posix_fadvise 这个系统调用配合相应的参数将所需要的文件预取到PageCache里。 - 数据库的表是如何存放数据的？
数据库中的数据主要是从codepath，和 遍历pid 中的fd所指向的链接的信息进行筛选存入数据库的表中，iop用到的表主要存app的.apk，vdex，.so等文件，并记录被使用的时间，和使用到的次数。
| 预期日期 | 截止日期 | 结果 |
|---|---|---|
| 2022/11/14 | 2022/11/25 | 理解Iop底层机制，输出一篇关于Iop底层原理的文档 |
第三阶段
第三阶段属于测试阶段，在这个阶段，我应该已经成功启动了Iop机制，且已经掌握了90%的Iop原理，可以比较灵活的使用BoostFrame了。这阶段的主要目标是测试在Iop开启的条件下，App冷启动和热启动时间的效率比，当然测试的话还是使用自动化脚本比较方便，第三阶段的主要目标是输出一个相关脚本，并输出一篇性能分析文档。
第三阶段结果展望：
| 预期日期 | 截止日期 | 结果 |
|---|---|---|
| 2022/11/24 | 2022/12/5 | 输出测试文档 |
集成之后结果不如人意，从原理上讲应该可以优化，但集成之后并没有优化。心累…