android P OTA （MTK）初探 —— 3、基于块（Block）的OTA：升级包的制作流程

上文简单介绍了Target包，本文重点分析完整升级包。

一、OTA的流程图（转）

网上看到的一份不错的流程图。

二、升级包制作命令

制作升级包需要用到alps/build/tools/releasetools/ota_from_target_files.py这个脚本文件。可以配置很多参数。
但此命令必然有两个不带前缀的参数：要升级到的版本对应的Target包 和 要生成的升级包的名字。

一般需要的命令格式如下：

./build/tools/releasetools/ota_from_target_files -v --block -k ./build/target/product/security/releasekey -i ./out/ota_old.zip ./out/ota_new.zip ./out/update.zip

其中：

-v              表示显示出当前执行的代码的行号。
--block         代码生成基于块的升级包，其实已经没有意义了。android P的代码，不再支持基于文件的升级包。Keeping this flag here to not break existing callers.这是google给的解释。
-k              表示用后面紧跟的密钥重新签名升级包。
-i              表示后面紧跟的文件是旧版本的Target包，即 此命令是要生成增量升级包，而不是完整升级包。

其实还有很多参数可以添加，到脚本中查看一下就一目了然了。比如做增量升级包的时候添加–log_diff，做完增量升级包后，运行脚本target_files_diff.py 打印出差异的log。

三、升级包主要涉及的文件

首先就是之前文件提到的Target包涉及的文件：
1、alsp/build/core/Makefile
2、target包中的ota_update_list.txt

做升级包需要的：
1、alps/build/tools/releasetools/ota_from_target_files (这其实是链向同目录下的ota_from_target_files.py的软链接)
2、alps/build/tools/releasetools/common.py
3、alps/build/tools/releasetools/edify_generator.py
4、过程中生成的脚本文件updater-script，最终在升级包的META-INF/com/google/android目录下
5、从alps/vendor/mediatek/proprietary/scripts/releasetools/releasetools.py拷贝到Target包中的releasetools.py

四、重点函数分析：

1、ota_from_target_files.py中的main函数：

def main(argv):

  def option_handler(o, a):
    if o in ("-k", "--package_key"):
      OPTIONS.package_key = a
    elif o in ("-i", "--incremental_from"):
      OPTIONS.incremental_source = a
    elif o == "--full_radio":
      OPTIONS.full_radio = True
    elif o == "--full_bootloader":
      OPTIONS.full_bootloader = True
    elif o == "--wipe_user_data":
      OPTIONS.wipe_user_data = True
    elif o == "--downgrade":
      OPTIONS.downgrade = True
      OPTIONS.wipe_user_data = True
    elif o == "--override_timestamp":
      OPTIONS.downgrade = True
    elif o in ("-o", "--oem_settings"):
      OPTIONS.oem_source = a.split(',')
    elif o == "--oem_no_mount":
      OPTIONS.oem_no_mount = True
    elif o in ("-e", "--extra_script"):
      OPTIONS.extra_script = a
    elif o in ("-t", "--worker_threads"):
      if a.isdigit():
        OPTIONS.worker_threads = int(a)
      else:
        raise ValueError("Cannot parse value %r for option %r - only "
                         "integers are allowed." % (a, o))
    elif o in ("-2", "--two_step"):
      OPTIONS.two_step = True
    elif o == "--include_secondary":
      OPTIONS.include_secondary = True
    elif o == "--no_signing":
      OPTIONS.no_signing = True
    elif o == "--verify":
      OPTIONS.verify = True
    elif o == "--block":
      OPTIONS.block_based = True
    elif o in ("-b", "--binary"):
      OPTIONS.updater_binary = a
    elif o == "--stash_threshold":
      try:
        OPTIONS.stash_threshold = float(a)
      except ValueError:
        raise ValueError("Cannot parse value %r for option %r - expecting "
                         "a float" % (a, o))
    elif o == "--log_diff":
      OPTIONS.log_diff = a
    elif o == "--payload_signer":
      OPTIONS.payload_signer = a
    elif o == "--payload_signer_args":
      OPTIONS.payload_signer_args = shlex.split(a)
    elif o == "--extracted_input_target_files":
      OPTIONS.extracted_input = a
    elif o == "--skip_postinstall":
      OPTIONS.skip_postinstall = True
    else:
      return False
    return True

##lyc
#进行参数解析，调用common.py中的方法解析，并用上面的函数option_handler添加一些ParseOptions本身不支持的参数。
  args = common.ParseOptions(argv, __doc__,
                             extra_opts="b:k:i:d:e:t:2o:",
                             extra_long_opts=[
                                 "package_key=",
                                 "incremental_from=",
                                 "full_radio",
                                 "full_bootloader",
                                 "wipe_user_data",
                                 "downgrade",
                                 "override_timestamp",
                                 "extra_script=",
                                 "worker_threads=",
                                 "two_step",
                                 "include_secondary",
                                 "no_signing",
                                 "block",
                                 "binary=",
                                 "oem_settings=",
                                 "oem_no_mount",
                                 "verify",
                                 "stash_threshold=",
                                 "log_diff=",
                                 "payload_signer=",
                                 "payload_signer_args=",
                                 "extracted_input_target_files=",
                                 "skip_postinstall",
                             ], extra_option_handler=option_handler)

##必然是2，target.zip  和 update.zip
  if len(args) != 2:
    common.Usage(__doc__)
    sys.exit(1)

#上面部分都是在处理我们调用此脚本时带的参数信息。

#如果是要做降级的OTA，则只能做差分包；不然，任意的版本都能用此包降级
  if OPTIONS.downgrade:
    # We should only allow downgrading incrementals (as opposed to full).
    # Otherwise the device may go back from arbitrary build with this full
    # OTA package.
    if OPTIONS.incremental_source is None:
      raise ValueError("Cannot generate downgradable full OTAs")

  # Load the build info dicts from the zip directly or the extracted input
  # directory. We don't need to unzip the entire target-files zips, because they
  # won't be needed for A/B OTAs (brillo_update_payload does that on its own).
  # When loading the info dicts, we don't need to provide the second parameter
  # to common.LoadInfoDict(). Specifying the second parameter allows replacing
  # some properties with their actual paths, such as 'selinux_fc',
  # 'ramdisk_dir', which won't be used during OTA generation.
#从压缩包里解压缩出一个文件的方法是使用ZipFile的read方法
#import zipfile  
#z = zipfile.ZipFile(filename, 'r')  
#print z.read(z.namelist()[0]) 
#这样就读取出z.namelist()中的第一个文件，并且输出到屏幕，当然也可以把它存储到文件。
  if OPTIONS.extracted_input is not None:
    OPTIONS.info_dict = common.LoadInfoDict(OPTIONS.extracted_input)
  else:
    with zipfile.ZipFile(args[0], 'r') as input_zip:
	# 主要是解析例如以下三个文件，并将文件内容以(k,v)键值对形式保存到OPTIONS.info_dict中
    # 三个文件各自是：
    # 1. META/misc_info.txt
    # 2. SYSTEM/build.prop
    # 3. RECOVERY/RAMDISK/etc/recovery.fstab
    #这个info_dict贯穿全场
      OPTIONS.info_dict = common.LoadInfoDict(input_zip)

  if OPTIONS.verbose:
    print("--- target info ---")
    common.DumpInfoDict(OPTIONS.info_dict)

##lyc
#如果是做差分包，就把old target包的info_dict也解析出来
  # Load the source build dict if applicable.
  if OPTIONS.incremental_source is not None:
    OPTIONS.target_info_dict = OPTIONS.info_dict
    with zipfile.ZipFile(OPTIONS.incremental_source, 'r') as source_zip:
      OPTIONS.source_info_dict = common.LoadInfoDict(source_zip)

    if OPTIONS.verbose:
      print("--- source info ---")
      common.DumpInfoDict(OPTIONS.source_info_dict)

  # Load OEM dicts if provided. 这里指定的文件的内容应该都是param=value的形式（除了#开头的注释）
  OPTIONS.oem_dicts = _LoadOemDicts(OPTIONS.oem_source)

#是否是制作A/B系统的包
  ab_update = OPTIONS.info_dict.get("ab_update") == "true"


##lyc
#获取签名
  # Use the default key to sign the package if not specified with package_key.
  # package_keys are needed on ab_updates, so always define them if an
  # ab_update is getting created.
  if not OPTIONS.no_signing or ab_update:
    if OPTIONS.package_key is None:
      OPTIONS.package_key = OPTIONS.info_dict.get(
          "default_system_dev_certificate",
          "build/target/product/security/testkey")
    # Get signing keys
#GetKeyPasswords的作用是：
#Given a list of keys, prompt the user to enter passwords for
#  those which require them（有的签名文件是被加密了的，此时需要用户输入密码）.  Return a {key: password} dict.  password
#  will be None if the key has no password.
    OPTIONS.key_passwords = common.GetKeyPasswords([OPTIONS.package_key])
    
#A/B系统的，到这里就结束了。
  if ab_update:
    WriteABOTAPackageWithBrilloScript(
        target_file=args[0],
        output_file=args[1],
        source_file=OPTIONS.incremental_source)

    print("done.")
    return


  # Sanity check the loaded info dicts first.
  if OPTIONS.info_dict.get("no_recovery") == "true":
    raise common.ExternalError(
        "--- target build has specified no recovery ---")

#检测cache分区，其实像广升OTA，下载升级包用的都是data分区，cache分区实在太小了。不过，cache分区在升级过程中也会用到。
  # Non-A/B OTAs rely on /cache partition to store temporary files.
  cache_size = OPTIONS.info_dict.get("cache_size")
  if cache_size is None:
    print("--- can't determine the cache partition size ---")
  OPTIONS.cache_size = cache_size


##lyc
#这里的extra script，开发者自己添加的脚本，在更新脚本末尾（各种img都添加完毕，但还没有unremount的时候执行）会追加这个文件的内容。
#  -e  (--extra_script)  
#      Insert the contents of file at the end of the update script.
  if OPTIONS.extra_script is not None:
    OPTIONS.extra_script = open(OPTIONS.extra_script).read()


  if OPTIONS.extracted_input is not None:
    OPTIONS.input_tmp = OPTIONS.extracted_input
  else:
    print("unzipping target target-files...")
    OPTIONS.input_tmp = common.UnzipTemp(args[0], UNZIP_PATTERN)
  OPTIONS.target_tmp = OPTIONS.input_tmp


##lyc
#获取releasetools.py
#是一个比较重要的文件，也可以用 参数 -s 指定，或者用默认的。下面是具体的获取策略。
  # If the caller explicitly specified the device-specific extensions path via
  # -s / --device_specific, use that. Otherwise, use META/releasetools.py if it
  # is present in the target target_files. Otherwise, take the path of the file
  # from 'tool_extensions' in the info dict and look for that in the local
  # filesystem, relative to the current directory.
  if OPTIONS.device_specific is None:
    from_input = os.path.join(OPTIONS.input_tmp, "META", "releasetools.py")
    if os.path.exists(from_input):
      print("(using device-specific extensions from target_files)")
      OPTIONS.device_specific = from_input
    else:
      OPTIONS.device_specific = OPTIONS.info_dict.get("tool_extensions")
  if OPTIONS.device_specific is not None:
    OPTIONS.device_specific = os.path.abspath(OPTIONS.device_specific)



#去做整包
  # Generate a full OTA.
  if OPTIONS.incremental_source is None:
    with zipfile.ZipFile(args[0], 'r') as input_zip:
      WriteFullOTAPackage(
          input_zip,
          output_file=args[1])



#去做差分包
  # Generate an incremental OTA.
  else:
    print("unzipping source target-files...")
    OPTIONS.source_tmp = common.UnzipTemp(
        OPTIONS.incremental_source, UNZIP_PATTERN)
    with zipfile.ZipFile(args[0], 'r') as input_zip, \
        zipfile.ZipFile(OPTIONS.incremental_source, 'r') as source_zip:
      WriteBlockIncrementalOTAPackage(
          input_zip,
          source_zip,
          output_file=args[1])
#如果有参数--log_diff，做完差分包后，运行脚本target_files_diff.py 打印出差异的log。
    if OPTIONS.log_diff:
      with open(OPTIONS.log_diff, 'w') as out_file:
        import target_files_diff
        target_files_diff.recursiveDiff(
            '', OPTIONS.source_tmp, OPTIONS.input_tmp, out_file)


##lyc
#疑问 在哪儿签名的？
#在具体做包函数的最后的FinalizeMetadata函数中会调用SignOutput，进行签名。

  print("done.")

2、制作完整升级包用的WriteFullOTAPackage函数：

def WriteFullOTAPackage(input_zip, output_file):
  target_info = BuildInfo(OPTIONS.info_dict, OPTIONS.oem_dicts)

  # We don't know what version it will be installed on top of. We expect the API
  # just won't change very often. Similarly for fstab, it might have changed in
  # the target build.
  target_api_version = target_info["recovery_api_version"]
  
  #这里引入了一个新的模块edify_generator,并且抽象一个脚本生成器，用来生成edify脚本。
  #这里的脚本指的就是updater-script安装脚本，它是一个文本文件。
  #edify有两个主要的文件。这些文件可以在最终的升级包update.zip文件内的META-INF/com/google/android文件夹中找到。①update-binary -- 当用户选择刷入update.zip（通常是在恢复模式中）时所执行的二进制解释器，这个文件就是生成target包时提到的放在OTA/bin/目录下的那个updater文件。②updater-script -- 安装脚本，它是一个文本文件。
  #那么edify是什么呢？
  #edify是用于从.zip文件中安装CyanogenMod和其它软件的简单脚本语言。edify脚本不一定是用于更新固件。它可以用来替换/添加/删除特定的文件，甚至格式分区。通常情况下，edify脚本运行于用户在恢复模式中选择“刷写zip”时。
  
  script = edify_generator.EdifyGenerator(target_api_version, target_info)

  if target_info.oem_props and not OPTIONS.oem_no_mount:
    target_info.WriteMountOemScript(script)

#创建一个元数据字典用来封装更新包的相关系统属性，如ro.build.fingerprint（系统指纹），"ro.build.date.utc",（系统编译的时间(数字版),没必要修改）等。即最终update.zip包中的META-INF/com/android/metadata. 
  metadata = GetPackageMetadata(target_info)

  if not OPTIONS.no_signing:
    staging_file = common.MakeTempFile(suffix='.zip')
  else:
    staging_file = output_file

#这就是我们要生成的升级包文件
  output_zip = zipfile.ZipFile(
      staging_file, "w", compression=zipfile.ZIP_DEFLATED)

#返回一个以这些参数为属性的类DeviceSpecificParams的对象device_specific
#即：获得一些环境变量，封装在DEviceSpecificParams类当中，这是一个封装了设备特定属性的类；
  device_specific = common.DeviceSpecificParams(
      input_zip=input_zip,
      input_version=target_api_version,
      output_zip=output_zip,
      script=script,
      input_tmp=OPTIONS.input_tmp,
      metadata=metadata,
      info_dict=OPTIONS.info_dict)

#还没看懂这个是什么作用，竟然用assert，为什么必须要有recovery？。坑（4）
  assert HasRecoveryPatch(input_zip)

#下面这段代码我们可以理解为不允许降级，也就是说在脚本中的这段Assert语句，使得update zip包只能用于升级旧版本。其实，是不允许使用旧的软件包进行ota，如果要降级，也要重新编译低等级（比如，基于之前的某个git提交点）的软件包。
  # Assertions (e.g. downgrade check, device properties check).
  ts = target_info.GetBuildProp("ro.build.date.utc")
  ts_text = target_info.GetBuildProp("ro.build.date")
  script.AssertOlderBuild(ts, ts_text)

#下面的Assert语句，表示update zip包只能用于同一设备，即目标设备的 ro.product.device 必须跟update.zip中的相同。
  target_info.WriteDeviceAssertions(script, OPTIONS.oem_no_mount)
  
  
  #回调函数，用于调用设备相关代码。经过跟踪查看，这里是调用脚本releasetools.py中的对应函数。当然，如果没定义这个函数就什么也不做。
  #比如后面的device_specific.FullOTA_InstallBegin()和device_specific.FullOTA_InstallEnd()
  
  device_specific.FullOTA_Assertions()

#这里的策略涉及到如何更新recovery才能用新的recovery进行系统更新：
#如果要更新recovery分区，就先更新recovery，在重启，用新的recovery进行升级操作。升级recovery需要先将recovery放入/boot分区中，再写入到Recovery分区，然后重启。
#至于，这里的代码逻辑为啥是先2/3、3/3 最后再处理1/3，这最好通过查看最终生成的update-script文件来分析。因为有if else控制，所以是正确的。坑（5）
  # Two-step package strategy (in chronological order, which is *not*
  # the order in which the generated script has things):
  #
  # if stage is not "2/3" or "3/3":
  #    write recovery image to boot partition
  #    set stage to "2/3"
  #    reboot to boot partition and restart recovery
  # else if stage is "2/3":
  #    write recovery image to recovery partition
  #    set stage to "3/3"
  #    reboot to recovery partition and restart recovery
  # else:
  #    (stage must be "3/3")
  #    set stage to ""
  #    do normal full package installation:
  #       wipe and install system, boot image, etc.
  #       set up system to update recovery partition on first boot
  #    complete script normally
  #    (allow recovery to mark itself finished and reboot)


#从target包中获取recovery.img，boot.img也是用这个方法获取的
  recovery_img = common.GetBootableImage("recovery.img", "recovery.img",
                                         OPTIONS.input_tmp, "RECOVERY")
										
#用户输入了-2 这个参数，则先更新Recovery，在用新的Recovery更新主系统。					
  if OPTIONS.two_step:
    if not target_info.get("multistage_support"):
      assert False, "two-step packages not supported by this build"
    fs = target_info["fstab"]["/misc"]
    assert fs.fs_type.upper() == "EMMC", \
        "two-step packages only supported on devices with EMMC /misc partitions"
    bcb_dev = {"bcb_dev": fs.device}
	#将recovery_img.data写入到output_zip文件文件名为recovery.img。如果不更新recovery，就不会写进去。
    common.ZipWriteStr(output_zip, "recovery.img", recovery_img.data)
    script.AppendExtra("""
if get_stage("%(bcb_dev)s") == "2/3" then
""" % bcb_dev)

    # Stage 2/3: Write recovery image to /recovery (currently running /boot).
    script.Comment("Stage 2/3")
	#输出脚本，用于将Recovery.img写入到挂载点为/recovery的分区。recovery 和 boot 都是这样更新的。
    script.WriteRawImage("/recovery", "recovery.img")
    script.AppendExtra("""
set_stage("%(bcb_dev)s", "3/3");
reboot_now("%(bcb_dev)s", "recovery");
else if get_stage("%(bcb_dev)s") == "3/3" then
""" % bcb_dev)

    # Stage 3/3: Make changes.
    script.Comment("Stage 3/3")
#end of Two-step package strategy


  # Dump fingerprints
  script.Print("Target: {}".format(target_info.fingerprint))


#开始整包升级
  device_specific.FullOTA_InstallBegin()

  system_progress = 0.75
  if OPTIONS.wipe_user_data:
    system_progress -= 0.1
  if HasVendorPartition(input_zip):
    system_progress -= 0.1
  script.ShowProgress(system_progress, 0)

  # See the notes in WriteBlockIncrementalOTAPackage().
  allow_shared_blocks = target_info.get('ext4_share_dup_blocks') == "true"

  # Full OTA is done as an "incremental" against an empty source image. This
  # has the effect of writing new data from the package to the entire
  # partition, but lets us reuse the updater code that writes incrementals to
  # do it.
  #处理system分区，后面的vendor分区也是这么处理的。
  system_tgt = common.GetSparseImage("system", OPTIONS.input_tmp, input_zip,
                                     allow_shared_blocks)
  system_tgt.ResetFileMap()
  system_diff = common.BlockDifference("system", system_tgt, src=None)
  system_diff.WriteScript(script, output_zip)

#处理boot分区
  boot_img = common.GetBootableImage(
      "boot.img", "boot.img", OPTIONS.input_tmp, "BOOT")

  if HasVendorPartition(input_zip):
    script.ShowProgress(0.1, 0)

    vendor_tgt = common.GetSparseImage("vendor", OPTIONS.input_tmp, input_zip,
                                       allow_shared_blocks)
    vendor_tgt.ResetFileMap()
    vendor_diff = common.BlockDifference("vendor", vendor_tgt)
    vendor_diff.WriteScript(script, output_zip)

  AddCompatibilityArchiveIfTrebleEnabled(input_zip, output_zip, target_info)

  common.CheckSize(boot_img.data, "boot.img", target_info)
  common.ZipWriteStr(output_zip, "boot.img", boot_img.data)
  script.ShowProgress(0.05, 5)
  script.WriteRawImage("/boot", "boot.img")
  script.ShowProgress(0.2, 10)
  
  
  
  #这里调用了releasetools.py中的FullOTA_InstallEnd函数。像tee.img lk.img vbmeta.img preloader.img 等都是在这里添加进升级包的。
  device_specific.FullOTA_InstallEnd()


#执行额外的脚本
  if OPTIONS.extra_script is not None:
    script.AppendExtra(OPTIONS.extra_script)

  script.UnmountAll()

#是否要清理data分区？
  if OPTIONS.wipe_user_data:
    script.ShowProgress(0.1, 10)
    script.FormatPartition("/data")
	

  if OPTIONS.two_step:
    script.AppendExtra("""
set_stage("%(bcb_dev)s", "");
""" % bcb_dev)
    script.AppendExtra("else\n")
    # Stage 1/3: Nothing to verify for full OTA. Write recovery image to /boot.
    script.Comment("Stage 1/3")
	#这里是将recovery.img写入到了boot.img,是为了：
	#  In two-step OTAs, we write recovery image to /boot as the first step so that we can reboot to there and install a new recovery image to /recovery.
	#可是这个为什么是在后面才做？？？同坑（5）
    _WriteRecoveryImageToBoot(script, output_zip)
    script.AppendExtra("""
set_stage("%(bcb_dev)s", "2/3");
reboot_now("%(bcb_dev)s", "");
endif;
endif;
""" % bcb_dev)



  script.SetProgress(1)
  
  
  #在OTA包中写入updater-script 和 update-binary
  script.AddToZip(input_zip, output_zip, input_path=OPTIONS.updater_binary)
  metadata["ota-required-cache"] = str(script.required_cache)


  # We haven't written the metadata entry, which will be done in
  # FinalizeMetadata.
  common.ZipClose(output_zip)
  needed_property_files = (
      NonAbOtaPropertyFiles(),
  )
  
  ##将metadata信息写入到output_zip的META-INF/com/android/metadata文件里，并进行签名
  FinalizeMetadata(metadata, staging_file, output_file, needed_property_files)