8.0,9.0平台system.img生成差分包解析

一、问题来源

二、问题探究

1、之前的固件查分流程

2、make生成system.img和make otapackage 生成 system.img的差别

3、一些简单的命令尝试

三、问题解决

四、问题优化

 

一、问题来源

    7.0之后system分区默认使用dm-verity，只能使用block做包，而出现了不能固件查分的情况，因为使用之前的方法获取的system.map文件跟我们的system源文件时不匹配的，block做包的关键是需要对应的system.map,如客户出现编译流程问题或者fota包丢失，而只能使用整包进行升级，所以这个问题还是值得探究

 

二、问题探究

首先我们还是跑一遍之前使用过的使用img文件制作差分包的流程

1、system.img sparse转换为ext4

sparse 和 ext4 是system.img的两种格式，目前默认编译出来的是sparse格式

执行命令：simg2img system.img system.raw

2、创建文件夹： mkdir system

3、执行mount挂载拿到file system: sudo mount -t ext4 system.raw ./system

这时候我们拿到了文件格式的system:

4、将拿到的system文件放入到ota_target_file.zip/SYSTEM/

删除ota_target_file.zip/IMAGES/

执行：./build/tools/releasetools/add_img_to_target_files.py -v ota_target_files.zip

将重新生成IMAGES文件夹和system.img文件

5、将刷机的system.img替换IMAGES/system.img

6、执行做包命令直接出现错误提示AssertionError

 

这种流程说明这种方式拿到的system.map是不对的，问题的关键在于需要拿到对应的map，接下来开始挖坑和填坑

首先我们先分析system.img的生成

1、make时候生成

# $(1): output file
define build-systemimage-target
  @echo "Target system fs image: $(1)"
  $(call create-system-vendor-symlink)
  $(call create-system-product-symlink)
  @mkdir -p $(dir $(1)) $(systemimage_intermediates) && rm -rf $(systemimage_intermediates)/system_image_info.txt
  $(call generate-userimage-prop-dictionary, $(systemimage_intermediates)/system_image_info.txt, \
      skip_fsck=true)
  $(hide) PATH=$(foreach p,$(INTERNAL_USERIMAGES_BINARY_PATHS),$(p):)$$PATH \
      build/make/tools/releasetools/build_image.py \
      $(TARGET_OUT) $(systemimage_intermediates)/system_image_info.txt $(1) $(TARGET_OUT) \
      || ( echo "Out of space? the tree size of $(TARGET_OUT) is (MB): " 1>&2 ;\
           du -sm $(TARGET_OUT) 1>&2;\
           if [ "$(INTERNAL_USERIMAGES_EXT_VARIANT)" == "ext4" ]; then \
               maxsize=$(BOARD_SYSTEMIMAGE_PARTITION_SIZE); \
               echo "The max is $$(( maxsize / 1048576 )) MB." 1>&2 ;\
           else \
               echo "The max is $$(( $(BOARD_SYSTEMIMAGE_PARTITION_SIZE) / 1048576 )) MB." 1>&2 ;\
           fi; \
           mkdir -p $(DIST_DIR); cp $(INSTALLED_FILES_FILE) $(DIST_DIR)/installed-files-rescued.txt; \
           exit 1 )
endef

核心方法为build-systemimage-target

build/make/tools/releasetools/build_image.py \

     $(TARGET_OUT) $(systemimage_intermediates)/system_image_info.txt $(1) $(TARGET_OUT) \

调用build_image.py文件传入四个参数

1、所需的文件格式system：./out/target/product/k39tv1_64_bsp/system/system/

2、所需要的一些参数：./out/target/product/k39tv1_64_bsp/obj/PACKAGING/systemimage_intermediates/system_image_info.txt

ext_mkuserimg=mkuserimg_mke2fs.sh
fs_type=ext4
system_size=2684354560
userdata_size=3221225472
cache_fs_type=ext4
cache_size=452984832
vendor_fs_type=ext4
vendor_size=578813952
extfs_sparse_flag=-s
squashfs_sparse_flag=-s
selinux_fc=out/target/product/k39tv1_64_bsp/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin
boot_signer=true
verity=true
verity_key=build/target/product/security/verity
verity_signer_cmd=verity_signer
verity_fec=true
system_verity_block_device=/dev/block/platform/bootdevice/by-name/system
vendor_verity_block_device=/dev/block/platform/bootdevice/by-name/vendor
system_root_image=true
ramdisk_dir=out/target/product/k39tv1_64_bsp/root
skip_fsck=true

 

3、生成指向文件：./out/target/product/k39tv1_64_bsp/obj/PACKAGING/systemimage_intermediates/system.img

4、用于读取一些特殊的配置信息：./out/target/product/k39tv1_64_bsp/system/

 

在build_image.py中执行main方法，进而执行，生成system.img

BuildImage(in_dir, image_properties, out_file, target_out)

 

 

2、make otapackage生成system.img

也就是我们使用的命令 ./build/tools/releasetools/add_img_to_target_files.py -v ota_target_files.zip

AddImagesToTargetFiles(args[0]) ----->>

AddSystem( output_zip, recovery_img=recovery_image, boot_img=boot_image)------>>

CreateImage(OPTIONS.input_tmp, OPTIONS.info_dict, "system", img,block_list=block_list)

核心方法CreateImage

1、OPTIONS.input_tmp：传入的tmp源文件

2、OPTIONS.info_dict： 解析的misc_info.txt中的信息作为生成固件的参数

3、"system", img 对应需要生成的固件

4、block_list=block_list  system，userdata，cache都要通过这个方法生成，所以这里设置block_list参数，如果需要生成map 我们传入对应的路径

 

带着问题进行分析

1、为什么同样都调用了BuildImage，但是make ota输出了system.map

首先这个方法一共三个参数中，输入，输出，参数，那么输入，输出可以理解为是相同的，那么关键就在于参数

的差异，没什么好的方法，打log和看代码

make 流程：

vendor_size=578813952

system_verity_block_device=/dev/block/platform/bootdevice/by-name/system

fs_type=ext4

vendor_verity_block_device=/dev/block/platform/bootdevice/by-name/vendor

boot_signer=true

skip_fsck=true

ext_mkuserimg=mkuserimg_mke2fs.sh

squashfs_sparse_flag=-s

system_root_image=true

selinux_fc=out/target/product/k39tv1_64_bsp/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin

system_size=2684354560

verity_signer_cmd=verity_signer

extfs_sparse_flag=-s

userdata_size=3221225472

verity=true

cache_fs_type=ext4

vendor_fs_type=ext4

verity_fec=true

ramdisk_dir=out/target/product/k39tv1_64_bsp/root

verity_key=build/target/product/security/verity

cache_size=45298483

 

def main(argv):
  if len(argv) != 4:
    print(__doc__)
    sys.exit(1)
  //指定传入参数
  in_dir = argv[0]
  glob_dict_file = argv[1]
  out_file = argv[2]
  target_out = argv[3]
  #把传入的system_info放入字典中
  glob_dict = LoadGlobalDict(glob_dict_file)
  #mount_point在system_info中是没有的，所以会进入下面的判断
  if "mount_point" in glob_dict:
    # The caller knows the mount point and provides a dictionay needed by
    # BuildImage().
    image_properties = glob_dict
  else:
    image_filename = os.path.basename(out_file)
    mount_point = ""
    if image_filename == "system.img":
      mount_point = "system"
    elif image_filename == "system_other.img":
      mount_point = "system_other"
    elif image_filename == "userdata.img":
      mount_point = "data"
    elif image_filename == "cache.img":
      mount_point = "cache"
    elif image_filename == "vendor.img":
      mount_point = "vendor"
    elif image_filename == "oem.img":
      mount_point = "oem"
    elif image_filename == "product.img":
      mount_point = "product"
    else:
      print("error: unknown image file name ", image_filename, file=sys.stderr)
      sys.exit(1)
    #获得当前system.img的image_prop属性
    image_properties = ImagePropFromGlobalDict(glob_dict, mount_point)

  if not BuildImage(in_dir, image_properties, out_file, target_out):
    print("error: failed to build %s from %s" % (out_file, in_dir),
          file=sys.stderr)
    sys.exit(1)

ImagePropFromGlobalDict

  def copy_prop(src_p, dest_p):
    """Copy a property from the global dictionary.
  ......
  ......
      
  d["mount_point"] = mount_point
  #执行了copy的动作如果我们手动放入system_info不在这里个参数中，是不会加到最后的image_prop中的
  if mount_point == "system":
    copy_prop("avb_system_hashtree_enable", "avb_hashtree_enable")
    copy_prop("avb_system_add_hashtree_footer_args",
              "avb_add_hashtree_footer_args")
    copy_prop("avb_system_key_path", "avb_key_path")
    copy_prop("avb_system_algorithm", "avb_algorithm")
    copy_prop("fs_type", "fs_type")
    # Copy the generic system fs type first, override with specific one if
    # available.
    copy_prop("system_fs_type", "fs_type")
    copy_prop("system_headroom", "partition_headroom")
    copy_prop("system_size", "partition_size")
    if not copy_prop("system_journal_size", "journal_size"):
      d["journal_size"] = "0"
    copy_prop("system_verity_block_device", "verity_block_device")
    copy_prop("system_root_image", "system_root_image")
    copy_prop("ramdisk_dir", "ramdisk_dir")
    copy_prop("ramdisk_fs_config", "ramdisk_fs_config")
    copy_prop("ext4_share_dup_blocks", "ext4_share_dup_blocks")
    copy_prop("system_squashfs_compressor", "squashfs_compressor")
    copy_prop("system_squashfs_compressor_opt", "squashfs_compressor_opt")
    copy_prop("system_squashfs_block_size", "squashfs_block_size")
    copy_prop("system_squashfs_disable_4k_align", "squashfs_disable_4k_align")
    copy_prop("system_base_fs_file", "base_fs_file")
    copy_prop("system_extfs_inode_count", "extfs_inode_count")
    if not copy_prop("system_extfs_rsv_pct", "extfs_rsv_pct"):
      d["extfs_rsv_pct"] = "0"
    ......
    ......

 

make otapackage流程:

creating system.img...

fs_config=/tmp/targetfiles-9pZIhQ/META/filesystem_config.txt

fs_type=ext4

hash_seed=12f7f181-06f3-5f0b-9930-e25c81f9a170

ramdisk_fs_config=/tmp/targetfiles-9pZIhQ/META/root_filesystem_config.txt

mount_point=system

uuid=8bb05621-b4e4-578a-9772-f45980d305ea

ext_mkuserimg=mkuserimg_mke2fs.sh

squashfs_sparse_flag=-s

system_root_image=true

verity=true

selinux_fc=/tmp/targetfiles-9pZIhQ/META/file_contexts.bin

timestamp=1230739200

verity_signer_cmd=verity_signer

extfs_sparse_flag=-s

block_list=/tmp/system-fhRUcs.map

skip_fsck=true

partition_name=system

journal_size=0

partition_size=2684354560

verity_block_device=/dev/block/platform/bootdevice/by-name/system

verity_fec=true

ramdisk_dir=/tmp/targetfiles-9pZIhQ/ROOT

extfs_rsv_pct=0

verity_key=build/target/product/security/verity

 

def CreateImage(input_dir, info_dict, what, output_file, block_list=None):
  print("creating " + what + ".img...")

  image_props = build_image.ImagePropFromGlobalDict(info_dict, what)
  fstab = info_dict["fstab"]
  mount_point = "/" + what
  if fstab and mount_point in fstab:
    image_props["fs_type"] = fstab[mount_point].fs_type

  # Use a fixed timestamp (01/01/2009) when packaging the image.
  # Bug: 24377993
  epoch = datetime.datetime.fromtimestamp(0)
  timestamp = (datetime.datetime(2009, 1, 1) - epoch).total_seconds()
  image_props["timestamp"] = int(timestamp)

  if what == "system":
    fs_config_prefix = ""
  else:
    fs_config_prefix = what + "_"

  fs_config = os.path.join(
      input_dir, "META/" + fs_config_prefix + "filesystem_config.txt")
  if not os.path.exists(fs_config):
    fs_config = None

  # Override values loaded from info_dict.
  if fs_config:
    image_props["fs_config"] = fs_config
  #block_list=/tmp/system-fhRUcs.map
  if block_list:
    image_props["block_list"] = block_list.name

  # Use repeatable ext4 FS UUID and hash_seed UUID (based on partition name and
  # build fingerprint).
  # 增加uuid 和 hash_seed参数给到image_prop
  uuid_seed = what + "-"
  if "build.prop" in info_dict:
    build_prop = info_dict["build.prop"]
    if "ro.build.fingerprint" in build_prop:
      uuid_seed += build_prop["ro.build.fingerprint"]
    elif "ro.build.thumbprint" in build_prop:
      uuid_seed += build_prop["ro.build.thumbprint"]
  image_props["uuid"] = str(uuid.uuid5(uuid.NAMESPACE_URL, uuid_seed))
  hash_seed = "hash_seed-" + uuid_seed
  image_props["hash_seed"] = str(uuid.uuid5(uuid.NAMESPACE_URL, hash_seed))
 
  for key,value in image_props.items():
    print('{key}={value}'.format(key = key, value = value))
   #调用BuildImage方法
  succ = build_image.BuildImage(os.path.join(input_dir, what.upper()),
                                image_props, output_file.name)
  assert succ, "build " + what + ".img image failed"

  output_file.Write()
  if block_list:
    block_list.Write()

  # Set the 'adjusted_partition_size' that excludes the verity blocks of the
  # given image. When avb is enabled, this size is the max image size returned
  # by the avb tool.
  is_verity_partition = "verity_block_device" in image_props
  verity_supported = (image_props.get("verity") == "true" or
                      image_props.get("avb_enable") == "true")
  is_avb_enable = image_props.get("avb_hashtree_enable") == "true"
  if verity_supported and (is_verity_partition or is_avb_enable):
    adjusted_blocks_value = image_props.get("partition_size")
    if adjusted_blocks_value:
      adjusted_blocks_key = what + "_adjusted_partition_size"
      info_dict[adjusted_blocks_key] = int(adjusted_blocks_value)/4096 - 1

 

对比我们发现，make ota会有map 是因为参数中有block_list，而如果我们直接加在system_image_info.txt中，根据原始代码也是无法生效的

 

2、尝试使用命令直接生成

 

def BuildImage(in_dir, prop_dict, out_file, target_out=None):
  """Build an image to out_file from in_dir with property prop_dict.
    ......
    ......
    
    # Run e2fsck on the inflated image file
    e2fsck_command = ["e2fsck", "-f", "-n", unsparse_image]
    (e2fsck_output, exit_code) = RunCommand(e2fsck_command)

    os.remove(unsparse_image)

    if exit_code != 0:
      print("Error: '%s' failed with exit code %d:\n%s" % (
          e2fsck_command, exit_code, e2fsck_output))
      return False

  return True

对应log:

可以看出真正执行的脚本为mkuserimg_mke2fs.sh生成system.img和map

 

所以我用命令做了尝试：

 ./out/host/linux-x86/bin/mkuserimg_mke2fs.sh  ./out/target/product/k39tv1_64_bsp/obj/PACKAGING/target_files_intermediates/full_k39tv1_64_bsp-target_files-eng.adups/SYSTEM/ system.img ext4 /system 2642440192 -j 0 -T 1230739200 -C out/target/product/k39tv1_64_bsp/obj/PACKAGING/target_files_intermediates/full_k39tv1_64_bsp-target_files-eng.adups/META/filesystem_config.txt -B system.map -L / -M 0 ./out/target/product/k39tv1_64_bsp/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin

 

发现可以直接生成system.map文件 而输出对应的system.img为ext4格式,也就是说，map文件的生成跟dm-verity无关，因为map是对应ext4格式中文件所占用的区间，而dm-verity是加在sprase img最后的，

我们使用这个生成的map尝试做包发现报错基本上全部的system下的文件 bad offset，也就是区间范围不对，但是这次没有报错system.map不匹配，所以我们可以尝试修改build_image.py文件重新生成对应file system的 system.map

 

三、问题解决

1、更改build_image.py

这段修改对流程没什么影响，只是可以使用common的option方法，这样我们单独执行或者编译调用可以看到log

def main(argv):
  #adupsfota start by libiaobiao 
  def option_handler(o, a):
    return True

  args = common.ParseOptions(
      argv, __doc__, extra_opts="",
      extra_long_opts=[],
      extra_option_handler=option_handler)

  if len(args) != 4:
    print(__doc__)
    sys.exit(1)

  in_dir = args[0]
  glob_dict_file = args[1]
  out_file = args[2]
  target_out = args[3]
  #adupsfota end by libiaobiao
  glob_dict = LoadGlobalDict(glob_dict_file)
  for key,value in glob_dict.items():
    print('{key}={value}'.format(key = key, value = value))

copy_prop加入copy_prop("block_list", "block_list")

def ImagePropFromGlobalDict(glob_dict, mount_point):
  """Build an image property dictionary from the global dictionary.

  d["mount_point"] = mount_point
  if mount_point == "system":
    #adupsfota start by libiaobiao 
    copy_prop("block_list", "block_list")
    #adupsfota end by libiaobiao 
    copy_prop("avb_system_hashtree_enable", "avb_hashtree_enable")
    copy_prop("avb_system_add_hashtree_footer_args",
              "avb_add_hashtree_footer_args")
    copy_prop("avb_system_key_path", "avb_key_path")
    copy_prop("avb_system_algorithm", "avb_algorithm")
    copy_prop("fs_type", "fs_type")
    # Copy the generic system fs type first, override with specific one if
    # available.
    copy_prop("system_fs_type", "fs_type")
    copy_prop("system_headroom", "partition_headroom")
    copy_prop("system_size", "partition_size")
    if not copy_prop("system_journal_size", "journal_size"):
      d["journal_size"] = "0"
    copy_prop("system_verity_block_device", "verity_block_device")
    copy_prop("system_root_image", "system_root_image")
    copy_prop("ramdisk_dir", "ramdisk_dir")
    copy_prop("ramdisk_fs_config", "ramdisk_fs_config")
    copy_prop("ext4_share_dup_blocks", "ext4_share_dup_blocks")
    copy_prop("system_squashfs_compressor", "squashfs_compressor")
    copy_prop("system_squashfs_compressor_opt", "squashfs_compressor_opt")
    copy_prop("system_squashfs_block_size", "squashfs_block_size")
    copy_prop("system_squashfs_disable_4k_align", "squashfs_disable_4k_align")
    copy_prop("system_base_fs_file", "base_fs_file")
    copy_prop("system_extfs_inode_count", "extfs_inode_count")
    if not copy_prop("system_extfs_rsv_pct", "extfs_rsv_pct"):
      d["extfs_rsv_pct"] = "0"

2、重新执行build_image.py生成system.map

./build/make/tools/releasetools/build_image.py -v ./out/target/product/k39tv1_64_bsp/system/system/ ./out/target/product/k39tv1_64_bsp/obj/PACKAGING/systemimage_intermediates/system_image_info.txt ./out/target/product/k39tv1_64_bsp/obj/PACKAGING/systemimage_intermediates/system.img ./out/target/product/k39tv1_64_bsp/system/

因为我们上面更改了代码 所以这里加入-v可以看到输出log

3、重新做包

把生成的system.map和源system.img放入到ota_target_file.zip/IMAGES中，生成差分包

4、测试验证升级OK