在Android环境下编译 bsdiff/bspatch 增量更新库
一、 概述
什么增量更新呢？比如：应用市场省流量更新软件，一个100M的apk可能只需要下载一个20M的增量包就能完成更新，不需要下载整个Apk。增量更新不仅限于apk。
本篇博客主要记录bsdiff/bspatch增量更新编译为so库的过程。分为2个部分提取增量文件和合并增量文件
准备工作
- bsdiff/bspatch下载路径
- bzip2下载路径
很多资料提到bzip2官方网站，现在未发现下载链接。不是本文重点，不再关注。 - Android Studio 含NDK
二、创建项目
1.bsdiff/bspatch
[外链图片转存失败(img-ZmBQU5rD-1567734596093)(https://github.com/zhangzhian/Bsdiff_Bspatch/blob/master/art/bsdiff:bspatch.jpeg?raw=true)]
解压bsdiff/bspatch下载好的文件，如上图所示。
重点关注1和2两个文件
- bsdiff.c 提取增量文件
- bspatch.c 合并增量文件
2. 创建Android 项目
首先是用Android Studio创建一个“include c++ support”带有c++支持的项目。需要用到NDK，NDK部分的内容若有疑问请自行搜索。
[外链图片转存失败(img-e7HwgzeZ-1567734596094)(https://github.com/zhangzhian/Bsdiff_Bspatch/blob/master/art/android.png?raw=true)]
将我们下载的bzip2的源文件和bsdiff/bspatch的源文件，放到Android项目中，如上图。
diff-patch-lib.cpp是项目自己生成的cpp，改了个名字。
3.修改CMakeList.txt
# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html
# Sets the minimum version of CMake required to build the native library.
cmake_minimum_required(VERSION 3.4.1)
# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.
add_library( # Sets the name of the library.
diff-patch
# Sets the library as a shared library.
SHARED
# Provides a relative path to your source file(s).
src/main/cpp/diff-patch-lib.cpp
src/main/cpp/bsdiff.c
src/main/cpp/bspatch.c
src/main/cpp/bzip2/blocksort.c
src/main/cpp/bzip2/bzip2.c
src/main/cpp/bzip2/bzip2recover.c
src/main/cpp/bzip2/bzlib.c
src/main/cpp/bzip2/compress.c
src/main/cpp/bzip2/crctable.c
src/main/cpp/bzip2/decompress.c
src/main/cpp/bzip2/huffman.c
src/main/cpp/bzip2/randtable.c
)
# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.
find_library( # Sets the name of the path variable.
log-lib
# Specifies the name of the NDK library that
# you want CMake to locate.
log )
# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.
target_link_libraries( # Specifies the target library.
diff-patch
# Links the target library to the log library
# included in the NDK.
${log-lib} )
我把要生成的库的名字改为diff-patch
add_library中添加下载的c源文件
4. 添加JNI接口
设计到jni的知识，不是本文重点，如果不了解请自行学习。
- 加载diff-patch库
- 提供了diff和patch两个方法
package com.yodosmart.bsdiffbspatch;
/**
* @Author: 张志安
* @Mail: [email protected]
* @Date: 2019/3/6 15:44
*/
public class DiffPatchUtil {
static {
System.loadLibrary("diff-patch");
}
/**
* native方法 比较路径为oldPath的文件与newPath的文件之间差异，并生成patch包，存储于patchPath
*
* 返回：0，说明操作成功
*
* @param oldPath 示例:/sdcard/old.apk
* @param newPath 示例:/sdcard/new.apk
* @param patchPath 示例:/sdcard/xx.patch
* @return
*/
public static native int diff(String oldPath, String newPath, String patchPath);
/**
* native方法 使用路径为oldPath的文件与路径为patchPath的补丁包，合成新的文件，并存储于newPath
*
* 返回：0，说明操作成功
*
* @param oldPath 示例:/sdcard/old.apk
* @param newPath 示例:/sdcard/new.apk
* @param patchPath 示例:/sdcard/xx.patch
* @return
*/
public static native int patch(String oldPath, String newPath,
String patchPath);
}
5、注意
后续会遇到的一些问题
- 添加权限
- diff-patch-lib.cpp include头文件的时候需要添加extern “C”，如下
extern "C"
{
#include "bsdiff.h"
#include "bspatch.h"
}
- bsdiff.c和bspatch.c中的
#if defined(BSDIFF_EXECUTABLE) #endif去掉 - bsdiff.c
#include "bzip2/bzlib.h" - bspatch.c
#include "bspatch.h" #include "bzip2/bzlib.h"
三、提取增量文件
首先我们来看如何提取增量文件
extern "C"
JNIEXPORT jint JNICALL
Java_com_yodosmart_bsdiffbspatch_DiffPatchUtil_diff(JNIEnv *env, jclass type, jstring oldPath_,
jstring newPath_, jstring patchPath_) {
const char *oldPath = env->GetStringUTFChars(oldPath_, 0);
const char *newPath = env->GetStringUTFChars(newPath_, 0);
const char *patchPath = env->GetStringUTFChars(patchPath_, 0);
int argc = 4;
char * argv[argc];
argv[0] = (char *)"bsdiff";
argv[1] = (char *)oldPath;
argv[2] = (char *)newPath;
argv[3] = (char *)patchPath;
int ret = bsdiff_main(argc,argv);
env->ReleaseStringUTFChars(oldPath_, oldPath);
env->ReleaseStringUTFChars(newPath_, newPath);
env->ReleaseStringUTFChars(patchPath_, patchPath);
return ret;
}
主要是用bsdiff_main(argc,argv)调用bsdiff库
源文件中含有main函数，我这里主要是把main函数改了名称，添加了注释
int bsdiff_main(int argc, char *argv[]) {
//文件句柄
int fd;
//bz2错误
int bz2err;
//老版本文件 新版本文件
uint8_t *old, *new;
//旧版本文件和新版本文件的大小
off_t oldsize, newsize;
//大小为8的buff
uint8_t buf[8];
//增量文件
FILE *pf;
//差分结构体
struct bsdiff_stream stream;
//bz2文件
BZFILE *bz2;
memset(&bz2, 0, sizeof(bz2));
stream.malloc = malloc;
stream.free = free;
stream.write = bz2_write;
if (argc != 4) errx(1, "usage: %s oldfile newfile patchfile\n", argv[0]);
/*
* Allocate oldsize+1 bytes instead of oldsize bytes to ensure
* that we never try to malloc(0) and get a NULL pointer
*
* 旧版本文件分配内存，读出文件内容
*
*
* int open(const char * pathname, int flags, mode_t mode);
*
* 参数 pathname 指向欲打开的文件路径字符串
* 参数 flags 为文件的打开方式 O_RDONLY 以只读方式打开文件
* 参数 mode 只有在建立新版本文件时才会生效, 真正建文件时的权限会受到umask值所影响, 因此该文件权限应该为 (mode-umaks).
*
* 返回值：成功则返回文件句柄，否则返回-1
*
*
* off_t lseek(int filedes, off_t offset, int whence);
*
* 参数 offset 的含义取决于参数 whence：
* 1. 如果 whence 是 SEEK_SET，文件偏移量将被设置为 offset。
* 2. 如果 whence 是 SEEK_CUR，文件偏移量将被设置为 cfo 加上 offset，
* offset 可以为正也可以为负。
* 3. 如果 whence 是 SEEK_END，文件偏移量将被设置为文件长度加上 offset，
* offset 可以为正也可以为负。
*
* 返回值：新的偏移量（成功），-1（失败）
*
*
* ssize_t read(int fd, void * buf, size_t count);
*
* 参数 void *buf 读上来的数据保存在缓冲区buf中，同时文件的当前读写位置向后移
* 参数 size_t count 是请求读取的字节数。若参数count 为0, 则read()不会有作用并返回0.
*
* 返回值：为实际读取到的字节数
*
*/
if (((fd = open(argv[1], O_RDONLY, 0)) < 0) ||
((oldsize = lseek(fd, 0, SEEK_END)) == -1) ||
((old = malloc(oldsize + 1)) == NULL) ||
(lseek(fd, 0, SEEK_SET) != 0) ||
(read(fd, old, oldsize) != oldsize) ||
(close(fd) == -1))
err(1, "%s", argv[1]);
/*
* Allocate newsize+1 bytes instead of newsize bytes to ensure
* that we never try to malloc(0) and get a NULL pointer
*
* 新版本文件分配内存，读取文件内存
*
*/
if (((fd = open(argv[2], O_RDONLY, 0)) < 0) ||
((newsize = lseek(fd, 0, SEEK_END)) == -1) ||
((new = malloc(newsize + 1)) == NULL) ||
(lseek(fd, 0, SEEK_SET) != 0) ||
(read(fd, new, newsize) != newsize) ||
(close(fd) == -1))
err(1, "%s", argv[2]);
/*
* Create the patch file
*
* 创建一个patch文件
*/
if ((pf = fopen(argv[3], "w")) == NULL)
err(1, "%s", argv[3]);
/**
* Write header (signature+newsize)
*
* 写头部
*
* size_t fwrite(const void* buffer, size_t size, size_t count, FILE* stream);
*
* 返回值：返回实际写入的数据块数目
*（1）buffer：是一个指针，对fwrite来说，是要获取数据的地址；
*（2）size：要写入内容的单字节数；
*（3）count:要进行写入size字节的数据项的个数；
*（4）stream:目标文件指针；
*
*/
offtout(newsize, buf);
if (fwrite("ENDSLEY/BSDIFF43", 16, 1, pf) != 1 ||
fwrite(buf, sizeof(buf), 1, pf) != 1)
err(1, "Failed to write header");
/**
* 以bz2的方式打开增量文件
*/
if (NULL == (bz2 = BZ2_bzWriteOpen(&bz2err, pf, 9, 0, 0)))
errx(1, "BZ2_bzWriteOpen, bz2err=%d", bz2err);
/**
* 差分算法
*/
stream.opaque = bz2;
if (bsdiff(old, oldsize, new, newsize, &stream))
err(1, "bsdiff");
/**
* 关闭文件
*/
BZ2_bzWriteClose(&bz2err, bz2, 0, NULL, NULL);
if (bz2err != BZ_OK)
err(1, "BZ2_bzWriteClose, bz2err=%d", bz2err);
if (fclose(pf))
err(1, "fclose");
/*
* Free the memory we used
*
* 释放使用的内存
*/
free(old);
free(new);
return 0;
}
调用提取增量文件
int result = DiffPatchUtil.patch(oldpath,patchNewPath,patchPath);
四、合并增量文件
接下来我们看如何合并增量文件
extern "C"
JNIEXPORT jint JNICALL
Java_com_yodosmart_bsdiffbspatch_DiffPatchUtil_patch(JNIEnv *env, jclass type, jstring oldPath_,
jstring newPath_, jstring patchPath_) {
const char *oldPath = env->GetStringUTFChars(oldPath_, 0);
const char *newPath = env->GetStringUTFChars(newPath_, 0);
const char *patchPath = env->GetStringUTFChars(patchPath_, 0);
int argc = 4;
char * argv[argc];
argv[0] = (char *)"bspatch";
argv[1] = (char *)oldPath;
argv[2] = (char *)newPath;
argv[3] = (char *)patchPath;
int ret = bspatch_main(argc,argv);
env->ReleaseStringUTFChars(oldPath_, oldPath);
env->ReleaseStringUTFChars(newPath_, newPath);
env->ReleaseStringUTFChars(patchPath_, patchPath);
return ret;
主要是调用bspatch库中的bspatch_main(argc,argv);合并增量文件
int bspatch_main(int argc, char * argv[])
{
//增量文件
FILE * f;
// 老版本文件
int fd;
//bz2错误
int bz2err;
//16+8
uint8_t header[24];
uint8_t *old, *new;
int64_t oldsize, newsize;
BZFILE* bz2;
struct bspatch_stream stream;
struct stat sb;
if(argc!=4) errx(1,"usage: %s oldfile newfile patchfile\n",argv[0]);
/**
* Open patch file
* 打开增量文件
*/
if ((f = fopen(argv[3], "r")) == NULL)
err(1, "fopen(%s)", argv[3]);
/**
* Read header
* 读头部
*/
if (fread(header, 1, 24, f) != 24) {
if (feof(f))
errx(1, "Corrupt patch\n");
err(1, "fread(%s)", argv[3]);
}
/**
* Check for appropriate magic
*
* int memcmp(const void *buf1, const void *buf2, unsigned int count);
* 比较内存区域buf1和buf2的前count个字节。
*/
if (memcmp(header, "ENDSLEY/BSDIFF43", 16) != 0)
errx(1, "Corrupt patch\n");
/**
* Read lengths from header
* 与差分offtout对应
* 读新版本文件的长度
*/
newsize=offtin(header+16);
if(newsize<0)
errx(1,"Corrupt patch\n");
/**
* 打开老版本文件
*/
if(((fd=open(argv[1],O_RDONLY,0))<0) ||
((oldsize=lseek(fd,0,SEEK_END))==-1) ||
((old=malloc(oldsize+1))==NULL) ||
(lseek(fd,0,SEEK_SET)!=0) ||
(read(fd,old,oldsize)!=oldsize) ||
(fstat(fd, &sb)) ||
(close(fd)==-1)) err(1,"%s",argv[1]);
/**
* 给新版本文件分配内存
*/
if((new=malloc(newsize+1))==NULL) err(1,NULL);
/**
* Close patch file and re-open it via libbzip2 at the right places
* 关闭增量文件并且通过libbzip2重新打开
*/
if (NULL == (bz2 = BZ2_bzReadOpen(&bz2err, f, 0, 0, NULL, 0)))
errx(1, "BZ2_bzReadOpen, bz2err=%d", bz2err);
stream.read = bz2_read;
stream.opaque = bz2;
/**
* 增量算法
*/
if (bspatch(old, oldsize, new, newsize, &stream))
errx(1, "bspatch");
/**
* Clean up the bzip2 reads
* 关闭bzip2的读
*/
BZ2_bzReadClose(&bz2err, bz2);
fclose(f);
/**
* Write the new file
* 写新文件
*/
if(((fd=open(argv[2],O_CREAT|O_TRUNC|O_WRONLY,sb.st_mode))<0) ||
(write(fd,new,newsize)!=newsize) || (close(fd)==-1))
err(1,"%s",argv[2]);
free(new);
free(old);
return 0;
}
同样是修改源码中的mian函数。
调用增量文件
int result = DiffPatchUtil.patch(oldpath,patchNewPath,patchPath);
五、测试验证
我们使用了ipc021501.apk和ipc030601.apk两个安装包进行测试
String path = Environment.getExternalStorageDirectory().getPath();
final String oldpath = path + "/新文件夹/" + "ipc021501.apk" ;
final String newpath = path + "/新文件夹/" + "ipc030601.apk";
final String patchPath = path + "/新文件夹/" + "ipc.patch";
final String patchNewPath = path + "/新文件夹/" + "ipc030602.apk";
依次调用
DiffPatchUtil.diff和DiffPatchUtil.patch
Thread thread = new Thread(new Runnable() {
@Override
public void run() {
int result = DiffPatchUtil.diff(oldpath,newpath,patchPath);
Log.e("zza",result+"");
}
});
thread.start();
Thread thread = new Thread(new Runnable() {
@Override
public void run() {
int result = DiffPatchUtil.patch(oldpath,patchNewPath,patchPath);
Log.e("zza",result+"");
}
});
thread.start();
生成文件ipc.patch和ipc030602.apk
如何对比ipc030601.apk和ipc030602.apk是否一致呢，我们可以使用MD5
[外链图片转存失败(img-q7FvOI9Y-1567734596095)(https://github.com/zhangzhian/Bsdiff_Bspatch/blob/master/art/apk%E5%B7%AE%E5%88%86md5.png?raw=true)]
可以看到两个文件的md5至一致。我们可以认为两个apk包完全一样。
六、总结
如果只是单纯的要使用该功能，可以直接将生成的so文件拷入，直接loadLibrary使用即可。
文本源码
生成的so库
欢迎关注我的公众号，持续分析优质技术文章
