Tensorflow C++ so库 for Android 编译全记录

关于tf C++ so库 for Android 编译全记录

0. 前言

本文涉及自己电脑路径的名称未改动（懒得改了）。另外本文于2018年初撰写，当时迫于调参爸爸用的是tf，所以硬着头皮去啃这块骨头，当然我是Pytorch党。一年过去，不知道这篇文章还值不值钱（？）

1. 清理bazel工作环境。

bazel clean --expunge 

bazel clean只清理BUILD等配置文件，不清理、不重置源码文件（*.cc,*.h）

2. 配置ndk版本为15

因为ndk15的编译器为clang5.0.300080，如果clang版本<5，头文件就有点偏老，会出很多bug。而且目前bazel版本最高支持ndk15。

配置./configure文件：

WARNING: Running Bazel server needs to be killed, because the startup options are different.
You have bazel 0.14.1 installed.
Please specify the location of python. [Default is /home/unaguo/anaconda2/bin/python]: 


Found possible Python library paths:
  /home/unaguo/anaconda2/lib/python2.7/site-packages
Please input the desired Python library path to use.  Default is [/home/unaguo/anaconda2/lib/python2.7/site-packages]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: Y
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Apache Kafka Platform support? [Y/n]: n
No Apache Kafka Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: n
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: n
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: n
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
No OpenCL SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: n
No CUDA support will be enabled for TensorFlow.

Do you wish to download a fresh release of clang? (Experimental) [y/N]: n
Clang will not be downloaded.

Do you wish to build TensorFlow with MPI support? [y/N]: n
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: -march=armv7-a


The WORKSPACE file has at least one of ["android_sdk_repository", "android_ndk_repository"] already set. Will not ask to help configure the WORKSPACE. Please delete the existing rules to activate the helper.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
	--config=mkl         	# Build with MKL support.
	--config=monolithic  	# Config for mostly static monolithic build.
Configuration finished

对于WORKSPACE文件中没有配置ndk信息和sdk信息的，执行./configure在最后会让配置ndk信息。可以不在命令行配置，然后手动在WORKSPACE文件中配置。

android_ndk_repository(
  name="androidndk",
  path="/home/unaguo/env/android-ndk-r15c",
  api_level=15)


android_sdk_repository(
  name="androidsdk",
  api_level=16,
  path="/home/unaguo/env/android-ndk-r15c",
  build_tools_version="ndk-build")

需要注意的是：bazel对于ndk里面的build无法识别，必须要重命名为build-tools才行。

3. 配置version_script.lds

version_script.lds文件用来设置最终output的so库文件中的symbol table以及各个symbol的版本号。这里不解释具体为啥。

在tensorflow/tensorflow/cc/路径下，创建version_script.lds文件，其中写上：

VERS_1.0 {
  global:
    *TF_*;
    *TFE_*;
    extern "C++" {
	tensorflow::*;
	google::protobuf::io::*;
	#google::protobuf::io::CopyingInputStream;
	google::protobuf::MessageLite::*;
	google::protobuf::Arena;
	google::protobuf::Message;
	*CopyingInputStream*;
    };
  # Hide everything else.
  local:
    *;

具体是参照tensorflow/tensorflow/c/下的version_script.lds编写的。

（1） *TF_*的意思是暴露含TF_的符号函数。

（2） tensorflow::*的意思是暴露以tensorflow::开头的符号函数，也即是暴露命名空间tensorflow中所有的符号函数。

（3） extern "C++" {的意思是当作C++的符号函数暴露。

4. 编写BUILD模块配置文件

cc_binary(
    name = "libtensorflow_cc.so",
    srcs = [],
    copts = tf_copts() + [
        "-ffunction-sections",
        "-fdata-sections",
    ],
    linkopts = if_android([
        "-landroid",
        "-llog",
        "-lm",
        "-z defs",
        "-s",
        "-Wl,--gc-sections",
        "-Wl,-soname=libtensorflow_cc.so",
        "-Wl,--version-script",
	"//tensorflow/cc:version_script.lds",
    ]),
    linkshared = 1,
    linkstatic = 1,
    tags = [
        "manual",
        "notap",
    ],
    deps = [
        "//tensorflow/c:c_api",
        "//tensorflow/cc:cc_ops",
        "//tensorflow/cc:client_session",
        "//tensorflow/cc:scope",
        #"//tensorflow/cc/profiler",
	"//tensorflow/cc:version_script.lds",
	"//tensorflow/core:android_tensorflow_lib",
    ],
)

具体里面的意思不解释了。
注意一个是deps字段中//tensorflow/cc/profiler的文件模块不能加。

5.Terminal编译libtensorflow_cc.sofor Android

bazel build
--config=monolithic
-c opt 
//tensorflow/contrib/android:libtensorflow_cc.so    
--verbose_failures    
--crosstool_top=//external:android/crosstool
--host_crosstool_top=@bazel_tools//tools/cpp:toolchain    
--cpu="armeabi-v7a" 
--cxxopt="-std=c++11" 
--copt="-DMDB_USE_ROBUST=0"  
--cxxopt="-DEIGEN_HAS_C99_MATH" 
--copt="-D__STDC_LIMIT_MACROS" 
--copt="-D__STDC_CONSTANT_MACROS"   
--compiler="clang5.0.300080"

输入上面的指令编译。

附录一：编译Bug种类及debug方法

（1） atomic相关的错误：

invalid failure memory model for '__atomic_compare_exchange'

似乎和optimization种类有关。似乎是对于gcc<5的编译器支持不好。说是在gcc5.4中已经解决这个问题。 应该是要明确指定optimization级别。

解决办法：升级NDK版本到15。

（2） undeclared inclusion相关的错误：

undeclared inclusion(s) in rule '@jpeg//:simd_armv7a':
this rule is missing dependency declarations for the following files included by 'external/jpeg/simd/jsimd_arm.c':
  'external/jpeg/jpegint.h'
  'external/jpeg/jerror.h'

意思是少了模块的在相关BUILD文件中的定义。

解决办法：

…1 在编译的目标文件(e.g. libtensorflow_cc.so)的BUILD文件中对应的模块定义(e.g. cc_library...)中,deps字段中添加依赖，上例bug应在cc_library的deps中添加：

@jpeg//:simd_armv7a

…2 同时添加一个新字段visibility，用来提供全局可见性：

visibility = ["//visibility:public"],

…3 然后修改这个@jpeg第三方库BUILD下的simd_armv7a模块。

注意Tensorflow的第三方库都放在tensorflow/bazel-tensorflow/external/中。

所以在tensorflow/bazel-tensorflow/external/jpeg下找到BUILD.bazel文件，修改simd_armv7a模块。

上述error log的意思就是在这个simd_armv7a字段中，缺少'external/jpeg/jpegint.h'和'external/jpeg/jerror.h'两个源码依赖。因此需要在name字段为simd_armv7a的cc_library模块中的srcs字段中添加：

"jpegint.h",
"jerror.h",

这里因为jpegint.h和jerror.h两个文件和BUILD在一个目录下，故没有前面的路径（external/jpeg/）。

…4 然后给这个cc_library模块添加新字段visibility，用来提供全局可见性：

visibility = ["//visibility:public"],

（3） 'posix_fallocate' undeclared here错误：

external/org_sqlite/sqlite3.c:31693:42: error: 'posix_fallocate' undeclared here (not in a function)
   { "fallocate",    (sqlite3_syscall_ptr)posix_fallocate,  0 },

一般这个错误指向sqlite3.c。

…1 打开sqlite3.c文件。（注意路径在```tensorflow/bazel-tensorflow/external/org_sqlite/sqlite3.c）

• 将#define HAVE_POSIX_FALLOCATE 1注释掉
• 加上#define DHAVE_STRERROR_R 1

（4） 'SIZE_MAX' undeclared错误：

external/gif_archive/lib/openbsd-reallocarray.c:33:19: error: 'SIZE_MAX' undeclared (first use in this function)
      nmemb > 0 && SIZE_MAX / nmemb < size)

是缺少SIZE_MAX定义的结果，目前没有一个比较好的办法，只能是：
在这个external/gif_archive/lib/openbsd-reallocarray.c文件首行添加：

#if defined(i386) | defined(arm) | defined(powerpc)
#define SIZE_MAX 0xffffffff
#endif
#if defined(x86_64)
#define SIZE_MAX 0xffffffffffffffff
#else
#define SIZE_MAX 0xffffffff         //最后会走这行
#endif

网友说的：1）添加标志位#define __STDC_LIMIT_MACROS和#define __STDC_CONSTANT_MACROS2）加上头文件#include来解决这个bug。

在我的电脑环境中，这两条都具备，所以采取了自己定义的方法来debug。

（5） use of deleted function

暂未解决。

（6）关于-lpthread找不到的问题：

https://github.com/tensorflow/tensorflow/issues/2775

附录二：bazel实际编译指令样例

  external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -shared -o bazel-out/armeabi-v7a-opt/bin/tensorflow/contrib/android/libtensorflow_cc.so -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/cc/libcc_ops.lo -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/cc/libconst_op.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/cc/libclient_session.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/cc/libscope.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/cc/libops.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/cc/profiler/libprofiler.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_stats.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_code.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_graph.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_op.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_show_multi.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_scope.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_show.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_tensor.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_timeline.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_node_show.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_node.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/jsoncpp_git/libjsoncpp.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/internal/libtfprof_utils.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/c/libcheckpoint_reader.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/c/libtf_status_helper.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/c/libc_api.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/util/tensor_bundle/libtensor_bundle.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/util/tensor_bundle/libnaming.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/libcore_cpu_base.lo -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/libgraph.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/grappler/libgrappler_item.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/grappler/libop_types.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/grappler/libutils.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/libtfprof_options.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/gif_archive/libgif.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/jpeg/libjpeg.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/jpeg/libsimd_armv7a.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/com_googlesource_code_re2/libre2.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/farmhash_archive/libfarmhash.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/highwayhash/libsip_hash.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/highwayhash/libarch_specific.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/png_archive/libpng.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/zlib_archive/libzlib.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/profiler/libprotos_all_cc.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/libandroid_tensorflow_lib.lo -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/kernels/libandroid_tensorflow_kernels.lo -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/libandroid_tensorflow_lib_lite.lo -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/double_conversion/libdouble-conversion.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/nsync/libnsync_cpp.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/libprotos_all_proto_cc_impl.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/tensorflow/core/liberror_codes_proto_cc_impl.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/fft2d/libfft2d.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/protobuf_archive/libprotobuf.a -Wl,-no-whole-archive -Wl,-whole-archive bazel-out/armeabi-v7a-opt/bin/external/protobuf_archive/libprotobuf_lite.a -Wl,-no-whole-archive external/androidndk/ndk/sources/cxx-stl/gnu-libstdc++/4.9/libs/armeabi-v7a/libgnustl_static.a external/androidndk/ndk/sources/cxx-stl/gnu-libstdc++/4.9/libs/armeabi-v7a/libsupc++.a -landroid -llog -lm -z defs -s '-std=c++11' -Wl,--gc-sections '-Wl,-soname=libtensorflow_cc.so' -Wl,--version-script tensorflow/c/version_script.lds -ldl -lz -lm -ldl -lpthread -pthread -lm -lm -static-libgcc -gcc-toolchain external/androidndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -no-canonical-prefixes -target armv7-none-linux-androideabi -Wl,--fix-cortex-a8 -Wl,-S '--sysroot=external/androidndk/ndk/platforms/android-12/arch-arm')