scons_交叉编译arm64_sysroot基于根文件rootfs编译方法

1.问题现象

1. 目前使用docker + qemu +arm64 容器编译做编译.
2. 之前做嵌入式开发, 使用的是传统的arm-linux-gcc 交叉编译, 使用SDK环境, 添加程序和库, 按模版添加Makefile, cmake, config, 具体原理不清.
3. 最近使用scons替换Makefile, cmake; 发现gcc的 --sysroot 选项, 可以指定rootfs路径,替换默认搜索头文件, 库路径
   • cc编译选项: -I /usr/include -I /usr/include/aarch64-linux-gnu
   • ld链接选项: -L /usr/lib -L /usrlib
4. 尝试使用传统的交叉编译方法, 编译一下项目.
5. 解决遗留疑问: 如果交叉编译依赖opencv, openssl, ros等第3方库, 如何配置的问题.

2.环境变量

• 交叉编译工具路径: /home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/
• 交叉编译前缀: aarch64-none-linux-gnu-
• rootfs根文件系统路径: /home/liuj/3_work/6_rk3588_sdk/debian/binary
• ubuntu软件中心, arm64 交叉编译工具: g++-aarch64-linux-gnu gcc-aarch64-linux-gnu

3.实例1:交编译arm64 hello.c

代码:
1_hello.c

#include 
 
int main()
{
    printf("hello world\n");
    return 0;
}

SConstruct

# 交叉编译工具路径
cross_Tools_Path="/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/"
# 交叉编译工具前缀
cross_Tool_Prefix="aarch64-none-linux-gnu-"
# rootfs路径
rootdir="/home/liuj/3_work/6_rk3588_sdk/debian/binary"

# scons官方交叉编译示例
env_options = {
   "CC"    : cross_Tool_Prefix+"gcc",
   "CXX"   : cross_Tool_Prefix+"g++",
   "LD"    : cross_Tool_Prefix+"ld",
   "AR"    : cross_Tool_Prefix+"ar",
   "STRIP" : cross_Tool_Prefix+"strip",
}
env = Environment(**env_options)
env.AppendENVPath('PATH', cross_Tools_Path)

# 编译选项
env['CCFLAGS'] = "--sysroot=%s "%(rootdir)
# env['CCFLAGS'] += "-I %s/usr/include \
#     -I %s/usr/include/aarch64-linux-gnu \
#     -I %s/usr/include/freetype2 \
#     -I %s//usr/local/include" \
#     %(rootdir,rootdir,rootdir,rootdir)

env.Program('1_hello.c')

运行结果: scons

aarch64-none-linux-gnu-gcc -o 1_hello.o -c --sysroot=/home/liuj/3_work/6_rk3588_sdk/debian/binary 1_hello.c
In file included from 1_hello.c:1:
/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory
27 | #include
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

编译错误原因:

• rk3588默认自带编译工具链aarch64-none-linux-gnu-gcc 不支持–sysroot选项

解决方法1: 指定rootfs下的include头文件

SConstruct 取消如下注释

env['CCFLAGS'] += "-I %s/usr/include \
    -I %s/usr/include/aarch64-linux-gnu \
    -I %s/usr/include/freetype2 \
    -I %s//usr/local/include" \
    %(rootdir,rootdir,rootdir,rootdir)

scons编译成功

aarch64-none-linux-gnu-gcc -o 1_hello.o -c --sysroot=/home/liuj/3_work/6_rk3588_sdk/debian/binary -I /home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include -I /home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include/aarch64-linux-gnu -I /home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include/freetype2 -I /home/liuj/3_work/6_rk3588_sdk/debian/binary//usr/local/include 1_hello.c
aarch64-none-linux-gnu-gcc -o 1_hello 1_hello.o

解决方法2: 下载开源arm64-linux-gnu-gcc

ubuntu软件仓库安装arm64交叉编译工具

sudo apt install g++-aarch64-linux-gnu gcc-aarch64-linux-gnu

SConstruct 修改之后如下

import os

cross_Tool_Prefix="aarch64-linux-gnu-"
rootdir="/home/liuj/3_work/6_rk3588_sdk/debian/binary"
env_options = {
   "CC"    : cross_Tool_Prefix+"gcc",
   "CXX"   : cross_Tool_Prefix+"g++",
   "LD"    : cross_Tool_Prefix+"ld",
   "AR"    : cross_Tool_Prefix+"ar",
   "STRIP" : cross_Tool_Prefix+"strip",
}

env = Environment(**env_options)
env['CCFLAGS'] = "--sysroot=%s "%(rootdir)

env.Program('1_hello.c')

编译 scons

aarch64-linux-gnu-gcc -o 1_hello.o -c --sysroot=/home/liuj/3_work/6_rk3588_sdk/debian/binary 1_hello.c
aarch64-linux-gnu-gcc -o 1_hello 1_hello.o

小结

1. 解决方法1 有一个问题, 编译C++文件错误, 提示找不到头文件, 暂未找到解决方法.
2. 后续使用 ** 解决方法2** , 目的是验证gcc --sysroot选项做交叉编译功能

4.交叉编译依赖第3方库

1.前言

之前的文章: 搭建交叉编译环境–RK3588示范
遗留一个问题: 如果交叉编译依赖opencv, openssl, ros等第3方库, 改如何配置.

2.小知识: gcc默认搜索与支持的库

• 交叉编译工具路径: /home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/

查看gcc默认搜索lib库
./aarch64-none-linux-gnu-gcc --print-search-dirs

install:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/
programs:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/libexec/gcc/aarch64-none-linux-gnu/10.3.1/:/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/libexec/gcc/:/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/…/…/…/…/aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu/10.3.1/:/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/…/…/…/…/aarch64-none-linux-gnu/bin/
libraries:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/…/…/…/…/aarch64-none-linux-gnu/lib/aarch64-none-linux-gnu/10.3.1/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/…/…/…/…/aarch64-none-linux-gnu/lib/…/lib64/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/aarch64-none-linux-gnu/libc/lib/aarch64-none-linux-gnu/10.3.1/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/aarch64-none-linux-gnu/libc/lib/…/lib64/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/aarch64-none-linux-gnu/libc/usr/lib/aarch64-none-linux-gnu/10.3.1/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/aarch64-none-linux-gnu/libc/usr/lib/…/lib64/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/lib/gcc/aarch64-none-linux-gnu/10.3.1/…/…/…/…/aarch64-none-linux-gnu/lib/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/aarch64-none-linux-gnu/libc/lib/:
/home/liuj/3_work/6_rk3588_sdk/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/…/aarch64-none-linux-gnu/libc/usr/lib/

大家可以查看每一个lib库路径;
通过查看, 发现主要是支持libc, libgnuc, stdc++库.

故默认aarch64-linux-gnu-gcc 只能编译: 标准c/c++, kernel, linux系统编程等API 代码

3.实例: 交叉编译依赖ROS的程序

代码
test_uint8_array.cc

#include 
#include 

#include 
#include 
#include 
using namespace std;

int main(int argc, char* argv[])
{
	ros::init(argc, argv, "test_topic");
    ros::NodeHandle n;


    ros::Publisher pub_uint8_array = n.advertise("/test/UInt8MultiArray", 5);
    std_msgs::UInt8MultiArray msg_uint8_array;
    // uint8_t data[4] = { 0x1, 0x2, 0x4, 0x8 };
    vector data = { 0x1, 0x2, 0x4, 0x8 };
    while (ros::ok())
    {
        msg_uint8_array.data = data;
        pub_uint8_array.publish(msg_uint8_array);
        sleep(1);
        ros::spinOnce();
    }

    return 0;
}

SConstruct

import os

cross_Tool_Prefix="aarch64-linux-gnu-"
rootdir="/home/liuj/3_work/6_rk3588_sdk/debian/binary"
env_options = {
   "CC"    : cross_Tool_Prefix+"gcc",
   "CXX"   : cross_Tool_Prefix+"g++",
   "LD"    : cross_Tool_Prefix+"ld",
   "AR"    : cross_Tool_Prefix+"ar",
   "STRIP" : cross_Tool_Prefix+"strip",
   "LINKCOM" : "$LINK -o $TARGET $SOURCES $LD_PREFLAGS $_LIBDIRFLAGS $__RPATH $LINKFLAGS $_LIBFLAGS",
}
env = Environment(**env_options)

# 编译选项
env['CCFLAGS'] = "--sysroot=%s "%(rootdir)
env['CCFLAGS'] += "-I %s/opt/ros/noetic/include "%(rootdir)
env['CPPPATH'] = ["%s/usr/include"%(rootdir),
    "%s/usr/include/aarch64-linux-gnu"%(rootdir),
    "%s/usr/include/freetype2"%(rootdir),
    "%s/usr/local/include"%(rootdir)
    ]

# 链接选项
env['LD_PREFLAGS'] = "--sysroot=%s " %(rootdir)
sys_lib = [rootdir+"/lib",
    rootdir+"/usr/lib",
    rootdir+"/usr/lib/aarch64-linux-gnu",
    rootdir+"/usr/local/lib",
    rootdir+"/lib/aarch64-linux-gnu",
]
ros_lib = [rootdir+'/opt/ros/noetic/lib']

env['LIBPATH'] = sys_lib+ros_lib

dep_lib = ['roscpp','rosconsole']
env['LIBS'] = dep_lib

# env['RPATH'] = ros_lib
env['LINKFLAGS'] = "-Wl,--rpath-link={} ".format(" ".join(ros_lib))
env['LINKFLAGS'] += "-Wl,--copy-dt-needed-entries "

env.Program('test_uint8_array.cc')

scons编译成功

arch64-linux-gnu-g++ -o test_uint8_array.o -c --sysroot=/home/liuj/3_work/6_rk3588_sdk/debian/binary -I /home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/include -I/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include -I/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include/aarch64-linux-gnu -I/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/include/freetype2 -I/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/local/include test_uint8_array.cc
aarch64-linux-gnu-g++ -o test_uint8_array test_uint8_array.o --sysroot=/home/liuj/3_work/6_rk3588_sdk/debian/binary -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/lib -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/lib -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/lib/aarch64-linux-gnu -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/local/lib -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/lib/aarch64-linux-gnu -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib -Wl,--rpath-link=/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib -Wl,--copy-dt-needed-entries -lroscpp -lrosconsole
scons: done building targets.

美化的流程显示

gcc/g++ 编译流程

gcc/g++ 链接流程

运行测试:
拷贝文件到arm64 开发板, 运行正常.

5.遇到的问题

1.链接选项–rpath --rpath-link区别

一开始使用 env[‘RPATH’], 对应的gcc链接选项为 -Wl,--rpath=xxx_lib_dir
在ld链接时提示错误:

/usr/bin/ld: warning: libxxx.so.6, needed by /a/b/c/libyyy.so, not found (try using -rpath or -rpath-link)

网上搜索到的区别: GCC 中 -L、-rpath和-rpath-link的区别

• –rpath 1.链接时指定依赖库路径 2.运行时指定链接路径
• –rpath-link 1.链接时指定依赖库路径

尝试修改为–rpath-link, 如下

-- -Wl,--rpath=/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib 
++ -Wl,--rpath-link=/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib \

手动运行g++链接指令, 命令如下

aarch64-linux-gnu-g++ -o test_uint8_array.out test_uint8_array.o \
 --sysroot=/home/liuj/3_work/6_rk3588_sdk/debian/binary   \
 -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/lib   \
 -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/lib   \
 -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/lib/aarch64-linux-gnu   \
 -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/local/lib   \
 -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/lib/aarch64-linux-gnu   \
 -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib \
 -Wl,--rpath-link=/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib \
 -Wl,--copy-dt-needed-entries \
 -lroscpp -lrosconsole

结果:
手动链接成功

2.scons如何添加–rpath-link

查看官方文档, github下载源代码查看, 没有找到scons设置–rpath-link的选项
解决方法:
使用 LINKFKAGS 设置 --rpath-link

ros_lib = [rootdir+'/opt/ros/noetic/lib']
env['LINKFLAGS'] = "-Wl,--rpath-link={} ".format(" ".join(ros_lib))

编译结果: -->链接错误

aarch64-linux-gnu-g++ -o test_uint8_array -Wl,--rpath-link=/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib -Wl,--copy-dt-needed-entries test_uint8_array.o -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/lib -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/lib -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/lib/aarch64-linux-gnu -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/usr/local/lib -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/lib/aarch64-linux-gnu -L/home/liuj/3_work/6_rk3588_sdk/debian/binary/opt/ros/noetic/lib -lroscpp -lrosconsole
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cannot find /lib/aarch64-linux-gnu/libc.so.6
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cannot find /usr/lib/aarch64-linux-gnu/libc_nonshared.a
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cannot find /lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
collect2: error: ld returned 1 exit status
scons: *** [test_uint8_array] Error 1

解决方法:
-Wl选项放在-L选项之后, -->链接通过.

新问题:
如何设置scons ld链接选项顺序

3.scons如何修改链接选项顺序

官方文档 修改编译/链接选项顺序

# cc/gcc 编译选项顺序
CCCOM       = '%CC %CFLAGS %_IFLAGS -c %< -o %>',
# c++/g++ 编译选项顺序
CXXCOM      = '%CXX %CXXFLAGS %_IFLAGS -c %< -o %>',
# 链接选项顺序
LINKCOM     = '%LINK %LDFLAGS -o %> %< %_LDIRS %LIBS',

修改LINKCOM 即可修改scons ld链接选项顺序

我的修改如下
SConstruct

# 1.修改 ld选项顺序: 使LINKFLAGS 在 _LIBDIRFLAGS 之后
# 2.自定义 ld前缀选项 LD_PREFLAGS
env_options = {
   "LINKCOM" : "$LINK -o $TARGET $SOURCES $LD_PREFLAGS $_LIBDIRFLAGS $__RPATH $LINKFLAGS $_LIBFLAGS",
}

# 3.定义LD_PREFLAGS
env['LD_PREFLAGS'] = "--sysroot=%s " %(rootdir)

运行scons, 编译链接通过,如文章 3.实例: 交叉编译依赖ROS的程序

6.总结

• make, cmake, scons本质上是调用gcc/g++ 命令进行编译, 链接. 各种参数选项, 是为了配置gcc/g++选项.
  如本次使用scons, 与手动执行 gcc/g++ 交叉编译链接成功.
  那cmake各种参数设置, 也是为了调整编译/链接选项, 实现手动执行gcc命令的效果.
• 又一次熟悉了gcc/g++ 编译,链接流程, 交叉编译环境配置.
• 猜测–rpath-link , 是gcc检查到host x86_64, target arch64, 不能使用–rpath包含路径.

7.未解决疑问, 问题

Q1: 如何打印显示gcc默认include路径?
Q2: 为什么gcc/g++ ld链接选项顺序, 对链接结果有影响?

• 默认ld链接选项顺序 -Wl,--rpath-link -Llib_dir -llib 失败
• 而-Llib_dir -Wl,--rpath-link -llib 成功.
  知道的小伙伴留言一下.

参考

• gcc中文手册-选项分类
• gcc-sysroot选项详细说明
• gcc-link链接选项

签名
一个喜欢机器人行业的 嵌入式-内核驱动-系统-中间件-网络-开发工程师
微信: liuj1637664504

---