dpdk20.11.9 编译arm版本以及在arm 应用中引用dpdk20.11.9

以往19版本的dpdk 都是可以直接用make 的方式进行编译, e.g,

make T=x86_64-native-linux-gcc install

为了和客户那边用的DPDK 版本一致, 这次要用dpdk20.11.9, 并且要把之前跑在X86 版本的服务器上的程序跑在ARM 版本上.

目前有两个问题:

1. 编译出arm 版本的dpdk.

2. 把编译出来的dpdk lib 引用到arm版本的 testArmApp中.

-------------------------------------------------------------

1. 编译出arm 版本的dpdk.

1). 下载源码:

dpdk.or -> core ->download -> 选择20.11.9 下载:

DPDK

dpdk-20.11.9.tar.xz

2). 从dpdk.org中搜索相关的步骤可以得到以下信息:

4. Cross compiling DPDK for aarch64 and aarch32 — Data Plane Development Kit 23.11.0 documentation

前边的步骤, 官网写得很清楚了, 最后两个步骤的关键步骤说一下:

# 这一步是准备一些编译相关的配置.
meson setup aarch64-build-gcc --cross-file config/arm/arm64_armv8_linux_gcc

# 这里是按之前生成的配置文件正式编译
ninja -C aarch64-build-gcc

3). 以前步骤很快就把dpdk 编译出来了.

4.). 把dpdk 引用到testArmApp 中. 

这个时候发现dpdk 20中的makefile变了, "include $(RTE_SDK)/mk/rte.extapp.mk"  这些语句没有了. 加上也会报错. 这行的意思就是初始化一些dpdk的lib 依赖, 以及确定gcc 版本, Ld 版本等.

那么在makefile中就要手动加入Lib的引用了. 

开始使用的全是-lrte_eal ... 这种参数来编译的, 也编译成功了. 不过testArmApp在服务器上跑的时候发现.so 文件(动态链接库)找不到.

所以把所有dpdk lib 相关的都改成了 .a的形式, 如下所示:

我们先注意一下51 行这里,有问题, 后边再说.

按以上新的.a 的依赖方式, 发现编译到最后的时候, 始终有两个函数找不到定义:

dpdk-stable-20.11.9/arm64-build-retry2/lib/librte_eal.a(librte_eal_common_eal_common_options.c.o): In function `handle_eal_info_request':

eal_common_options.c:(.text+0x34): undefined reference to `rte_tel_data_start_array'

eal_common_options.c:(.text+0x54): undefined reference to `rte_tel_data_add_array_string'

/data/yubo/dpdk-stable-20.11.9/arm64-build-retry2/lib/librte_eal.a(librte_eal_common_eal_common_options.c.o): In function `eal_save_args':

eal_common_options.c:(.text+0x90c): undefined reference to `rte_telemetry_register_cmd'

eal_common_options.c:(.text+0x924): undefined reference to `rte_telemetry_register_cmd'

/data/yubo/dpdk-stable-20.11.9/arm64-build-retry2/lib/librte_eal.a(librte_eal_linux_eal.c.o): In function `rte_eal_init':

eal.c:(.text+0x1e48): undefined reference to `rte_telemetry_init'

/data/yubo/dpdk-stable-20.11.9/arm64-build-retry2/lib/librte_eal.a(librte_eal_common_eal_common_devargs.c.o): In function `rte_devargs_layers_parse':

eal_common_devargs.c:(.text+0x2e8): undefined reference to `rte_kvargs_parse_delim'

eal_common_devargs.c:(.text+0x34c): undefined reference to `rte_kvargs_free'

再三确认了代码和函数, 用arch -t 查看相关的.a 文件也发现定义的, 但是还是在Link这一步会出问题.

最后才发现. 比如A.a 中调用了B.a 那, 依赖时, A.a 要放到前边, 所以.a的依赖顺序改成以下:

lib 引用没有问题了, 又报了以下问题:

/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_common_eal_common_log.c.o): In function `rte_log_get_stream':
eal_common_log.c:(.text+0x20): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_common_eal_common_log.c.o): In function `eal_log_set_default':
eal_common_log.c:(.text+0x800): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_interrupts.c.o): In function `eal_intr_thread_main':
eal_interrupts.c:(.text+0x23c): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_interrupts.c.o): In function `rte_intr_callback_register':
eal_interrupts.c:(.text+0x8ac): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_interrupts.c.o): In function `rte_intr_callback_unregister':
eal_interrupts.c:(.text+0xb98): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_interrupts.c.o): In function `rte_eal_intr_init':
eal_interrupts.c:(.text+0x1684): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_interrupts.c.o): In function `rte_thread_is_intr':
eal_interrupts.c:(.text.experimental+0x4): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_interrupts.c.o): In function `rte_intr_callback_unregister_pending':
eal_interrupts.c:(.text.experimental+0x38): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_memalloc.c.o): In function `huge_sigbus_handler':
eal_memalloc.c:(.text+0x4): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'
/data//dpdk-stable-20.11.9/arm64-build/lib/librte_eal.a(librte_eal_linux_eal_memalloc.c.o): In function `huge_wrap_sigsetjmp':
eal_memalloc.c:(.text+0x1c): relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21 against `.bss'

查了很多网页发现是因为testArmApp中定义的数组太大, 和arm 版本默认的栈大小兼容. 需要加上以下参数进行编译:

-fno-PIC mcmodel=lagre

开始尝试在testArmApp 编译的时候加上这两个参数, 编译时发现并不行.

那估计是被引用 的dpdk Lib 也需要重新编译. 加上这两个参数. 那问题来了, dpdk 的编译都不知道参数哪里配置的.

经过多次尝试发现, dpdk的的编译参数是在以下这一步生成的:

meson setup aarch64-build-gcc --cross-file config/arm/arm64_armv8_linux_gcc

发现相关的编译参数都在文件夹 aarch64-build-gcc中, 经过多次编译发 这个只是一个文件夹名字, 是可以随便取的.  所以在这个文件夹中, 我找到了相关的配置文件:build.ninja

把这个文件中所有参数: fPIC 修改为 fno-pic mcmodel=large

然后再执行: 

ninja -C aarch64-build-gcc

这个时候发现testArmApp 就正常生成了.

arm服务器上测试, testArmApp  能正常工作. 

=======================================================

5). 以下是一些解决问题过程的工具记录:

5.1).cd /data/dpdk-stable-20.11.9

meson build

提示要用python3.7 以上.

==>安装 python 3.7:

sudo apt-get install python3.7

From

=> 版本确认

python --version是3.8.0

python3 --version 是3.6.9

看来python3 的默认版本不对, 要把phyton3 改成3.8.0

whereis python3   找到bin 位置, 然后修改相关链接:

sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1

test@test-accd100012:/data/dpdk-stable-20.11.9$ python3 --version

Python 3.8.0

这样就对了.

=>再重启meson build :

提示没有 ModuleNotFoundError: No module named 'mesonbuild'

===>

安装meson build 相关东西. 按: DPDK 交叉编译基本流程_dpdk交叉编译-CSDN博客  尝试:

下载交叉编译工具链

$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz

====>这一步就下不下来.   让IT修改外网配置后,能下载了.

将交叉编译工具的路径加入 PATH环境变量中

$ export PATH=$PATH:/home/zhoumin/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/bin

=>numactl 手动下载:

Releases · numactl/numactl · GitHub   2.0.13 的src code.

(wget 下载不了. 只有手动下载)

=>生成配置文件

meson setup arm64-build --cross-file config/arm/arm64_armv8_linux_gcc

这一步报: 'meson==0.52.1' distribution was not found and is required by the application

按:https://blog.csdn.net/a50095949/article/details/127289702  步骤试下:

sudo apt-get install python3 python3-pip ninja-build

pip3 install --user meson==0.52.1

到此, 准备编译dpdk 编译的工具链时遇到的问题就完了.