交叉编译tcpreplay

前言

最近项目上出现个问题，其中一个报文发出去后，在互联网中被丢包。后续的重传也被丢包。因为该连接只有该数据包被丢弃，前面和后面的数据包都正常传送。怀疑是该报文被网络中某个设备拦截并丢弃。
为了验证猜想，准备对被丢弃的报文进行重放测试。设备是arm64的openwrt定制系统，需要交叉编译tcpreplay。
在编译过程中颇费了一些周章，所以记录下踩过的这些雷，希望能够帮助到有同样困扰的人。

1. 预备工作

1.1 libpcap库

编译tcpreplay使用libpcap库,需要提前下载并编译,在此不再赘述.

2. 下载tcpreplay源码

去github下载tcpreplay的最新源码。

root@localhost:~## git clone https://github.com/appneta/tcpreplay

3. 配置编译环境

3.1 autoreconf生成配置脚本

root@localhost:~/tcpreplay# autoreconf -vif

3.2 configure生成Makefile

大多数人在这一步常常会遇到各种问题，一部分是编译所需的依赖不满足，另一部分是configure的参数未使用正确，导致执行失败。

3.2.1 第一次配置失败

root@localhost:~/tcpreplay# ./configure --host=aarch64-openwrt-linux --with-libpcap=/root/libpcap/ --prefix=/root/tcpreplay
checking whether to enable maintainer-specific portions of Makefiles... yes
...//省略
checking for libpcap... configure: error: "Unable to find matching library for header file in /root/libpcap/"

提示在指定的路径下找不到匹配的libpcap库。
查看/root/libpcap/下确实有相关头文件和库文件。后分析得知，configure默认查找动态库文件，而我们提供的是静态libpcap.a文件，所以提示找不到。

root@appex:~/libpcap# ls lib/
libpcap.a  pkgconfig

为了让,我增加了让configure去查找静态libpcap.a --enable-static参数,结果还是提示同样错误。
难道不支持使用静态libpcap库？于是查看configure -h，发现需要使用 --enable-static-link 来使用静态libpcap库。

root@localhost:~/tcpreplay# ./configure -h
...//省略
--enable-static-link    Use static libraries ( .a or .A.tbd ) - default no
--enable-dynamic-link   Use shared libraries ( .so .dylib or .tbd ) -
...//省略

3.2.2 第二次配置成功

root@localhost:~/tcpreplay# ./configure --host=aarch64-openwrt-linux --with-libpcap=/root/libpcap/ --prefix=/root/tcpreplay --enable-static_link
checking whether to enable maintainer-specific portions of Makefiles... yes
...//省略
##########################################################################
             TCPREPLAY Suite Configuration Results (4.3.3)
##########################################################################
libpcap:                    /root/libpcap/ (>= 0.9.6)
PF_RING libpcap             no   
libdnet:                    no   
autogen:                    /usr/bin/autogen (5.18.12)
Use libopts tearoff:        yes
64bit counter support:      yes
tcpdump binary path:        /usr/sbin/tcpdump
fragroute support:          no
tcpbridge support:          yes
tcpliveplay support:        yes

Supported Packet Injection Methods (*):
Linux TX_RING:              no
Linux PF_PACKET:            yes
BSD BPF:                    no
libdnet:                    no
pcap_inject:                yes
pcap_sendpacket:            yes **
pcap_netmap                 no
Linux/BSD netmap:           no
Tuntap device support:      yes

* In order of preference; see configure --help to override
** Required for tcpbridge

4. 编译tcpreplay

到这一步一般都是make && make install了。大多数情况都能正确编译，少部分情况需要手动做一些调整。

4.1 编译错误1

root@localhost:~/tcpreplay# make
Making all in scripts
make[1]: Entering directory '/root/tcpreplay/scripts'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/scripts'
Making all in lib
...//省略
make  all-am
make[4]: Entering directory '/root/tcpreplay/src/common'
  CC       cidr.o
In file included from ../../src/defines.h:57:0,
                 from cidr.c:22:
/root/libpcap//include/pcap.h:43:10: fatal error: pcap/pcap.h: No such file or directory
 #include 
          ^~~~~~~~~~~~~
compilation terminated.
Makefile:456: recipe for target 'cidr.o' failed
make[4]: *** [cidr.o] Error 1
make[4]: Leaving directory '/root/tcpreplay/src/common'
Makefile:385: recipe for target 'all' failed
make[3]: *** [all] Error 2
make[3]: Leaving directory '/root/tcpreplay/src/common'
Makefile:1194: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/root/tcpreplay/src'
Makefile:574: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/root/tcpreplay/src'
Makefile:450: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

错误提示无法找到 pcap/pcap.h 文件。
明明库文件目录下存在对应文件啊。

root@localhost:~/libpcap# ls include/pcap/pcap.h 
include/pcap/pcap.h

后分析得知，gcc的 <> 查找路径里 不包含 /root/libpcap/include,所以差找不到。

#include "..." search starts here:
#include <...> search starts here:
 /home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/bin/../lib/gcc/aarch64-openwrt-linux-musl/7.3.0/../../../../aarch64-openwrt-linux-musl/sys-include
 /home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/bin/../lib/gcc/aarch64-openwrt-linux-musl/7.3.0/../../../../aarch64-openwrt-linux-musl/include
 /home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/bin/../lib/gcc/aarch64-openwrt-linux-musl/7.3.0/include
 /home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/usr/include

我在这里直接将include目录拷贝到交叉编译器的usr目录下了。
其实有个更优雅的办法，就是编译libpcap库的时候指定 PKG_CONFIG_PATH为编译器的usr/lib/pkgconfig, 这样编译出来的 头文件和库文件自动在编译器的查找路径内。后续就不会遇到这个问题了。

4.2 编译错误2

root@localhost:~/tcpreplay# make
Making all in scripts
make[1]: Entering directory '/root/tcpreplay/scripts'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/scripts'
Making all in lib
...//省略
  CCLD     tcpreplay
../libopts/.libs/libopts.a(libopts_la-libopts.o): In function `ao_realloc':
/root/tcpreplay/libopts/autoopts.c:78: undefined reference to `rpl_realloc'
./common/libcommon.a(utils.o): In function `_our_safe_realloc':
/root/tcpreplay/src/common/utils.c:76: undefined reference to `rpl_realloc'
collect2: error: ld returned 1 exit status
Makefile:691: recipe for target 'tcpreplay' failed
make[3]: *** [tcpreplay] Error 1
make[3]: Leaving directory '/root/tcpreplay/src'
Makefile:1194: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/root/tcpreplay/src'
Makefile:574: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/root/tcpreplay/src'
Makefile:450: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

提示 undefined reference to `rpl_realloc’ 。这是肿么回事，查看configure，发现有ac_cv_func_realloc_0_nonnull 参数，如果为no的话，则会使用rpl_realloc。

if test $ac_cv_func_realloc_0_nonnull = yes; then :

$as_echo "#define HAVE_REALLOC 1" >>confdefs.h

else
  $as_echo "#define HAVE_REALLOC 0" >>confdefs.h

   case " $LIBOBJS " in
  *" realloc.$ac_objext "* ) ;;
  *) LIBOBJS="$LIBOBJS realloc.$ac_objext"
 ;;
esac


$as_echo "#define realloc rpl_realloc" >>confdefs.h

fi

于是重新修改configure命令如下

root@localhost:~/tcpreplay# ac_cv_func_realloc_0_nonnull=yes ./configure --host=aarch64-openwrt-linux --with-libpcap=/root/libpcap/ --prefix=/root/tcpreplay --enable-static_link

重新生成Makefile后，再make编译就直接成功了。

root@localhost:~/tcpreplay# make
Making all in scripts
make[1]: Entering directory '/root/tcpreplay/scripts'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/scripts'
Making all in lib
make[1]: Entering directory '/root/tcpreplay/lib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/lib'
Making all in libopts
make[1]: Entering directory '/root/tcpreplay/libopts'
make  all-am
make[2]: Entering directory '/root/tcpreplay/libopts'
...//省略
make[3]: Leaving directory '/root/tcpreplay/src'
make[2]: Leaving directory '/root/tcpreplay/src'
make[1]: Leaving directory '/root/tcpreplay/src'
make[1]: Entering directory '/root/tcpreplay'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/root/tcpreplay'

后记

交叉编译中的configure命令是门学问，需要多钻研多学习，欢迎大家一起交流成长。