安装bpftrace和bcc的踩坑记录
最后在Ubuntu22.04使用Ubuntu提供的安装命令完成了安装。这里是记录尝试在Ubuntu18.04和Ubuntu22.04使用源码安装未果的过程。
文章目录
-
- 22版本安装bcc
-
- 准备工具
- 安装命令
- 使用报错：
- iovisor封装的安装方式
- ubuntu的安装方式
- For Bionic (18.04 LTS)官方提供的源码安装
-
- 准备工具安装
- 下载和安装
- 使用报错：
- 安装bpftrace
-
- Ubuntu22自带安装
- Ubuntu22源码安装
- Ubuntu18源码安装
- 参考：
- by the way
22版本安装bcc
准备工具
sudo apt install -y zip bison build-essential cmake flex git libedit-dev \
libllvm14 llvm-14-dev libclang-14-dev python3 zlib1g-dev libelf-dev libfl-dev python3-setuptools \
liblzma-dev libdebuginfod-dev arping netperf iperf
安装命令
wget https://github.com/iovisor/bcc/releases/download/v0.28.0/bcc-src-with-submodule.tar.gz
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd
注意，上面那段安装命令不是直接使用官网提供的git clone命令，因为那不是release版本缺少头文件，会产生如下报错：
CMake Warning at CMakeLists.txt:73 (message):
Failed to update submodule libbpf
bcc/src/cc/frontends/clang/b_frontend_action.cc:37:
/home/xmu3/bcc/src/cc/bcc_libbpf_inc.h:8:11: fatal error: libbpf/src/bpf.h: No such file or directory
8 | # include "libbpf/src/bpf.h"
| ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/cc/frontends/clang/CMakeFiles/clang_frontend-objects.dir/build.make:90: src/cc/frontends/clang/CMakeFiles/clang_frontend-objects.dir/b_frontend_action.cc.o] Error 1
使用报错：
sudo ./execsnoop.py
Traceback (most recent call last):
File "/home/bcc/tools/./execsnoop.py", line 23, in <module>
from bcc import BPF
File "/usr/lib/python3/dist-packages/bcc-EAD_HASH_NOTFOUND_GITDIR_N-py3.10.egg/bcc/__init__.py", line 26, in <module>
File "/usr/lib/python3/dist-packages/bcc-EAD_HASH_NOTFOUND_GITDIR_N-py3.10.egg/bcc/libbcc.py", line 17, in <module>
File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libbcc.so.0: cannot open shared object file: No such file or directory
可以查到很多关于libbcc.so的报错，但是后面的错误各有差异，一直解决未果。（谁找到了希望滴滴我）
iovisor封装的安装方式
没有尝试过，但是听所也有坑。
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
ubuntu的安装方式
这种方式安装的目录是在/usr/sbin，那些文件都是*-bpfcc的样式。和源码安装相比有的功能这里是没有的。以及Ubuntu18.04之后的版本才能使用这个命令，之前以及18.04都没有。
sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
For Bionic (18.04 LTS)官方提供的源码安装
准备工具安装
sudo apt-get -y install zip bison build-essential cmake flex git libedit-dev \
libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev libfl-dev python3-setuptools \
liblzma-dev arping netperf iperf
下载和安装
git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd
使用报错：
yi@PC:~/bcc/build$ sudo /usr/share/bcc/tools/opensnoop
Traceback (most recent call last):
File "/usr/share/bcc/tools/opensnoop", line 24, in <module>
from bcc import ArgString, BPF
File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 27, in <module>
from .libbcc import lib, _CB_TYPE, bcc_symbol, bcc_symbol_option, _SYM_CB_TYPE
File "/usr/lib/python2.7/dist-packages/bcc/libbcc.py", line 20, in <module>
lib.bpf_module_create_b.restype = ct.c_void_p
File "/usr/lib/python2.7/ctypes/__init__.py", line 379, in __getattr__
func = self.__getitem__(name)
File "/usr/lib/python2.7/ctypes/__init__.py", line 384, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/x86_64-linux-gnu/libbcc.so.0: undefined symbol: bpf_module_create_b
修改脚本开头，把python换成python3，或者使用命令时加上python3
好的，接下来这个bug成为每一次源码安装都跨不过的坎了。
sudo /usr/share/bcc/tools/execsnoop
LLVM ERROR:
最开始以为LLVM9以下bcc好像不支持了。其实和LLVM没关系，官方也是用的LLVM-7。安装完LLVM15之后，源码安装过程中另一个问题，关于python
[ 36%] Building sdist for python3
Traceback (most recent call last):
File "/home/yi/下载/bcc/build/src/python/bcc-python3/setup.py", line 3, in <module>
from setuptools import setup
File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 23, in <module>
from setuptools.dist import Distribution
File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 34, in <module>
from setuptools import windows_support
File "/usr/local/lib/python3.9/site-packages/setuptools/windows_support.py", line 2, in <module>
import ctypes
File "/usr/local/lib/python3.9/ctypes/__init__.py", line 8, in <module>
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
src/python/CMakeFiles/bcc_py_python3.dir/build.make:71: recipe for target 'src/python/bcc-python3/dist/bcc-0.28.0+9596edb9.tar.gz' failed
make[2]: *** [src/python/bcc-python3/dist/bcc-0.28.0+9596edb9.tar.gz] Error 1
CMakeFiles/Makefile2:616: recipe for target 'src/python/CMakeFiles/bcc_py_python3.dir/all' failed
make[1]: *** [src/python/CMakeFiles/bcc_py_python3.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
查了查大家都说是要重装python3，结果系统给我重装没了。
python2运行以下报错，让用python3即可
sudo ./execsnoop
Traceback (most recent call last):
File "./execsnoop", line 23, in <module>
from bcc import BPF
ImportError: No module named bcc
python3运行又是LLVM ERROR
单步调试在b = BPF(text=bpf_text)处
sudo python3 ./hello_world.py
bpf: Argument list too long. Program too large (0 insns), at most 4096 insns
Traceback (most recent call last):
File "./hello_world.py", line 12, in <module>
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
File "/usr/lib/python3/dist-packages/bcc-0.28.0+9596edb9-py3.6.egg/bcc/__init__.py", line 487, in __init__
File "/usr/lib/python3/dist-packages/bcc-0.28.0+9596edb9-py3.6.egg/bcc/__init__.py", line 1465, in _trace_autoload
File "/usr/lib/python3/dist-packages/bcc-0.28.0+9596edb9-py3.6.egg/bcc/__init__.py", line 527, in load_func
Exception: Failed to load BPF program b'kprobe__sys_clone': Argument list too long
再debug下去感觉没完没了，干了一天后就换Ubuntu22了，两个系统的使用感像从农村到了城里。
安装bpftrace
Ubuntu22自带安装
官方安装说明
安装的目录是在/usr/sbin，那些文件都是*-bt的样式
sudo apt-get install -y bpftrace
但是也会报错：
ERROR: Could not resolve symbol: /proc/self/exe:BEGIN_trigger
这个报错在github上有很长的讨论，不过解决方案就一句命令：
sudo apt-get install bpftrace-dbgsym
然而，安装说无法定位，也是报错。这时候去Launchpad上面有发布的软件包安装bpftrace-dbgsym_0.14.0-1_amd64.ddeb这个调试符号包。下载后
sudo dpkg -i bpftrace-dbgsym_0.14.0-1_amd64.ddeb
在Ubuntu下查找某个软件包的PPA源的方法主要有以下几种:
1、在软件包的官网或者GitHub仓库的说明中查找,维护者通常会提供建议的PPA源。
2、在Launchpad网站上搜索软件包名称,查看是否存在相关的PPA源。Launchpad托管了Ubuntu的大量PPA源。
3、使用Google搜索"软件包名 + ppa",看是否有相关讨论帖子推荐的PPA源。
4、使用命令行工具apt-cache search,看搜索结果中是否有类似“ppa.launchpad.net”的源,这表示系统已知的PPA。
5、使用命令行工具apt-add-repository,它可以直接通过ppa:user/ppa-name的格式添加某PPA源。
6、检查/etc/apt/sources.list.d目录下的配置文件,是否已有该软件包相关的PPA源,文件名称通常是ppa的名称。
7、使用诸如aptly、repoman等第三方工具也可以搜索本地已知的所有PPA源列表。
一般来说,维护者提供的PPA源是首选,查找时注意区分体系架构、Ubuntu版本、软件包版本等差异。
Ubuntu22源码安装
从源码安装的话：bpftrace 二进制文件将安装在 /usr/local/bin/bpftrace 中，工具将安装在 /usr/local/share/bpftrace/tools 中。可以使用 cmake 的参数更改安装位置，默认位置为 -DCMAKE_INSTALL_PREFIX=/usr/local。
sudo apt-get update
sudo apt-get install -y libbpfcc-dev
sudo apt-get install -y \
bison \
cmake \
flex \
g++ \
git \
libelf-dev \
zlib1g-dev \
libfl-dev \
systemtap-sdt-dev \
binutils-dev \
libcereal-dev \
llvm-dev \
llvm-runtime \
libclang-dev \
clang \
libpcap-dev \
libgtest-dev \
libgmock-dev \
asciidoctor \
libdw-dev \
pahole
git clone https://github.com/iovisor/bpftrace --recurse-submodules
mkdir bpftrace/build; cd bpftrace/build;
../build-libs.sh
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j8
sudo make install
也是有报错，但是忘记记录了。
Ubuntu18源码安装
bpftrace将llvm降到7，并且源码安装0.11.1之后，make报错：
bpftrace/src/attached_probe.cpp:801:51: error: too few arguments to function ‘void* bpf_attach_uprobe(int, bpf_probe_attach_type, const char*, const char*, uint64_t, pid_t, int, int, perf_reader_cb, void*)’
probe_.pid);
^
In file included from /home/xmu2/bpftrace/src/attached_probe.h:10:0,
from /home/xmu2/bpftrace/src/attached_probe.cpp:16:
/usr/include/bcc/libbpf.h:80:8: note: declared here
void * bpf_attach_uprobe(int progfd, enum bpf_probe_attach_type attach_type,
^~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp: In member function ‘void bpftrace::AttachedProbe::attach_usdt(int)’:
/home/xmu2/bpftrace/src/attached_probe.cpp:911:77: error: too few arguments to function ‘void* bpf_attach_uprobe(int, bpf_probe_attach_type, const char*, const char*, uint64_t, pid_t, int, int, perf_reader_cb, void*)’
eventname().c_str(), probe_.path.c_str(), offset_, pid == 0 ? -1 : pid);
^
In file included from /home/xmu2/bpftrace/src/attached_probe.h:10:0,
from /home/xmu2/bpftrace/src/attached_probe.cpp:16:
/usr/include/bcc/libbpf.h:80:8: note: declared here
void * bpf_attach_uprobe(int progfd, enum bpf_probe_attach_type attach_type,
^~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp: In member function ‘void bpftrace::AttachedProbe::attach_tracepoint()’:
/home/xmu2/bpftrace/src/attached_probe.cpp:927:26: error: too few arguments to function ‘void* bpf_attach_tracepoint(int, const char*, const char*, int, int, int, perf_reader_cb, void*)’
eventname().c_str());
^
In file included from /home/xmu2/bpftrace/src/attached_probe.h:10:0,
from /home/xmu2/bpftrace/src/attached_probe.cpp:16:
/usr/include/bcc/libbpf.h:87:8: note: declared here
void * bpf_attach_tracepoint(int progfd, const char *tp_category,
^~~~~~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp: In member function ‘void bpftrace::AttachedProbe::attach_watchpoint(int, const string&)’:
/home/xmu2/bpftrace/src/attached_probe.cpp:1120:23: error: ‘bpf_attach_perf_event_raw’ was not declared in this scope
int perf_event_fd = bpf_attach_perf_event_raw(progfd_, &attr, pid, -1, -1, 0);
^~~~~~~~~~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp:1120:23: note: suggested alternative: ‘bpf_attach_perf_event’
int perf_event_fd = bpf_attach_perf_event_raw(progfd_, &attr, pid, -1, -1, 0);
^~~~~~~~~~~~~~~~~~~~~~~~~
bpf_attach_perf_event
src/CMakeFiles/bpftrace.dir/build.make:75: recipe for target 'src/CMakeFiles/bpftrace.dir/attached_probe.cpp.o' failed
make[2]: *** [src/CMakeFiles/bpftrace.dir/attached_probe.cpp.o] Error 1
CMakeFiles/Makefile2:1160: recipe for target 'src/CMakeFiles/bpftrace.dir/all' failed
make[1]: *** [src/CMakeFiles/bpftrace.dir/all] Error 2
Makefile:145: recipe for target 'all' failed
make: *** [all] Error 2
bpftrace安装时报错
CMake Error at /usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Please run /home/xmu2/bpftrace/build-libs.sh from the build folder first
(missing: LIBBCC_LIBRARIES LIBBCC_INCLUDE_DIRS)
Call Stack (most recent call first):
/usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
cmake/FindLibBcc.cmake:84 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:116 (find_package)
按照其他说法安装libbpfcc后仍然报错：
CMake Error at /usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Please run /home/xmu2/bpftrace/build-libs.sh from the build folder first
(missing: LIBBCC_INCLUDE_DIRS)
Call Stack (most recent call first):
/usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
cmake/FindLibBcc.cmake:84 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:116 (find_package)
变成版本llvm-7之后：
/virtual/main.c:138:1: error: expected identifier or '('
段错误
参考：
https://blog.csdn.net/Longyu_wlz/article/details/109900096
https://github.com/bpftools/linux-observability-with-bpf.git
https://blog.csdn.net/qq_34258344/article/details/104400738
https://yaoyao.io/posts/how-to-setup-ebpf-env-on-ubuntu.html
作者回复: libbpf-dev只包含在比较新的发行版中，其他发行版可以从源码安装，具体步骤可以参考 https://github.com/libbpf/libbpf#build。
另外，我们案例的Github中也有源码编译的详细步骤：https://github.com/feiskyer/ebpf-apps/blob/main/bpf-apps/Makefile#L21-L22
by the way
按道理这些bug都可以去github开个issue的，不过比较赶时间，也不确定可不可以复现，有谁又遇到了可以去开一个。