[containerd] 在Windows上使用IDEA远程调试containerd, ctr, containerd-shim

1. containerd安装

[Ubuntu 22.04] 安装containerd

2. 源码编译

  主要步骤如下：

• 1、从github下载containerd源码
  • git clone [email protected]:containerd/containerd.git
• 2、切换到想要debug的分支或者tag，我这里选择切换到1.7.2版本的tag
  • git branch v1.7.2 v1.7.2 && git checkout v1.7.2
• 3、下载contaienrd的依赖
  • cd containerd && go mod tidy
• 4、编译containerd源码
  • make build GODEBUG=true all

root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# make build GODEBUG=true all
+ build
+ bin/ctr
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie  -o bin/ctr -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd  ' -tags "urfave_cli_no_docs static_build"  ./cmd/ctr
+ bin/containerd
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie  -o bin/containerd -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd  ' -tags "urfave_cli_no_docs static_build"  ./cmd/containerd
+ bin/containerd-stress
go build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie  -o bin/containerd-stress -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd  ' -tags "urfave_cli_no_docs static_build"  ./cmd/containerd-stress
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# 
root@containerd:~/workspace/containerd# ls -lhtr bin/
total 158M
-rwxr-xr-x 1 root root  31M Jul 29 14:34 ctr
-rwxr-xr-x 1 root root  61M Jul 29 14:34 containerd
-rwxr-xr-x 1 root root  29M Jul 29 14:34 containerd-stress
-rwxr-xr-x 1 root root 9.4M Jul 29 14:34 containerd-shim
-rwxr-xr-x 1 root root  12M Jul 29 14:34 containerd-shim-runc-v1
-rwxr-xr-x 1 root root  17M Jul 29 14:34 containerd-shim-runc-v2
root@containerd:~/workspace/containerd# 

  注意，编译的时候containerd会依赖btrfs文件系统，如果没有安装，编译的时候会报错（提示：linux/btrfs_tree.h: No such file or directory）。按照文档第一步安装containerd不会遇到这个问题。

3. 验证编译的二进制文件是否含有调试需要的信息

  注意，以下三种验证方式，选择其中的一种验证成功即可。

3.1. objdump工具验证

  正确的现象如下，执行objdump --syms bin/containerd是可以看到调试信息的

root@containerd:~/workspace/containerd# objdump --syms bin/containerd

bin/containerd:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000000 l    df *ABS*  0000000000000000              Scrt1.o
00000000000003b4 l     O .note.ABI-tag  0000000000000020              __abi_tag
0000000000000000 l    df *ABS*  0000000000000000              go.go
00000000003ad4c0 l     F .text  0000000000000000              runtime.text
00000000003ad4c0 l     F .text  0000000000000059              internal/cpu.Initialize
00000000003ad520 l     F .text  0000000000000537              internal/cpu.processOptions
00000000003ada60 l     F .text  0000000000000026              internal/cpu.indexByte
00000000003adaa0 l     F .text  0000000000000925              internal/cpu.doinit
00000000003ae3e0 l     F .text  0000000000000006              internal/cpu.isSet
00000000003ae400 l     F .text  000000000000001b              internal/cpu.cpuid.abi0
00000000003ae420 l     F .text  0000000000000011              internal/cpu.xgetbv.abi0
00000000003ae440 l     F .text  0000000000000009              internal/cpu.getGOAMD64level.abi0
00000000003ae460 l     F .text  000000000000007a              type:.eq.internal/cpu.option
00000000003ae4e0 l     F .text  00000000000000e6              type:.eq.[6]internal/cpu.option
00000000003ae5e0 l     F .text  0000000000000003              runtime/internal/atomic.(*Int32).Load
00000000003ae600 l     F .text  0000000000000003              runtime/internal/atomic.(*Int32).Store
00000000003ae620 l     F .text  000000000000000d              runtime/internal/atomic.(*Int32).CompareAndSwap
00000000003ae640 l     F .text  000000000000000a              runtime/internal/atomic.(*Int32).Add
00000000003ae660 l     F .text  0000000000000004              runtime/internal/atomic.(*Int64).Load
00000000003ae680 l     F .text  0000000000000004              runtime/internal/atomic.(*Int64).Store
00000000003ae6a0 l     F .text  000000000000000f              runtime/internal/atomic.(*Int64).CompareAndSwap
00000000003ae6c0 l     F .text  0000000000000007              runtime/internal/atomic.(*Int64).Swap
00000000003ae6e0 l     F .text  000000000000000d              runtime/internal/atomic.(*Int64).Add
00000000003ae700 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint8).Load
00000000003ae720 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint8).Store
00000000003ae740 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint8).And
00000000003ae760 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint8).Or
00000000003ae780 l     F .text  0000000000000024              runtime/internal/atomic.(*Bool).Load
00000000003ae7c0 l     F .text  000000000000001f              runtime/internal/atomic.(*Bool).Store
00000000003ae7e0 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).Load
00000000003ae800 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).LoadAcquire
00000000003ae820 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).Store
00000000003ae840 l     F .text  0000000000000003              runtime/internal/atomic.(*Uint32).StoreRelease
00000000003ae860 l     F .text  000000000000000d              runtime/internal/atomic.(*Uint32).CompareAndSwap
00000000003ae880 l     F .text  000000000000000d              runtime/internal/atomic.(*Uint32).CompareAndSwapRelease
00000000003ae8a0 l     F .text  0000000000000005              runtime/internal/atomic.(*Uint32).Swap
00000000003ae8c0 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint32).And
00000000003ae8e0 l     F .text  0000000000000004              runtime/internal/atomic.(*Uint32).Or

  错误的信息如下，会提示no symbols，这种二进制无法进行debug

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# objdump --syms /usr/local/bin/containerd

/usr/local/bin/containerd:     file format elf64-x86-64

SYMBOL TABLE:
no symbols


root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#

3.2. file工具验证

  正确信息如下，会显示with debug_info, not stripped

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# file bin/containerd
bin/containerd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9bb291e61e1eceb23359dc29100845e5c1edf763, for GNU/Linux 3.2.0, with debug_info, not stripped
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#

  错误信息如下，会显示stripped，说明不包含调试信息，无法进行debug

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# file /usr/local/bin/containerd
/usr/local/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=322f89b7e351fe2ccfaa0fe30de79c76d49d6e26, for GNU/Linux 3.2.0, stripped
root@containerd:~/workspace/containerd#

3.3. dlv工具验证

  正确的调试信息如下：

root@containerd:~/workspace/containerd# dlv exec  bin/containerd
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv)
(dlv)
(dlv)
(dlv) exit
root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd#

  错误的调试信息如下，会提示：no debug info found

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# dlv exec  /usr/local/bin/containerd
Warning: no debug info found, some functionality will be missing such as stack traces and variable evaluation.
Type 'help' for list of commands.
(dlv)
(dlv)
(dlv)
(dlv)
(dlv) exit

4. debug

  debug containerd步骤如下：

• 1、利用ps -ef|gerp containerd查看containerd启动所需要的参数，这一步特别重要，尤其是在debug k8s源码的时候，k8s的每一个组件都带了很多命令行参数，想要调试这些组件，必须把这些组件的启动参数原封不动的加入到dlv调试命令当中
  • 注意，实际上执行上述命令之后会发现，containerd并没有启动参数，因此无需关心。
• 2、通过dlv命令启动containerd，启动命令我们可以从IDEA remote debug功能拷贝过来
  • dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd
  • 注意：这里在启动contaienrd的时候没有指定任何参数，实际上也可以根据自己的情况加入containerd参数，譬如指定containerd的配置文件的位置（containerd默认配置文件为：/etc/containerd/config.toml），也可以指定调试的debug级别，譬如：
    • dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --config=/root/mycontainerd/config.toml --log-level=debug
• 3、在IDEA启动debug，连接到远程调试
  • 注意：在启动IDEA调试之前，你需要在想要debug的位置增加断点，否则程序启动会直接运行起来，等你这个时候打断点，很可能就晚了。
  • 注意：在启动IDEA调试之前，你需要修改IDEA操作系统标识为Linux

  执行dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug命令之后，此时会阻塞在这里，千万不要使用ctrl + c，只有当IDEA连接上来的时候才会开始执行

root@containerd:~/workspace/containerd#
root@containerd:~/workspace/containerd# dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
API server listening at: [::]:12345
2023-07-29T15:06:45+08:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)