calico构建镜像和宿主机dlv调试容器内felix进程方法

总体流程

image.png

1. 拉取代码到指定位置，并切换分支
2. 执行make build LOCAL_BUILD=true编译二进制文件
3. 执行make image LOCAL_BUILD=true生成容器镜像

依赖分析

构建calico-node镜像依赖的镜像列表

• calico/go-build:v0.54
• registry.access.redhat.com/ubi8/ubi-minimal:8.4
• calico/bird:v0.3.3-182-g4b493986-amd64
• calico/bpftool:v5.3-amd64
• centos:8
• registry.access.redhat.com/ubi8/ubi-minimal:8.4

软件版本依赖

• go
• git
• gcc

在宿主机上调试容器内felix进程

1、 修改node代码的Makefile

// 去掉优化选项
GCFLAGS=-gcflags "all=-N -l"   
// 增加gc编译选项
$(DOCKER_GO_BUILD_CGO) sh -c '$(GIT_CONFIG_SSH) go build -v -o $@ $(BUILD_FLAGS) $(GCFLAGS) $(LDFLAGS) ./cmd/calico-node/main.go'
// 强制写死cgo_enable=0，cgo_enable=1在调试的时候可能有问题，还未确认。
CGO_ENABLED=0

2、修改calico-node的ds，将readness和liveness关闭。
3、在宿主机上下载安装dlv，https://github.com/derekparker/delve
4、make build LOCAL_BUILD=true && make image LOCAL_BUILD=true
5、替换calico-node的镜像为新编译的镜像
6、宿主机执行dlv attach pgrep -f "calico-node -felix"
7、felix由runsv管理，长时间断点，runsv会重启felix，单独调试的话，可以将felix脱离runsv的管理。

[root@master-9 node]# dlv attach `pgrep -f "calico-node -felix"`   
Type 'help' for list of commands.
(dlv) b neigh_linux.go:119
Breakpoint 1 set at 0x2297573 for github.com/vishvananda/netlink.(*Handle).neighAdd() /go/pkg/mod/github.com/vishvananda/[email protected]/neigh_linux.go:119
(dlv) c
> github.com/vishvananda/netlink.(*Handle).neighAdd() /go/pkg/mod/github.com/vishvananda/[email protected]/neigh_linux.go:119 (hits goroutine(788):1 total:1) (PC: 0x2297573)
   114:         return pkgHandle.neighAdd(neigh, mode)
   115: }
   116:
   117: // NeighAppend will append an entry to FDB
   118: // Equivalent to: `bridge fdb append...`
=> 119: func (h *Handle) neighAdd(neigh *Neigh, mode int) error {
   120:         req := h.newNetlinkRequest(unix.RTM_NEWNEIGH, mode|unix.NLM_F_ACK)
   121:         return neighHandle(neigh, req)
   122: }
   123:
   124: // NeighDel will delete an IP address from a link device.
(dlv) c
> github.com/vishvananda/netlink.(*Handle).neighAdd() /go/pkg/mod/github.com/vishvananda/[email protected]/neigh_linux.go:119 (hits goroutine(788):2 total:2) (PC: 0x2297573)
   114:         return pkgHandle.neighAdd(neigh, mode)
   115: }
   116:
   117: // NeighAppend will append an entry to FDB
   118: // Equivalent to: `bridge fdb append...`
=> 119: func (h *Handle) neighAdd(neigh *Neigh, mode int) error {
   120:         req := h.newNetlinkRequest(unix.RTM_NEWNEIGH, mode|unix.NLM_F_ACK)
   121:         return neighHandle(neigh, req)
   122: }
   123:
   124: // NeighDel will delete an IP address from a link device.
(dlv) c
> github.com/vishvananda/netlink.(*Handle).neighAdd() /go/pkg/mod/github.com/vishvananda/[email protected]/neigh_linux.go:119 (hits goroutine(788):3 total:3) (PC: 0x2297573)
   114:         return pkgHandle.neighAdd(neigh, mode)
   115: }
   116:
   117: // NeighAppend will append an entry to FDB
   118: // Equivalent to: `bridge fdb append...`
=> 119: func (h *Handle) neighAdd(neigh *Neigh, mode int) error {
   120:         req := h.newNetlinkRequest(unix.RTM_NEWNEIGH, mode|unix.NLM_F_ACK)
   121:         return neighHandle(neigh, req)
   122: }
   123:
   124: // NeighDel will delete an IP address from a link device.
(dlv) c
> github.com/vishvananda/netlink.(*Handle).neighAdd() /go/pkg/mod/github.com/vishvananda/[email protected]/neigh_linux.go:119 (hits goroutine(788):4 total:4) (PC: 0x2297573)
   114:         return pkgHandle.neighAdd(neigh, mode)
   115: }
   116:
   117: // NeighAppend will append an entry to FDB
   118: // Equivalent to: `bridge fdb append...`
=> 119: func (h *Handle) neighAdd(neigh *Neigh, mode int) error {
   120:         req := h.newNetlinkRequest(unix.RTM_NEWNEIGH, mode|unix.NLM_F_ACK)
   121:         return neighHandle(neigh, req)
   122: }
   123:
   124: // NeighDel will delete an IP address from a link device.
(dlv) c

一些细节

1. 在centos宿主机上执行使用go build 编译 calico-node，需要升级go，git，gcc（编译bpf需要高版本的gcc）等，这些软件都需要编译安装，centos没有现有可用的rpm包。
   https://www.cnblogs.com/hi-blog/p/how-to-update-git-on-centos7.html。
2. 编译过程需要联网下载好多东西，需要宿主机配置好代理。
   mkdir /root/go/src/github.com/projectcalico/ -p
   git clone https://github.com/projectcalico/node.git
   git ls-remote --tags 获取远端tag号
   git checkout -b v3.20.1 1cf251788746268ebd0ebb6bc5b55f1d7c682f81
3. 在构建中也是需要下载好多文件，因为在docker run中也需要加上代理，修改Makefile，docker run 增加代理 -e http_proxy=http://9.9.9.134:8118 -e https_proxy=http://9.9.9.134:8118
   注释掉获取Makefile.common，不需要重复获取
   4、使用make build LOCAL_BUILD=true，修改calico-node代码，可能需要同时修改felix，libcalico等，所以需要加上LOCAL_BUILD来进行go.mod中的依赖替换。

// 编译时将变量进行赋值，这个技巧挺好
LDFLAGS=-ldflags "\
    -X $(PACKAGE_NAME)/pkg/lifecycle/startup.VERSION=$(GIT_VERSION) \
    -X $(PACKAGE_NAME)/buildinfo.GitVersion=$(GIT_DESCRIPTION) \
    -X $(PACKAGE_NAME)/buildinfo.BuildDate=$(DATE) \
    -X $(PACKAGE_NAME)/buildinfo.GitRevision=$(GIT_COMMIT)"

内网编译修改项

calico-node的整个编译和构建镜像的过程还是挺复杂，如果需要在内网进行编译的话，需要适配修改的内容还是挺多的。

• 提前下载编译使用的相关镜像。
• 提前使用go mod下载好依赖。
• 提前准备好Makefile中需要下载的文件。
• Dockerfile.amd64分两部分，写一个Dockerfile将前一部分制作成一个基础镜像，例如叫calico/node:base，避免重复编译，提升效率。
• Dockerfile.amd64第二部分再写一个Dockerfile，只拷贝calico-node二进制到容器内即可。
• 修改Makefile和Dockerfile进行构建。
• 修改gomodcache的路径，映射到容器内。

第二部分的Dockerfile

# Copy everything into a fresh scratch image so that naive CVE scanners don't pick up binaries and libraries
# that have been removed in our later layers.
FROM calico/node:base as calico_base
FROM scratch
COPY --from=calico_base / /

# Add in top-level license file
COPY LICENSE /licenses
 
# Copy in the calico-node binary
COPY dist/bin/calico-node-amd64 /bin/calico-node
 
CMD ["start_runit"]
 
# Required labels for certification
LABEL name="Calico node" \
       vendor="Project Calico" \
       version=$GIT_VERSION \
       release="1" \
       summary="Calico node handles networking and policy for Calico" \
       description="Calico node handles networking and policy for Calico" \
       maintainer="[email protected]"
 
 # Tell sv where to find the services.
 ENV SVDIR=/etc/service/enabled