如何在docker中进行gdb调试

环境准备

install nsenter

nsenter 工具在 util-linux 包2.23版本后包含。 nsenter 可以访问另一个进程的名字空间。nsenter 要正常工作需要有 root 权限。很不幸，Ubuntu 14.04 仍然使用的是 util-linux 2.20。

$ cd /tmp; curl https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz | tar -zxf-; cd util-linux-2.24;
$ ./configure --without-ncurses
$ make nsenter && sudo cp nsenter /usr/local/bin

pull ubuntu image

$ docker pull ubuntu

ubuntu:gdb镜像制作

sources.list

启动ubuntu镜像的容器

$ docker run -it --rm ubuntu bash
root@27b0644624a9:/#

查看ubuntu版本：

root@27b0644624a9:/# cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

因为容器里还没有vi等基础软件，所以在虚机上编辑sources.list：

deb http://mirrors.zte.com.cn/ubuntu/ xenial main multiverse restricted universe
deb http://mirrors.zte.com.cn/ubuntu/ xenial-backports main multiverse restricted universe
deb http://mirrors.zte.com.cn/ubuntu/ xenial-proposed main multiverse restricted universe
deb http://mirrors.zte.com.cn/ubuntu/ xenial-security main multiverse restricted universe
deb http://mirrors.zte.com.cn/ubuntu/ xenial-updates main multiverse restricted universe
deb-src http://mirrors.zte.com.cn/ubuntu/ xenial main multiverse restricted universe
deb-src http://mirrors.zte.com.cn/ubuntu/ xenial-backports main multiverse restricted universe
deb-src http://mirrors.zte.com.cn/ubuntu/ xenial-proposed main multiverse restricted universe
deb-src http://mirrors.zte.com.cn/ubuntu/ xenial-security main multiverse restricted universe
deb-src http://mirrors.zte.com.cn/ubuntu/ xenial-updates main multiverse restricted universe

拷贝sources.list到容器的/etc/apt/目录下：

$ docker cp ./sources.list 27:/etc/apt/

install some software for gdb

更新源的索引：

root@27b0644624a9:/# apt-get update

gdb依赖python的3.x版本。在ubuntu14.04中，gdb依赖的python版本是3.4，那么在ubuntu16.04中，gdb依赖的python版本应该不低于3.4。
搜索一下python3.x版本的程序名：

root@27b0644624a9:/# apt-cache search python

安装python:

root@27b0644624a9:/# apt-get install python3

查询python版本号，符合预期：

root@27b0644624a9:/# python3 --version
Python 3.5.2

安装gdb:

root@27b0644624a9:/# apt-get install gdb

验证gdb是否可以run:

root@27b0644624a9:/# gdb
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb)

vi是很基础的编辑工具，也建议安装：

root@27b0644624a9:/# apt-get install vim

save ubuntu:gdb image

$ docker commit 27 ubuntu:gdb
$ docker save -o ubuntu-gdb.tar ubuntu:gdb

ubuntu:gdb镜像验证

Dockerfile

Benchmark是笔者用C++开发的一个关于网络性能的测试工具，它同时支持linux socket和dpdk socket，所以对dpdk的动态库有依赖。
Dockerfile文件描述：

FROM ubuntu:gdb

COPY ./libdpdk.so /lib/
COPY ./Benchmark /Benchmark

ENTRYPOINT ["/Benchmark"]

制作镜像：

$ docker build -t ubuntu-gdb:test .
$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
ubuntu-gdb          test                10f840213452        About a minute ago   294 MB

run image

$ docker run -it --rm ubuntu-gdb:test bash
is dpdk 0
self ip is 0
self port is 10008
socket fd is 3
msgReceive: now begin receive msg!

查看容器Id:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
fd97fcd35a67        ubuntu-gdb:test     "/Benchmark bash"   About a minute ago   Up About a minute                       priceless_bell

进入容器：

$ docker exec -it fd bash
root@fd97fcd35a67:/#

gdb debug

查看进程号：

root@fd97fcd35a67:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1 46.7  0.0  33016  1964 ?        Ssl+ 10:00   7:17 /Benchmark bash
root         7  0.0  0.0  18244  3340 ?        Ss   10:08   0:00 bash
root        18  0.0  0.0  34424  2912 ?        R+   10:16   0:00 ps aux

查看该容器对应的Pid:

$ docker inspect -f {{.State.Pid}} fd
15099

先输入nsenter命令使gdb attach成功，然后再分别输入info threads和bt命令确认响应没毛病：

$  sudo nsenter -t 15099 -m -p gdb -p 1
[sudo] password for zte: 
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 1
[New LWP 6]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fc7f369298d in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7fc7f41b1940 (LWP 1) "Benchmark" 0x00007fc7f369298d in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x7fc7f281b700 (LWP 6) "Benchmark" 0x00007fc7f369a8f3 in recvfrom () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0  0x00007fc7f369298d in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000000000407e57 in main (argc=2, argv=0x7fffe3029a08) at /home/zte/gitlab/benchmark/src/Main.cpp:135
(gdb)

应用场景

gdb调试一般是不得已的选择。我们一般先是通过代码走查和测试来找bug，其次是通过日志分析找bug，最后的最后才是通过gdb找bug。话虽如此，但gdb调试仍然很重要，比如程序“卡死”等场景。
每个应用容器都依赖gdb并不是能在容器中进行gdb调试的唯一选择。在有kubernetes集群的系统中，gdb所在的容器可以在一个pod中挂一个，当出现“顽疾”时通过共享文件系统来调试程序。

小结

本文以ubuntu系统为例，先介绍了环境准备，然后带着读者一起制作了ubuntu:gdb镜像，最后以C++程序为例验证了ubuntu:gdb镜像的可用性。本文所讲的对于docker的gdb调试方法具有一定的通用性，大家可以在其他的linux系统上如法炮制，同时支持gdb调试的语言都可以通过本文介绍的方法在docker中调试，比如Golang，它和C/C++的gdb调试稍有不同，具体请参考官方文档《Debugging Go Code with GDB》。在虚拟化时代，掌握docker中的gdb调试方法是一个程序员的基本技能之一。