gitlab-runner 失联了

有的时候我们会遇到这么一个问题，gitlab 的ci 跑不了了。 原因是runner全都不活跃了 —— 翻译成人话，runner按钮全都灰了，这个时候怎么办呢

故障排查

• 进container docker exec -it gitlab-runner
• 查看runners的状态

  root@7cffd231aff6:/# gitlab-runner status
  gitlab-runner: Service is not running.
  

• 重启runners

  root@7cffd231aff6:/# gitlab-runner run
  Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
  Running in system-mode.
  
  Configuration loaded                                builds=0
  Metrics server disabled
  WARNING: Checking for jobs... failed                runner=1c7783cc status=couldn't execute POST against http://gitlab.xx.com/api/v4/jobs/request: Post http://gitlab.xx.com/api/v4/jobs/request: dial tcp: i/o timeout
  

• 发现可能是 runner 这台服务器连不上 gitlab.xx.com 了， 经验证果然是这个问题

  gitlab-runner# ping git.jd.com
  ping: unknown host git.jd.com
  

解决方案

初步判断是docker container 访问不了外部网络

• docker container 网络访问的几种方式
  • bridge: 默认模式
  • host: 与宿主机共享IP和端口
  • container: 共享其他container的网络
  • none: 不联网
• 验证其他container是否能够正常访问
  • 拉取一个 hello-world image

    gitlab-runner# docker run -it hello-world
    Unable to find image 'hello-world:latest' locally
    Trying to pull repository docker.io/library/hello-world ...
    latest: Pulling from docker.io/library/hello-world
    9db2ca6ccae0: Pull complete
    Digest: sha256:135a30bd414bd8d23e386763e36f3dc4ee8ed25d2d6068082c0af796513d9d0d
    Status: Downloaded newer image for docker.io/hello-world:latest
    WARNING: IPv4 forwarding is disabled. Networking will not work.
    

  • bingo 服务器上的 ip_forward 被限制无法访问
• 设置 ip_forward=1 sysctl -w net.ipv4.ip_forward=1

参考文章

• 容器间网络通信设置
• ip_forward与路由转发