Docker----gitlab-runner自动化部署

一、gitlab-runner简介

GitLab Runner是一个开源项目，用于运行您的作业并将结果发送回GitLab。它与GitLab CI一起使用，GitLab CI是GitLab随附的开源持续集成服务，用于协调作业。

特征：

• 允许运行：
  • 多个工作同时进行。
  • 使用多个令牌与多个服务器（甚至每个项目）。
  • 限制每个令牌的并发作业数。
• 可以运行工作：
  • 本地。
  • 使用Docker容器。
  • 使用Docker容器并通过SSH执行作业。
  • 使用Docker容器在不同的云和虚拟化管理程序上进行自动扩展。
  • 连接远程SSH服务器。
• 用Go编写并以单个二进制形式分发，没有任何其他要求。
• 支持Bash，Windows Batch和Windows PowerShell。
• 适用于GNU / Linux，macOS和Windows（几乎可以在任何地方运行Docker）。
• 允许自定义作业运行环境。
• 自动配置重新加载而不重启。
• 易于使用的设置，支持Docker，Docker-SSH，Parallels或SSH运行环境。
• 启用Docker容器的缓存。
• 易于安装作为GNU / Linux，macOS和Windows的服务。
• 嵌入式Prometheus指标HTTP服务器

二、docker搭建gitlab-runner

docker hub 镜像仓库地址

不太懂docker的可以参考我的一篇关于docker常用命令总结的文章

cyf@KobeBryant:~$ mkdir -p docker/gitlab/runner && cd docker/gitlab/runner
cyf@KobeBryant:~/docker/gitlab/runner$ docker run -d --rm --name runner gitlab/gitlab-runner:v11.0.0
cyf@KobeBryant:~/docker/gitlab/runner$ docker cp gitlab-runner:/etc/gitlab-runner && mv gitlab-runner config
cyf@KobeBryant:~/docker/gitlab/runner$ docker stop runner
cyf@KobeBryant:~$ docker run -d --add-host gitlab.cyf.com:192.168.10.24 -v /home/cyf/docker/gitlab/runner/config:/etc/gitlab-runner -v /home/cyf/docker/gitlab/runner/scripts/:/home/gitlab-runner/scripts -v /var/run/docker.sock:/var/run/docker.sock --name runner gitlab/gitlab-runner:v11.1.0

映射目录说明：

• scripts脚本目录映射：添加一些脚本供运行runner时使用，可以根据需要自行定义
• config配置文件映射：便于修改config.toml文件，对不同注册的runner做一些配置修改
• docker.sock：Docker守护进程默认监听的Unix域套接字，容器中的进程可以通过它与Docker守护进程进行通信

三、集成gitlab CI/CD流水线

1、注册runner到gitlab服务器（以我目前应用的runner为例）

gitlab服务器搭建请参考我的另一篇文章：docker容器搭建----gitlab代码仓库

maven：用于java程序打包、跑单元测试、代码审核（sonarqube）

cyf@KobeBryant:~/docker/gitlab$ docker exec -it runner gitlab-runner register
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://192.168.10.24/
Please enter the gitlab-ci token for this runner:
maGJhGKD4UgLSxJ-Z1E_
Please enter the gitlab-ci description for this runner:
[cfb1f6f983f3]: maven-java
Please enter the gitlab-ci tags for this runner (comma separated):
maven
Registering runner... succeeded                     runner=maGJhGKD
Please enter the executor: kubernetes, docker, virtualbox, docker+machine, ssh, docker-ssh+machine, docker-ssh, parallels, shell:
docker
Please enter the default Docker image (e.g. ruby:2.1):
cuiyf/maven3-jdk8
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

docker：用于构建服务镜像、推送镜像到远程仓库

cyf@KobeBryant:~/docker/gitlab$ docker exec -it runner gitlab-runner register
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://192.168.10.24/
Please enter the gitlab-ci token for this runner:
maGJhGKD4UgLSxJ-Z1E_
Please enter the gitlab-ci description for this runner:
[cfb1f6f983f3]: docker-run
Please enter the gitlab-ci tags for this runner (comma separated):
docker
Registering runner... succeeded                     runner=maGJhGKD
Please enter the executor: parallels, virtualbox, docker+machine, kubernetes, docker-ssh, shell, ssh, docker-ssh+machine, docker:
docker
Please enter the default Docker image (e.g. ruby:2.1):
cuiyf/docker:18.05.0
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

shell：用于登录、部署服务到远程docker主机

cyf@KobeBryant:~/docker/gitlab$ docker exec -it runner gitlab-runner register
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://192.168.10.24/
Please enter the gitlab-ci token for this runner:
maGJhGKD4UgLSxJ-Z1E_
Please enter the gitlab-ci description for this runner:
[cfb1f6f983f3]: shell
Please enter the gitlab-ci tags for this runner (comma separated):
shell
Registering runner... succeeded                     runner=maGJhGKD
Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, parallels, shell, virtualbox, docker, docker-ssh:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

2、注册结果如下图

3、注册完config.toml配置如下

concurrent = 1
check_interval = 0

[[runners]]
  name = "shell"
  url = "http://gitlab.cyf.com/"
  token = "65ad803959a1cdb6d9f9b89d374afb"
  executor = "shell"
  [runners.cache]

[[runners]]
  name = "docker"
  url = "http://gitlab.cyf.com/"
  token = "3d8e1a1c0b56ee6a952ab8513ad0a7"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "cuiyf/docker:18.05.0"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

[[runners]]
  name = "maven"
  url = "http://gitlab.cyf.com/"
  token = "4cbc8676e2fedd9a199c32f707fca2"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "cuiyf/maven3-jdk8"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

四、config-toml配置简介

1、全局配置

详情见：https://docs.gitlab.com/runner/configuration/advanced-configuration.html

设置	描述
concurrent	限制全局可以同时运行的作业数
check_interval	定义新作业检查之间的间隔长度（以秒为单位

2、[[runners]]部分

设置	描述
name	Runner的描述，只是提供信息
url	gitlab服务器地址
token	Runner的特殊令牌（不要与注册令牌混淆）
executor	选择应如何构建项目（ssh, docker+machine, docker-ssh+machine, kubernetes, parallels, shell, virtualbox, docker, docker-ssh）
limit	限制此类型runner可同时处理的作业数

3、[runners.docker]部分

设置	描述
tls_verify	启用或禁用与Docker守护程序的连接的TLS验证。默认情况下禁用
image	使用此图像运行构建
privileged	使容器以特权模式运行（不安全）
disable_cache	禁用使用自动创建（未映射到主机目录）缓存卷的本地标志
volumes	指定应安装的其他卷
shm_size	指定映像的共享内存大小（以字节为单位）

五、使用gitlab-runner部署项目

1、上传项目到gitlab（使用一个特殊方法）

在gitlab上新建一个eureka项目，拉去github上的eureka项目到本地，然后推送到gitlab上，git命令不懂请参考我的另一篇博客：git常用命令总结

cyf@KobeBryant:~/git$ git clone http://192.168.10.24/root/eureka.git
cyf@KobeBryant:~/git$ git clone https://github.com/github-cyf/eureka_server.git
cyf@KobeBryant:~/git$ mv eureka_server/* eureka
cyf@KobeBryant:~/git$ mv eureka_server/.* eureka
cyf@KobeBryant:~/git$ cd eureka_server/
cyf@KobeBryant:~/git/eureka_server$ git add .
cyf@KobeBryant:~/git/eureka_server$ git commit -m "上传eureka项目"
cyf@KobeBryant:~/git/eureka_server$ git push

2、上传.gitlab-ci.yml文件到项目/目录，配置如下

stages:
  - test
  - build
  - deploy
cache:
  paths:
    - .m2/
    
variables:
  MAVEN_OPTS: "-Dmaven.repo.local=.m2"
  MAVEN_IMAGE: cuiyf/maven3-jdk8:latest
  CONTAINER_NAME: eureka-cyf
  PROD_REPO: registry.cn-beijing.aliyuncs.com/cyf/all
  
build image:
  image: $MAVEN_IMAGE
  stage: test
  script:
    - mvn  package docker:build -q -Dmaven.test.skip=false
  tags:
    - maven
  only:
    - master

push image prod:
  stage: build
  before_script:
    - docker login --username=$USERNAME --password=$PASSWORD registry.cn-beijing.aliyuncs.com
  script:
    - docker tag $DEV_REPO:$CONTAINER_NAME  $PROD_REPO:$CONTAINER_NAME
    - docker push $PROD_REPO:$CONTAINER_NAME
  tags:
    - docker
  only:
    - master
  when: on_success
      
ssh run on remote machine prod:
  stage: deploy
  script:
    - bash ~/scripts/ssh-deploy.sh $CONTAINER_NAME 1025 1025 $PROD_REPO:$CONTAINER_NAME cyf 192.168.10.24
  tags:
    - shell
  only:
    - master
  when: on_success

注意：

1. ssh-deploy.sh此脚本在容器内部的/home/gitlab-runner/scripts目录下，内容如下：

#!/bin/bash      
ssh -p 24  $5@$6 <<EOF
cd;bash ~/docker/gitlab/runner/deploy.sh $1 $2 $3 $4                                                                                                                                                                            
EOF

2. deploy.sh此脚本在远程部署主机~/docker/gitlab/runner/目录下，内容如下：

#!/bin/bash
conName=$1
eonPort=$2
conPort=$3
images=$4
count=`docker ps -a  |grep "$conName"|wc -l `
if [ $count -eq 0 ];then
	echo "$conName container  is not exit"
else 
	id=$(docker ps -a |grep "$conName" |awk '{print $1}')
for i in $id 
do  
	docker stop $i
	docker rm $i
done						     
fi
docker rmi $images
docker run --restart=always -d --name $conName -p $eonPort:$conPort $images

3. ssh登录远程主机，需要配置ssh公钥，配置过程如下：

cyf@KobeBryant:~$ docker exec -it gitlab-runner bash
root@6af0ce3a3254:/# su gitlab-runner
gitlab-runner@6af0ce3a3254:/$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/gitlab-runner/.ssh/id_rsa): 
Created directory '/home/gitlab-runner/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/gitlab-runner/.ssh/id_rsa.
Your public key has been saved in /home/gitlab-runner/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:0ZzqZODcTcUhmHBC+lE71BmBz0UTOAh3Gck+xl4QIcg gitlab-runner@6af0ce3a3254
The key's randomart image is:
+---[RSA 2048]----+
|     o=+*BO&*o   |
|     .E*=*X+o.   |
|    . o +==+     |
|     + + *B .    |
|      + So.o     |
|       +  .      |
|        .        |
|                 |
|                 |
+----[SHA256]-----+
gitlab-runner@6af0ce3a3254:/$ ssh-copy-id -i ~/.ssh/id_rsa.pub -p24 [email protected]
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/gitlab-runner/.ssh/id_rsa.pub"
The authenticity of host '[192.168.10.24]:24 ([192.168.10.24]:24)' can't be established.
ECDSA key fingerprint is SHA256:vPilmOy8x6qiFv6zfl47vD2st2gI6sIkHXdijb2qAu0.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -p '24' '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

3、运行流水线（添加.gitlab-ci.yml将会自动运行）

首次运行test阶段时遇到了一个docker守护进程的问题，问题如下：

修改gitlab-runner配置文件，volumes字段配置，配置如下

    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]

当所有阶段均通过即部署完毕如下图：

六、构建镜像自动清理

1、需要清理的原因

频繁构建项目，会产生好多悬挂镜像和停止状态的容器，占用磁盘空间，需要定时清理

2、搭建gitlab-runner-docker-cleanup，自动清理缓存和映像，

详细地址说明：https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup

docker run -d \
    -e LOW_FREE_SPACE=10G \
    -e EXPECTED_FREE_SPACE=20G \
    -e LOW_FREE_FILES_COUNT=1048576 \
    -e EXPECTED_FREE_FILES_COUNT=2097152 \
    -e DEFAULT_TTL=10m \
    -e USE_DF=1 \
    --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --name=gitlab-runner-docker-cleanup \
    quay.io/gitlab/gitlab-runner-docker-cleanup

各个环境变量介绍：

变量	默认值	描述
CHECK_PATH	/	检查磁盘使用情况时使用的路径
LOW_FREE_SPACE	1GB	触发缓存和图像删除时
EXPECTED_FREE_SPACE	2GB	清理多少可用空间
LOW_FREE_FILES_COUNT	131072	当空闲文件（i节点）的数量低于此值时，将触发缓存和图像删除
EXPECTED_FREE_FILES_COUNT	262144	要清理多少个可用文件（i-nodes）
USE_DF	false	使用命令行df工具检查磁盘空间。设置为false连接到远程Docker Engine时。设置为true与本地安装的Docker Engine一起使用时
CHECK_INTERVAL	10s	多久检查一次磁盘空间
retry_interval的	30s	在失败的情况下重试之前需要等待多长时间
DEFAULT_TTL	1m	保留新下载的映像或创建的缓存的最短时间

3、设置定时任务清理

设置每天0时0分清理一下悬挂镜像和无用存储卷，0 0 */1 * * /bin/bash /home/cyf/scripts/clean-docker.sh，简单脚本如下：

#!/bin/bash
docker image prune -f && docker volume prune -f