CI/CD学习笔记-Jenkins综合项目Github+Jenkins+Harbor+Docker

实验说明

实验拓扑

部署流程

1. 研发push到github代码库

2. Jenkins 构建，pull git代码 使用maven进行编译打包

3. 打包生成的代码，生成一个新版本的镜像，push到本地docker仓库harbor

4. 发布，测试机器 pull 新版本的镜像，并删除原来的容器，重新运行新版本镜像。

实验参数

项目	描述	备注
代码仓库	https://github.com
容器镜像仓库	Harbor V2.3	harbor.corp.tanzu
CI/CD服务器	Jenkins 2.289.2-1.1	jenkins.corp.tanzu; \n Installed:JDK/Jenkins/git/maven/docker
测试服务器	安装Docker	192.168.110.27

部署服务

部署Harbor镜像仓库

参考：基于SSL的Harbor镜像仓库的搭建，创建用户zyi
创建公有项目jenkins

测试服务器

安装Docker和jq

yum install docker jq -y //后面的脚本会用到,jq类似于sed/awk专门处理json格式的文件
systemctl start docker

预先配置

上配置：
#visudo
#Defaults requiretty
Defaults:root !requiretty

否则在机器业务机器上执行脚本时会报错：
[SSH] executing…
sudo: sorry, you must have a tty to run sudo
docker: invalid reference format.

Jenkins服务部署

See:CI/CD学习笔记-jenkins安装

预先配置

由于在Jenkins机器上docker是使用root用户运行的，而Jenkins是使用普通用户jenkins运行的，所以要先配置下jenkins用户可以使用docker命令。

[root@jenkins ~]# visudo
jenkins ALL=(root) NOPASSWD: /usr/bin/docker

另外j基于以下原因在Jenkins机器上配置visudo

#Disable "ssh hostname sudo ", because it will show the password in clear.
#You have to run "ssh -t hostname sudo ".

[root@jenkins ~]#visudo
#Defaults requiretty
Defaults:jenkins !requiretty

如果不配置这个，在执行下面脚本时，会报错误：

• cp -f /home/jenkins/.jenkins/workspace/godseyeBranchForNov/godseye-container/target/godseye-container-wisedu.war /home/jenkins/docker-file/godseye_war/godseye.war
• sudo docker login -u zyi -p W123 -e [email protected] jenkins.corp.tanzu
  sudo: sorry, you must have a tty to run sudo

安装插件

插件"Maven Integration"和"SSH"
点击“Manage Jenkins”–>“Plugin Manager”，搜索插件"Maven Integration"和"SSH"，进行安装。

配置远程接入测试服务器

点击“Manage Jenkins”–>“Manage Credentials ”


选择用户名密码方式

构建JOB

代码源

使用Git，代码地址： https://github.com/etaon/easy-springmvc-maven.git


Goals and options填写：clean package -Dmaven.test.skip=true

Post Step

需要完成以下任务：

1. Jenkins服务器：编译完成后，build生成一个新版本的镜像，push到远程harbor仓库；
2. 远程测试服务器：拉取镜像、发布

构建新镜像并push到Harbor

配置可执行脚本

脚本方式提交：

# Jenkins机器：编译完成后，build生成一个新版本的镜像，push到远程harbor仓库
 
# Variables
JENKINS_WAR_HOME='/var/lib/jenkins/workspace/maven-docker/target'
DOCKERFILE_HOME='/var/lib/jenkins/docker-file/maven-docker-test_war'
HARBOR_IP='harbor.corp.tanzu'
REPOSITORIES='jenkins/maven-docker'
HARBOR_USER='zyi'
HARBOR_USER_PASSWD='VMware1!'
HARBOR_USER_EMAIL='[email protected]'
 
# Copy the newest war to docker-file directory.
\cp -f ${JENKINS_WAR_HOME}/easy-springmvc-maven.war ${DOCKERFILE_HOME}/maven-docker.war
 
# Delete image early version.
sudo docker login -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} ${HARBOR_IP} 
IMAGE_ID=`sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'`
if [ -n "${IMAGE_ID}" ];then
    sudo docker rmi ${IMAGE_ID}
fi

sudo docker login -u dewinals -p xxxxxx
# Build image.
cd ${DOCKERFILE_HOME}
TAG=`date +%Y%m%d-%H%M%S`
sudo docker build -t ${HARBOR_IP}/${REPOSITORIES}:${TAG} . &>/dev/null
 
sudo docker login -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} ${HARBOR_IP}  
# Push to the harbor registry.
sudo docker push ${HARBOR_IP}/${REPOSITORIES}:${TAG} &>/dev/null

注意：
1.在*DOCKERFILE_HOME=’/var/lib/jenkins/docker-file/maven-docker-test_war’*下构建Dockerfile

# cd /root/jenkins/docker-file/maven-docker-test_war
# vim Dockerfile
# Version 1.0
# Base images.
FROM tomcat:8.0.36-alpine

# Author.
MAINTAINER zyi >

# Add war.
ADD maven-docker.war /usr/local/tomcat/webapps/

# Define working directory.
WORKDIR /usr/local/tomcat/bin/

# Define environment variables.
ENV PATH /usr/local/tomcat/bin:$PATH

# Define default command.
CMD ["catalina.sh", "run"]

# Expose ports.
EXPOSE 8080

2. 赋权文件夹docker-file

chmod 777 docker-file/ -R

3. Dockerfile所需的base images：tomcat需要从dockers.io下载，构建之前登录

sudo docker login -u dewinals -p xxxxxx

Build Control output

Started by user yizhao
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/maven-docker
The recommended git tool is: NONE
No credentials specified
/usr/bin/git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/maven-docker/.git # timeout=10
Fetching changes from the remote Git repository
/usr/bin/git config remote.origin.url https://github.com/etaon/easy-springmvc-maven.git # timeout=10
Fetching upstream changes from https://github.com/etaon/easy-springmvc-maven.git
/usr/bin/git --version # timeout=10
git --version # ‘git version 2.27.0’
/usr/bin/git fetch --tags --force --progress – https://github.com/etaon/easy-springmvc-maven.git +refs/heads/:refs/remotes/origin/ # timeout=10
/usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 67604f7f9f30505e3bb3e8935c745154f04aa372 (refs/remotes/origin/master)
/usr/bin/git config core.sparsecheckout # timeout=10
/usr/bin/git checkout -f 67604f7f9f30505e3bb3e8935c745154f04aa372 # timeout=10
Commit message: “修改standard/1.1.2的依赖”
/usr/bin/git rev-list --no-walk 67604f7f9f30505e3bb3e8935c745154f04aa372 # timeout=10
Parsing POMs
Established TCP socket on 33405
[maven-docker] $ /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64//bin/java -cp /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven35-agent-1.13.jar:/usr/share/maven/boot/plexus-classworlds-2.5.2.jar:/usr/share/maven/conf/logging jenkins.maven3.agent.Maven35Main /usr/share/maven /var/cache/jenkins/war/WEB-INF/lib/remoting-4.7.jar /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven35-interceptor-1.13.jar /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.13.jar 33405
<=[JENKINS REMOTING CAPACITY]=>channel started
Executing Maven: -B -f /var/lib/jenkins/workspace/maven-docker/pom.xml clean package -Dmaven.test.skip=true
[INFO] Scanning for projects…
[WARNING]
[WARNING] Some problems were encountered while building the effective model for springmvc-maven:easy-springmvc-maven:war:0.0.1-SNAPSHOT
[WARNING] ‘build.plugins.plugin.version’ for org.apache.maven.plugins:maven-war-plugin is missing. @ line 22, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ----------------< springmvc-maven:easy-springmvc-maven >----------------
[INFO] Building springmvc-maven 0.0.1-SNAPSHOT
[INFO] --------------------------------[ war ]---------------------------------
[INFO]
[INFO] — maven-clean-plugin:2.5:clean (default-clean) @ easy-springmvc-maven —
[INFO] Deleting /var/lib/jenkins/workspace/maven-docker/target
[INFO]
[INFO] — maven-resources-plugin:2.6:resources (default-resources) @ easy-springmvc-maven —
[INFO] Using ‘UTF-8’ encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /var/lib/jenkins/workspace/maven-docker/src/main/resources
[INFO]
[INFO] — maven-compiler-plugin:3.1:compile (default-compile) @ easy-springmvc-maven —
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /var/lib/jenkins/workspace/maven-docker/target/classes
[INFO]
[INFO] — maven-resources-plugin:2.6:testResources (default-testResources) @ easy-springmvc-maven —
[INFO] Not copying test resources
[INFO]
[INFO] — maven-compiler-plugin:3.1:testCompile (default-testCompile) @ easy-springmvc-maven —
[INFO] Not compiling test sources
[INFO]
[INFO] — maven-surefire-plugin:2.12.4:test (default-test) @ easy-springmvc-maven —
[INFO] Tests are skipped.
[INFO]
[INFO] — maven-war-plugin:2.2:war (default-war) @ easy-springmvc-maven —
[INFO] Packaging webapp
[INFO] Assembling webapp [easy-springmvc-maven] in [/var/lib/jenkins/workspace/maven-docker/target/easy-springmvc-maven]
[INFO] Processing war project
[INFO] Copying webapp resources [/var/lib/jenkins/workspace/maven-docker/src/main/webapp]
[INFO] Webapp assembled in [105 msecs]
[INFO] Building war: /var/lib/jenkins/workspace/maven-docker/target/easy-springmvc-maven.war
[INFO] WEB-INF/web.xml already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.080 s
[INFO] Finished at: 2021-07-23T01:03:44-07:00
[INFO] ------------------------------------------------------------------------
Waiting for Jenkins to finish collecting data
[JENKINS] Archiving /var/lib/jenkins/workspace/maven-docker/pom.xml to springmvc-maven/easy-springmvc-maven/0.0.1-SNAPSHOT/easy-springmvc-maven-0.0.1-SNAPSHOT.pom
[JENKINS] Archiving /var/lib/jenkins/workspace/maven-docker/target/easy-springmvc-maven.war to springmvc-maven/easy-springmvc-maven/0.0.1-SNAPSHOT/easy-springmvc-maven-0.0.1-SNAPSHOT.war
channel stopped
[maven-docker] $ /bin/sh -xe /tmp/jenkins711386174496231458.sh

• JENKINS_WAR_HOME=/var/lib/jenkins/workspace/maven-docker/target
• DOCKERFILE_HOME=/var/lib/jenkins/docker-file/maven-docker-test_war
• HARBOR_IP=harbor.corp.tanzu
• REPOSITORIES=jenkins/maven-docker
• HARBOR_USER=zyi
• HARBOR_USER_PASSWD=‘VMware1!’
• [email protected]
• cp -f /var/lib/jenkins/workspace/maven-docker/target/easy-springmvc-maven.war /var/lib/jenkins/docker-file/maven-docker-test_war/maven-docker.war
• sudo docker login -u zyi -p ‘VMware1!’ harbor.corp.tanzu
  WARNING! Using --password via the CLI is insecure. Use --password-stdin.
  WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
  Configure a credential helper to remove this warning. See
  https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
++ sudo docker images
++ grep jenkins/maven-docker
++ awk ‘{print $3}’

• IMAGE_ID=d0dcd354b02b
• ‘[’ -n d0dcd354b02b ‘]’
• sudo docker rmi d0dcd354b02b
  Untagged: harbor.corp.tanzu/jenkins/maven-docker:20210723-005858
  Untagged: harbor.corp.tanzu/jenkins/maven-docker@sha256:a759dc8c7beb0f3a918c9dcba9a7032236714a6c10db61655536b489f96933ee
  Deleted: sha256:d0dcd354b02bf2a48925498b6abed56a42152e6a5fa0eddd40f4fc4a60180466
  Deleted: sha256:9b645e1a904b6a46adddb7c2ab169918d2b76000cb9482a23e7d8a64bd6b8f56
  Deleted: sha256:9877d1031b7a5adbf70df56ce3f16174da7e289a47e412cd0d83bbd352f45c30
  Deleted: sha256:19d261fd72b302179fb46184d6cbc26528478bd047daa3f4db6ad32e5327e380
  Deleted: sha256:38e37a14460f2dc6b9eb9794635107d9ad13e6ee7e0e4be229066d59dabbcd91
  Deleted: sha256:2cc14fc784b435ec741c3077be2a13fd9c7b1456430ae6b528c451797949395d
  Deleted: sha256:eeb150de68bf106cb41e0262d03911f278bc81e3f31b84aacf84445705a42e65
• sudo docker login -u dewinals -p xxxxxx
  WARNING! Using --password via the CLI is insecure. Use --password-stdin.
  WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
  Configure a credential helper to remove this warning. See
  https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

• cd /var/lib/jenkins/docker-file/maven-docker-test_war
  ++ date +%Y%m%d-%H%M%S
• TAG=20210723-010352
• sudo docker build -t harbor.corp.tanzu/jenkins/maven-docker:20210723-010352 .
• sudo docker login -u zyi -p ‘VMware1!’ harbor.corp.tanzu
  WARNING! Using --password via the CLI is insecure. Use --password-stdin.
  WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
  Configure a credential helper to remove this warning. See
  https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

• sudo docker push harbor.corp.tanzu/jenkins/maven-docker:20210723-010352
  [SSH] script:
  USER=“jenkins”

在Harbor上查看上传结果

测试服务器拉取镜像，发布（ssh）

配置ssh脚本

在Post Step选择

脚本

#!/bin/bash

# 拉取镜像，发布
HARBOR_IP='harbor.corp.tanzu'
REPOSITORIES='jenkins/maven-docker'
HARBOR_USER='admin'
HARBOR_USER_PASSWD='VMware1!'
 
echo $HARBOR_IP

# 登录harbor
docker login -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} ${HARBOR_IP}
 
# Stop container, and delete the container.
CONTAINER_ID=`docker ps | grep "maven-docker" | awk '{print $1}'`
if [ -n "$CONTAINER_ID" ]; then
    docker stop $CONTAINER_ID
    docker rm $CONTAINER_ID
else #如果容器启动时失败了，就需要docker ps -a才能找到那个容器
    CONTAINER_ID=`docker ps -a | grep "maven-docker" | awk '{print $1}'`
    if [ -n "$CONTAINER_ID" ]; then  # 如果是第一次在这台机器上拉取运行容器，那么docker ps -a也是找不到这个容器的
        docker rm $CONTAINER_ID
    fi
fi
 
# Delet docker-maven image early version.
IMAGE_ID=`sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'`
if [ -n "${IMAGE_ID}" ];then
    docker rmi ${IMAGE_ID}
fi
 
# Pull image.
 wget  $HARBOR_IP/api/v2.0/projects/jenkins/repositories/maven-docker/artifacts --no-check-certificate
 TAG=`awk -F 'name":"'  '{print $2}' artifacts |cut -d '"' -f1`

docker pull ${HARBOR_IP}/${REPOSITORIES}:${TAG} &>/dev/null

# Run.
docker run -d --name maven-docker-test -p 8080:8080 ${HARBOR_IP}/${REPOSITORIES}:${TAG}

注意：

1. 本shell分为两部分:删除旧本地镜像和pull新镜像并部署
2. 为了得到harbor镜像的tag，使用了以下代码

wget  $HARBOR_IP/api/v2.0/projects/jenkins/repositories/maven-docker/artifacts --no-check-certificate #下载包含tag的文件
TAG=`awk -F 'name":"'  '{print $2}' artifacts |cut -d '"' -f1`#awk提取tag

Build Control output

#!/bin/bash
#拉取镜像，发布
HARBOR_IP=‘harbor.corp.tanzu’
REPOSITORIES=‘jenkins/maven-docker’
HARBOR_USER=‘admin’
HARBOR_USER_PASSWD=‘VMware1!’
echo $HARBOR_IP
#登录harbor
docker login -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD} ${HARBOR_IP}
#Stop container, and delete the container.
CONTAINER_ID=docker ps | grep "maven-docker" | awk '{print $1}'
if [ -n “$CONTAINER_ID” ]; then
docker stop $CONTAINER_ID
docker rm $CONTAINER_ID
else #如果容器启动时失败了，就需要docker ps -a才能找到那个容器
CONTAINER_ID=docker ps -a | grep "maven-docker" | awk '{print $1}'
if [ -n “$CONTAINER_ID” ]; then # 如果是第一次在这台机器上拉取运行容器，那么docker ps -a也是找不到这个容器的
docker rm $CONTAINER_ID
fi
fi
#Delet docker-maven image early version.
IMAGE_ID=sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'
if [ -n “${IMAGE_ID}” ];then
docker rmi ${IMAGE_ID}
fi
#Pull image.
wget $HARBOR_IP/api/v2.0/projects/jenkins/repositories/maven-docker/artifacts --no-check-certificate
TAG=awk -F 'name":"' '{print $2}' artifacts |cut -d '"' -f1
docker pull $HARBO{R}_{I}P/${REPOSITORIES}{TAG} &>/dev/null
#Run.
docker run -d --name maven-docker-test -p 8080:8080 $HARBO{R}_{I}P/${REPOSITORIES}{TAG}

[SSH] executing…
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

harbor.corp.tanzu
Login Succeeded
URL transformed to HTTPS due to an HSTS policy
–2021-07-23 03:12:42-- https://harbor.corp.tanzu/api/v2.0/projects/jenkins/repositories/maven-docker/artifacts
Resolving harbor.corp.tanzu (harbor.corp.tanzu)… 192.168.110.2
Connecting to harbor.corp.tanzu (harbor.corp.tanzu)|192.168.110.2|:443… connected.
WARNING: The certificate of ‘harbor.corp.tanzu’ is not trusted.
WARNING: The certificate of ‘harbor.corp.tanzu’ hasn’t got a known issuer.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/json]
Saving to: ‘artifacts’ 0K .......... ........ 72.6M=0s
2021-07-23 03:12:42 (72.6 MB/s) - ‘artifacts’ saved [19341]
Untagged: harbor.corp.tanzu/jenkins/maven-docker:20210723-020658
Untagged: harbor.corp.tanzu/jenkins/maven-docker@sha256:6eeff5ee0e17aa4fba5f39cf667a35d05c271b45c52d96a1671ee8cdcc3bbb73
Deleted: sha256:3e3d1ecab48f251264c6538fd6459ed01af8a92010943b2eb61ffc14a3ffc954
Deleted: sha256:b1c870aac2cc56f68b22a5daadee04ee0e114b6f6d473cf38bf380200255c025
Deleted: sha256:085a15bd8efb301e5b1242d2c0199f6d7a0dff5c6de4b092756458da8e39696d
Deleted: sha256:c6f2dcb1332e84764d578eb547a281721e21bca7bf4e87e86ce3727aa4412a32
Deleted: sha256:10c3de769b3e3fcb7448e105c5d3232e3581093e33ee3c8098b68fe9361b7243
Deleted: sha256:f67b645b1e3e41d3acf3a4a1099fcf5c5106265ff0e54c99638d00f4b6f63c28
Deleted: sha256:de2f333ab91a13c109652b25b79965c017f3e4f0f6cd8f706662b7ff221c05cc
Deleted: sha256:9857c81f9ea282d7dac1546a918b0f9e43a06817459a8ef3d2ea98bc00a97afe
Deleted: sha256:f129b789c58f529b932863a829528e08c6e9f930ebfd43bbdd35cddb7004b8ab
Deleted: sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f
9cd6278866644fb7db29e7186b270d36696b65212fbf213b0c5e35e415329e72
[SSH] completed
[SSH] exit-status: 0
Finished: SUCCESS

查看新image和服务

[root@client-8 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
harbor.corp.tanzu/jenkins/maven-docker 20210723-031450 6c8e836bc53c 2 days ago 153MB

[root@client-8 ~]# lsof -i:8080
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 41188 root 4u IPv4 563367 0t0 TCP *:webcache (LISTEN)
docker-pr 41194 root 4u IPv6 563373 0t0 TCP *:webcache (LISTEN)

部署结果

http到测试服务器192.168.110.27的8080端口