【k8s实战一】Jenkins 部署应用到 Kubernetes
阅读目录
- 01 本文主旨
- 02 CI/CD流程
- 03 准备虚拟机
- 04 部署registry
- 05 配置maven
- 06 Tomcat部署Jenkins
- 07 部署Kubernetes集群
- 08 准备Demo
- 09 创建Jenkins流水线项目
- 部署与验证
- 总结
- 参考
【k8s实战一】Jenkins 部署应用到 Kubernetes
01 本文主旨
目标是演示整个Jenkins从源码构建镜像到部署镜像到Kubernetes集群过程。
为了简化流程与容易重现文中效果，做出如下操作：
- 使用
vmware创建两台虚拟机 - 使用 初始化k8s集群
- 使用代码仓库，本文内容所有配置文件均会上传至此仓库 http://172.30.30.253/DefaultCollection/cicd_demo/_git/cicd_demo
02 CI/CD流程
根据我画的这张图，Jenkins任务整体的流程分为五个步骤：
- 获取源码
- 构建制品
- 构建镜像并推送
- 发起部署请求
- k8s调度pod，拉取镜像，部署完成
03 准备虚拟机
准备两个虚拟机，均安装好docker，参考Ubuntu 16.04及以上 安装/卸载 Docker-CE
| VM用途 | 系统版本 | IP |
|---|---|---|
| Jenkins | CentOS Linux release 7.9.2009 | 192.168.137.64 |
| k8s master | CentOS Linux release 7.9.2009 | 192.168.137.61 |
设置Jenkins机器免密登录k8s master主机
ssh-keygen -t rsa
ssh-copy-id [email protected]
验证免密效果
ssh [email protected]
exit
04 部署registry
部署registry之前，可以先将私有仓库配置到 /etc/docker/daemon.json，两台虚拟机均需要配置,K8S用containerd 参考最下面FAQ。
sudo mkdir -p /etc/docker
sudo vim /etc/docker/daemon.json
内容如下：
{
"insecure-registries": ["192.168.137.61:5000"]
}
接着在 192.168.137.61 虚拟机上部署registry，生产环境推荐使用Harbor
docker run -d --name registry -p 5000:5000 registry
05 配置maven
在jenkins虚拟机安装maven
wget https://mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar zxvf apache-maven-3.6.3-bin.tar.gz
mv apache-maven-3.6.3 maven-3.6.3
修改 /etc/profile，添加 M2_HOME 环境变量映射到 maven 目录，重写 PATH 添加 maven 的 bin 目录，如图31、32行
06 Tomcat部署Jenkins
在jenkins虚拟机 192.168.137.64上执行命令下载 tomcat 与 jenkins.war
#下载tomcat
wget https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.60/bin/apache-tomcat-9.0.60.tar.gz
wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz
#解压并重命名
tar -zxf apache-tomcat-9.0.41.tar.gz
mv apache-tomcat-9.0.41 /home/hellxz/jenkins
#进入jenkins安装目录
cd ~/jenkins/webapps
#删除webapps下示例项目
rm -rf *
#下载最新版jenkins.war
wget https://get.jenkins.io/war-stable/2.332.1/jenkins.war
wget https://mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war
#安装openjdk、git
sudo apt-update
sudo apt-get install -y default-jdk default-jdk-headless git
设置Jenkins家目录，编辑 /etc/profile，添加 JENKINS_HOME 环境变量
刷新配置
source /etc/profile
修改tomcat下 conf/context.xml，加大静态资源缓存，提前解决启动 Jenkins 时的报错
启动Jenkins
cd ~/jenkins/bin
./startup.sh
访问 http://192.168.137.64:8080，安装推荐插件，安装加速可参考 Jenkins安装插件提速
07 部署Kubernetes集群
部署一个可以通过 kubectl 命令部署应用的k8s环境即可，部署k8s环境不是本文重点，以下为参考
使用kubeadm部署Kubernetes集群，点击展开查看
使用Minikube部署单节点测试k8s环境，点击展开查看
本文使用Minikube做测试，坑比较多，想趟坑可以先参考下 开发者如何快速搭建本地 Kubernetes 集群？Minikube趟坑记录_Kubernetes中文社区
关于Minikube的吐槽
08 准备Demo
Demo分五部分：测试代码、Jenkins构建脚本、docker镜像构建脚本、k8s部署配置yaml、部署脚本，均处在同一仓库中
仓库地址：http://172.30.30.253/DefaultCollection/cicd_demo/_git/cicd_demo
目录结构：
demo
使用start.spring.io生成的只添加spring-boot-starter-web依赖，在启动类上加了个 /test 的接口
Dockerfile
FROM openjdk:8-jdk
#设置工作目录
WORKDIR /root
#复制制品jar到/root/app.jar位置
ADD target/*.jar app.jar
#JVM参数，给后期调优使用
ENV JVM_OPTS=""
#启动服务
CMD java ${JVM_OPTS} \
-Djava.security.egd=file:/dev/./urandom \
-jar app.jar
shell
artifact2image.sh 制品转镜像Shell脚本，本文为了简单未安装Jenkins关于docker的插件
#!/bin/bash
# -- 构建镜像并推送私有仓库 --
set -eu #如有报错或取不到变量情况停止执行
#声名常量
IMG_REGISTRY="192.168.137.61:5000" #镜像仓库
IMG_NAME="cicd-demo"
IMG_TAG=`date "+%Y%M%d_%H%M"` #镜像标签，如 20201223_1351
IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #镜像上传与拉取的名称
#构建镜像
docker build -t ${IMG_FULL_NAME} .
#推送镜像
docker push ${IMG_FULL_NAME}
#删除本地镜像
docker rmi ${IMG_FULL_NAME}
#修改deploy.yaml的镜像标签
sed -i "s#{{IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml
deploy2k8s.sh 部署镜像到k8s环境脚本
#!/bin/bash
# -- 部署image到k8s --
# !注意：需要提前做ssh免密登录
set -eu
#定义常量
PROJECT_NAME="cicd-demo"
UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}"
FILE_NAME="${UPLOAD_DIR}/deploy.yaml"
SSH_USER="hellxz"
SSH_IP="192.168.137.61"
#首先删除待上传目录的同名文件
ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}"
#确保部署文件目录存在
ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}"
#远程复制部署文件
scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME}
#远程执行部署命令
ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}"
Jenkinsfile
pipeline{
agent any
stages {
//由于源码和Jenkinsfile处于同一仓库，在Jenkins项目执行时，会先将Jenkinsfile所在的仓库克隆下来，为了简单，这里就不重复添加拉取源码的操作了
//其它情况，如使用多个仓库一些构建的，这种就需要额外添加拉取代码的stage了。
stage('Build Artifact') {
steps{
sh label:'maven building', script: 'mvn clean package -DskipTests'
}
}
stage('Build Image'){
steps{
sh label:'image building', script: '/bin/bash artifact2image.sh'
}
}
stage('Deploy k8s'){
steps{
sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh'
}
}
}
post {
success{
//成功清理工作空间，失败保留现场
cleanWs()
}
}
}
deploy.yaml
apiVersion: apps/v1 #api版本
kind: Deployment #对象类型
metadata: #元信息标识
name: cicd-demo
namespace: default
labels: #自定义Deployment绑定的标签
app: cicd-demo
spec: #期待运行状态
replicas: 1 #部署实例数
selector:
matchLabels:
app: cicd-demo
template: #pod
metadata:
labels: #与spec.selector.matchLabels相同
app: cicd-demo
spec: #pod容器期望的状态
containers:
- name: cicd-demo
image: {{IMAGE_NAME}}
imagePullPolicy: IfNotPresent
env:
- name: JVM_OPTS
value: "-Xms128m -Xmx256m"
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: cicd-demo-svc
namespace: default
labels:
service: cicd-demo-svc
spec:
selector: #匹配pod标签
app: cicd-demo
type: NodePort #svc类型，ExternalName, ClusterIP, NodePort, and LoadBalancer
ports:
- name: cicd-demo-port
protocol: TCP
port: 8080
nodePort: 30000
09 创建Jenkins流水线项目
访问 http://192.168.137.64:8080/jenkins 并登录，以下创建新流水线demo构建项目
部署与验证
执行构建demo程序
日志输出：
Started by user admin
Obtained Jenkinsfile from git https://github.com/hellxz/cicd-demo.git
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/hellxz/jenkins/home/workspace/demo
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/hellxz/cicd-demo.git
> git init /home/hellxz/jenkins/home/workspace/demo # timeout=10
Fetching upstream changes from https://github.com/hellxz/cicd-demo.git
> git --version # timeout=10
> git --version # 'git version 2.17.1'
> git fetch --tags --progress -- https://github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/hellxz/cicd-demo.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
Commit message: "update"
> git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build Artifact)
[Pipeline] sh (maven building)
+ mvn clean package -DskipTests
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< online.hellxz:cicd-demo >-----------------------
[INFO] Building cicd-demo 0.0.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo ---
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo ---
[INFO] Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.248 s
[INFO] Finished at: 2020-12-23T11:51:20Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build Image)
[Pipeline] sh (image building)
+ /bin/bash artifact2image.sh
Sending build context to Docker daemon 17.26MB
Step 1/5 : FROM openjdk:8-jdk
---> 89f100fa8f9f
Step 2/5 : WORKDIR /root
---> Using cache
---> 17670bfa01f1
Step 3/5 : ADD target/*.jar app.jar
---> b7e4292e4f80
Step 4/5 : ENV JVM_OPTS=""
---> Running in 0be8c16c9c8a
Removing intermediate container 0be8c16c9c8a
---> 3ef66b100082
Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom ${JVM_OPTS} -jar app.jar
---> Running in c8209a642a2d
Removing intermediate container c8209a642a2d
---> bc7859f9b4fe
Successfully built bc7859f9b4fe
Successfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151
The push refers to repository [192.168.87.129:5000/cicd-demo]
44cbb90cc2c3: Preparing
f85e383859a1: Preparing
ffb4778f8a52: Preparing
e528f2c31deb: Preparing
c5f4367d4a59: Preparing
ceecb62b2fcc: Preparing
193bc1d68b80: Preparing
f0e10b20de19: Preparing
ceecb62b2fcc: Waiting
193bc1d68b80: Waiting
f0e10b20de19: Waiting
ffb4778f8a52: Layer already exists
e528f2c31deb: Layer already exists
f85e383859a1: Layer already exists
193bc1d68b80: Layer already exists
c5f4367d4a59: Layer already exists
ceecb62b2fcc: Layer already exists
f0e10b20de19: Layer already exists
44cbb90cc2c3: Pushed
20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006
Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151
Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4
Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291
Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4
Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9
Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Deploy k8s)
[Pipeline] sh (deploy image to k8s)
+ /bin/bash deploy2k8s.sh
deployment.apps/cicd-demo created
service/cicd-demo-svc created
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
可以看到以下输出，说明程序已部署到k8s集群中了
deployment.apps/cicd-demo created
service/cicd-demo-svc created
ssh登录k8s的机器验证
总结
简单总结下，Jenkins与Kubernetes的集成，更多地是Jenkins如何控制Kubernetes部署的流程，文中例子是通过 构建并上传镜像 和 ssh发送配置和命令 实现的。
本文是参考慕课网上的教程，快速扫了下视频实践所得，是了解其思路后自行实现的。脚本还有一点不完美，比如替换部署yaml内容是一次性的，这里只是演示大家知道整体流程就好。
实际操作上才发现，脑子说会了，然后双手却没有跟上，还是要动手啊！
操作了一天半左右，最开始还想着用vagrant初始化VM，由于工作上有个虚拟机是VBox5的，升级新版不能用。搞了半天放弃了Vagrant（VBox6.1升级了命令参数不兼容旧版，导致Vagrant为兼容它把几乎所有版本Vagrant都改了……）。
文中难免因为安装部分的描述而过于冗长，感谢大家能看到这里，最后再次感谢慕课网的鹿哥
参考
Jenkins+K8s实现持续集成-慕课网
Deployment.spec.selector.matchLables实验解释 - 云+社区 - 腾讯云
markdown折叠内容语法 - 简书
【k8s学习笔记】使用 kubeadm 部署 v1.18.5 版本 Kubernetes集群 - 东北小狐狸 - 博客园
【K8s学习笔记】K8s是如何部署应用的？ - 东北小狐狸 - 博客园
Minikube - Kubernetes本地实验环境-阿里云开发者社区
minikube start | minikube
Jenkins教程（一）安装Jenkins - 东北小狐狸 - 博客园
Jenkins安装插件提速 - 东北小狐狸 - 博客园
FAQ:
Failed to pull image "192.168.137.61:5000/cicd-demo:20220717_1507": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.137.61:5000/cicd-demo:20220717_1507": failed to resolve reference "192.168.137.61:5000/cicd-demo:20220717_1507": failed to do request: Head "https://192.168.137.61:5000/v2/cicd-demo/manifests/20220717_1507": http: server gave HTTP response to HTTPS client
kubernetes1.20.x 默认容器采用的是containerd，不是docker，因此配置有所不同。
无配置时报错如下
http: server gave HTTP response to HTTPS client
x509: certificate signed by unknown authority在每个node上修改containerd的配置文件
vi /etc/containerd/config.tomlhttp访问方式写入
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."core.harbor.service.com:80"]
endpoint = ["http://core.harbor.service.com:80"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."core.harbor.service.com".tls]
insecure_skip_verify = truehttps访问方式写入
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."core.harbor.service.com"]
endpoint = ["https://core.harbor.service.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."core.harbor.service.com".tls]
insecure_skip_verify = true重启containerd
systemctl restart containerd测试
# http
crictl pull --creds admin:Harbor12345 core.harbor.service.com:80/library/nginx
# https
crictl pull --creds admin:Harbor12345 core.harbor.service.com/library/nginx创建secret
# http
kubectl create secret docker-registry regcred --docker-server=http://core.harbor.service.com:80 --docker-username=admin --docker-password=Harbor12345 [email protected]
# https
kubectl create secret docker-registry regcred --docker-server=https://core.harbor.service.com --docker-username=admin --docker-password=Harbor12345 [email protected]