Jenkins 基于Kubernetes动态创建pod
1、概述
DevOps 是一系列做法和工具，可以使 IT 和软件开发团队之间的流程实现自动化。其中，随着敏捷软件开发日趋流行，持续集成 (CI) 和持续交付 (CD) 已经成为该领域一个理想的解决方案。在 CI/CD 工作流中，每次集成都通过自动化构建来验证，包括编码、发布和测试，从而帮助开发者提前发现集成错误，团队也可以快速、安全、可靠地将内部软件交付到生产环境。
不过，传统的 Jenkins Master-Agent 架构（即多个 Agent 为一个 Master 工作）有以下不足。
- 如果 Master 宕机，整个 CI/CD 流水线会崩溃。
- 资源分配不均衡，一些 Agent 的流水线任务 (Job) 出现排队等待，而其他 Agent 处于空闲状态。
- 不同的 Agent 可能配置环境不同，并需要使用不同的编码语言。这种差异会给管理和维护带来不便。
KubeSphere CI/CD 流水线工作流（和下面实践原理一样）
KubeSphere CI/CD 流水线基于底层 Kubernetes Jenkins Agent 而运行。这些 Jenkins Agent 可以动态扩缩，即根据任务状态进行动态供应或释放。Jenkins Master 和 Agent 以 Pod 的形式运行在 KubeSphere 节点上。Master 运行在其中一个节点上，其配置数据存储在一个存储卷 (Volume) 中。Agent 运行在各个节点上，但可能不会一直处于运行状态，而是根据需求动态创建并自动删除。
当 Jenkins Master 收到构建请求，会根据标签动态创建运行在 Pod 中的 Jenkins Agent 并注册到 Master 上。当 Agent 运行完任务后，将会被释放，相关的 Pod 也会被删除。
动态供应 Jenkins Agent
动态供应 Jenkins Agent 有以下优势：
- 资源分配合理：KubeSphere 动态分配已创建的 Agent 至空闲节点，避免因单个节点资源利用率高而导致任务排队等待。
- 高可扩缩性：当 KubeSphere 集群因资源不足而导致任务长时间排队等待时，您可以向集群新增节点。
- 高可用性：当 Jenkins Master 故障时，KubeSphere 会自动创建一个新的 Jenkins Master 容器，并将存储卷挂载至新创建的容器，保证数据不会丢失，从而实现集群高可用。
2、安装插件 在Kubernetes平台部署Jenkins
在部署Jenkins的时候，也需要注意数据的持久化，存储在远端。
[root@master cicd]# cat jenkins.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: ops
labels:
name: jenkins
spec:
replicas: 1
selector:
matchLabels:
name: jenkins
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
serviceAccountName: jenkins
containers:
- name: jenkins
image: jenkins/jenkins
ports:
- containerPort: 8080
- containerPort: 50000
resources:
limits:
cpu: 1
memory: 1.5Gi
requests:
cpu: 1
memory: 1Gi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
securityContext:
fsGroup: 1000
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins
namespace: ops
spec:
storageClassName: "managed-nfs-storage"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: ops
spec:
selector:
name: jenkins
type: NodePort
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
nodePort: 30008
- name: agent
port: 50000
protocol: TCP
---
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
namespace: ops
rules:
- apiGroups: [""]
resources: ["pods","events"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets","events"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins
namespace: ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins[root@master secrets]# pwd
/ifs/kubernetes/ops-jenkins-pvc-921620d5-6d2c-4aaa-aa46-287fbd2fdb17/secrets
[root@master secrets]# ls /ifs/kubernetes/
default-nfs-pvc-pvc-9e696088-2834-43d0-baef-f9d5a2820425 ops-grafana-pvc-c8291750-10d6-4501-961f-c351d72cec8f
ops-es-pvc-pvc-d451bfeb-8dab-4c4a-84f9-bdb6bf6ce53f ops-jenkins-pvc-921620d5-6d2c-4aaa-aa46-287fbd2fdb17默认从国外网络下载插件，会比较慢，建议修改国内源：
# 进入到nfs共享目录
cd /ifs/kubernetes/ops-jenkins-pvc-ea1e0744-e147-4150-9b05-3e38bc5feaa8/updates
sed -i
's/https:\/\/updates.jenkins.io\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g'
default.json && \
sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
# 删除pod重建，pod名称改成你实际的
kubectl delete pod jenkins-dccd449c7-vx6sj -n ops管理Jenkins->系统配置-->管理插件-->分别搜索Git Parameter/Git/Pipeline/kubernetes/Config
File Provider，选中点击安装。
- Git Parameter：Git参数化构建
- Git：拉取代码
- Pipeline：流水线
- kubernetes：连接Kubernetes动态创建Slave代理
- Config File Provider：存储kubectl用于连接k8s集群的kubeconfig配置文件
3、添加kubernetes集群
管理Jenkins->Manage Nodes and Clouds->configureClouds->Add
Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理
插件介绍：https://github.com/jenkinsci/kubernetes-plugin
Kubernetes 地址：可以通过这个service地址去连接到,这样Jenkins pod就能够连接到kubernetes
[root@master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app-log-stdout ClusterIP 10.233.1.6 80/TCP 9d
kubernetes ClusterIP 10.233.0.1 443/TCP 11d 
Jenkins 地址：这个是slave pod连接Jenkins的地址，就是让slave pod连接过来的地址。用这个地址去连接Jenkins master。
[root@master ~]# kubectl get svc -n ops
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins NodePort 10.233.60.174 80:30008/TCP,50000:30561/TCP 21h 4、Jenkins在K8S中动态创建代理
Jenkins Master/Slave架构:当需要创建agent的时候，Jenkins调用k8s去创建，不用的话就将其销毁，这样就更加自动化了，并且更大化的去利用资源。需要发布项目就创建一个slave pod，不需要的话就将其销毁，释放资源，动态化。
这个pod也是通过镜像启动，这个镜像也需要一些工具，比如maven和jdk这些，所以先要构建slave镜像。
目录里涉及四个文件：
- Dockerfile：构建镜像
- jenkins-slave：shell脚本启动slave.jar
- settings.xml：修改maven官方源为阿里云源
- slave.jar：agent程序，接受master下发的任务
构建并推送到镜像仓库：
docker build -t reg.ctnrs.com/library/jenkins-slave-jdk:1.8 .
docker push reg.ctnrs.com/library/jenkins-slave-jdk:1.8
[root@master jenkins-slave]# ls
Dockerfile jenkins-slave jenkins-slave.tar jenkins.tar kubectl settings.xml slave.jar
[root@master jenkins-slave]# cat Dockerfile
FROM centos:7
LABEL maintainer lizhenliang
RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \
yum clean all && \
rm -rf /var/cache/yum/* && \
mkdir -p /usr/share/jenkins
COPY slave.jar /usr/share/jenkins/slave.jar
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave
COPY kubectl /usr/bin/
ENTRYPOINT ["jenkins-slave"]pipeline {
agent {
kubernetes {
label "jenkins-slave"
yaml '''
apiVersion: v1
kind: Pod
metadata:
name: jenkins-slave
spec:
containers:
- name: jnlp
image: jenkins-slave-jdk:1.8
imagePullPolicy: Never
'''
}
}
stages {
stage('TestAgent') {
steps {
sh 'hostname'
}
}
}
}注意，这个容器名字 - name: jnlp不能写错，只能为jnlp，否者会创建失败
Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] echo
[WARNING] label option is deprecated. To use a static pod template, use the 'inheritFrom' option.
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes ops/jenkins-slave-v3zqc-zkrw4
[Normal][ops/jenkins-slave-v3zqc-zkrw4][Scheduled] Successfully assigned ops/jenkins-slave-v3zqc-zkrw4 to node2
[Normal][ops/jenkins-slave-v3zqc-zkrw4][Pulled] Container image "jenkins-slave-jdk:1.8" already present on machine
Still waiting to schedule task
‘jenkins-slave-v3zqc-zkrw4’ is offline
[Normal][ops/jenkins-slave-v3zqc-zkrw4][Created] Created container jnlp
[Normal][ops/jenkins-slave-v3zqc-zkrw4][Started] Started container jnlp
Agent jenkins-slave-v3zqc-zkrw4 is provisioned from template jenkins-slave-v3zqc
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins.ops/job/tomcat-web/18/"
runUrl: "job/tomcat-web/18/"
labels:
jenkins: "slave"
jenkins/label-digest: "03ddc3eddf95d5470d5c7fb6d2937abaeca3b79e"
jenkins/label: "jenkins-slave"
name: "jenkins-slave-v3zqc-zkrw4"
spec:
containers:
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_AGENT_NAME"
value: "jenkins-slave-v3zqc-zkrw4"
- name: "JENKINS_NAME"
value: "jenkins-slave-v3zqc-zkrw4"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.ops/"
image: "jenkins-slave-jdk:1.8"
imagePullPolicy: "Never"
name: "jnlp"
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- emptyDir:
medium: ""
name: "workspace-volume"
Running on jenkins-slave-v3zqc-zkrw4 in /home/jenkins/agent/workspace/tomcat-web
[Pipeline] {
[Pipeline] stage
[Pipeline] { (TestAgent)
[Pipeline] sh
+ hostname
jenkins-slave-v3zqc-zkrw4
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS