kubernetes基于helm部署jenkins
helm部署jenkins
基于kubernetes之上运行jenkins可以实现jenkins工作节点的动态调用伸缩，更好的提高资源利用率。通过Jenkins的kubernetes-plugin来实现将Jenkins运行在Kubernetes上的功能。
参考：
https://hub.helm.sh/charts/stable/jenkins
https://github.com/helm/charts/tree/master/stable/jenkins
环境信息
[root@master01 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master01 Ready master 39d v1.18.6 192.168.93.60 <none> CentOS Linux 7 (Core) 3.10.0-1127.el7.x86_64 docker://19.3.0
node01 Ready <none> 39d v1.18.6 192.168.93.61 <none> CentOS Linux 7 (Core) 3.10.0-1127.el7.x86_64 docker://19.3.0
node02 Ready <none> 39d v1.18.6 192.168.93.62 <none> CentOS Linux 7 (Core) 3.10.0-1127.el7.x86_64 docker://19.3.0
准备支持动态存储的storageclass，这里使用rancher longhorn，也可以使用nfs、ceph、localpv等。
#所有节点安装longhorn依赖
yum install -y iscsi-initiator-utils
systemctl enable --now iscsid
#部署longhorn
helm repo add longhorn https://charts.longhorn.io
helm repo update
kubectl create namespace longhorn-system
helm install longhorn \
--namespace longhorn-system \
--set defaultSettings.defaultDataPath="/var/lib/longhorn/" \
--set defaultSettings.defaultReplicaCount=2 \
--set service.ui.type=NodePort \
--set service.ui.nodePort=30880 \
longhorn/longhorn
查看就绪的storageclass
[root@master01 ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 18h
helm部署jenkins
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
kubectl create ns jenkins
helm install jenkins \
--namespace jenkins \
--set master.image=jenkins/jenkins \
--set master.tag=2.248 \
--set master.adminUser=admin \
--set master.adminPassword="jenkins@123" \
--set master.serviceType=NodePort \
--set master.nodePort=30080 \
--set master.initContainerEnv[0].name=JENKINS_UC \
--set master.initContainerEnv[0].value="https://updates.jenkins-zh.cn/update-center.json" \
--set master.initContainerEnv[1].name=JENKINS_UC_DOWNLOAD \
--set master.initContainerEnv[1].value="https://mirrors.tuna.tsinghua.edu.cn/jenkins" \
--set persistence.storageClass=longhorn \
--set persistence.size=10Gi \
stable/jenkins
参数说明：
- initContainerEnv参数：jenkins初始化时initContainer中会执行/var/jenkins_config/apply_config.sh脚本，并调用install-plugins.sh脚本下载kubernetes插件，指定为国内地址；
- persistence.storageClass参数：用于动态申请pv，持久化jenkins数据。
查看创建的pod
[root@master01 ~]# kubectl -n jenkins get pods
NAME READY STATUS RESTARTS AGE
jenkins-6c59c7486c-h2847 2/2 Running 0 149m
修改插件源为国内地址
[root@master01 ~]# kubectl -n jenkins exec -it jenkins-6c59c7486c-h2847 sh
mkdir $JENKINS_HOME/update-center-rootCAs
cat > $JENKINS_HOME/update-center-rootCAs/jenkins-update-center-cn-root-ca.crt <<END
-----BEGIN CERTIFICATE-----
MIICcTCCAdoCCQD/jZ7AgrzJKTANBgkqhkiG9w0BAQsFADB9MQswCQYDVQQGEwJD
TjELMAkGA1UECAwCR0QxCzAJBgNVBAcMAlNaMQ4wDAYDVQQKDAV2aWhvbzEMMAoG
A1UECwwDZGV2MREwDwYDVQQDDAhkZW1vLmNvbTEjMCEGCSqGSIb3DQEJARYUYWRt
aW5AamVua2lucy16aC5jb20wHhcNMTkxMTA5MTA0MDA5WhcNMjIxMTA4MTA0MDA5
WjB9MQswCQYDVQQGEwJDTjELMAkGA1UECAwCR0QxCzAJBgNVBAcMAlNaMQ4wDAYD
VQQKDAV2aWhvbzEMMAoGA1UECwwDZGV2MREwDwYDVQQDDAhkZW1vLmNvbTEjMCEG
CSqGSIb3DQEJARYUYWRtaW5AamVua2lucy16aC5jb20wgZ8wDQYJKoZIhvcNAQEB
BQADgY0AMIGJAoGBAN+6jN8rCIjVkQ0Q7ZbJLk4IdcHor2WdskOQMhlbR0gOyb4g
RX+CorjDRjDm6mj2OohqlrtRxLGYxBnXFeQGU7wWjQHyfKDghtP51G/672lXFtzB
KXukHByHjtzrDxAutKTdolyBCuIDDGJmRk+LavIBY3/Lxh6f0ZQSeCSJYiyxAgMB
AAEwDQYJKoZIhvcNAQELBQADgYEAD92l26PoJcbl9GojK2L3pyOQjeeDm/vV9e3R
EgwGmoIQzlubM0mjxpCz1J73nesoAcuplTEps/46L7yoMjptCA3TU9FZAHNQ8dbz
a0vm4CF9841/FIk8tsLtwCT6ivkAi0lXGwhX0FK7FaAyU0nNeo/EPvDwzTim4XDK
9j1WGpE=
-----END CERTIFICATE-----
END
cp $JENKINS_HOME/hudson.model.UpdateCenter.xml{,.bak}
sed -i 's#https://updates.jenkins.io/update-center.json#https://mirrors.huaweicloud.com/jenkins/updates/update-center.json#g' \
$JENKINS_HOME/hudson.model.UpdateCenter.xml
如果要清理环境执行以下命令
helm -n jenkins uninstall jenkins
访问jenkins
查看暴露端口
[root@master01 ~]# kubectl -n jenkins get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins NodePort 10.108.37.198 <none> 8080:30080/TCP 17h
jenkins-agent ClusterIP 10.102.141.194 <none> 50000/TCP 17h
浏览器访问：http://192.168.93.60:30080
默认jenkins已经添加kubernetes cloud，选择管理jenkins—>管理node和cloud，确认cloud配置即可。
安装以下常用插件
- pipeline
- docker pipeline
- blueocean
创建以下凭证
- 创建用户名密码类型的github代码仓库凭证，ID为github
- 创建用户名密码类型的aliyun registry镜像仓库凭证，ID为aliyun
- 创建Secret file类型的kubeconfig集群凭证，上传kubernetes master节点/root/.kube/config文件,ID为kubeconfig
部署流水线
创建pipeline类型流水线，复制粘贴以下pipeline脚本：
pipeline {
environment {
REGISTRY='registry.cn-shenzhen.aliyuncs.com/willspace/spring-demo'
VERSION = "${env.BUILD_ID}"
REGISTRY_CREDENTIAL = 'aliyun'
}
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
metadata:
labels:
app: jenkins-agent
spec:
containers:
- name: maven
image: willdockerhub/maven:mirror
args:
- cat
tty: true
- name: docker
image: docker:dind
args:
- "--registry-mirror=https://uyah70su.mirror.aliyuncs.com"
securityContext:
privileged: true
tty: true
- name: kubectl
image: lachlanevenson/k8s-kubectl
command:
- cat
tty: true
- name: helm
image: lachlanevenson/k8s-helm
command:
- cat
tty: true
"""
}
}
stages {
stage('SCM Checkout') {
steps {
git branch: "master", credentialsId: 'github', url: 'https://github.com/willzhang/spring-demo.git'
}
}
stage('source build') {
steps {
container('maven') {
sh 'mvn package'
}
}
}
stage('docker build') {
steps {
container('docker') {
sh 'docker build -t ${REGISTRY}:${VERSION} .'
}
}
}
stage('docker publish') {
steps {
container('docker') {
withDockerRegistry([credentialsId: "${REGISTRY_CREDENTIAL}", url: "https://registry.cn-shenzhen.aliyuncs.com"]) {
sh "docker push ${REGISTRY}:${VERSION}"
}
}
}
}
stage('app deploy with kubectl') {
steps {
container('kubectl') {
withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
sh """
mkdir -p /root/.kube/ && echo $KUBECONFIG >/root/.kube/config
sed -i "s#spring-boot-demo-image#${REGISTRY}:${VERSION}#g" k8s/deployment.yaml
kubectl apply -f k8s/deployment.yaml
"""
}
}
}
}
stage('app deploy with helm') {
steps {
container('helm') {
withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
sh """
mkdir -p /root/.kube/ && echo $KUBECONFIG >/root/.kube/config
helm -n default upgrade --install spring-demo \
--set service.type=NodePort \
--set service.nodePort=30085 \
--set image.repository=${REGISTRY}:${VERSION} \
helm/
"""
}
}
}
}
}
}
以上pipeline脚本执行了以下步骤：
- 动态创建agent pod，该pod运行maven、docker、kubectl以及默认的jenkins agent容器
- 拉取github代码仓库spring demo源码，也包含Dockerfiel、yaml及helm chart
- 执行maven编译生成构建物
- 构建docker镜像推送阿里云镜像仓库（dockerhub仓库推送太慢）
- 部署应用到kubernes集群
- 流水线运行完成agent pod被销毁
流水线运行时查看创建的动态pod，该pod包含了4个容器maven、docker、kubectl以及默认的jenkins agent，agent容器默认用于连接master，其他容器如maven、docker、kubectl用于构建和发布阶段
[root@master01 ~]# kubectl -n jenkins get pods
NAME READY STATUS RESTARTS AGE
jenkins-6c59c7486c-h2847 2/2 Running 0 49m
spring-demo-3-d1c4f-nr71g-q6s25 4/4 Running 0 80s
流水线运行完成时查看流水线运行状态：
blueocean
查看部署的应用
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
spring-boot-demo-75b58bbbc4-z4w5f 1/1 Running 0 6m39s
spring-demo-66f458db45-26p48 1/1 Running 0 6m38s
[root@master01 ~]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
spring-demo default 1 2020-09-03 06:54:28.533946871 +0000 UTC deployed spring-boot-demo-0.1.0 v1.0
查看应用service
[root@master01 ~]# kubectl get svc -l app=spring-boot-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
spring-boot-service NodePort 10.109.182.171 <none> 8080:31064/TCP 22h
浏览器访问应用
