kubernetes部署jenkins及配置Kubernetes plugin测试发布流程（干货）

一、部署环境

• minikube：v1.25.2
• Kubernetes：v1.23.3

二、安装部署

2.1创建namespace、sa、pv

kubectl apply -f namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: jenkins

kubectl apply -f sa.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: jenkins
rules:
- apiGroups:
  - '*'
  resources:
  - statefulsets
  - services
  - replicationcontrollers
  - replicasets
  - podtemplates
  - podsecuritypolicies
  - pods
  - pods/log
  - pods/exec
  - podpreset
  - poddisruptionbudget
  - persistentvolumes
  - persistentvolumeclaims
  - jobs
  - endpoints
  - deployments
  - deployments/scale
  - daemonsets
  - cronjobs
  - configmaps
  - namespaces
  - events
  - secrets
  verbs:
  - create
  - get
  - watch
  - delete
  - list
  - patch
  - update
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:jenkins

创建存储时根据自己的实际情况进行创建方式与存储大小

kubectl apply -f pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
  namespace: jenkins
spec:
  storageClassName: jenkins-pv
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/jenkins-volume
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  storageClassName: jenkins-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

2.2部署jenkins

kubectl apply -f deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:2.346.1-lts
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:   #资源限制根据自己实际情况配置
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: jenkins
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  ports:
  - name: web
    port: 8080
    targetPort: web
  - name: agent
    port: 50000
    targetPort: agent

2.3 启动状态查询

kubectl get pods -n jenkins

NAME                       READY   STATUS    RESTARTS   AGE
jenkins-7b56b4d997-44jlg   1/1     Running   0          7m15s

使用dashboard查询

三、配置jenkins

3.1 配置web访问

kubectl apply -f ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: jenkins
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: "jenkins.test.com"   #使用自己映射的域名
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: jenkins
            port:
              number: 8080

3.2 WEB登陆配置

查询配置管理密码：kubectl logs -f $(kubectl get pods -n jenkins|grep jenkins|awk ‘{print $1}’) -n jenkins

安装推荐插件（推荐插件有些并不需要，这里可以选择需要的插件安装来节约时间）

配置账号信息，后面默认即可

配置完成

3.3 安装kubernetes插件

kubernetes插件官方文档：https://plugins.jenkins.io/kubernetes/#plugin-content-configuration-reference

配置插件：Dashboard–>Manage Jenkins–>插件管理

搜索kubernetes插件并安装

等待安装完成

3.4 配置kubernetes插件

系统管理—>节点管理—>configureClouds

如果kubrenetes上安装的jenkins，信息都可不填，直接测试都能连接上

配置pod Templates（以maven为例）

说明：

• pod模板自定义配置自己需要的名称
• 容器配置配置需要的镜像，示例中使用开源项目kubesphere的devops的maven镜像（镜像内置了打包使用到的基础工具）
• 容器配置高级配置里面有资源限制配置，根据需求自行配置
• volume配置，根据自己需求配置，示例配置将maven本地仓库挂载到host
• pod的yaml配置，主要配置了maven的配置文件，将配置文件使用cm的方式挂载，方便自定义maven的配置
• sa配置是给启动的maven容器授权

kubectl apply -f cm.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: jenkins-maven
  namespace: jenkins
data:
  MavenSetting: |
    

    <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this worwalkmank for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
#默认配置文件内容省略，请参考maven官网

四、测试jenkins发布

4.1 安装pipeline插件

参考安装kubernetes插件方法

4.2 创建测试任务

因条件有限，此测试任务只测试配置的maven容器是否可以使用，不会完整的发布一个项目到kubernetes中

创建任务

测试发布

会单独创建一个slave节点来运行



发布测试完成

4.3 发布到kubernetes介绍

如果需要发布到kubernetes环境，主要注意注意以下几点

• 下载插件“Kubernetes Continuous Deploy”

• 配置的pod模板镜像内需要安装kubectl工具

• 添加凭据，让容器获取kubectl权限
  

pipeline配置例如

    stage('deploy') {
      agent none
      steps {
        kubernetesDeploy(configs: 'Deployment.yaml', enableConfigSubstitution: true, kubeconfigId: 'admin')
      }
    }

最后，如果有发布问题，可留言交流

------------------日常记录------------------