jenkins介绍以及安装部署【包含k8s动态代理slave的方式】

概念

jenkins作为当下一款流行的CI/CD工具，拥有非常丰富的插件以及活跃的社区，其主要特点有：
● 一款广泛用于持续集成的自动化工具。
● 很好的集成了gitlab、sonar、jacoco等工具。
● 支持多种构建工具【maven、grade】。
● 拥有完整的权限管控系统。
● 代码生成器自动生成各种流水线语法。
● 跨平台运行，可以在windows、macos、linux上运行。

安装部署

Jenkins的安装：
● 可以使用rpm/war/docker/k8s安装；
● 由于jenkins是java语言写的，需要提前安装好JDK环境。

Jenkins采用分布式架构，分为master节点和slave节点。
● master节点负责任务的调度也可以运行任务。
● agent节点负责具体任务的执行。

在k8s集群中部署的分布式jenkins,可以动态的创建和销毁slave-pod，当有运行任务的请求时，根据提前设定好的pod模板，自动生成slave-pod，当运行任务结束后，slave-pod自动销毁，节省了很多的服务器资源，下面会有具体的部署介绍

环境准备

环境名称	环境验证	作用
jdk	java -version	jenkins运行所需环境&&java项目编译环境
maven	mvn -v	java项目构建工具
node.js&npm	node -v	服务端的JS

根据具体情况安装对应的环境，比如python、go等。

rpm安装

#下载
https://mirrors.tuna.tsinghua.edu.cn/jenkins/redhat-stable/jenkins-2.289.3-1.1.noarch.rpm
#解压
npm -ivh jenkins-2.289.3-1.1.noarch.rpm
#配置文件编辑
/etc/init.d/jenkins中修改jdk路径为自己实际的路径
/etc/sysconfig/jenkins/config.xml修改workspace路径，自定义路径
<workspaceDir>/data/jenkins/workspace/${ITEM_FULLNAME}</workspaceDir>
<buildsDir>/data/jenkins/builds/${ITEM_FULLNAME}/builds</buildsDir>
##启停
systemctl start jenkins #启动命令
systemctl stop jenkins  #停止命令
systemctl enable jenkins #开机自启动
#访问路径
http://ip:8080
#获取管理员密码
/var/lib/jenkins/secrets/initialAdminPassword

slave节点的安装运行

mangerJenkins - Manger Node and Clouds


下载agent.jar，启动命令如下所示

docker方式安装

#docker环境安装，参照博主docker介绍的文章即可，此处不在详述
#镜像拉取
docker pull jenkins/jenkins:2.360-centos7
#创建目录
mkdir -p /data/jenkins
chmod +x /data/jenkins
chmod 777 -R /data/jenkins
#启动运行，8080为jenkins服务端口,5000端口为master与slave通讯的端口,可以自定义设置
docker run -itd --name jenkins \
-p 8080:8080 \
-p 5000:5000 \
-e JAVA_OPTS="-Dorg.apache.commons.jelly.tags.fmt.timeZone='Asia/Shanghai'" \
--privileged=true  \
-v /data/jenkins:/var/jenkins_home jenkins/jenkins:2.360-centos7 

k8s方式安装

deployment资源文件清单：

apiVersion: apps/v1
kind: Deployment 
metadata:
  name: jenkins
  labels:
    name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      name: jenkins 
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: reg.crc.com.cn/library/jenkins:lts 
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
      securityContext:
        fsGroup: 1000
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-home
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-home
spec:
  storageClassName: "managed-nfs-storage"
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi

service资源清单：

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  selector:
    name: jenkins
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP
      nodePort: 30006
    - name: agent
      port: 50000
      protocol: TCP

service-account资源清单：

# In GKE need to get RBAC permissions first with
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=|--group=]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

rbac资源清单文件：

---
# 创建名为jenkins的ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
# 创建名为jenkins的Role，授予允许管理API组的资源Pod
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
# 将名为jenkins的Role绑定到名为jenkins的ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

ingress资源清单文件：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
spec:
  rules:
  - host: jenkins.test.com
    http:
      paths:
      - backend:
          service:
            name: jenkins
            port:
              number: 80
        path: /
        pathType: Prefix 

k8s-slave安装部署

资源清单文件：

agent.jar/Dockerfile/helm/jenkins-slave/kubectl/settings.xml

agent.jar #装好jenkins master后，从jenkins master上获取jar

helm下载helm工具获取

Dockerfile:

FROM reg.crc.com.cn/library/centos_tool:7.9
LABEL maintainer zhangjunwei

# RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \ 
RUN yum clean all && \
    rm -rf /var/cache/yum/* && \
    mkdir -p /usr/share/jenkins

COPY agent.jar /usr/share/jenkins/slave.jar  
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave
COPY helm kubectl /usr/bin/

ENTRYPOINT ["jenkins-slave"]

jenkins-slave:

#!/usr/bin/env sh

if [ $# -eq 1 ]; then

	# if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
	exec "$@"

else

	# if -tunnel is not provided try env vars
	case "$@" in
		*"-tunnel "*) ;;
		*)
		if [ ! -z "$JENKINS_TUNNEL" ]; then
			TUNNEL="-tunnel $JENKINS_TUNNEL"
		fi ;;
	esac

	# if -workDir is not provided try env vars
	if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
		case "$@" in
			*"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
			*)
			WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
		esac
	fi

	if [ -n "$JENKINS_URL" ]; then
		URL="-url $JENKINS_URL"
	fi

	if [ -n "$JENKINS_NAME" ]; then
		JENKINS_AGENT_NAME="$JENKINS_NAME"
	fi  

	if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
		echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
		JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
	fi

	# If both required options are defined, do not pass the parameters
	OPT_JENKINS_SECRET=""
	if [ -n "$JENKINS_SECRET" ]; then
		case "$@" in
			*"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
			*)
			OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
		esac
	fi
	
	OPT_JENKINS_AGENT_NAME=""
	if [ -n "$JENKINS_AGENT_NAME" ]; then
		case "$@" in
			*"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
			*)
			OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
		esac
	fi

	#TODO: Handle the case when the command-line and Environment variable contain different values.
	#It is fine it blows up for now since it should lead to an error anyway.

	exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
fi

settings.xml:

  

  

  

  
  
    
  

  
  
    
  

  
  
    

    
  

  
  
    
         
      central     
      central     
      aliyun maven
      https://maven.aliyun.com/repository/public     
    
  

  
  
    

    
  

  

k8sPV自动供给

先准备一台NFS服务器为K8S提供存储支持。

# yum install nfs-utils
# vi /etc/exports
/ifs/kubernetes *(rw,no_root_squash)
# mkdir -p /ifs/kubernetes
# systemctl start nfs
# systemctl enable nfs

并且要在每个Node上安装nfs-utils包，用于mount挂载时用。

由于K8S不支持NFS动态供给，还需要先安装上图中的nfs-client-provisioner插件：

# cd nfs-client
# vi deployment.yaml # 修改里面NFS地址和共享目录为你的
# kubectl apply -f .
# kubectl get pods
NAME                                     READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-df88f57df-bv8h7   1/1     Running   0          49m

deployment资源文件：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: reg.crc.com.cn/library/nfs-subdir-external-provisioner:v4.0.1
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 10.200.204.19
            - name: NFS_PATH
              value: /ifs/kubernetes
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.200.204.19
            path: /ifs/kubernetes
                                                                                           

class.yaml资源文件

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true"

rbac.yaml资源文件

kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

jenkins在kubernetes创建动态代理

1、在jenkins插件安装中，安装如下插件：
Git Parameter
Git
Pipeline
Config file Provider
kubernetes
Extended Choice Parameter
Active choice
Generic Webhook Trigger
rebuild

2、在全局配置中，添加配置连接kubernetes集群
3、构建jenkins slave镜像

jenkins备份

vim jenkins_backup.sh(仅供参照)
#！/bin/bash
cd /var/lib
tar -zcf /bak/uat_jenkins/uat-jenkins$(date +"%Y%m%d").tar.gz  jenkins

find . +mtime +3 |xargs rm -f