将master部署到k8s中，并通过Kubernetes插件，动态生成slave节点，保证有job时动态创建slave，构建完成之后删除slave，充分利用k8s的资源，同时也节约Jenkins集群单独部署时，slave节点资源的浪费。
-
部署nfs网络文件系统，来持久化Jenkins的job和构建时的依赖
-
新建一台虚拟机，IP地址分配为:
192.168.241.144 参照进行安装部署,注意必须全用户具有写的权限
-
-
部署安装Jenkins
安装所需要的文件可以从GitHub获取
-
创建na-rbac.yaml文件，用于创建namespace和serviceAcount
apiVersion: v1 kind: Namespace metadata: name: kube-ops --- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: kube-ops --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins namespace: kube-ops rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] - apiGroups: ["apps"] resources: ["deployments"] # 相对于GitHub中的文件，添加了deployments资源的权限，主要用于后期部署，不然会报错 verbs: ["create","delete","get","list","patch","update","watch"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: jenkins namespace: kube-ops roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: kube-ops -
创建pvc.yaml文件，主要用过nfs来进行网络存储
apiVersion: v1 kind: PersistentVolume # 定义资源，创建pv(persistence volume) metadata: name: opspv namespace: kube-ops spec: capacity: storage: 10Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete mountOptions: - hard - nfsvers=4.1 nfs: server: 192.168.241.144 path: /nfs-share # 改文件路径必须再nfs得服务器中存在 --- kind: PersistentVolumeClaim # 向pv申请资源 apiVersion: v1 metadata: name: opspvc namespace: kube-ops spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi -
创建deployment.yaml文件,部署deployment和service
--- apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: kube-ops spec: selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080 # web服务暴露的端口 name: web protocol: TCP - containerPort: 50000 # 用于和agent通信的端口 name: agent protocol: TCP resources: # 使用的资源 limits: cpu: 2000m memory: 2Gi requests: cpu: 1000m memory: 1Gi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 volumeMounts: - name: jenkinshome subPath: jenkins mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: opspvc # 引用前面创建的pvc --- apiVersion: v1 kind: Service # 部署service metadata: name: jenkins namespace: kube-ops labels: app: jenkins spec: selector: app: jenkins ports: - name: web port: 8080 targetPort: web - name: agent port: 50000 targetPort: agent -
创建ingress.yaml文件，这里使用得是nginx-ingress-controller部署得ingress-controller，主要得目的是每个域名分配不同得IP，则必须部署一个新得ingress-controller
-
ingress-controller配置文件
kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - update - list - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "- " # Here: " - " # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - list - apiGroups: - "" resources: - configmaps verbs: - create - update - list - apiGroups: - "" resources: - endpoints verbs: - get - list --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-devops-binding # 注意该名字最好修改一下，保证唯一性 namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-ops --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-devops-binding # 注意该名字最好修改一下，保证唯一 labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole # 这里使用的不是dashboard时创建的ingress的集群角色 subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-ops --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount containers: - name: nginx-ingress-controller image: wistiaanders/nginx-ingress-controller:0.25.1 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io - --ingress-class=devops-nginx-ingress # 这里配置类别，用于安装运维的ingress，都注册到这个上面 - --enable-ssl-passthrough securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 lifecycle: preStop: exec: command: - /wait-shutdown --- --- kind: Service apiVersion: v1 metadata: name: ingress-nginx-jenkins namespace: kube-ops labels: app.kubernetes.io/name: ingress-nginx-jenkins app.kubernetes.io/part-of: ingress-nginx-jenkins spec: externalTrafficPolicy: Local type: LoadBalancer selector: app.kubernetes.io/name: ingress-nginx-jenkins app.kubernetes.io/part-of: ingress-nginx-jenkins ports: - name: http port: 80 targetPort: http - name: https port: 443 targetPort: https --- -
ingress-controller配置文件
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: jenkins-ingress namespace: kube-ops annotations: kubernetes.io/ingress.class: "devops-nginx-ingress" # 需要和上面创建ingress-controller时配置启动参数中的ingress-class一致 spec: rules: - host: jenkins.tlh.com http: paths: - path: / backend: serviceName: jenkins servicePort: 8080
-
-
对上面的文件依次执行
kuberctl apply -f *.yaml -
配置csrf
jenkins-csrf
-
安装Jenkins插件
- kubernetes插件
该插件的目的是用于动态生成slave节点，优化资源，提示构建的性能
-
配置
k8s-plugin-conf 配置中的地址是通过k8s提供的服务发现来实现的
- Git Parameter Plug-In
- Extended Choice Parameter Plug-In
- kubernetes插件