基于kubernetes的jenkins持续集成
发布流程设计
在kubernetes中部署jenkins
-参考：https://github.com/jenkinsci/kubernetes
- yml文件
jenkins
├── deployment.yml #jenkins 部署
├── rbac.yml #权限准入文件
└── service.yml #服务暴露文件- 部署jenkins
$ kubectl apply -f jenkins/
$ kubectl get pod -n ops
NAME READY STATUS RESTARTS AGE
jenkins-9cc69b868-zms8w 1/1 Running 0 22h
$ kubectl logs jenkins-9cc69b868-zms8w -n ops #从日志中找到默认的admin密码-
登陆http://NodeIP:30008
- 使用日志中的临时密码解锁jenkins
- 选择插件
- 选择无
- 创建admin与密码完成登陆
- 配置插件源
默认从国外网络下载插件，会比较慢，建议修改国内源：
# 找到NFS服务器，修改卷中的数据
$ cd /ifs/kubernetes/ops-jenkins-pvc-8947582f-11d3-47ed-92c0-bfdbf8aae813/updates/
$ sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json
$ sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
# 删除pod重建，pod名称改成你实际的
kubectl delete pod jenkins-dccd449c7-vx6sj -n ops- 安装插件
- 管理Jenkins->系统配置-->管理插件-->分别搜索Git Parameter/Git/Pipeline/kubernetes/Config File Provider，选中点击安装。
- Git Parameter：Git参数化构建
- Git：拉取代码
- Pipeline：流水线
- kubernetes：连接Kubernetes动态创建Slave代理
- Config File Provider：存储kubectl用于连接k8s集群的kubeconfig配置文件
添加kubernetes集群到jenkins
管理Jenkins->Manage Nodes and Clouds->configureClouds->Add
在kubernetes中构建动态slave
构建slave镜像
Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理
插件介绍：https://github.com/jenkinsci/kubernetes-plugin
https://github.com/jenkinsci/docker-inbound-agent
$ tree jenkins-slave/
jenkins-slave/
├── Dockerfile
├── jenkins-slave
├── kubectl
├── settings.xml
└── slave.jar课件目录里涉及四个文件：
- Dockerfile 构建镜像
- jenkins-slave shell脚本启动slave.jar
- settings.xml 修改maven官方源为阿里云源
- slave.jar agent程序，接受master下发的任务
构建并推送到镜像仓库：
$ cd jenkins-slave/
$ docker build -t hub.cropy.cn/library/jenkins-slave-jdk:1.8 .
$ docker push hub.cropy.cn/library/jenkins-slave-jdk:1.8二进制部署的k8s需要kubeconfig文件
-
如果是kubeadm搭建的集群则不需要考虑这一步
- 签发CA证书
$ cat > ca-config.json < admin-csr.json < - 生成kubeconfig授权文件
# kubeconfig
kubectl config set-cluster kubernetes \
--certificate-authority=/data/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.56.14:6443 \
--kubeconfig=admin.kubeconfig
# 设置客户端认证
kubectl config set-credentials admin \
--client-key=admin-key.pem \
--client-certificate=admin.pem \
--embed-certs=true \
--kubeconfig=admin.kubeconfig
# 设置默认上下文
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin \
--kubeconfig=admin.kubeconfig
# 设置当前使用配置
kubectl config use-context kubernetes --kubeconfig=admin.kubeconfig- 绑定权限到admin
$ vim admin.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: admin
$ kubectl apply -f admin.yml
$ kubectl get pod --kubeconfig=admin.kubeconfig -n ops # 测试权限
$ cp admin.kubeconfig ~/.kube/configjenkins PIPELINE
Jenkins Pipeline是一套插件，支持在Jenkins中实现集成和持续交付管道；
- Pipeline通过特定语法对简单到复杂的传输管道进行建模；
- 声明式：遵循与Groovy相同语法。pipeline { }
- 脚本式：支持Groovy大部分功能，也是非常表达和灵活的工具。node { }
- Jenkins Pipeline的定义被写入一个文本文件，称为Jenkinsfile。
PIpeline 流水线流程
- 新建pipeline job
- 选择kubernetes 声明式pipeline
pipline脚本内容如下：
pipeline {
agent {
kubernetes {
label "jenkins-slave"
yaml '''
apiVersion: v1
kind: Pod
metadata:
name: jenkins-slave
spec:
containers:
- name: jnlp
image: hub.cropy.cn/library/jenkins-slave-jdk:1.8
command:
- sleep
args:
- infinity
'''
// Can also wrap individual steps:
// container('shell') {
// sh 'hostname'
// }
defaultContainer 'shell'
}
}
stages {
stage('Main') {
steps {
sh 'hostname'
}
}
}
}jenkins常见的认证信息保存
保存git和harbor的凭据ID
1bbda277-a0e0-42c9-b4f3-be22e6344d66
6020966e-3469-44e7-baff-e9945fbaa4f3
k8s认证配置文件放到jenkins
- 添加配置
- 选择自定义的加入即可
拷贝好ID： 4f0526c1-18c8-48c7-bd22-eeb8b179a950
基于参数化构建
- 创建pipelinejob
- 选择参数
- 选择参数示例使用
-
修改Git中的yml文件参数
- pipeline脚本
// 公共
def registry = "hub.cropy.cn"
// 项目
def project = "dev"
def app_name = "java-demo"
def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
def git_address = "http://192.168.56.19:9999/root/java-demo.git"
// 认证
def secret_name = "registry-pull-secret"
def harbor_auth = "6020966e-3469-44e7-baff-e9945fbaa4f3"
def git_auth = "1bbda277-a0e0-42c9-b4f3-be22e6344d66"
def k8s_auth = "4f0526c1-18c8-48c7-bd22-eeb8b179a950"
pipeline {
agent {
kubernetes {
label "jenkins-slave"
yaml """
kind: Pod
metadata:
name: jenkins-slave
spec:
containers:
- name: jnlp
image: "${registry}/library/jenkins-slave-jdk:1.8"
imagePullPolicy: Always
volumeMounts:
- name: docker-cmd
mountPath: /usr/bin/docker
- name: docker-sock
mountPath: /var/run/docker.sock
- name: maven-cache
mountPath: /root/.m2
volumes:
- name: docker-cmd
hostPath:
path: /usr/bin/docker
- name: docker-sock
hostPath:
path: /var/run/docker.sock
- name: maven-cache
hostPath:
path: /tmp/m2
"""
}
}
parameters {
gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH'
choice (choices: ['1', '3', '5', '7'], description: '副本数', name: 'ReplicaCount')
choice (choices: ['dev','test','prod'], description: '命名空间', name: 'Namespace')
}
stages {
stage('拉取代码'){
steps {
checkout([$class: 'GitSCM',
branches: [[name: "${params.Branch}"]],
doGenerateSubmoduleConfigurations: false,
extensions: [], submoduleCfg: [],
userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]
])
}
}
stage('代码编译'){
steps {
sh """
mvn clean package -Dmaven.test.skip=true
"""
}
}
stage('构建镜像'){
steps {
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
sh """
echo '
FROM ${registry}/base/tomcat:v1
LABEL maitainer wanghui
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
' > Dockerfile
docker build -t ${image_name} .
docker login -u ${username} -p '${password}' ${registry}
docker push ${image_name}
"""
}
}
}
stage('部署到K8S平台'){
steps {
configFileProvider([configFile(fileId: "${k8s_auth}", targetLocation: "admin.kubeconfig")]){
sh """
sed -i 's#IMAGE_NAME#${image_name}#' deploy.yaml
sed -i 's#NAMESPACE#${Namespace}#' deploy.yaml
sed -i 's#REPLICAS#${ReplicaCount}#' deploy.yaml
kubectl apply -f deploy.yaml -n ${Namespace} --kubeconfig=admin.kubeconfig
"""
}
}
}
}
}- 部署到k8s这里面的kubeconfig需要使用生成的对应namespace的配置文件
- docker镜像仓库地址加入secret：docker-regsitry-auth
$ kubectl create namespace test
$ kubectl create secret docker-registry docker-regsitry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=hub.cropy.cn -n test- deploy中的需要替换的大写参数也需要替换，然后提交之后再做jenkins编译
$cat deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: java-demo
namespace: NAMESPACE
spec:
replicas: REPLICAS
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
imagePullSecrets:
- name: "docker-regsitry-auth"
containers:
- image: IMAGE_NAME
name: java-demo
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
protocol: TCP
resources:
requests:
cpu: 0.5
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
labels:
app: java-demo
name: java-demo
namespace: NAMESPACE
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8080
nodePort: 30018
selector:
app: java-demo
type: NodePort