jenkins发布k8s应用
因为还没有完成控制台开发，我们k8s应用的发布目前只能直接在jenkins中实现了。
1. 基础镜像
按层次制作基础镜像
centos -> centos-sshd -> centos74-jre[678] -> centos74-jre[678]-[tomcat6|tomcat7|tomcat8|springboot]
发布应用时，根据jre\middleware参数，选择基础镜像
2. 资源申请参数说明
必填
| project |
项目的名称，纯数字前要加'p' |
| app |
应用名称，以项目名称作为前缀，app在k8s集群是唯一的，作为应用的标识 |
| domain |
域名，无填'null' |
| sessionSticky |
会话保持，类似nginx的ip_hash，'true'或’false’ |
| replicate |
副本数，即节点数 |
| javaVersion |
jre版本,支持jre6\jre7\jre8 |
| middleware |
中间件，支持tomcat6\tomcat7\tomcat8\springboot |
| rwFileList |
Tomcat项目文件如需要写权限，多个用逗号分隔，无填’null’ |
| codeRepo |
git地址 |
| warDir |
war包相对路径，如’target’ |
| mvnCommand |
mvn打包命令 |
| sbStartCommand |
Springboot项目启动文件指定profile的命令 |
| dingToken |
如需要发dingding群通知，提供token，无则’null’ |
可选（运维有默认配置）
| cpuReq |
Cpu初始分配核数目 |
| cpuLmt |
Cpu分配核数目上限 |
| memReq |
内存初始分配大小（M） |
| memLmt |
内存分配大小上限（M），一般和memReq相同 |
| jvmSize |
Jvm大小，默认’null’，公式根据memReq自动计算 |
| directMemSize |
对外内存最大大小，默认’null’，公式根据memReq自动计算 |
| mvnVersion |
Mvn版本 |
| mvnJavaVersion |
Mvn环境的jdk版本 |
| imageVersion |
基础镜像版本（研发不需了解） |
3. jenkins pipeline
所有变量抽出至environment下，便于配置，也便于后续自研平台调用；
整个过程分为checkout、mvn、build、deploy、post action几个部分；
checkout：拉取代码
mvn：调用mvn命令打包
build：执行make_dockerfile.sh，生成dockerfile，调用docker命令build镜像、push镜像
deploy：执行make_deploymentfile.sh，make_ingressfile.sh，调用kubectl命令创建deployment、ingress（含service）
post action：发送钉钉通知完整jenkinsfile如下
pipeline {
environment {
project = 'myproj'
app = 'myproc-myapp'
domain = 'mydomain.ali.com'
replicate = '4'
sessionSticky = 'false'
javaVersion = 'jre8'
middleware = 'springboot'
rwFileList = 'null'
codeRepo = 'http://git.mydomain.com/usergroup/myapp.git'
warDir = 'target'
mvnCommand = 'clean install -DskipTests=true -Ptest'
sbStartCommand = '-Dspring.profiles.active=test'
dingToken = '8826e85c02c796aa5aea0a39d292e1111111111111111111111'
cpuReq = '0.2'
cpuLmt = '2'
memReq = '1024'
memLmt = '1024'
jvmSize = 'null'
directMemSize = 'null'
mvnVersion = 'maven-3.6.0'
mvnJavaVersion = 'jdk1.8.0_221'
imageVersion = 'v1'
}
agent any
options {
timeout(time: 10, unit: 'MINUTES')
}
stages {
stage('checkout') {
steps {
script {
def codeRepoCred
def reg1 = /.+git\.mydomain\.com.+/
if ("${codeRepo}".matches(reg1)) {
codeRepoCred = 'mycred'
}
checkout([$class: 'GitSCM', branches: [[name: '$tag']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${codeRepoCred}", url: "${codeRepo}"]]])
}
}
}
stage('mvn') {
environment {
PATH = "/usr/local/java/${mvnJavaVersion}/bin:$PATH"
}
steps {
sh "/usr/local/maven/${mvnVersion}/bin/mvn -f pom.xml ${mvnCommand}"
}
}
stage('build') {
steps {
sh "/data1/template/make_dockerfile.sh ${WORKSPACE} ${javaVersion} ${middleware} ${cpuLmt} ${memLmt} ${rwFileList} ${imageVersion} ${sbStartCommand} ${jvmSize} ${directMemSize}"
script {
tagnew = "${tag}".replaceAll("/","_");
sh "/usr/bin/docker build -f ${WORKSPACE}/deploy/dockerfile -t harbor.mydomain.com/$project/${app}:${tagnew} ${WORKSPACE}/${warDir}"
sh "/usr/bin/docker push harbor.mydomain.com/$project/${app}:${tagnew}"
sh "/usr/bin/docker rmi harbor.mydomain.com/$project/${app}:${tagnew}"
}
}
}
stage('deploy') {
steps {
script {
tagnew = "${tag}".replaceAll("/","_");
sh "/data1/template/make_deploymentfile.sh ${WORKSPACE} ${app} ${project} ${replicate} ${BUILD_NUMBER} ${cpuReq} ${cpuLmt} ${memReq} ${memLmt} ${tagnew}"
sh "/usr/bin/kubectl apply -f ${WORKSPACE}/deploy/deployment.yaml"
sh "/usr/bin/kubectl delete ingress ${app} > /dev/null 2>&1 || true"
sh "/usr/bin/kubectl delete service ${app} > /dev/null 2>&1 || true"
if ("${domain}" != "null") {
sh "/data1/template/make_ingressfile.sh ${WORKSPACE} ${app} ${domain} ${sessionSticky}"
sh "/usr/bin/kubectl apply -f ${WORKSPACE}/deploy/ingress.yaml"
}
sh "/usr/bin/kubectl rollout status deployment/${app}"
}
}
}
}
post {
success {
dingTalk accessToken:"https://oapi.dingtalk.com/robot/send?access_token=${dingToken}",
imageUrl:'',
jenkinsUrl:'http://10.40.16.212/',
message:' deploy success',
notifyPeople:''
}
failure {
dingTalk accessToken:'https://oapi.dingtalk.com/robot/send?access_token=${dingToken}',
imageUrl:'',
jenkinsUrl:'http://10.40.16.212/',
message:' deploy failed',
notifyPeople:''
}
}
}4. shell与模板文件
jenkins服务器上的模板相关文件：*.template
make_*.sh传递pipeline中的变量，根据template文件生成各个项目特定的dockerfile、deployment.yaml、ingress.yaml
dockerfile.template
将应用启动需要的参数，输出至用户.bash_profile中，生成环境变量；后续脚本直接调用环境变量
FROM harbor.mydomain.com/baseimage/imageTag
LABEL type="appimage"
COPY --chown=appuser:appuser *.war /apps/deploy/
RUN echo "export CPU_LIMIT=cpuLmtTag" >> /home/appuser/.bash_profile \
&& echo "export MEM_LIMIT=memLmtTag" >> /home/appuser/.bash_profile \
&& echo "export JVM_SIZE=jvmSizeTag" >> /home/appuser/.bash_profile \
&& echo "export DIRECT_MEM_SIZE=directMemSizeTag" >> /home/appuser/.bash_profile \
&& echo "export PERM_NAME=permNameTag" >> /home/appuser/.bash_profile \
&& echo "export FILE_LIST=rwFileListTag" >> /home/appuser/.bash_profile \
&& echo "export SB_START_COMMAND=sbStartCommandTag" >> /home/appuser/.bash_profile \
&& echo -e "LIBSYSCONFCPUS=\${CPU_LIMIT}" >> /home/appuser/.bash_profile \
&& echo -e "if [ \${LIBSYSCONFCPUS} -lt 2 ]; then LIBSYSCONFCPUS=2; fi" >> /home/appuser/.bash_profile \
&& echo "export LIBSYSCONFCPUS" >> /home/appuser/.bash_profile \
&& echo -e "export LD_PRELOAD=/usr/local/lib/libsysconfcpus.so:\${LD_PRELOAD}" >> /home/appuser/.bash_profile
CMD ["/apps/script/pod_start.sh"]deployment.template
apiVersion: apps/v1
kind: Deployment
metadata:
name: appTag
labels:
app: appTag
spec:
replicas: replicateTag
selector:
matchLabels:
app: appTag
template:
metadata:
labels:
app: appTag
project: projectTag
buildnum: "buildnumTag"
spec:
dnsPolicy: Default
containers:
- name: appTag
image: harbor.mydomain.com/projectTag/appTag:tagTag
imagePullPolicy: Always
ports:
- containerPort: 9999
securityContext:
privileged: false
resources:
requests:
cpu: cpuReqTag
memory: memReqTagMi
ephemeral-storage: 2Gi
limits:
cpu: cpuLmtTag
memory: memLmtTagMi
ephemeral-storage: 5Gi
readinessProbe:
#httpGet:
# path: /healthProbe.jsp
tcpSocket:
port: 9114
initialDelaySeconds: 30
periodSeconds: 3
livenessProbe:
#httpGet:
# path: /healthProbe.jsp
tcpSocket:
port: 9114
initialDelaySeconds: 120
periodSeconds: 3
lifecycle:
preStop:
exec:
command: ["/apps/script/pod_pre_stop.sh"]
volumeMounts:
- mountPath: /apps/heapdump
name: dump-vol
volumes:
- name: dump-vol
hostPath:
path: /var/lib/docker/dumpvol
type: Directoryingress.template
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: appTag
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: domainTag
http:
paths:
- path: /
backend:
serviceName: appTag
servicePort: http
---
kind: Service
apiVersion: v1
metadata:
name: appTag
annotations:
traefik.ingress.kubernetes.io/affinity: "sessionStickyTag"
traefik.ingress.kubernetes.io/session-cookie-name: "lbcookie"
spec:
clusterIP: None
selector:
app: appTag
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9999make_dockerfile.sh
#!/bin/bash
workspace=$1
jre=$2
middleware=$3
cpulmt=$4
memlmt=$5
rwfilelist=$6
imageVersion=$7
sbstartCommand=$8
jvmSize=$9
directMemSize=${10}
if [ $# -ne 10 ]; then
echo "Usage: $0 [workspace] [jre] [middlewre] [cpulmt] [memlmt] [rwfilelist] [imageVersion] [sbstartCommand] [jvmSize] [directMemSize]"
exit 1
fi
if [ $9 == "null" ]; then
if [ $memlmt -le 3072 ]; then
jvmSize=`expr $memlmt / 2`
else
jvmSize=`expr $memlmt - 1500`
fi
fi
if [ ${10} == "null" ]; then
if [ $memlmt -le 2048 ]; then
directMemSize=256
else
directMemSize=512
fi
fi
if [ $jre == "jre6" || $jre == "jre7" ]; then
permName="PermSize"
else
permName="MetaspaceSize"
fi
image="centos74-"${jre}"-"${middleware}":"${imageVersion}
mydockerfile=$workspace/deploy/dockerfile
mkdir -p $workspace/deploy
cp /data1/template/dockerfile.template $mydockerfile
/usr/bin/sed -i "s/imageTag/${image}/g" $mydockerfile
/usr/bin/sed -i "s/cpuLmtTag/${cpulmt}/g" $mydockerfile
/usr/bin/sed -i "s/memLmtTag/${memlmt}/g" $mydockerfile
/usr/bin/sed -i "s/rwFileListTag/${rwfilelist}/g" $mydockerfile
/usr/bin/sed -i "s/sbStartCommandTag/${sbstartCommand}/g" $mydockerfile
/usr/bin/sed -i "s/jvmSizeTag/${jvmSize}/g" $mydockerfile
/usr/bin/sed -i "s/directMemSizeTag/${directMemSize}/g" $mydockerfile
/usr/bin/sed -i "s/permNameTag/${permName}/g" $mydockerfilemake_deploymentfile.sh
#!/bin/bash
workspace=$1
app=$2
project=$3
replicate=$4
buildnum=$5
cpuReq=$6
cpuLmt=$7
memReq=$8
memLmt=$9
tag=${10}
if [ $# -ne 10 ]; then
echo "Usage: $0 [workspace] [app] [project] [replicate] [buildnum] [cpuReq] [cpuLmt] [memReq] [memLmt] [tag]"
exit 1
fi
mydeploymentfile=$workspace/deploy/deployment.yaml
cp /data1/template/deployment.template $mydeploymentfile
/usr/bin/sed -i "s/appTag/${app}/g" $mydeploymentfile
/usr/bin/sed -i "s/projectTag/${project}/g" $mydeploymentfile
/usr/bin/sed -i "s/replicateTag/${replicate}/g" $mydeploymentfile
/usr/bin/sed -i "s/buildnumTag/${buildnum}/g" $mydeploymentfile
/usr/bin/sed -i "s/cpuReqTag/${cpuReq}/g" $mydeploymentfile
/usr/bin/sed -i "s/cpuLmtTag/${cpuLmt}/g" $mydeploymentfile
/usr/bin/sed -i "s/memReqTag/${memReq}/g" $mydeploymentfile
/usr/bin/sed -i "s/memLmtTag/${memLmt}/g" $mydeploymentfile
/usr/bin/sed -i "s/tagTag/${tag}/g" $mydeploymentfilemake_ingressfile.sh
#!/bin/bash
workspace=$1
app=$2
domain=$3
sessionSticky=$4
if [ $# -ne 4 ]; then
echo "Usage: $0 [workspace] [app] [domain] [sessionSticky]"
exit 1
fi
myingressfile=$workspace/deploy/ingress.yaml
cp /data1/template/ingress.template $myingressfile
/usr/bin/sed -i "s/appTag/${app}/g" $myingressfile
/usr/bin/sed -i "s/domainTag/${domain}/g" $myingressfile
/usr/bin/sed -i "s/sessionStickyTag/${sessionSticky}/g" $myingressfile