jenkins+kubernetes（k8s）发布Springboot项目
jenkins+kubernetes（k8s）发布Java项目
- 准备工作
- 机器列表
- 镜像列表
- 创建一个java项目
- 创建jenkins镜像
- Jenkins服务配置
- 启动jenkins
- 访问jenkins
- 系统配置和权限配置
- 发布Java服务
- 查看java服务运行状态
记录一下学习k8s+jenkins发布项目的过程，期间遇到很多坑。希望能让刚接触的童鞋少走弯路。
准备工作
本篇只介绍发布的过程，在这之前有很多必要的工作要做：
- k8s集群搭建 ，我自己搭的是3个节点的集群；
- harbor搭建 ，docker镜像私仓；
机器列表
| host | ip | 描述 |
|---|---|---|
| centos7docker | 192.168.147.129 | harbor私仓机器 |
| k8smaster | 192.168.147.130 | k8smaster节点 |
| k8sworker1 | 192.168.147.131 | k8s节点 |
| k8sworker2 | 192.168.147.132 | k8s节点 |
镜像列表
| 镜像名称 | 依赖的镜像 | 描述 |
|---|---|---|
| jenkins/jenkins:lts | 无 | jenkins官方镜像 |
| centos7docker:443/aliang-xyl/jenkins:lts | jenkins/jenkins:lts | 自制jenkins镜像 |
| centos:7 | 无 | centos7官方镜像 |
| centos7docker:443/aliang-xyl/java8:jdk-8u231 | centos:7 | 自制java服务基础镜像 |
| centos7docker:443/aliang-xyl/provider-docker-demo:e161850 | centos7docker:443/aliang-xyl/java8:jdk-8u231 | java服务镜像 |
创建一个java项目
创建一个Java项目，结构如下:
项目Dockerfile文件内容, 用于项目打包成镜像
# 基于哪个镜像, 这个是我自定义的镜像，可根据实际情况更改
FROM centos7docker:443/aliang-xyl/java8:jdk-8u231
ARG JAR_FILE
ADD target/provider-docker-demo.jar /provider-docker-demo.jar
# 配置容器启动后执行的命令
ENTRYPOINT ["java","-Ddemo.test=test", "-Xmn256M", "-Xms1G", "-Xmx1G", "-XX:MetaspaceSize=512M","-jar","/provider-docker-demo.jar"]
k8s.ymal文件内容
apiVersion: v1
kind: Service
metadata:
name: provider-docker-demo
namespace: default
labels:
app: provider-docker-demo
spec:
type: NodePort
ports:
- port: 30001
nodePort: 30001 #service对外开放端口
selector:
app: provider-docker-demo
---
apiVersion: apps/v1
kind: Deployment #对象类型
metadata:
name: provider-docker-demo #名称
labels:
app: provider-docker-demo #标注
spec:
replicas: 1 #运行容器的副本数，修改这里可以快速修改分布式节点数量
selector:
matchLabels:
app: provider-docker-demo
template:
metadata:
labels:
app: provider-docker-demo
spec:
containers: #docker容器的配置
- name: provider-docker-demo
image: centos7docker:443/aliang-xyl/provider-docker-demo:> # pull镜像的地址 ip:prot/dir/images:tag
imagePullPolicy: IfNotPresent #pull镜像时机，
ports:
- containerPort: 30001 #容器对外开放端口
volumeMounts:
- mountPath: logs/privider-demo
name: provider-docker-demo-logs
volumes:
- name: provider-docker-demo-logs
hostPath:
path: /home/logs/privider-docker-demo
自定义基础docker镜像Dockerfile，定制一个Java服务运行的基础镜像
# centos:7是官方镜像
FROM centos:7
# jdk信息
ADD jdk-8u231-linux-x64.tar.gz /usr/local
ENV JAVA_HOME /usr/local/jdk1.8.0_231
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH $PATH:$JAVA_HOME/bin
# 更改系统语言，不更改的话java服务中中文会出现乱码
RUN yum -y install kde-l10n-Chinese telnet && \
yum -y install glibc-common &&\
yum clean all && \
localedef -c -f UTF-8 -i zh_CN zh_CN.utf8
ENV LC_ALL=zh_CN.utf8
ENV LANG=zh_CN.utf8
ENV LANGUAGE=zh_CN.utf8
# 更改时区
RUN echo 'Asia/Shanghai' >/etc/timezone
创建基础镜像
[root@centos7docker home]# docker build -t centos7docker:443/aliang-xyl/java8:jdk-8u231 . -f DockerFile
推送至harbor私仓
这里需要登录至私仓，否则推不上去
[root@centos7docker home]# docker login -uadmin -p123456 centos7docker:443
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@centos7docker home]# docker push centos7docker:443/aliang-xyl/java8:jdk-8u231
基础镜像构建完毕
创建jenkins镜像
官方的镜像可以直接使用,目前我使用的Jenkins版本是2.204.5，但是如果要发布k8s服务的话，会报错：
kubectl command not found
因为没有安装kubectl,所以需要定制jenkins镜像，Dockerfile如下:
# 这个镜像是官方镜像
FROM jenkins/jenkins:lts
# 因为默认不是使用root账号，无法完成下面的操作，所以必须使用root
USER root
# 直接删除自带的jdk，不同的jenkins可自行运行docker run -it [imageid] /bin/sh 然后查看jdk版本和路径，然后删除
RUN rm -rf /usr/local/openjdk-8
# 使用自己的jdk
ADD jdk-8u231-linux-x64.tar.gz /usr/local
ENV JAVA_HOME /usr/local/jdk1.8.0_231
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH $PATH:$JAVA_HOME/bin
# 使用自己的maven,注意先把压缩包里的settings.xml改好,比如本地仓库路径和私服配置信息
# 这里我把仓库路径配置为：/var/jenkins_home/repository
ADD apache-maven-3.6.3-bin.tar.gz /usr/local
ENV MAVEN_HOME /usr/local/apache-maven-3.6.3
ENV PATH $MAVEN_HOME/bin:$PATH
# 安装kubeclt工具
ADD kubernetes-client-linux-amd64.tar.gz /
RUN chmod +x ./kubernetes/client/bin/kubectl
RUN mv ./kubernetes/client/bin/kubectl /usr/local/bin/kubectl
RUN ln -s /usr/local/bin/kubectl /usr/bin/kubectl
# 需要找到k8s集群里master节点的/etc/kubernetes/admin.conf配置拷贝出来
ADD admin.conf /root
RUN mkdir -p /root/.kube
RUN mv /root/admin.conf /root/.kube/config
RUN chown $(id -u):$(id -g) /root/.kube/config
# 更改jenkins镜像的时区
RUN echo 'Asia/Shanghai' >/etc/timezone
准备文件
[root@k8smaster jenkins]# ll
总用量 211704
-rw-r--r-- 1 root root 5449 3月 6 16:16 admin.conf
-rw-r--r-- 1 root root 9506321 3月 21 02:58 apache-maven-3.6.3-bin.tar.gz
-rw-r--r-- 1 root root 726 3月 21 03:29 Dockerfile
-rw-r--r-- 1 root root 194151339 3月 3 16:04 jdk-8u231-linux-x64.tar.gz
-rw-r--r-- 1 root root 13108955 3月 21 02:38 kubernetes-client-linux-amd64.tar.gz
kubectl下载地址:
注意：我使用的是v1.17.3版本，可根据自己的实际版本下载
https://storage.googleapis.com/kubernetes-release/release/v1.17.3/kubernetes-client-linux-amd64.tar.gz
接下来开始构建jenkins镜像
镜像名称:centos7docker:443/aliang-xyl/jenkins:lts
[root@centos7docker home]# docker build -t centos7docker:443/aliang-xyl/jenkins:lts . -f DockerFile
[root@centos7docker home]# docker push centos7docker:443/aliang-xyl/jenkins:lts
Jenkins服务配置
k8s账号角色信息，文件名：jenkins-service-account.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-admin
secrets:
- name: jenkins-admin-token-bcv7t
---
kind: Role
#rbac的版本为官方定义的版本，可去官网查询
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins-admin
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: ["","extensions", "apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["", "services", "endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins-admin
roleRef:
kind: Role
name: jenkins-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: jenkins-admin
Jenkins的Deployment和Service，文件:jenkins.ymal
注意：我直接使用的是root权限启动的jenkins(yml里有备注)，可以省去很多权限不足的麻烦
#-----Deployment----------------
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
labels:
app: jenkins
spec:
replicas: 1 #副本数为1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
serviceAccountName: jenkins-admin
imagePullSecrets:
- name: jenkins-admin-token-bcv7t
containers:
- name: jenkins
image: centos7docker:443/aliang-xyl/jenkins:lts
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0 #直接使用root权限启动，可以减少很多权限带来的问题
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- name: docker #docker的挂载不能少，否则在运行脚本时会找不到docker命令
mountPath: /usr/bin/docker
- name: docker-sock
mountPath: /var/run/docker.sock
- name: date-config
mountPath: /etc/localtime
- name: kube-config
mountPath: /root/.kube
hostAliases: # 在spec下的hostAliases 处添加，此处为虚拟机的host配置，可根据实际情况来配置
- ip: "192.168.147.130"
hostnames:
- "k8s.apiserver"
- ip: "192.168.147.131"
hostnames:
- "k8sworker1"
- ip: "192.168.147.132"
hostnames:
- "k8sworker2"
volumes:
- name: jenkins-home
hostPath:
path: /home/data/jenkins_home
- name: docker
hostPath:
path: /usr/bin/docker
- name: docker-sock
hostPath:
path: /var/run/docker.sock
- name: date-config
hostPath:
path: /etc/localtime
- name: kube-config
hostPath:
path: /root/.kube
---
#------service---------------
apiVersion: v1
kind: Service
metadata:
name: jenkins
labels:
name: jenkins
spec:
type: NodePort
ports:
- name: jenkins
port: 8080
targetPort: 8080
nodePort: 30009 #开启nodeport
- name: jenkins-agent
port: 50000
targetPort: 50000
nodePort: 30010
selector:
app: jenkins
启动jenkins
在k8s集群的master节点机器上运行以下命令来发布jenkins服务
[root@k8smaster k8s]# kubectl apply -f jenkins-service-account.yml
[root@k8smaster k8s]# kubectl apply -f jenkins.yaml
查看运行状态
[root@k8smaster k8s]# kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-75cb68d55f-7qkbz 1/1 Running 0 4h26m
查看日志获取初始化密码
[root@k8smaster k8s]# kubectl logs jenkins-75cb68d55f-7qkbz
初始化的时候日志里会有一个uuid的字符串，该字符串为初始密码，用来登录jenkins
访问jenkins
可以访问k8s的任意一个子节点来访问jenkins:
jenkins初始化，根据上一步拿到的初始密码来登录jenkins，并获取推荐的插件，
注意：版本过低的jenkins无法安装部分插件，目前我所使用的jenkins为最新版本
系统配置和权限配置
系统设置添加kubernetes云
进入 系统管理/系统配置
如果这里连接失败，绝大部分原因是权限的问题，可能是k8s发布jenkins-service-account.yml的时候除了问题
添加git权限
进入凭据
添加git权限
添加harbor账号信息
注意：权限里的id定义之后在后面会用到
发布Java服务
新建一个流水线任务
这里我直接使用的是脚本
// harbor镜像仓库地址信息
def registry = "centos7docker:443"
// 项目
def project = "aliang-xyl"
def app_name = "provider-docker-demo"
node(){
stage('1.拉取Git代码'){
checkout([$class: 'GitSCM', branches: [[name: '*/master'], [name: '*/develop']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanBeforeCheckout']], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '2e599a5b-f64d-4ef6-992b-f2bff64bb7a9', url: '[email protected]:aLiang-xyl/provider-docker-demo.git']]])
script {
build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
}
image_name = "${registry}/${project}/${app_name}:${build_tag}"
}
stage('2.java代码编译'){
echo "------maven版本、JDK版本-------"
sh "mvn --version"
sh "mvn clean package -Dmaven.test.skip=true"
}
stage('3.镜像打包docker build') {
echo "镜像: ${image_name}"
sh "docker --version"
sh "docker build -t ${image_name} . -f src/main/docker/Dockerfile"
}
stage('4.推镜像仓库Push') {
withCredentials([usernamePassword(credentialsId: 'jenkins-harbor-creds', passwordVariable: 'dockerHubPassword', usernameVariable: 'dockerHubUser')]) {
sh "docker login -u ${dockerHubUser} -p ${dockerHubPassword} ${registry}"
sh "docker push ${image_name}"
}
}
stage('5.部署到k8s') {
sh "sed -i 's//${build_tag}/' src/main/docker/k8s.yaml"
sh "kubectl apply -f src/main/docker/k8s.yaml"
echo "项目部署成功"
}
}
或者也可以把脚本文件Jenkinsfile放在项目的根目录
注意：脚本里的checkout可以自动生成:
接下来就是见证奇迹的时刻
查看java服务运行状态
[root@k8smaster k8s]# clear
[root@k8smaster k8s]# kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-75cb68d55f-7qkbz 1/1 Running 0 5h13m
provider-docker-demo-7f66999f78-jphn7 1/1 Running 0 144m
至此整个发布流程结束