在前面的三篇文章中，已经实现了使用Jenkins拉取GitHub的项目，并进行构建，最后在推送到采用Harbor搭建的私有镜像仓库中。CI的流程基本上可以说已经实现完了，这节将介绍如何将构建好的服务部署到k8s集群中。
以下的操作全部在namespace是tlh的命名空间执行，所以之前先创建namespace。
kubectl create namespace tlh
1、部署MySQL
1.1 创建pv用户挂载初始化脚本
-
创建pvc.yaml文件
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mysql-init-storage namespace: tlh spec: accessModes: - ReadWriteMany resources: requests: storage: 50M -
创建pvc
这里也是使用前面章节创建的默认的storageclass来动态创建pv
kubectl apply -f pvc.yaml -n tlh -
上传初始化脚本
-
查看创建好的pv
1.png -
将源码中的tlh.sql文件上传到对应的文件夹中
2.png
-
1.2 创建部署
-
创建deployment.yaml文件
--- apiVersion: apps/v1 kind: Deployment metadata: name: mysql spec: selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - image: mysql:5.7 name: mysql env: - name: MYSQL_ROOT_PASSWORD # 设置root密码 value: "123456" - name: MYSQL_DATABASE # 初始化创建的数据库 value: "tlh" ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage # 挂载数据存储 mountPath: /var/lib/mysql - name: mysql-init mountPath: /docker-entrypoint-initdb.d # 挂载初始化脚本目录 volumes: - name: mysql-persistent-storage # 声明使用的pv persistentVolumeClaim: claimName: mysql-pv-claim - name: mysql-init persistentVolumeClaim: claimName: mysql-init-storage --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pv-claim # 创建存储的pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi -
创建deployment
kubectl apply -f deployment.yaml -n tlh -
查看deployment
kubectl get deployment mysql -n tlh3.png
1.3 创建服务
-
创建service.yaml文件
apiVersion: v1 kind: Service metadata: name: mysql spec: ports: - port: 3306 selector: app: mysql -
创建service
kubectl apply -f service.yaml -n tlh -
查看服务
kubectl get service mysql -n tlh4.png
2、部署Tlh服务
因为最终在Jenkins构建完之后是将docker镜像上传到的harbor搭建的私有仓库中，而k8s不能直接从私有仓库拉取镜像，所以先要配置密钥。
2.1 配置k8s登陆harbor的密钥
-
在master主机执行docker login，这里填写上一章中在harbor中创建的jenkins账户的信息。
docker login -
生成密钥
kubectl create secret generic regcred \ --from-file=.dockerconfigjson=.docker/config.json \ --type=kubernetes.io/dockerconfigjson \ -n tlh # 注意命名空间 -
查看密钥
kubectl get secret regcred --output=yaml -n tlh
2.2 创建configmap
在部署的服务中需要使用到MySQL服务，而其配置信息需要数据库驱动、用户名和密码，在以往docker方式部署的时候通过环境变量的方式注入到应用中。这里先采用configmap来存储应用需要的配置信息，方便进行管理，最后也通过环境变量的方式注入到pod中。
- 创建tlh-env-file.properties文件
mysql.user=root
mysql.pwd=123456
mysql.url=jdbc:mysql://mysql.tlh.svc.cluster.local:3306/tlh # 这里使用k8s的dns来填写数据库的url地址
-
创建configmap
kubectl create configmap tlh-configmap --from-env-file=tlh-env-file.properties -n tlh -
查看configmap
kubectl get configmaps tlh-configmap -o yaml -n tlh5.png
2.3 创建deployment
-
创建deploymen.yaml文件
apiVersion: apps/v1 kind: Deployment metadata: name: tlh-deployment namespace: tlh spec: selector: matchLabels: app: tlh-server template: metadata: labels: app: tlh-server spec: containers: - image: harbor.tlh.com/tlhhup/jks:1.2 name: tlh env: - name: MYSQL_USER valueFrom: configMapKeyRef: # 这里引用configmap中的key name: tlh-configmap key: mysql.user - name: MYSQL_PASSWORD valueFrom: configMapKeyRef: name: tlh-configmap key: mysql.pwd - name: MYSQL_URL valueFrom: configMapKeyRef: name: tlh-configmap key: mysql.url ports: - containerPort: 8089 name: tlh imagePullSecrets: - name: regcred -
创建deployment
kubectl apply -f deployment.yaml -n tlh -
查看
kubectl get deployment tlh-deployment -n tlh6.png
2.4 创建service
-
创建service.yaml
apiVersion: v1 kind: Service metadata: name: tlh spec: ports: - port: 8089 selector: app: tlh-server -
创建service
kubectl apply -f service.yaml -n tlh -
查看
kubectl get service tlh -n tlh7.png
3、部署Ingress
为了达到单独的IP地址的效果，这里重新部署一个ingress-controller，最终由metallab进行重新分配一个新得IP地址。
3.1 部署ingress-controller
-
创建deployement.yaml文件
kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - update - list - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "- " # Here: " - " # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - list - apiGroups: - "" resources: - configmaps verbs: - create - update - list - apiGroups: - "" resources: - endpoints verbs: - get - list --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-tlh-binding namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: tlh --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-tlh-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: tlh --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount containers: - name: nginx-ingress-controller image: wistiaanders/nginx-ingress-controller:0.25.1 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io - --ingress-class=tlh-nginx-ingress # 注意这个值的设置，必需唯一 - --enable-ssl-passthrough securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 lifecycle: preStop: exec: command: - /wait-shutdown --- -
创建deployment
kubectl apply -f deployment.yaml -n tlh -
创建service.yaml文件
kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: tlh labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: externalTrafficPolicy: Local type: LoadBalancer selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ports: - name: http port: 80 targetPort: http - name: https port: 443 targetPort: https --- -
创建service
kubectl apply -f service.yaml -n tlh
3.2 部署Tlh的ingress
-
创建ingress.yaml文件
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: tlh-ingress namespace: tlh annotations: kubernetes.io/ingress.class: "tlh-nginx-ingress" # 设置使用的ingress-controller spec: rules: - host: dev.tlh.com # 通过host来匹配规则 http: paths: - path: / backend: serviceName: tlh servicePort: 8089 -
创建ingress
kubectl apply -f ingress.yaml -n tlh -
查看ingress
kubectl describe ingress -n tlh8.png -
将查看得到的IP地址配置到宿主机的hosts文件中，打开浏览器http://dev.tlh.com/tlh,输入用户名和密码(admin/admin)
9.png