K8s平台部署企业级项目案例

目录

1、架构图

2、部署一个java网站项目实践

2.1 编译java

2.2、构建与上传镜像

2.3、k8s中部署项目

访问

3、资源配置

3.2、加入配置文件挂载

4、部署数据库并连接

恢复数据

项目连接数据库

5、更新

执行

6、ingress暴露（前提需要运行ingress服务）

7、配置外网负载均衡

---

1、架构图

在没有k8s前，一个项目的开发部署架构大概是这样的

在有k8s后，这个架构变成了这样

在k8s中部署项目的流程

2、部署一个java网站项目实践

2.1 编译java

第一步：先将代码拉取到本地（这里有一个实例代码拉取），下载好的代码就是一个包，没有的可以通过此命令拉取

git clone https://github.com/lizhenliang/tomcat-java-demo

java项目需要下载maven依赖和java-1.8环境编译，yum安装

yum install java-1.8.0-openjdk maven git -y

安装成功

进入java包内

cd tomcat-java-demo-master

修改mvn为阿里云镜像地址

vi /etc/maven/settings.xml

在mirror位置加入阿里云地址

阿里云仓库地址为

  aliyunmaven

  *

  阿里云公共仓库

  https://maven.aliyun.com/repository/public

清理过去的依赖并跳过测试并编译代码

mvn clean package -Dmaven.test.skip=ture

成功，成功后会生成一个target目录

target目录里面会多一个war包

ls target/

这个war包就是要运行的程序

解压这个war包内容到ROOT目录

unzip target/ly-simple-tomcat-0.0.1-SNAPSHOT.war -d ROOT

root内就是可运行的项目

2.2、构建与上传镜像

制作一个dockerfile

vi Dockerfile

FROM lizhenliang/tomcat

LABEL maintainer af

COPY target/ROOT /usr/local/tomcat/webapps/ROOT

这个dockerfile意思是引入一个tomcat镜像，在将运行项目放入这个tomcat的运行目录，注意，这个docker要在tomcat-java-demo-master目录执行

执行打包命令，将此镜像命名为java-demo:v1 

docker build -t java-demo:v1 .

在harbor创建一个demo私有仓库

注意：在harbor中，公开仓库是免凭据拉取，推送需要凭据

                               私有仓库拉取与推送都需要凭据

给刚刚新建的镜像打上tag(镜像名前面对应仓库服务器ip与要推送的仓库名)

docker tag java-demo:v1 192.168.209.100/demo/java-demo:v1

配置可信任（如果仓库是HTTPS访问不用配置，加入仓储ip）（集群内所有服务器都需要配置）

vi /etc/docker/daemon.json

{

  "insecure-registries": ["192.168.209.100"]

}

在服务器上加入域名解析

vi /etc/hosts

192.168.209.100 reg.ctnrs.com

登录harbor

docker login 192.168.209.100

账号admin，密码harbor

上传镜像

docker push 192.168.209.100/demo/java-demo:v1

2.3、k8s中部署项目

创建一个测试yaml

kubectl create deployment java-demo --image=192.168.209.100/demo/java-demo:v1 --dry-run=client -o yaml > deployment.yaml

此时如果直接执行yaml会报仓库没有授权

所以，此时需要k8s中的Secret来进行操作，创建一个k8s的Secret来保存凭据，kubectl调用这个凭据

kubectl create secret docker-registry harbor-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=192.168.209.100

查看是否创建成功

kubectl get secret

在deployment中引用这个secret（他与containers同级）

vi deployment

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    app: java-demo

  name: java-demo

spec:

  replicas: 1

  selector:

    matchLabels:

      app: java-demo

  strategy: {}

  template:

    metadata:

      creationTimestamp: null

      labels:

        app: java-demo

    spec:

      imagePullSecrets:

      - name: harbor-auth

      containers:

      - image: 192.168.209.100/demo/java-demo:v1

        name: java-demo

        resources: {}

执行

kubectl apply -f deployment.yaml

成功

访问

为这个deployment创建service访问

kubectl expose deployment java-demo --port=80 --target-port=8080 --type=NodePort

成功，对外端口为32575

访问

访问成功

3、资源配置

使用控制器部署镜像

给项目加上资源限制与健康检查

vi deployment

一般项目部署都需要配置这些

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    app: java-demo

  name: java-demo

spec:

  replicas: 1

  selector:

    matchLabels:

      app: java-demo

  strategy: {}

  template:

    metadata:

      creationTimestamp: null

      labels:

        app: java-demo

    spec:

      imagePullSecrets:

      - name: harbor-auth

      containers:

      - image: 192.168.209.100/demo/java-demo:v1

        name: java-demo

        resources:

          limits:

            cpu: 1

            memory: 1Gi

          requests:

            cpu: 0.7

            memory: 700Mi

        livenessProbe:

          httpGet:

            path: /

            port: 8080

          initialDelaySeconds: 50

          periodSeconds: 30

        readinessProbe:

          httpGet:

            path: /

            port: 8080

          initialDelaySeconds: 50

          periodSeconds: 30

3.2、加入配置文件挂载

先将配置文件存入configMap

vi configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: java-demo-config

data:

  application.yaml: |

      server:

        port: 8080

      spring:

        datasource:

          url: jdbc:mysql://localhost:3306/test?characterEncoding=utf-8

          username: root

          password: 12345

          driver-class-name: com.mysql.jdbc.Driver

        freemarker:

          allow-request-override: false

          cache: true

          check-template-location: true

          charset: UTF-8

          content-type: text/html; charset=utf-8

          expose-request-attributes: false

          expose-session-attributes: false

          expose-spring-macro-helpers: false

          suffix: .ftl

          template-loader-path:

            - classpath:/templates/

kubectl apply -f configmap.yaml

kubectl get configmap

成功

在deployment中通过configmap挂载配置文件到项目

vi deployment

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    app: java-demo

  name: java-demo

spec:

  replicas: 1

  selector:

    matchLabels:

      app: java-demo

  strategy: {}

  template:

    metadata:

      creationTimestamp: null

      labels:

        app: java-demo

    spec:

      imagePullSecrets:

      - name: harbor-auth

      containers:

      - image: 192.168.209.100/demo/java-demo:v1

        name: java-demo

        resources:

          limits:

            cpu: 1

            memory: 1Gi

          requests:

            cpu: 0.7

            memory: 700Mi

        livenessProbe:

          httpGet:

            path: /

            port: 8080

          initialDelaySeconds: 50

          periodSeconds: 30

        readinessProbe:

          httpGet:

            path: /

            port: 8080

          initialDelaySeconds: 50

          periodSeconds: 30

        volumeMounts:

        - name: config

          mountPath: "/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/application.yml"

          subPath: "application.yml"

      volumes:

      - name: config

        configMap:

          name: java-demo-config

          items:

          - key: "application.yaml"

            path: "application.yml"

kubectl applyment -f deployment

4、部署数据库并连接

一般情况下不建议k8s中部署mysql

将mysql.yaml传入服务器

此mysql的部署用到了数据卷nfs动态供给（pv、pvc）没有做的参考前面的nfs部署与pv、pvc设置文档

如果没有nfs动态供给环境，可以删除标红内容改为emptyDir: {}，与name同缩进，做临时数据库存储

vi mysql.yaml

apiVersion: v1

kind: Secret

metadata:

  name: java-demo-db

type: Opaque

data:

  mysql-root-password: "MTIzNDU2"

  mysql-password: "MTIzNDU2"

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: java-demo-db

spec:

  selector:

    matchLabels:

      project: www

      app: mysql

  template:

    metadata:

      labels:

        project: www

        app: mysql

    spec:

      containers:

      - name: db

        image: mysql:5.7.30

        resources:

          requests:

            cpu: 500m

            memory: 512Mi

          limits:

            cpu: 500m

            memory: 512Mi

        env:

        - name: MYSQL_ROOT_PASSWORD

          valueFrom:

            secretKeyRef:

              name: java-demo-db

              key: mysql-root-password

        - name: MYSQL_PASSWORD

          valueFrom:

            secretKeyRef:

              name: java-demo-db

              key: mysql-password

        - name: MYSQL_USER

          value: "aliang"

        - name: MYSQL_DATABASE

          value: "k8s"

        ports:

        - name: mysql

          containerPort: 3306

        livenessProbe:

          exec:

            command:

            - sh

            - -c

            - "mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"

          initialDelaySeconds: 30

          periodSeconds: 10

        readinessProbe:

          exec:

            command:

            - sh

            - -c

            - "mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"

          initialDelaySeconds: 5

          periodSeconds: 10

        volumeMounts:

        - name: data

          mountPath: /var/lib/mysql

      volumes:

      - name: data

        persistentVolumeClaim:

          claimName: java-demo-db

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: java-demo-db

spec:

  storageClassName: "managed-nfs-storage"

  accessModes:

    - "ReadWriteOnce"

  resources:

    requests:

      storage: "8Gi"

---

apiVersion: v1

kind: Service

metadata:

  name: java-demo-db

spec:

  type: ClusterIP

  ports:

  - name: mysql

    port: 3306

    targetPort: mysql

  selector:

    project: www

    app: mysql

kubectl apply -f mysql.yaml

恢复数据

将sql文件导入到k8s中的mysql容器中

sql文件所在位置

cd /root/tomcat-java-demo-master/db

将sql文件拷贝进容器

kubectl cp tables_ly_tomcat.sql java-demo-db-76d66777b6-bcwx4:/

进入mysql容器，sql文件已传入

kubectl exec -it java-demo-db-76d66777b6-bcwx4 -- bash

连接数据库，密码为123456，用show databases可以看到数据库内有k8s库，变量生效

mysql -uroot -p123456

在mysql中创建一个数据库test并切换到这个数据库

create database test;

use test；

在test库中执行拷贝进来的sql

source /tables_ly_tomcat.sql;

可以看到表已经导入

show tables；

项目连接数据库

修改之前创建的与deployment绑定的configmap，修改他的mysql连接地址为mysql的地址，使用mysql的service中CLUSTER-IP的ip与端口，账号密码也进行对应修改

vi configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: java-demo-config

data:

  application.yaml: |

      server:

        port: 8080

      spring:

        datasource:

          url: jdbc:mysql://10.108.140.66:3306/test?characterEncoding=utf-8

          username: root

          password: 123456

          driver-class-name: com.mysql.jdbc.Driver

        freemarker:

          allow-request-override: false

          cache: true

          check-template-location: true

          charset: UTF-8

          content-type: text/html; charset=utf-8

          expose-request-attributes: false

          expose-session-attributes: false

          expose-spring-macro-helpers: false

          suffix: .ftl

          template-loader-path:

            - classpath:/templates/

执行

kubectl apply -f configmap.yaml

5、更新

修改了configmap但程序也不会动态更新，需要人工来更新，此时我们进行一次更新的全套模拟操作

cd /root/tomcat-java-demo-master

随便修改项目主页一点内容，模拟一次更新

vi src/main/resources/templates/index.ftl

用maven重新编译

mvn clean package -Dmaven.test.skip=ture

重新解压放入ROOT

cd target/

unzip ly-simple-tomcat-0.0.1-SNAPSHOT.war -d ROOT

回到docker位置重新打包

cd ..

docker build -t java-demo:v2 .

修改tag

docker tag java-demo:v2 192.168.209.100/demo/java-demo:v2

上传镜像到仓库

docker push 192.168.209.100/demo/java-demo:v2

修改deployment的yaml，改用新的镜像v2版本

vi deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  labels:

    app: java-demo

  name: java-demo

spec:

  replicas: 1

  selector:

    matchLabels:

      app: java-demo

  strategy: {}

  template:

    metadata:

      creationTimestamp: null

      labels:

        app: java-demo

    spec:

      imagePullSecrets:

      - name: harbor-auth

      containers:

      - image: 192.168.209.100/demo/java-demo:v2

        name: java-demo

        resources:

          limits:

            cpu: 1

            memory: 1Gi

          requests:

            cpu: 0.7

            memory: 700Mi

        livenessProbe:

          httpGet:

            path: /

            port: 8080

          initialDelaySeconds: 50

          periodSeconds: 30

        readinessProbe:

          httpGet:

            path: /

            port: 8080

          initialDelaySeconds: 50

          periodSeconds: 30

        volumeMounts:

        - name: config

          mountPath: "/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/application.yml"

          subPath: "application.yml"

      volumes:

      - name: config

        configMap:

          name: java-demo-config

          items:

          - key: "application.yaml"

            path: "application.yml"

kubectl apply -f deployment.yaml

执行

访问项目，可以看见页面已经修改，数据库也连接成功

6、ingress暴露（前提需要运行ingress服务）

vi ingress.yaml

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: java-demo

spec:

  ingressClassName: nginx

  rules:

  - host: java.aliangedu.cn

    http:

      paths:

      - path: /

        pathType: Prefix

        backend:

          service:

            name: java-demo

            port:

              number: 80

kubectl apply -f ingress.yaml

ingress代理成功

7、配置外网负载均衡

使用nginx模拟外网负载均衡

先安装epel源

yum install epel-release

安装nginx

yum install nginx

修改nginx配置

vi /etc/nginx/nginx.conf

加入这两段对ingress进行外网暴露与负载均衡

upstream ingress-controller {

        server 192.168.209.111:30761;

        server 192.168.209.112:30761;

    }

    server {

        listen       80;

        listen       [::]:80;

        server_name  _;

        root         /usr/share/nginx/html;

        location / {

            proxy_pass http://ingress-controller;

            proxy_set_header Host $Host;

        }

启动nginx

systemctl start nginx

这时访问nginx的端口会被代理到ingress服务上

监听nginx访问日志可以看到我们访问信息

代理成功