.Net Core 微服务实战 - 部署到 Kubernetes

Kubernetes

Kubernetes 是一个用于自动部署、扩展和管理容器化应用程序的开源系统。

Kubernetes 的优势

• 自动化容器部署
• 资源管理与容器调度
• 服务注册发现与负载均衡
• 内置配置与密钥管理
• 丰富的社区组件
• 极强的可扩展性

环境搭建

安装 Docker Desktop

官网下载安装： Docker Desktop

安装后启动若提示：cannot enable hyper-v service，确保 Windows 功能启用了 Hyper-V :
并打开控制台输入 systeminfo 查看 Hyper-V 要求，若固件中已启用虚拟化为否，则需要进入 BIOS 面板，启用 CPU Configuration 的 Intel Virtualization Technology 。

完成后即可正常启动 Docker Desktop 。

安装 Kubernetes

Kubernetes 安装参考：https://github.com/AliyunContainerService/k8s-for-docker-desktop

这里说几个踩到的坑：
直接在 Docker Desktop 上启用 Kubernetes ，发现 Kubernetes 一直在启动状态。
卸载 Docker Desktop 后参考：https://github.com/AliyunContainerService/k8s-for-docker-desktop 安装后，发现 Kubernetes 还是一直处于启动状态，删除 ‘C:\Users\yourUserName\AppData\Local\Docker\pki’ 目录后同样还是不行。
最后再次卸载 Docker Desktop ，然后把所有 Docker 、 Kubernetes 相关目录全部删除掉，重新参考：https://github.com/AliyunContainerService/k8s-for-docker-desktop 安装后， Kubernetes 仍然一直处于启动状态，删除 ‘C:\Users\yourUserName\AppData\Local\Docker\pki’ 目录后重启发现 Kubernetes 可以正常启动了。

安装 Helm

安装 Helm 之前需要 使用 PowerShell 以管理员身份 安装 Chocolatey ：

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

使用 PowerShell 以管理员身份 安装 Helm ：

# Use Chocolatey on Windows
# 本行命令需要需要管理员身份
choco install kubernetes-helm

# Change helm repo
helm repo add stable http://mirror.azure.cn/kubernetes/charts/

# Update charts repo
helm repo update

安装 ingress 和 kubernetes-dashboard

使用 PowerShell 安装：

kubectl apply -f mandatory.yaml
kubectl apply -f ingress-port.yaml
kubectl apply -f kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard-ingress.yaml

kubernetes TOKEN 获取：

$TOKEN=((kubectl -n kube-system describe secret default | Select-String "token:") -split " +")[1]
kubectl config set-credentials docker-for-desktop --token="${TOKEN}"
echo $TOKEN

"Any key to exit"  ;
Read-Host | Out-Null ;
Exit

访问 kubernetes-dashboard ： http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

安装 prometheus

安装参考：https://github.com/AliyunContainerService/prometheus-operator 。
配置 values ：

helm upgrade prometheus-operator stable/prometheus-operator --version 8.10.0 --values .\prometheus-operator\values.yaml  --namespace kube-system

安装基础设施

kubectl create configmap fluentd-config --from-file=fluentd -o yaml --dry-run | kubectl apply -f - 
kubectl apply -f .

fluentd.conf ：

  @type  forward
  @id    input1
  @label @mainstream
  port  24224


  @id fluentd-containers.log
  @label @mainstream
  @type tail
  path /var/log/containers/ts*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag raw.kubernetes.*
  format json
  read_from_head true


  @type stdout



  
    @type elasticsearch
    host elasticsearch
    include_tag_key true
    type_name fluentd
    port 9200
    logstash_format true
  
  
    @type file
    @id   output_docker1
    path         /fluentd/log/docker.*.log
    symlink_path /fluentd/log/docker.log
    append       true
    time_slice_format %Y%m%d
    time_slice_wait   1m
    time_format       %Y%m%dT%H%M%S%z
  
  
    @type file
    @id   output1
    path         /fluentd/log/data.*.log
    symlink_path /fluentd/log/data.log
    append       true
    time_slice_format %Y%m%d
    time_slice_wait   10m
    time_format       %Y%m%dT%H%M%S%z
  

kubectl apply -f . : 执行目录下所有 yaml 文件

elasticsearch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: "elasticsearch:7.5.2"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/usr/share/elasticsearch/data"
            name: elasticsearch-storage
        ports:
        - containerPort: 9200
        - containerPort: 9300
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: elasticsearch-storage
          hostPath:
              path: "/g/k8s/volumes/elasticsearch/data"
              type: DirectoryOrCreate
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: "elasticsearch"
spec:
  type: NodePort
  ports:
  - nodePort: 30007
    port: 9200
    targetPort: 9200
    name: "9200"
    protocol: TCP
  - nodePort: 30008
    port: 9300
    targetPort: 9300
    name: "9300"
    protocol: TCP
  selector:
    app: elasticsearch

exceptionless.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: exceptionless-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: exceptionless-api
  template:
    metadata:
      labels:
        app: exceptionless-api
    spec:
      containers:
      - name: exceptionless-api
        image: "exceptionless/api:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:[email protected]:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-storage
          hostPath:
              path: "/g/k8s/volumes/exceptionless"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: "exceptionless-api"
spec:
  type: NodePort
  ports:
  - nodePort: 30012
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-api

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: exceptionless-job
spec:
  replicas: 1
  selector:
    matchLabels:
      app: exceptionless-job
  template:
    metadata:
      labels:
        app: exceptionless-job
    spec:
      containers:
      - name: exceptionless-job
        image: "exceptionless/job:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:[email protected]:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-job-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-job-storage
          hostPath:
              path: "/g/k8s/volumes/exceptionless"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: "exceptionless-job"
spec:
  type: NodePort
  ports:
  - nodePort: 30014
    port: 80
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: exceptionless-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: exceptionless-ui
  template:
    metadata:
      labels:
        app: exceptionless-ui
    spec:
      containers:
      - name: exceptionless-ui
        image: "exceptionless/ui:2.8.1497"
        imagePullPolicy: IfNotPresent
        env:
        - name: AppMode
          value: "Production"
        - name: EX_ApiUrl
          value: http://localhost:30012
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: "exceptionless-ui"
spec:
  type: NodePort
  ports:
  - nodePort: 30013
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-ui

fluentd.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: fluentd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: "witskeeper/fluentd-es:68"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/fluentd/etc/fluent.conf"
            name: fluentd-config
            subPath: fluent.conf
          - mountPath: "/var/log"
            name: containers-logs
          - mountPath: "/var/lib/docker/containers"
            name: docker-logs
        ports:
        - containerPort: 24224
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: fluentd-config
          configMap:
              name: fluentd-config
        - name: containers-logs
          hostPath:
              path: "/var/log"
              type: DirectoryOrCreate
        - name: docker-logs
          hostPath:
            path: /var/lib/docker/containers
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: "fluentd"
spec:
  type: NodePort
  ports:
  - nodePort: 30011
    port: 24224
    targetPort: 24224
    protocol: TCP
  selector:
    app: fluentd

grafana.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: default
  labels:
    tag: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: "grafana/grafana:6.5.2"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3000
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 256Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: default
  labels:
    tag: "grafana"
spec:
  type: NodePort
  ports:
  - nodePort: 30015
    port: 3000
    targetPort: 3000
    protocol: TCP
  selector:
    app: grafana

kibana.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: default
  labels:
    tag: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: "kibana:6.8.6"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        - name: XPACK_MONITORING_ENABLED
          value: "true"
        ports:
        - containerPort: 5601
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    tag: "kibana"
spec:
  type: NodePort
  ports:
  - nodePort: 30009
    port: 5601
    targetPort: 5601
    protocol: TCP
  selector:
    app: kibana

mysql.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: default
  labels:
    tag: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: "mysql:8.0.19"
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        args:
          - "--character-set-server=utf8mb4"
          - "--collation-server=utf8mb4_unicode_ci"
        volumeMounts:
          - mountPath: "/var/lib/mysql"
            name: mysql-pv-storage
        ports:
        - containerPort: 3306
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 256Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: mysql-pv-storage
          hostPath:
              path: "/g/k8s/volumes/mysql"
              type: DirectoryOrCreate
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: default
  labels:
    tag: "mysql"
spec:
  type: NodePort
  ports:
  - nodePort: 30001
    port: 3306
    targetPort: 3306
    protocol: TCP
  selector:
    app: mysql

rabbitmq.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq
  namespace: default
  labels:
    tag: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: "rabbitmq:3.8.2-management"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 4369
        - containerPort: 5671
        - containerPort: 5672
        - containerPort: 15671
        - containerPort: 15672
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 256Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: rabbitmq-storage
          hostPath:
              path: "/g/k8s/volumes/rabbitmq"
              type: DirectoryOrCreate
      restartPolicy: Always


---
  
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: default
  labels:
    tag: "redis"
spec:
  type: NodePort
  ports:
  - nodePort: 30003
    name: rabbitmq-manager
    port: 15672
    targetPort: 15672
    protocol: TCP
  - nodePort: 30004
    name: rabbitmq-service
    port: 5672
    targetPort: 5672
    protocol: TCP
  selector:
    app: rabbitmq

redis.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: default
  labels:
    tag: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: "redis:5.0.7"
        imagePullPolicy: IfNotPresent
        args:
          - "--appendonly yes"
        volumeMounts:
          - mountPath: "/data"
            name: redis-storage
        ports:
        - containerPort: 6379
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: redis-storage
          hostPath:
              path: "/g/k8s/volumes/redis"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: default
  labels:
    tag: "redis"
spec:
  type: NodePort
  ports:
  - nodePort: 30002
    port: 6379
    targetPort: 6379
    protocol: TCP
  selector:
    app: redis

部署应用

构建

为项目生成 Dockerfile ：右击项目 -> 添加 -> Docker 支持。
配置 Nuget ：COPY [“NuGet.config”,“NuGet.config”]
添加NuGet.config至项目根目录：

<configuration>
  <packageSources>
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
    <add key="skyapm-dotnet" value="https://www.myget.org/F/netcorepal/api/v3/index.json" />
  packageSources>
configuration>

使用 PowerShell 运行：

Param(
    [parameter(Mandatory=$true)][string]$tag
)

docker build --no-cache  -f  .\App\TS.Microservices.WebApi\Dockerfile -t ts-microservices-webapi:$tag .
docker build --no-cache  -f  .\ApiGateways\TS.Microservices.Mobile.Gateway\Dockerfile -t ts-microservices-mobile-gateway:$tag .
docker build --no-cache  -f  .\ApiGateways\TS.Microservices.Mobile.ApiAggregator\Dockerfile -t ts-microservices-mobile-apiaggregator:$tag .

输入版本号开始构建：

完成后可以使用 docker images 命令看到镜像：

部署

kubectl create configmap ts-microservices-webapi-config --from-file=ts-microservices-webapi/configs -o yaml --dry-run | kubectl apply -f - 
kubectl apply -f .\ts-microservices-webapi\ts-microservices-webapi.yaml

kubectl create configmap ts-microservices-mobile-apiaggregator-config --from-file=ts-microservices-mobile-apiaggregator/configs -o yaml --dry-run | kubectl apply -f - 
kubectl apply -f .\ts-microservices-mobile-apiaggregator\ts-microservices-mobile-apiaggregator.yaml

kubectl create configmap ts-microservices-config --from-env-file=env.txt -o yaml --dry-run | kubectl apply -f - 

kubectl create configmap ts-microservices-mobile-gateway-config --from-file=ts-microservices-mobile-gateway/configs -o yaml --dry-run | kubectl apply -f - 
kubectl apply -f .\ts-microservices-mobile-gateway\ts-microservices-mobile-gateway.yaml
kubectl apply -f .\ts-microservices-mobile-gateway\ts-microservices-mobile-gateway-ingress.yaml

通过 Helm 部署：

kubectl create configmap ts-microservices-webapi-config --from-file=ts-microservices-webapi/configs -o yaml --dry-run | kubectl apply -f - 
kubectl create configmap ts-microservices-mobile-gateway-config --from-file=ts-microservices-mobile-gateway/configs -o yaml --dry-run | kubectl apply -f - 
kubectl create configmap ts-microservices-config --from-env-file=env.txt -o yaml --dry-run | kubectl apply -f - 
kubectl create configmap ts-microservices-mobile-apiaggregator-config --from-file=ts-microservices-mobile-apiaggregator/configs -o yaml --dry-run | kubectl apply -f - 

helm install ts-microservices-webapi .\charts\ts-microservices-webapi -n default
helm install ts-microservices-mobile-gateway .\charts\ts-microservices-mobile-gateway -n default
helm install ts-microservices-mobile-apiaggregator .\charts\ts-microservices-mobile-apiaggregator -n default

第一次安装使用 install ，后续使用 upgrade 。

helm uninstall ts-microservices-webapi  -n default
helm uninstall ts-microservices-mobile-gateway  -n default
helm uninstall ts-microservices-mobile-apiaggregator  -n default