k8s安装配置nacos集群

1 在k8s中部署的难点

在k8s中部署nacos集群和在裸机器上直接部署nacos机器其实差别不大。
最主要的区别是k8s中部署的服务没有固定的ip地址，而nacos集群部署需要配置所有实例的ip

2 解决

在k8s中通过StatefulSet和Headless Service为每个nacos实例生成一个唯一的dns地址，
创建一个普通Service给可客户端使用

版本： nacos1.4.0

前提前提条件：

mysql 

Ingress Controller (用于ingress暴露方式)

具体实现如下。

3 实现方式

3.1 创建数据库配置

vim nacos-mysql-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
   name: nacos-uid
data:
   mysql.host: "192.168.199.201"
   mysql.db.name: "nacos_config"
   mysql.port: "3306"
   mysql.user: "root"
   mysql.password: "mysqlpassword"

kubectl apply   -f  nacos-mysql-config.yaml

注意：我是直接现有mysql的nacos库，如果要新建数据库必须先在数据库新建名称为nacos_config的库并导入对应nacos1.4.0下config/nacos-mysql.sql脚本导入到nacos_config库

3.2 部署Headless Service

Headless Service为每个pod（nacos实例）生成一个DNS地址，用作NACOS_SERVERS配置

vim nacos-headless.yaml

apiVersion: v1
kind: Service
metadata:
   name: nacos-headless
   labels:
     app: nacos
   annotations:
     service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
   ports:
     - port: 8848
       name: server
       targetPort: 8848
   clusterIP: None
   selector:
     app: nacos

kubectl apply   -f  nacos-headless.yaml

3.3 通过StatefulSet部署nacos

StatefulSet部署方式为每个POD生成固定的名称，如nacos-0、nacos-1、nacos-2等。

vim nacos-StatefulSet.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
   name: nacos
spec:
   serviceName: nacos-headless
   replicas: 3
   template:
     metadata:
       labels:
         app: nacos
       annotations:
         pod.alpha.kubernetes.io/initialized: "true"
     spec:
       affinity:
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             - labelSelector:
                 matchExpressions:
                   - key: "app"
                     operator: In
                     values:
                       - nacos-headless
               topologyKey: "kubernetes.io/hostname"
       containers:
         - name: k8snacos
           imagePullPolicy: Always
           image: nacos/nacos-server:1.4.0
           resources:
             requests:
               memory: "2Gi"
               cpu: "500m"
           ports:
             - containerPort: 8848
               name: client
           env:
             - name: NACOS_REPLICAS
               value: "3"
             - name: MYSQL_SERVICE_HOST
               valueFrom:
                 configMapKeyRef:
                   name: nacos-uid
                   key: mysql.host
             - name: MYSQL_SERVICE_DB_NAME
               valueFrom:
                 configMapKeyRef:
                   name: nacos-uid
                   key: mysql.db.name
             - name: MYSQL_SERVICE_PORT
               valueFrom:
                 configMapKeyRef:
                   name: nacos-uid
                   key: mysql.port
             - name: MYSQL_SERVICE_USER
               valueFrom:
                 configMapKeyRef:
                   name: nacos-uid
                   key: mysql.user
             - name: MYSQL_SERVICE_PASSWORD
               valueFrom:
`          configMapKeyRef:
                   name: nacos-uid
                   key: mysql.password
             - name: MODE
               value: "cluster"
             - name: NACOS_SERVER_PORT
               value: "8848"
             - name: PREFER_HOST_MODE
               value: "hostname"
             - name: NACOS_SERVERS
               value: "nacos-0.nacos-headless.default.svc.cluster.local:8848 nacos-1.nacos-headless.default.svc.cluster.local:8848 nacos-2.nacos-headless.default.svc.cluster.local:8848"
   selector:
     matchLabels:
       app: nacos

kubectl apply   -f  nacos-StatefulSet.yaml

3.4 部署普通Service

apiVersion: v1
kind: Service
metadata:
   name: nacos-service
   namespace: default
   annotations:
     nginx.ingress.kubernetes.io/affinity: "true"
     nginx.ingress.kubernetes.io/session-cookie-name: backend
     nginx.ingress.kubernetes.io/load-balancer-method: drr

spec:
   selector:
     app: nacos
   ports:
     - name: web
       port: 80
       targetPort: 8848
       nodePort:  8848
   type: NodePort

kubectl apply   -f  nacos-service.yaml

执行的时候报错误，报错是因为在 Kubernetes 创建 service 使用nodePort 暴露 外部端口8848 时报错端口权限限制

 The Service "nacos-service" is invalid: spec.ports[0].nodePort: Invalid value: 8848: provided port is not in the valid range. The range of valid ports is 30000-32767

 The Service "nacos-service" is invalid: spec.ports[0].nodePort: Invalid value: 8848: provided port is not in the valid range. The range of valid ports is 30000-32767
解决办法：

编辑 /etc/kubernetes/manifests/kube-apiserver.yaml文件

找到 --service-cluster-ip-range 这一行，在这一行的下一行增加 如下内容

- --service-node-port-range=1-65535

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.199.137:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --feature-gates=RemoveSelfLink=false
    - --advertise-address=192.168.199.137
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --service-node-port-range=1-65535
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.13
    imagePullPolicy: IfNotPresent
 livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.199.137
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.199.137
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.199.137
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

或者使用下面的ingress，也可以一起使用---------------------------------------------------------------------------------------------------

3.5 配置Ingress

vim ingress-nacos.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
   name: nacos-web
   namespace: default

spec:
   rules:
     - host: nacos-web.nacos-demo.com
       http:
         paths:
           - path: /
             backend:
               serviceName: nacos-service
               servicePort: web

kubectl apply   -f  ingress-nacos.yaml

结果图

最后，想通过域名 nacos-web.nacos-demo.com 可以访问，需要在dns服务器解析域名或者本地wind的hosts添加本地解析

访问

nacos-web.nacos-demo.com/nacos

或者

192.168.1.16:8848/nacos/

参考：





K8S安装部署Nacos集群_软件设计艺术的博客-CSDN博客_k8s nacos