k8s部署redis-cluster+predixy集群

部署redis-cluster+predixy代理集群

资料来源大佬：https://zhuanlan.zhihu.com/p/445316238

• 准备环境

192.168.0.27   master1

192.168.0.138 node1

192.168.0.52   node2

master1 NFS

其他大大小小配置参考我部署k8s集群那篇

• 部署redis-cluster集群

1. NFS文件挂载信息

挂载服务器信息

cat /etc/exports

/nfs  192.168.0.0/24(rw,no_root_squash,sync)

创建挂载目录

mkdir -p /nfs/redis/volumes/

1. 创建NFS的rbac(基于角色的访问控制)

创建命名空间

kubectl create namespace k8s-redis

创建rbac

cat nfs-rbac.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: nfs-redis-provisioner

  namespace: k8s-redis

---

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: nfs-redis-provisioner-runner

rules:

  - apiGroups: [""]

    resources: ["persistentvolumes"]

    verbs: ["get", "list", "watch", "create", "delete"]

  - apiGroups: [""]

    resources: ["persistentvolumeclaims"]

    verbs: ["get", "list", "watch", "update"]

  - apiGroups: ["storage.k8s.io"]

    resources: ["storageclasses"]

    verbs: ["get", "list", "watch"]

  - apiGroups: [""]

    resources: ["events"]

    verbs: ["create", "update", "patch"]

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: run-nfs-redis-provisioner

subjects:

  - kind: ServiceAccount

    name: nfs-redis-provisioner

    namespace: k8s-redis

roleRef:

  kind: ClusterRole

  name: nfs-redis-provisioner-runner

  apiGroup: rbac.authorization.k8s.io

---

kind: Role

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: leader-locking-nfs-redis-provisioner

  namespace: k8s-redis

rules:

  - apiGroups: [""]

    resources: ["endpoints"]

    verbs: ["get", "list", "watch", "create", "update", "patch"]

---

kind: RoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: leader-locking-nfs-redis-provisioner

subjects:

  - kind: ServiceAccount

    name: nfs-redis-provisioner

    namespace: k8s-redis

roleRef:

  kind: Role

  name: leader-locking-nfs-redis-provisioner

  apiGroup: rbac.authorization.k8s.io

加载：

kubectl apply -f nfs-rbac.yaml

1. 创建redis-cluster集群的storageclass

cat nfs-storgeclass.yaml

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: managed-redis-nfs-storage

  namespace: k8s-redis

provisioner: redis-nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致

parameters:

#  archiveOnDelete: "false"

  archiveOnDelete: "true"

reclaimPolicy: Retain

加载

kubectl apply -f nfs-storageClass.yaml

查看

kubectl get sc -n k8s-redis

1. 创建redis-cluster集群的provisioner

cat nfs-provisioner.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: nfs-redis-provisioner

  labels:

    app: nfs-redis-provisioner

  namespace: k8s-redis  #与RBAC文件中的namespace保持一致

spec:

  replicas: 1

  selector:

    matchLabels:

      app: nfs-redis-provisioner

  strategy:

    type: Recreate

  selector:

    matchLabels:

      app: nfs-redis-provisioner

  template:

    metadata:

      labels:

        app: nfs-redis-provisioner

    spec:

      serviceAccountName: nfs-redis-provisioner

      containers:

        - name: nfs-redis-provisioner

          image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0

          volumeMounts:

            - name: nfs-redis-root

              mountPath: /persistentvolumes

          env:

            - name: PROVISIONER_NAME

              value: redis-nfs-storage

            - name: NFS_SERVER

              value: 192.168.0.27   #NFS Server IP地址

            - name: NFS_PATH

              value: "/nfs/redis/volumes"    #NFS挂载卷

      volumes:

        - name: nfs-redis-root

          nfs:

            server: 192.168.0.27  #NFS Server IP地址

            path: "/nfs/redis/volumes"     #NFS 挂载卷

      imagePullSecrets:

      - name: registry-op.test.cn

注意：原文镜像获取不到，我换了个镜像获取

加载

kubectl apply -f nfs-provisioner.yaml

查看

1. 部署redis-cluster statefulset（有状态）服务

1. 下载redis-trib.rb工具

此工具用于初始化redis集群，打包在docker镜像中。

注意点：如果没有docker仓库，那就在node中建立镜像,此次选择使用node打镜像。建议在集群安装时就把harbor添加进去。

[root@node1 ~]# mkdir dockerfile

cd dockerfile

克隆

git clone -b master https://github.com/beebol/redis-trib.rb.git

dockerfile文件

注意点：redis-trib.rb文件不可以复制粘贴到本地，一定要下载！否则会有env ruby not found等问题。再给redis-trib.rb文件加上执行权限。血淋淋的教训

cat Dockerfile

FROM redis:6.2.4

RUN apt-get update -y

RUN apt-get install -y  ruby rubygems

RUN apt-get clean all

RUN gem install redis

RUN apt-get install dnsutils -y

COPY ./redis-trib.rb/redis-trib.rb /usr/local/bin/

RUN chmod 777 /usr/local/bin/redis-trib.rb

构建镜像

docker build -t redis:local .

1. 创建redis-cluster的configmap

回到master1

cd /data/redis-cluster/

cat redis-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: redis-cluster

  namespace: k8s-redis

data:

  fix-ip.sh: |

    #!/bin/sh

    CLUSTER_CONFIG="/data/nodes.conf"

    if [ -f ${CLUSTER_CONFIG} ]; then

      if [ -z "${POD_IP}" ]; then

        echo "Unable to determine Pod IP address!"

        exit 1

      fi

      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"

      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}

    fi

    exec "$@"

  redis.conf: |

    cluster-enabled yes

    cluster-config-file /data/nodes.conf

    cluster-node-timeout 10000

    protected-mode no

    daemonize no

    pidfile /var/run/redis.pid

    port 6379

    tcp-backlog 511

    bind 0.0.0.0

    timeout 3600

    tcp-keepalive 1

    loglevel verbose

    #requirepass 30MdvvbxPxt1yxgf

    logfile /data/redis.log

    databases 16

    save 900 1

    save 300 10

    save 60 10000

    stop-writes-on-bgsave-error yes

    rdbcompression yes

    rdbchecksum yes

    dbfilename dump.rdb

    dir /data

    appendonly yes

    appendfilename "appendonly.aof"

    appendfsync everysec

    no-appendfsync-on-rewrite no

    auto-aof-rewrite-percentage 100

    auto-aof-rewrite-min-size 64mb

    lua-time-limit 20000

    slowlog-log-slower-than 10000

    slowlog-max-len 128

    latency-monitor-threshold 0

    notify-keyspace-events ""

    hash-max-ziplist-entries 512

    hash-max-ziplist-value 64

    list-max-ziplist-entries 512

    list-max-ziplist-value 64

    set-max-intset-entries 512

    zset-max-ziplist-entries 128

    zset-max-ziplist-value 64

    hll-sparse-max-bytes 3000

    activerehashing yes

    client-output-buffer-limit normal 0 0 0

    client-output-buffer-limit slave 256mb 64mb 60

    client-output-buffer-limit pubsub 32mb 8mb 60

    hz 10

    aof-rewrite-incremental-fsync yes

根据需求自定义自己的redis配置。

kubectl apply -f redis-configmap.yaml

1. 创建Statefulset服务

cat redis-statefulset.yaml

apiVersion: v1

kind: Service

metadata:

  namespace: k8s-redis

  name: redis-cluster

spec:

  clusterIP: None

  ports:

  - port: 6379

    targetPort: 6379

    name: client

  - port: 16379

    targetPort: 16379

    name: gossip

  selector:

    app: redis-cluster

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  namespace: k8s-redis

  name: redis-cluster

spec:

  serviceName: redis-cluster

  podManagementPolicy: OrderedReady

  replicas: 6

  selector:

    matchLabels:

      app: redis-cluster

  template:

    metadata:

      labels:

        app: redis-cluster

    spec:

      containers:

      - name: redis

        image: redis:local

        imagePullPolicy: IfNotPresent #从node本地读取镜像

        ports:

        - containerPort: 6379

          name: client

        - containerPort: 16379

          name: gossip

        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]

        env:

        - name: POD_IP

          valueFrom:

            fieldRef:

              fieldPath: status.podIP

        volumeMounts:

        - name: conf

          mountPath: /etc/redis/

          readOnly: false

        - name: data

          mountPath: /data

          readOnly: false

      volumes:

      - name: conf

        configMap:

          name: redis-cluster

          defaultMode: 0755

#      imagePullSecrets:

#      - name: registry-op.test.cn 此处没有用到镜像凭证，所以不用加

  volumeClaimTemplates:

  - metadata:

      name: data

      annotations:

        volume.beta.kubernetes.io/storage-class: "managed-redis-nfs-storage"

    spec:

      accessModes:

        - ReadWriteMany

      resources:

        requests:

          storage: 10Gi

根据实际情况写yaml

加载

kubectl apply -f redis-statefulset.yaml

查看

kubectl get svc -n k8s-redis|grep redis-cluster

kubectl get pv -n k8s-redis|grep redis-cluster

kubectl get pvc -n k8s-redis|grep redis-cluster

查看挂载NFS文件情况

find /nfs/redis/volumes/k8s-redis-data-redis-cluster-*

1. 初始化redis-cluster集群

组成三组Redis服务master-->slave，三个master，三个slave

查看集群pod ip

kubectl get pods -n k8s-redis  -o wide|grep redis-cluster

筛选IP

kubectl get pods -l app=redis-cluster -n k8s-redis -o jsonpath='{range.items[*]}{.status.podIP}:6379 '

初始化

kubectl exec -it redis-cluster-0 -n k8s-redis -- redis-trib.rb create --replicas 1 $(kubectl get pods -l app=redis-cluster -n k8s-redis -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')

显示

1. 验证Redis-Cluster集群部署

显示ok

循环取得每台redis的角色信息

1. 部署redis-cluster集群代理predixy

同样的在node中建立镜像

mkdir predixy

cd predixy

1. 下载编译好的压缩包

wget https://github.com/joyieldInc/predixy/releases/download/1.0.5/predixy-1.0.5-bin-amd64-linux.tar.gz

tar xzf predixy-1.0.5-bin-amd64-linux.tar.gz

1. 创建dockerfile

[root@node1 predixy]# cat Dockerfile

FROM centos:7

RUN yum install -y epel-release net-tools

RUN yum install -y redis

RUN yum install -y libstdc++-static gcc gcc-c++ make

RUN mkdir /opt/predixy-1.0.5

RUN mkdir /etc/predixy

COPY ./predixy-1.0.5/bin/predixy  /usr/local/bin/ #此处是实际情况

COPY ./predixy-1.0.5/conf/*  /etc/predixy/

ENTRYPOINT ["/usr/local/bin/predixy","/etc/predixy/predixy.conf"]

建立镜像

docker build -t predixy:local .

1. 创建predixy-configmap

回到master1

[root@master1 redis-cluster]# cat predixy-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: predixy-config

  namespace: k8s-redis  #namespace按自已的修改

data:

  predixy.conf: |

    Name Predixy-DefaultNS

    Bind 0.0.0.0:7617

    WorkerThreads 4

    ClientTimeout 0

    Log /data/predixy.log

    LogRotate 1d

    LogVerbSample 0

    LogDebugSample 0

    LogInfoSample 100

    LogNoticeSample 1

    LogWarnSample 1

    LogErrorSample 1

    ClusterServerPool {

      MasterReadPriority 60

      StaticSlaveReadPriority 50

      DynamicSlaveReadPriority 60

      RefreshInterval 1

      ServerTimeout 1

      ServerFailureLimit 10

      ServerRetryTimeout 1

      KeepAlive 120

      Servers {

        + redis-cluster.k8s-redis.svc.cluster.local:6379

       }

     }

注意：原文有很多注释，可以自定义选择需求配置

1. 创建pv,pvc

cat pv-nfs.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

  name: redis-pv

  namespace: k8s-redis

  labels:

    type: nfs

spec:

  capacity:

    storage: 5Gi

  accessModes:

    - ReadWriteMany

  persistentVolumeReclaimPolicy: Recycle

  storageClassName: nfs

  nfs:

    path: "/nfs/redis/volumes"

    server: 192.168.0.27 #k8s-nfs matser

readOnly: false

加载

kubectl apply -f pv-nfs.yaml

cat pvc-nfs.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: redis-pvc

  namespace: k8s-redis

spec:

  accessModes:

  - ReadWriteMany

  resources:

     requests:

       storage: 5Gi

  storageClassName: nfs

加载

kubectl apply -f pvc-nfs.yaml

查看

kubectl get pv,pvc -n k8s-redis

1. 部署predixy-deployment（无状态）

cat predixy-deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  annotations:

    deployment.kubernetes.io/revision: '1'

  name: predixy

  namespace: k8s-redis

spec:

  progressDeadlineSeconds: 600

  replicas: 2

  revisionHistoryLimit: 5

  selector:

    matchLabels:

      app: predixy

  strategy:

    rollingUpdate:

      maxSurge: 25%

      maxUnavailable: 25%

    type: RollingUpdate

  template:

    metadata:

      labels:

        app: predixy

    spec:

      containers:

        - command:

            - predixy

            - /etc/predixy/predixy.conf

          image: predixy:local #node本地镜像

          imagePullPolicy: IfNotPresent #本地拉取

          name: predixy

          terminationMessagePath: /dev/termination-log

          terminationMessagePolicy: File

          resources:

            requests:

              cpu: 100m

              memory: 30Mi

            limits:

              cpu: 100m

              memory: 30Mi

          volumeMounts:

            - mountPath: /etc/predixy/

              name: predixy-config-dir

              readOnly: true

            - mountPath: /data/

              name: predixy-data-dir

      imagePullSecrets:

      - name: registry-op.test.cn

      dnsPolicy: ClusterFirst

      restartPolicy: Always

      schedulerName: default-scheduler

      terminationGracePeriodSeconds: 30

      volumes:

        - configMap:

            defaultMode: 420

            name: predixy-config

          name: predixy-config-dir

        - name: predixy-data-dir

          persistentVolumeClaim:

            claimName: redis-pvc

---

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

  name: predixy

  namespace: k8s-redis

spec:

  scaleTargetRef:

    apiVersion: apps/v1

    kind: Deployment

    name: predixy

  minReplicas: 1

  maxReplicas: 3

  metrics:

  - type: Resource

    resource:

       name: cpu

       target:

          type: Utilization

          averageUtilization: 50

  - type: Resource

    resource:

       name: memory

       target:

          type: AverageValue

          averageValue: 80Mi

---

apiVersion: v1

kind: Service

metadata:

  name: predixy

  namespace: k8s-redis

spec:

  externalTrafficPolicy: Cluster

  ports:

    - name: predixy-port

      nodePort: 30617

      port: 7617

      protocol: TCP

      targetPort: 7617

  selector:

    k8s.cn/name: predixy

  sessionAffinity: None

  type: NodePort

  selector:

    app: predixy

建立

kubectl apply -f predixy-deployment.yaml

查看

测试

master1安装redis

mkdir /usr/local/redis/

cd /usr/local/redis/

wget https://download.redis.io/releases/redis-6.2.4.tar.gz

tar xzf redis-6.2.4.tar.gz

cd redis-6.2.4/

yum install gcc gcc-c++ make -y

make

安装完毕

连接node

./src/redis-cli -h node1 -p 30617

输入INFO查看信息

压力测试

src/redis-benchmark -h node1 -p 30617 -c 50 -n 10000 -t get

src/redis-benchmark -h node2 -p 30617 -c 50 -n 10000 -t get

到此就完成了，但实际生产中redis必须需要密码

那就修改redis的configmap，把密码生成加入。再删除pod使其重启，这时候配置就生效了。