K8S搭建redis集群

此为Sunny 王苗苗同学的学习笔记，持续学习，持续分享，持续进步，向着大神之路前进~

之前看过挺多教程的，但是没能找到成功搭建，总有这样那样的问题，最终找到一个然后自己动手做一部分，搭建成功，开心~

1、拿了他人项目，使用的statefulSet，redis的pod运行起来了，但是查看集群的时候发现是不成功的
2、再此基础上，自己做一个tools镜像，然后初始化集群，查看集群状态为ok
3、测试主从切换，发现也ok~

1、
创建服务redis-headless-service.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster
  labels:
    app: redis
spec:
  #type: NodePort
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster

创建configmap和statefulSet命名为redis-sts.yaml
这里使用了storageClass，创建方式参考：K8S 自动按需自动创建nfs存储卷(pv)

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
data:
  update-node.sh: |
    #!/bin/sh
    REDIS_NODES="/data/nodes.conf"
    sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES} 
    exec "$@"   #把含有myself这一行的ip换成新的pod ip
  redis.conf: |+
    cluster-enabled yes  # 开启集群模式
    cluster-require-full-coverage no #no代表可以在slot没有全部分配的时候提供服务
    cluster-node-timeout 15000  #节点互连超时时间，单位为毫秒
    cluster-config-file /data/nodes.conf  #设置当前节点集群配置文件路径
    cluster-migration-barrier 1
    appendonly yes # 开启aof日志，即持久化
    protected-mode no  #关闭保护模式
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis
      appCluster: redis-cluster
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: cluster
        command: ["/conf/update-node.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: POD_IP  
          valueFrom:
            fieldRef:
              fieldPath: status.podIP  #获取pod的id，传给POD_IP
        - name: METADATA_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name #获取pod名
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: nfs-storage 

2、制作tools镜像，因为对dockfile语法还不是太熟悉，我是先运行一个centos镜像，然后下载相关的包，再打包成images的

#把redis5.0.3的包复制进容器
kubectl cp redis-5.0.3.tar.gz centos-6dbdb4689c-v59qc:/usr/local/
#进入容器
kubectl exec -it centos-6dbdb4689c-v59qc /bin/bash
#下载相关的包
yum install -y ruby ruby-devel rubygems rpm-build gcc make bind-utils

cd /usr/local/
#解压安装包
tar -xzvf redis-5.0.3.tar.gz
#编译安装
make && make install
#初始化集群dig为将域名解析为ip，+short是去掉多余的信息，只显示ip

再把上面的container打包为image
docker commit 容器id或名称 打包的镜像名称:标签

docker commit -a "sunny" -m "sunny make redis-tool" 16ab6df1c5b8 redis-cluster-tools

运行该容器

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: redis-test
  name: redis-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-test
  template:
    metadata:
      labels:
        app: redis-test
      name: pos-redis
    spec:
     containers:
     - name: pos-redis
       image: redis-cluster-tools:latest
       imagePullPolicy: Never
       args:
       - /bin/bash
       - -c
       - sleep 3600
     nodeName: node4

进入容器，初始化集群

[root@master redis]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-78cbf94495-4zwqj   1/1     Running   4          23h
redis-cluster-0                           1/1     Running   0          18h
redis-cluster-1                           1/1     Running   0          19h
redis-cluster-2                           1/1     Running   0          19h
redis-cluster-3                           1/1     Running   0          18h
redis-cluster-4                           1/1     Running   0          19h
redis-cluster-5                           1/1     Running   0          19h
redis-test-7d68b989b7-w9fsv               1/1     Running   28         28h
[root@master redis]# kubectl exec -it redis-test-7d68b989b7-w9fsv /bin/bash
[root@redis-test-7d68b989b7-w9fsv /]#/usr/local/bin/redis-cli --cluster create \
`dig +short redis-cluster-0.redis-cluster.default.svc.cluster.local`:6379 \
`dig +short redis-cluster-1.redis-cluster.default.svc.cluster.local`:6379 \
`dig +short redis-cluster-2.redis-cluster.default.svc.cluster.local`:6379 \
`dig +short redis-cluster-3.redis-cluster.default.svc.cluster.local`:6379 \
`dig +short redis-cluster-4.redis-cluster.default.svc.cluster.local`:6379 \
`dig +short redis-cluster-5.redis-cluster.default.svc.cluster.local`:6379 \
--cluster-replicas 1
#。。。结果省略，中途有个提示重新分配槽位，确认即可
#查看集群状态-h 是pod的ip，使用kubectl get pods -o wide 即可查看
[root@redis-test-7d68b989b7-w9fsv /]# /usr/local/bin/redis-cli -h 10.244.1.128 -p 6379
10.244.1.128:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:66338
cluster_stats_messages_pong_sent:67182
cluster_stats_messages_sent:133520
cluster_stats_messages_ping_received:67182
cluster_stats_messages_pong_received:66338
cluster_stats_messages_received:133520
10.244.1.128:6379> cluster nodes
0f43ee31fd242846438956b89e07ab293685a199 10.244.1.128:6379@16379 myself,master - 0 1589448764000 1 connected 0-5460
65e3317621ff37f9e302b65e44dcb59e6b048b18 10.244.3.106:6379@16379 slave 63b9268a6e3db4905a9c121d923231a2d4be184c 0 1589448767040 6 connected
383ae0b1796632c0d061fdd02ba2d2b8858f7c6b 10.244.2.100:6379@16379 slave 0f43ee31fd242846438956b89e07ab293685a199 0 1589448765033 4 connected
63b9268a6e3db4905a9c121d923231a2d4be184c 10.244.4.228:6379@16379 master - 0 1589448766037 3 connected 10923-16383
221e91c77729610aeaafd7d88b34c7ecd5e84a49 10.244.1.126:6379@16379 slave 096b0af04a4d9e7b84105bc2b3ebdf50fba175a7 0 1589448764029 5 connected
096b0af04a4d9e7b84105bc2b3ebdf50fba175a7 10.244.3.105:6379@16379 master - 0 1589448765000 2 connected 5461-10922
10.244.1.128:6379>

至此，redis集群初始化成功。

还需要见一个服务来对外提供服务redis-access-service.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: redis-access-service
  labels:
    app: redis
spec:
  type: NodePort
  ports:
  - port: 6379
    targetPort: 6379
    name: client
    protocol: "TCP"
  - port: 16379
    targetPort: 16379
    name: cluster
  #clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster

3、测试主从切换，删除一个pods，再次查看集群状态

[root@master redis-cluster-sts]# kubectl exec -it redis-cluster-0 /bin/bash
root@redis-cluster-0:/data# /usr/local/bin/redis-cli -c
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:66878
cluster_stats_messages_pong_sent:67737
cluster_stats_messages_sent:134615
cluster_stats_messages_ping_received:67737
cluster_stats_messages_pong_received:66878
cluster_stats_messages_received:134615
127.0.0.1:6379> cluster nodes
0f43ee31fd242846438956b89e07ab293685a199 10.244.1.128:6379@16379 myself,master - 0 1589449307000 1 connected 0-5460
65e3317621ff37f9e302b65e44dcb59e6b048b18 10.244.3.106:6379@16379 slave 63b9268a6e3db4905a9c121d923231a2d4be184c 0 1589449306000 6 connected
383ae0b1796632c0d061fdd02ba2d2b8858f7c6b 10.244.2.100:6379@16379 slave 0f43ee31fd242846438956b89e07ab293685a199 0 1589449306717 4 connected
63b9268a6e3db4905a9c121d923231a2d4be184c 10.244.4.228:6379@16379 master - 0 1589449308723 3 connected 10923-16383
221e91c77729610aeaafd7d88b34c7ecd5e84a49 10.244.1.126:6379@16379 slave 096b0af04a4d9e7b84105bc2b3ebdf50fba175a7 0 1589449305000 5 connected
096b0af04a4d9e7b84105bc2b3ebdf50fba175a7 10.244.3.105:6379@16379 master - 0 1589449307720 2 connected 5461-10922
#删除10.244.4.228这个pod,再查看集群，发现已经替换为新的pod ip
127.0.0.1:6379> cluster nodes
0f43ee31fd242846438956b89e07ab293685a199 10.244.1.128:6379@16379 myself,master - 0 1589449377000 1 connected 0-5460
65e3317621ff37f9e302b65e44dcb59e6b048b18 10.244.3.106:6379@16379 slave 63b9268a6e3db4905a9c121d923231a2d4be184c 0 1589449377984 6 connected
383ae0b1796632c0d061fdd02ba2d2b8858f7c6b 10.244.2.100:6379@16379 slave 0f43ee31fd242846438956b89e07ab293685a199 0 1589449378000 4 connected
63b9268a6e3db4905a9c121d923231a2d4be184c 10.244.4.230:6379@16379 master - 0 1589449378987 3 connected 10923-16383
221e91c77729610aeaafd7d88b34c7ecd5e84a49 10.244.1.126:6379@16379 slave 096b0af04a4d9e7b84105bc2b3ebdf50fba175a7 0 1589449375000 5 connected
096b0af04a4d9e7b84105bc2b3ebdf50fba175a7 10.244.3.105:6379@16379 master - 0 1589449378000 2 connected 5461-10922
#查看集群状态为正常
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:67112
cluster_stats_messages_pong_sent:67977
cluster_stats_messages_sent:135089
cluster_stats_messages_ping_received:67977
cluster_stats_messages_pong_received:67110
cluster_stats_messages_received:135087

集群搭建成功后可以把redis-test这个deploy删掉