记一次K8S部署redis过程（从集群到单机）

前言

任务：搭建测试环境的redis服务。网上翻阅各类资料后，决定搭建一个3主3从的集群环境。后因代码使用redis集群需要更改，又重新部署单机redis。

redis介绍：数据完全保存在内存中，速度非常快，可使用磁盘进行持久化。

redis相关词汇：缓存穿透、缓存击穿和缓存雪崩。

部署步骤梳理

1. 制作redis镜像（需要使用redis-trib.rb集群命令）；
2. redis配置文件和集群修复脚本准备
3. StatefulSet配置文件准备（数据持久使用动态供给）
4. 初始化redis集群
5. 验证redis集群
6. 部署单机redis

制作redis镜像

1. redis官方下载二进制安装包Download | Redis；
2. 编译安装后得到文件redis-trib.rb；
3. docker启动一个redis服务：docker run --name redis --restart=always -d redis:4.0.11
4. 将redis-trib.rb复制到镜像：docker cp redis-trib.rb redis:/usr/local/bin
5. 进入容器后做以下操作（安装redis-trib.rb命令执行环境ruby）：
   1. echo 'deb http://mirrors.aliyun.com/debian/ buster main contrib non-free' > /etc/apt/sources.list
   2. 执行：apt-get update（此处遇到报错（NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY DCC9EFBF77E11517）
   3. apt-get install software-properties-common dirmngr -y --allow-unauthenticated
   4. apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138 0E98404D386FA1D9 DCC9EFBF77E11517 （keys就是上面报错的NO_PUBKEY）
   5. 再次执行apt-get update
   6. 安装ruby并使用gem安装redis
   7. apt-get install -y ruby
   8. apt-get install -y rubygems
   9. apt-get clean all
   10. gem install redis
   11. apt-get install dnsutils -y
6. 使用docker commit生成新镜像：docker commit -a "author" -m "redis-cluster-image" 365d2c234ab9 redis-cluster:1.0

redis配置文件和集群修复脚本准备

vim redis-configmap.yaml

 kubectl apply -f redis-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: redis-cluster

  namespace: testing

data:

  fix-ip.sh: |

    #!/bin/sh

    CLUSTER_CONFIG="/data/nodes.conf"

    if [ -f ${CLUSTER_CONFIG} ]; then

      if [ -z "${POD_IP}" ]; then

        echo "Unable to determine Pod IP address!"

        exit 1

      fi

      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"

      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}

    fi

    exec "$@"

  redis.conf: |

    cluster-enabled yes

    cluster-config-file /data/nodes.conf

    cluster-node-timeout 10000

    protected-mode no

    daemonize no

    pidfile /var/run/redis.pid

    port 6379

    tcp-backlog 511

    bind 0.0.0.0

    timeout 3600

    tcp-keepalive 1

    loglevel verbose

    logfile /data/redis.log

    databases 16

    save 900 1

    save 300 10

    save 60 10000

    stop-writes-on-bgsave-error yes

    rdbcompression yes

    rdbchecksum yes

    dbfilename dump.rdb

    dir /data

    #requirepass yl123456

    appendonly yes

    appendfilename "appendonly.aof"

    appendfsync everysec

    no-appendfsync-on-rewrite no

    auto-aof-rewrite-percentage 100

    auto-aof-rewrite-min-size 64mb

    lua-time-limit 20000

    slowlog-log-slower-than 10000

    slowlog-max-len 128

    #rename-command FLUSHALL  ""

    latency-monitor-threshold 0

    notify-keyspace-events ""

    hash-max-ziplist-entries 512

    hash-max-ziplist-value 64

    list-max-ziplist-entries 512

    list-max-ziplist-value 64

    set-max-intset-entries 512

    zset-max-ziplist-entries 128

    zset-max-ziplist-value 64

    hll-sparse-max-bytes 3000

    activerehashing yes

    client-output-buffer-limit normal 0 0 0

    client-output-buffer-limit slave 256mb 64mb 60

    client-output-buffer-limit pubsub 32mb 8mb 60

    hz 10

    aof-rewrite-incremental-fsync yes

StatefulSet配置文件准备（数据持久使用动态供给）

vim redis-cluster.yaml

kubectl apply -f redis-cluster.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: wiseco
  name: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---

apiVersion: v1
kind: Service
metadata:
  namespace: testing
  name: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: testing
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis-cluster-4.0.11:1.0
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "course-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 50Gi

初始化redis集群

查看redis的各个节点都running后之下面命令进行集群初始化：

kubectl exec -it redis-cluster-0 -n testing -- redis-trib.rb create --replicas 1 $(kubectl get pods -l app=redis-cluster -n testing -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')

执行过程根据提示需要输入一次yes

验证redis集群

 for x in $(seq 0 5); do echo "redis-cluster-$x"; kubectl exec redis-cluster-$x -n testing -- redis-cli role; echo; done

部署单机redis

本想继续使用之前的redis镜像，发现怎么也无法启动，初步判断是缺少夯住进程的命令，使用Dockerfile重新编了一个redis镜像，增加了一个启动脚本

#!/bin/bash
# Redis启动命令
redis-server /etc/redis/redis.conf
# 使用tail -f 在pod内部构建守护进程
tail -f  /etc/hosts

创建redis的configmap

apiVersion: v1
data:
  redis.conf: |
    protected-mode no
    bind 0.0.0.0
    port 6379
    tcp-backlog 511
    timeout 0
    tcp-keepalive 300
    daemonize yes
    supervised no
    pidfile /var/run/redis.pid
    loglevel verbose
    logfile /data/redis.log
    databases 16
    save 900 1
    save 300 10
    save 60 1000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    slave-serve-stale-data yes
    slave-read-only yes
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    slave-priority 100
    requirepass hxcloud
    appendonly no
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    lua-time-limit 5000
    slowlog-log-slower-than 10000
    slowlog-max-len 1000
    latency-monitor-threshold 0
    notify-keyspace-events Ex
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes
kind: ConfigMap
metadata:
  name: redis-config
  namespace: testing

创建redis的deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: redis
  name: redis
  namespace: testing
spec:
  replicas: 1
  selector: #标签选择器，与上面的标签共同作用
    matchLabels: #选择包含标签app:redis的资源
       app: redis
  template: 
    metadata: 
      labels: #Pod的标签，上面的selector即选择包含标签
        app: redis
    spec: #期望Pod实现的功能（即在pod中部署）
      containers: #生成container，与docker中的container是同一种
      - name: redis
        image: redis-testing-4.0.11:1.2
        env:
        - name: TZ
          value: GMT
        ports:
        - containerPort: 6379
        volumeMounts:
        - mountPath: /data  
          name: data
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
      imagePullSecrets:
      - name: testing-harbor
      volumes:
        - name: conf
          configMap:
            name: redis-config
            defaultMode: 0755
        - name: data
          persistentVolumeClaim:
            claimName: redis-data

手动创建PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
  namespace: testing
spec:
  volumeName: redis-data
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-data
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: /data/nfs/testing-redis-data        #redis数据持久化的路径
    server: xxx.xxx.xxx.xxx        #你的nfs服务器

参考文档：https://www.cnblogs.com/cheyunhua/p/15619317.html