redis.jpeg
Kubernetes部署Redis高可用集群
Mongo介绍
Redis基础介绍请参考本博文Redis(一)之入门详解
Redis高可用集群方案介绍
笔者采用如下高可用方案：
redis-cluster.png
创建Cofigmap
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-conf
data:
redis.conf: |
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
create_cluster.sh: |
#!/bin/bash
PORT=6379
NODE_NUMBERS=6
REPLICAS=1
success=false
check_nodes_ready() {
while [ $success==false ]
do
sleep 1
success_num=0
maxNumber=$((NODE_NUMBERS-1))
for i in `seq 0 $maxNumber`
do
nodeAddress=redis-app-${i}.redis-service.localpv-redis.svc.cluster.local
check_results=`redis-cli -h $nodeAddress -p $PORT PING`
echo $i $check_results
if [ "$check_results" == "PONG" ]; then
success_num=$((success_num+1))
else
echo "redis node $nodeAddress hasn't ready "
fi
done
if [ $success_num -eq $NODE_NUMBERS ]
then
echo $success_num " nodes ,redis cluster has been ready"
break
else
unready=$((NODE_NUMBERS-success_num))
echo $unready " nodes not ready"
fi
done
}
check_nodes_ready
maxNumber=$((NODE_NUMBERS-1))
clusterNodeAddress=""
for i in `seq 0 $maxNumber`
do
nodeAddress=`dig +short redis-app-${i}.redis-service.localpv-redis.svc.cluster.local`:${PORT}
clusterNodeAddress="$nodeAddress $clusterNodeAddress"
done
echo $clusterNodeAddress
create_cluster=`redis-trib create --replicas ${REPLICAS} $clusterNodeAddress`
echo $create_cluster
创建Headless Service
apiVersion: v1
kind: Service
metadata:
name: redis-service
labels:
app: redis-app
spec:
ports:
- name: redis-port
port: 6379
clusterIP: None
selector:
app: redis-app
appCluster: redis-cluster
创建对外访问Service
apiVersion: v1
kind: Service
metadata:
name: redis-access-service
labels:
app: redis-app
spec:
ports:
- name: redis-port
protocol: "TCP"
port: 6379
targetPort: 6379
selector:
app: redis-app
appCluster: redis-cluster
创建集群节点
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis-app
spec:
serviceName: "redis-service"
replicas: 6
template:
metadata:
labels:
app: redis-app
appCluster: redis-cluster
spec:
terminationGracePeriodSeconds: 10
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis
topologyKey: kubernetes.io/hostname
containers:
- name: redis
image: "10.19.85.58:5000/service-brokers/redis:5.0.5"
command:
- "redis-server"
args:
- "/etc/redis/redis.conf"
- "--protected-mode"
- "no"
resources:
requests:
cpu: "100m"
memory: "100Mi"
ports:
- name: redis
containerPort: 6379
protocol: "TCP"
- name: cluster
containerPort: 16379
protocol: "TCP"
volumeMounts:
- name: "redis-conf"
mountPath: "/etc/redis"
- name: redis-volume
mountPath: "/var/lib/redis"
volumes:
- name: "redis-conf"
configMap:
name: "redis-conf"
items:
- key: "redis.conf"
path: "redis.conf"
- name: redis-volume
hostPath:
path: /ocdf_local_pv_vg0/redis
podAntiAffinity：Pod反亲和性，为了集群高可用每个redis服务尽可能的分散到不同记得节点上。具体Pod亲和性与反亲和性请参考Kubernetes(八)亲和性与返亲和性文章介绍。
节点存储方式采用hostpath模式。
创建redis-trib 工具服务
apiVersion: v1
kind: Pod
metadata:
name: redis-manager
labels:
app: redis-manager
spec:
terminationGracePeriodSeconds: 10
restartPolicy: OnFailure
containers:
- name: redis-trib
image: "redis-trib:1.0"
command:
- sh
- /configmap/create_cluster.sh
args:
- "-f"
- "/dev/null"
ports:
- name: redis
containerPort: 6379
protocol: "TCP"
- name: cluster
containerPort: 16379
protocol: "TCP"
volumeMounts:
- mountPath: /configmap
name: config
volumes:
- configMap:
defaultMode: 420
name: redis-conf
name: config
运用redis-trib集群管理工具，由于原镜像中redis-trib命令不支持域名解析，需要用到"dig +short 域名"将域名转换成IP。重新构建Docker镜像，Dockerfile如下所示：
From 314315960/redis-trib:latest
RUN apk add bind-tools; \
apk add redis;
ENTRYPOINT ["/usr/bin/redis-trib"]
Pod 服务集群创建
redis-trib 工具运用（create）
执行redis-trib命令创建redis集群
replicas：表示master设置从节点的个数
例子
redis-trib create --replicas 1 \
> `dig +short redis-app-0.redis-service.localpv-redis.svc.cluster.local`:6379 \
> `dig +short redis-app-1.redis-service.localpv-redis.svc.cluster.local`:6379 \
> `dig +short redis-app-2.redis-service.localpv-redis.svc.cluster.local`:6379 \
> `dig +short redis-app-3.redis-service.localpv-redis.svc.cluster.local`:6379 \
> `dig +short redis-app-4.redis-service.localpv-redis.svc.cluster.local`:6379 \
> `dig +short redis-app-5.redis-service.localpv-redis.svc.cluster.local`:6379
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
171.248.2.107:6379
171.248.3.228:6379
171.248.2.108:6379
Adding replica 171.248.3.230:6379 to 171.248.2.107:6379
Adding replica 171.248.3.231:6379 to 171.248.3.228:6379
Adding replica 171.248.3.229:6379 to 171.248.2.108:6379
M: 2ae472bd1708535edcd8aaaf3bf67efb14880174 171.248.2.107:6379
slots:0-5460 (5461 slots) master
M: 0a0d61759e5c89314b271152380cefcdda1dc9ac 171.248.3.228:6379
slots:5461-10922 (5462 slots) master
M: 9ec3557a72bbc89c3dfb899e84436399eeaac66c 171.248.2.108:6379
slots:10923-16383 (5461 slots) master
S: cc406ebc14ec2d7b240d2d02402cd808dde1479f 171.248.3.229:6379
replicates 9ec3557a72bbc89c3dfb899e84436399eeaac66c
S: 8a66da33a8a410dff879170c58573964dbeec3b7 171.248.3.230:6379
replicates 2ae472bd1708535edcd8aaaf3bf67efb14880174
S: 7583010551a9b6cd441ef8828832ce4423b8c5e1 171.248.3.231:6379
replicates 0a0d61759e5c89314b271152380cefcdda1dc9ac
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 171.248.2.107:6379)
M: 2ae472bd1708535edcd8aaaf3bf67efb14880174 171.248.2.107:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: cc406ebc14ec2d7b240d2d02402cd808dde1479f 171.248.3.229:6379
slots: (0 slots) slave
replicates 9ec3557a72bbc89c3dfb899e84436399eeaac66c
M: 0a0d61759e5c89314b271152380cefcdda1dc9ac 171.248.3.228:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 8a66da33a8a410dff879170c58573964dbeec3b7 171.248.3.230:6379
slots: (0 slots) slave
replicates 2ae472bd1708535edcd8aaaf3bf67efb14880174
M: 9ec3557a72bbc89c3dfb899e84436399eeaac66c 171.248.2.108:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 7583010551a9b6cd441ef8828832ce4423b8c5e1 171.248.3.231:6379
slots: (0 slots) slave
replicates 0a0d61759e5c89314b271152380cefcdda1dc9ac
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
此命令会根据设置的副本数和添加的节点IP来创建出一一对应的Master节点和Slave节点。此例子是单独在服务中进行手动创建演示。本博文方案为纯自动，交给Job来做，具体逻辑可以看configmap中的create_cluster.sh
2. 或者直接在集群中运用如下命令：
kubectl exec -it redis-manager-0 -c redis-trib -n localpv-redis -- redis-trib create --replicas 1 \
`dig +short redis-app-0.redis-service.localpv-redis.svc.cluster.local`:6379 \
`dig +short redis-app-1.redis-service.localpv-redis.svc.cluster.local`:6379 \
`dig +short redis-app-2.redis-service.localpv-redis.svc.cluster.local`:6379 \
`dig +short redis-app-3.redis-service.localpv-redis.svc.cluster.local`:6379 \
`dig +short redis-app-4.redis-service.localpv-redis.svc.cluster.local`:6379 \
`dig +short redis-app-5.redis-service.localpv-redis.svc.cluster.local`:6379
redis-trib 工具运用（info）
info 用来查看当前及集群的主从信息。
# redis-trib info `dig +short redis-app-0.redis-service.localpv-redis.svc.clu
ster.local`:6379
171.248.2.107:6379 (2ae472bd...) -> 0 keys | 5461 slots | 1 slaves.
171.248.3.228:6379 (0a0d6175...) -> 0 keys | 5462 slots | 1 slaves.
171.248.2.108:6379 (9ec3557a...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
redis-trib 工具运用（check）
check命令用来查看集群的详细信息。
# redis-trib check `dig +short redis-app-3.redis-service.localpv-redis.svc.cl
uster.local`:6379
>>> Performing Cluster Check (using node 171.248.3.229:6379)
S: cc406ebc14ec2d7b240d2d02402cd808dde1479f 171.248.3.229:6379
slots: (0 slots) slave
replicates 9ec3557a72bbc89c3dfb899e84436399eeaac66c
S: 8a66da33a8a410dff879170c58573964dbeec3b7 171.248.3.230:6379
slots: (0 slots) slave
replicates 2ae472bd1708535edcd8aaaf3bf67efb14880174
M: 2ae472bd1708535edcd8aaaf3bf67efb14880174 171.248.2.107:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 7583010551a9b6cd441ef8828832ce4423b8c5e1 171.248.3.231:6379
slots: (0 slots) slave
replicates 0a0d61759e5c89314b271152380cefcdda1dc9ac
M: 0a0d61759e5c89314b271152380cefcdda1dc9ac 171.248.3.228:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 9ec3557a72bbc89c3dfb899e84436399eeaac66c 171.248.2.108:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
连接节点验证
kubectl exec -it redis-app-2 /bin/bash -n localpv-redis
root@redis-app-2:/data# /usr/local/bin/redis-cli -c
127.0.0.1:6379> cluster nodes
2ae472bd1708535edcd8aaaf3bf67efb14880174 171.248.2.107:6379@16379 master - 0 1588071155633 1 connected 0-5460
9ec3557a72bbc89c3dfb899e84436399eeaac66c 171.248.2.108:6379@16379 myself,master - 0 1588071151000 3 connected 10923-16383
8a66da33a8a410dff879170c58573964dbeec3b7 171.248.3.230:6379@16379 slave 2ae472bd1708535edcd8aaaf3bf67efb14880174 0 1588071156135 5 connected
cc406ebc14ec2d7b240d2d02402cd808dde1479f 171.248.3.229:6379@16379 slave 9ec3557a72bbc89c3dfb899e84436399eeaac66c 0 1588071156636 4 connected
0a0d61759e5c89314b271152380cefcdda1dc9ac 171.248.3.228:6379@16379 master - 0 1588071157639 2 connected 5461-10922
7583010551a9b6cd441ef8828832ce4423b8c5e1 171.248.3.231:6379@16379 slave 0a0d61759e5c89314b271152380cefcdda1dc9ac 0 1588071157138 6 connected