kubernetes上部署Rabbitmq高可用记录

部署RabbitMq

RabbitMQ集群的两种模式
1）普通模式：默认的集群模式,队列消息只存在单个节点上
2）镜像模式：队列为镜像队列，队列消息存在每个节点上

镜像模式结构

官方解释：镜像模式是通过policy策略方式实现的

配置同步：
    1.Ha mode 同步模式,以下可选：
        1)all 所有的节点都将被同步
        2)exactly 指定个数的节点被同步
        3)nodes 指定的名称的节点被同步
    2.Ha params 上一项中的参数
    3.Ha async mode 同步方式，自动或手动

主要部署步骤

使用statefulset启动N个rabbitmq节点

mq-5c0f5af17b04e-0                  1/1       Running                 0          6m
mq-5c0f5af17b04e-1                  1/1       Running                 0          5m
mq-5c0f5af17b04e-2                  1/1       Running                 0          3m

每个节点执行以下初始化脚本

if [ -z "$(grep rabbitmq /etc/resolv.conf)" ]; then
    sed "s/^search \([^ ]\+\)/search ${MY_POD_NAME%-*}.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
    cat /etc/resolv.conf.new > /etc/resolv.conf;
    rm /etc/resolv.conf.new;
fi;

#检查集群状态
until rabbitmqctl node_health_check; do sleep 1; done;

# 加入集群，设置同步策略（镜像模式）
if [ -z "$(rabbitmqctl cluster_status | grep ${MY_POD_NAME%-*}-0)" ]; then
    rabbitmqctl stop_app;
    rabbitmqctl reset;
    rabbitmqctl join_cluster rabbit@${MY_POD_NAME%-*}-0;
    rabbitmqctl start_app;
    rabbitmqctl set_policy mirror-queues "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
fi;

进入容器，查看集群状态

root@mq-5c0f5af17b04e-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@mq-5c0f5af17b04e-0 ...
[{nodes,[{disc,['rabbit@mq-5c0f5af17b04e-0','rabbit@mq-5c0f5af17b04e-1',
                'rabbit@mq-5c0f5af17b04e-2']}]},
 {running_nodes,['rabbit@mq-5c0f5af17b04e-2','rabbit@mq-5c0f5af17b04e-1',
                 'rabbit@mq-5c0f5af17b04e-0']},
 {cluster_name,<<"rabbit@mq-5c0f5af17b04e-0.mq-5c0f5af17b04e.itfarm3.svc.cluster.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@mq-5c0f5af17b04e-2',[]},
          {'rabbit@mq-5c0f5af17b04e-1',[]},
          {'rabbit@mq-5c0f5af17b04e-0',[]}]}]

在web页面中查看集群状态

查看队列

测试高可用

镜像同步

生产一条消息，查看状态

//所有节点都存在消息
root@mq-5c0f5af17b04e-0:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

root@mq-5c0f5af17b04e-1:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

root@mq-5c0f5af17b04e-2:/# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

故障模拟

//删除多个节点，查看状态
$ kubectl.exe delete po mq-5c0f5af17b04e-0 mq-5c0f5af17b04e-1
pod "mq-5c0f5af17b04e-0" deleted
pod "mq-5c0f5af17b04e-1" deleted

//重启后，故障节点恢复队列消息
root@mq-5c0f5af17b04e-0:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1

root@mq-5c0f5af17b04e-1:/#  rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
mirror-admin    1