像 Rabbit MQ 这种消息中间件，一旦在项目中用到了，就说明项目的体量已经到了一定的级别，单机节点就显得捉襟见肘了。所以，一般都会搭建一个集群。下面就试着搭建三台机器的集群。
节点信息
节点一：192.168.0.116 centos1
节点二：192.168.0.117 centos2
节点三：192.168.0.118 centos3
配置 hosts 文件
配置各个节点的 hosts 文件，让各个节点都能互相识别对方。
# /etc/hosts
192.168.0.116 centos1
192.168.0.117 centos2
192.168.0.118 centos3复制 cookie 文件
编辑 Rabbit MQ 的 cookie 文件，以确保各个节点的 cookie 文件是同一个。这里使用 centos1 节点的 cookie 文件，将 centos1 的 cookie 文件复制到 centos2 和 centos3 的 /var/lib/rabbit/.erlang.cookie 或者 $HOME/.erlang.cookie下。
cookie 文件位置
cookie 文件默认路径为 /var/lib/rabbit/.erlang.cookie 或者 $HOME/.erlang.cookie。cookie 相当于密钥令牌，集群中的节点需要通过交换密钥令牌获得相互认证。
- 如果使用解压缩安装方式(二进制安装或者编译安装)，那么该文件存在位置为$HOME目录下。即
$HOME/.erlang.cookie。如果我们使用root安装，则位置为：/root/.erlang.cookie，其他用户为/home/用户名/.erlang.cookie。 - 如果使用rpm包方式安装，那么这个文件会存在于/var/lib/rabbitmq目录下。
查找 .erlang.cookie 文件
可以通过查看 Rabbit MQ log 日志的信息，如下面所示：所以我的 .erlang.cookie 文件在 /root/.erlang.cookie
[info] <0.270.0>
node : rabbit@centos1
home dir : /root
（我是 root 用户启动的）
config file(s) : (none)
cookie hash : tCXB8mlCcGEGGV1cYRkQCg==
log(s) : /usr/local/rabbitmq_server/var/log/rabbitmq/[email protected]
: /usr/local/rabbitmq_server/var/log/rabbitmq/rabbit@centos1_upgrade.log
database dir : /usr/local/rabbitmq_server/var/lib/rabbitmq/mnesia/rabbit@centos1配置集群
配置集群有三种方式，这里使用 rabbitmqctl 工具配置：
- 通过 rabbitmqctl 工具配置；
- 通过 rabbitmq.config 配置文件配置；
- 通过 rabbitmq-autocluster 插件配置；
启动三个节点的 Rabbit MQ 服务：
[root@centos1 ~]# rabbitmq-server -detached
[root@centos2 ~]# rabbitmq-server -detached
[root@centos3 ~]# rabbitmq-server -detached启动之后，目前三个节点还都是独立的，可以通过 rabbitmqctl cluster_status 命令查看状态。
加入集群
以 centos1 节点为基准，将 centos2 和 centos3 节点加入 centos1，以 centos2 为例步骤如下：
[root@centos2 ~]# rabbitmqctl stop_app
Stopping rabbit application on node rabbit@centos2 ...
[root@centos2 ~]# rabbitmqctl reset
Resetting node rabbit@centos2 ...
[root@centos2 ~]# rabbitmqctl join_cluster rabbit@centos1
Clustering node rabbit@centos2 with rabbit@centos1
[root@centos2 ~]# rabbitmqctl start_app
Starting node rabbit@centos2 ...在任意节点上使用命令 rabbitmqctl cluster_status
[
{nodes,[{disc,[rabbit@centos1,rabbit@centos2,rabbit@centos3]}]},
{running_nodes,[rabbit@centos1,rabbit@centos2,rabbit@centos3]},
{cluster_name,<<"rabbit@centos1">>},
{partitions,[]},
{alarms,[{rabbit@centos1,[]},{rabbit@centos2,[]},{rabbit@centos3,[]}]}
]Web 端展示
停掉某个节点
以上就是三个节点的集群的搭建，如果说停掉启动一个节点，会是什么结果？停掉 centos2 试试看。
[root@centos2 ~]# rabbitmqctl stop_apprabbitmqctl cluster_statua 查看
[
{nodes,[{disc,[rabbit@centos1,rabbit@centos2,rabbit@centos3]}]},
{running_nodes,[rabbit@centos1,rabbit@centos3]},
{cluster_name,<<"rabbit@centos1">>},
{partitions,[]},
{alarms,[{rabbit@centos1,[]},{rabbit@centos3,[]}]}
]Web 端展示
集群节点关闭和启动
如果关闭了集群中的所有节点，则需要确保在启动的时候最后关闭的那个节点是第一个启动。如果第一个启动的不是最后关闭的节点，那么这个节点会等待最后关闭的节点的启动。这个等待时间是 30 秒，如果没有等到，那么这个先启动的节点也会失败。
会有重试机制，默认重试 20 次，每次 30 秒以等待最后关闭节点的启动。当前版本：Rabbit MQ：RabbitMQ 3.8.9 on Erlang 23.0
在重试失败后，当前节点也会因为失败而关闭自身的应用。
重试日志：
2021-11-27 17:12:21.783 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-11-27 17:12:22.206 [debug] <0.2664.0> Lager installed handler lager_backend_throttle into lager_event
2021-11-27 17:12:51.784 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:12:51.784 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 8 retries left
2021-11-27 17:13:21.785 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:13:21.785 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 7 retries left
2021-11-27 17:13:51.786 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:13:51.786 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 6 retries left
2021-11-27 17:14:21.787 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:14:21.787 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 5 retries left
2021-11-27 17:14:51.788 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:14:51.788 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 4 retries left
2021-11-27 17:15:21.789 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:15:21.789 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 3 retries left
2021-11-27 17:15:51.790 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:15:51.790 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 2 retries left
2021-11-27 17:16:21.791 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:16:21.791 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 1 retries left
2021-11-27 17:16:51.792 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:16:51.792 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-11-27 17:17:21.793 [error] <0.2656.0> Feature flag `quorum_queue`: migration function crashed: {error,{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}}
[{rabbit_table,wait,3,[{file,"src/rabbit_table.erl"},{line,120}]},{rabbit_core_ff,quorum_queue_migration,3,[{file,"src/rabbit_core_ff.erl"},{line,60}]},{rabbit_feature_flags,run_migration_fun,3,[{file,"src/rabbit_feature_flags.erl"},{line,1602}]},{rabbit_feature_flags,'-verify_which_feature_flags_are_actually_enabled/0-fun-2-',3,[{file,"src/rabbit_feature_flags.erl"},{line,2269}]},{maps,fold_1,3,[{file,"maps.erl"},{line,233}]},{rabbit_feature_flags,verify_which_feature_flags_are_actually_enabled,0,[{file,"src/rabbit_feature_flags.erl"},{line,2267}]},{rabbit_feature_flags,sync_feature_flags_with_cluster,3,[{file,"src/rabbit_feature_flags.erl"},{line,2082}]},{rabbit_mnesia,ensure_feature_flags_are_in_sync,2,[{file,"src/rabbit_mnesia.erl"},{line,647}]}]
2021-11-27 17:17:21.793 [warning] <0.2656.0> Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/usr/local/rabbitmq_server/var/lib/rabbitmq/mnesia/rabbit@centos2-feature_flags`:
2021-11-27 17:17:21.793 [warning] <0.2656.0> Feature flags: - list of previously disabled feature flags now marked as such: [empty_basic_get_metric]
2021-11-27 17:17:21.800 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-11-27 17:17:51.801 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:17:51.801 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 8 retries left
2021-11-27 17:18:21.802 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:18:21.802 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 7 retries left
2021-11-27 17:18:51.803 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:18:51.803 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 6 retries left
2021-11-27 17:19:21.804 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:19:21.804 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 5 retries left
2021-11-27 17:19:51.805 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:19:51.805 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 4 retries left
2021-11-27 17:20:21.806 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:20:21.806 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 3 retries left
2021-11-27 17:20:51.807 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:20:51.807 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 2 retries left
2021-11-27 17:21:21.808 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:21:21.808 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 1 retries left
2021-11-27 17:21:51.809 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:21:51.809 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-11-27 17:22:21.813 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-11-27 17:22:21.813 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-11-27 17:22:21.817 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> BOOT FAILED
2021-11-27 17:22:21.818 [error] <0.2656.0> ===========
2021-11-27 17:22:21.818 [error] <0.2656.0> Timeout contacting cluster nodes: [rabbit@centos3,rabbit@centos1].
2021-11-27 17:22:21.818 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> BACKGROUND
2021-11-27 17:22:21.818 [error] <0.2656.0> ==========
2021-11-27 17:22:21.818 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> This cluster node was shut down while other nodes were still running.
2021-11-27 17:22:21.818 [error] <0.2656.0> To avoid losing data, you should start the other nodes first, then
2021-11-27 17:22:21.818 [error] <0.2656.0> start this one. To force this node to start, first invoke
2021-11-27 17:22:21.818 [error] <0.2656.0> "rabbitmqctl force_boot". If you do so, any changes made on other
2021-11-27 17:22:21.818 [error] <0.2656.0> cluster nodes after this one was shut down may be lost.
2021-11-27 17:22:21.818 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> DIAGNOSTICS
2021-11-27 17:22:21.818 [error] <0.2656.0> ===========
2021-11-27 17:22:21.819 [error] <0.2656.0>
2021-11-27 17:22:21.819 [error] <0.2656.0> attempted to contact: [rabbit@centos3,rabbit@centos1]
2021-11-27 17:22:21.819 [error] <0.2656.0>
2021-11-27 17:22:21.819 [error] <0.2656.0> rabbit@centos3:
2021-11-27 17:22:21.819 [error] <0.2656.0> * connected to epmd (port 4369) on centos3
2021-11-27 17:22:21.819 [error] <0.2656.0> * node rabbit@centos3 up, 'rabbit' application not running
2021-11-27 17:22:21.819 [error] <0.2656.0> * running applications on rabbit@centos3: [lager,observer_cli,
2021-11-27 17:22:21.819 [error] <0.2656.0> stdout_formatter,
2021-11-27 17:22:21.819 [error] <0.2656.0> gen_batch_server,aten,cuttlefish,
2021-11-27 17:22:21.819 [error] <0.2656.0> inets,credentials_obfuscation,
2021-11-27 17:22:21.820 [error] <0.2656.0> recon,ranch,jsx,goldrush,xmerl,
2021-11-27 17:22:21.820 [error] <0.2656.0> tools,syntax_tools,ssl,
2021-11-27 17:22:21.820 [error] <0.2656.0> public_key,asn1,crypto,compiler,
2021-11-27 17:22:21.820 [error] <0.2656.0> sasl,stdlib,kernel]
2021-11-27 17:22:21.820 [error] <0.2656.0> * suggestion: use rabbitmqctl start_app on rabbit@centos3
2021-11-27 17:22:21.820 [error] <0.2656.0> rabbit@centos1:
2021-11-27 17:22:21.820 [error] <0.2656.0> * connected to epmd (port 4369) on centos1
2021-11-27 17:22:21.923 [error] <0.2656.0> * node rabbit@centos1 up, 'rabbit' application not running
2021-11-27 17:22:21.924 [error] <0.2656.0> * running applications on rabbit@centos1: [lager,observer_cli,
2021-11-27 17:22:21.924 [error] <0.2656.0> stdout_formatter,
2021-11-27 17:22:21.924 [error] <0.2656.0> gen_batch_server,aten,cuttlefish,
2021-11-27 17:22:21.924 [error] <0.2656.0> inets,credentials_obfuscation,
2021-11-27 17:22:21.924 [error] <0.2656.0> recon,ranch,jsx,goldrush,xmerl,
2021-11-27 17:22:21.924 [error] <0.2656.0> tools,syntax_tools,ssl,
2021-11-27 17:22:21.924 [error] <0.2656.0> public_key,asn1,crypto,compiler,
2021-11-27 17:22:21.924 [error] <0.2656.0> sasl,stdlib,kernel]
2021-11-27 17:22:21.924 [error] <0.2656.0> * suggestion: use rabbitmqctl start_app on rabbit@centos1
2021-11-27 17:22:21.924 [error] <0.2656.0>
2021-11-27 17:22:21.924 [error] <0.2656.0> Current node details:
2021-11-27 17:22:21.924 [error] <0.2656.0> * node name: rabbit@centos2
2021-11-27 17:22:21.924 [error] <0.2656.0> * effective user's home directory: /root
2021-11-27 17:22:21.924 [error] <0.2656.0> * Erlang cookie hash: tCXB8mlCcGEGGV1cYRkQCg==
2021-11-27 17:22:21.924 [error] <0.2656.0>
2021-11-27 17:22:21.924 [error] <0.2656.0>
2021-11-27 17:22:22.926 [info] <0.2655.0> [{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.2655.0>},{registered_name,[]},{error_info,{exit,{{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.2654.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.2656.0>,normal}]},{links,[<0.2654.0>,<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,1598},{stack_size,28},{reductions,368}], []
2021-11-27 17:22:22.926 [error] <0.2655.0> CRASH REPORT Process <0.2655.0> with 0 neighbours exited with reason: {{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}} in application_master:init/4 line 138
2021-11-27 17:22:22.927 [info] <0.44.0> Application rabbit exited with reason: {{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}}
2021-11-27 17:22:22.927 [info] <0.44.0> Application rabbit exited with reason: {{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}}
2021-11-27 17:22:22.928 [info] <0.44.0> Application sysmon_handler exited with reason: stopped
2021-11-27 17:22:22.928 [info] <0.44.0> Application sysmon_handler exited with reason: stopped
2021-11-27 17:22:22.932 [info] <0.44.0> Application ra exited with reason: stopped
2021-11-27 17:22:22.932 [info] <0.44.0> Application ra exited with reason: stopped
2021-11-27 17:22:22.933 [info] <0.44.0> Application os_mon exited with reason: stopped
2021-11-27 17:22:22.933 [info] <0.44.0> Application os_mon exited with reason: stopped失败信息返回
[root@centos2 ~]# rabbitmqctl start_app
Starting node rabbit@centos2 ...
Error:
{:rabbit, {{:timeout_waiting_for_tables, [:rabbit@centos3, :rabbit@centos2, :rabbit@centos1], [:rabbit_user, :rabbit_user_permission, :rabbit_topic_permission, :rabbit_vhost, :rabbit_durable_route, :rabbit_durable_exchange, :rabbit_runtime_parameters, :rabbit_durable_queue]}, {:rabbit, :start, [:normal, []]}}}剔除节点
如果最后一个关闭的节点最终由于某些异常而无法启动，则可以通过 rabbitmqctl forget_cluster_node 命令将节点剔除集群。
如果集群中所有节点由于非正常因素关闭，那么集群中的节点都会认为自己不是最后关闭的，此时需要调用 rabbitmqctl force_boot 命令来启动一个节点，之后集群才能正常启动。
[root@centos2 ~]# rabbitmqctl force boot
Forcing boot for Mnesia dir /usr/local/rabbitmq_server/var/lib/rabbitmq/mnesia/rabbit@centos2
[root@centos2 ~]# rabbitmq server detached总结
以上就是多机多节点配置的流程。特别需要注意的是，牢记集群中节点的启动顺序，一定要先启动最后关闭的节点，避免启动失败。