rabbitmq集群搭建
rabbitmq集群
-
- 环境初始化
- 配置集群
-
- 配置集群节点通信
- 配置节点加入集群
- 常用命令
- 集群搭建避坑
| 节点 | IP |
|---|---|
| rabbitmq01（磁盘节点） | 192.168.200.80 |
| rabbitmq02（内存节点） | 192.168.200.81 |
| rabbitmq03（内存节点） | 192.168.200.82 |
环境初始化
虚拟机环境centos7.9，使用脚本环境初始化，三个节点分别执行
[root@localhost ~]# cat init.sh
#!/bin/bash
# 定义节点信息
NODES=("192.168.200.80 rabbitmq01" "192.168.200.81 rabbitmq02" "192.168.200.82 rabbitmq03")
# 定义当前节点的密码（默认集群统一密码）
HOST_PASS="000000"
# 时间同步的目标节点
TIME_SERVER=rabbitmq01
# 时间同步的地址段
TIME_SERVER_IP=192.160.200.0/24
# 欢迎界面
cat > /etc/motd <################################
# Welcome to mycluster #
################################
EOF
#禁用selinux
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
#firewalld
systemctl stop firewalld
systemctl disable firewalld >> /dev/null 2>&1
#关闭IPtables，清空规则
yum install iptables-services -y
if [ 0 -ne $? ]; then
echo -e "\033[31mThe installation source configuration errors\033[0m"
exit 1
fi
systemctl restart iptables
iptables -F
iptables -X
iptables -Z
/usr/sbin/iptables-save
systemctl stop iptables
systemctl disable iptables
#禁用NetworkManager
systemctl stop NetworkManager >> /dev/null 2>&1
systemctl disable NetworkManager >> /dev/null 2>&1
yum remove -y NetworkManager firewalld
systemctl restart network
# 优化ssh连接
sed -i -e 's/#UseDNS yes/UseDNS no/g' -e 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/g' /etc/ssh/sshd_config
systemctl reload sshd
# 修改主机名
for node in "${NODES[@]}"; do
ip=$(echo "$node" | awk '{print $1}')
hostname=$(echo "$node" | awk '{print $2}')
# 获取当前节点的主机名和 IP
current_ip=$(hostname -I | awk '{print $1}')
current_hostname=$(hostname)
# 检查当前节点与要修改的节点信息是否匹配
if [[ "$current_ip" == "$ip" && "$current_hostname" != "$hostname" ]]; then
echo "Updating hostname to $hostname on $current_ip..."
hostnamectl set-hostname "$hostname"
if [ $? -eq 0 ]; then
echo "Hostname updated successfully."
else
echo "Failed to update hostname."
fi
break
fi
done
# 遍历节点信息并添加到 hosts 文件
for node in "${NODES[@]}"; do
ip=$(echo "$node" | awk '{print $1}')
hostname=$(echo "$node" | awk '{print $2}')
# 检查 hosts 文件中是否已存在相应的解析
if grep -q "$ip $hostname" /etc/hosts; then
echo "Host entry for $hostname already exists in /etc/hosts."
else
# 添加节点的解析条目到 hosts 文件
sudo sh -c "echo '$ip $hostname' >> /etc/hosts"
echo "Added host entry for $hostname in /etc/hosts."
fi
done
if [[ ! -s ~/.ssh/id_rsa.pub ]]; then
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa -q -b 2048
fi
# 检查并安装 expect 工具
if ! which expect &> /dev/null; then
echo "expect 工具未安装，正在安装 expect..."
sudo yum install -y expect
fi
# 遍历所有节点
for node in "${NODES[@]}"; do
ip=$(echo "$node" | awk '{print $1}')
hostname=$(echo "$node" | awk '{print $2}')
expect -c "
set timeout -1
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub $hostname
expect {
\"*password:*\" { send -- \"$HOST_PASS\r\"; exp_continue }
\"*(yes/no)*\" { send -- \"yes\r\"; exp_continue }
eof { exit 1 }
}
"
done
# 时间同步
if [[ $name == $TIME_SERVER ]]; then
# 配置当前节点为时间同步源
sed -i '3,6s/^/#/g' /etc/chrony.conf
sed -i "7s/^/server $TIME_SERVER iburst/g" /etc/chrony.conf
echo "allow $TIME_SERVER_IP" >> /etc/chrony.conf
echo "local stratum 10" >> /etc/chrony.conf
else
# 配置当前节点同步到目标节点
sed -i '3,6s/^/#/g' /etc/chrony.conf
sed -i "7s/^/server $TIME_SERVER iburst/g" /etc/chrony.conf
fi
# 重启并启用 chrony 服务
systemctl restart chronyd
systemctl enable chronyd
echo "###############################################################"
echo "################# 集群初始化成功 #####################"
echo "###############################################################"
配置集群
配置离线源，安装软件包
[root@rabbitmq01 ~]# ls
anaconda-ks.cfg init.sh rabbitmq-repo.tar.gz
[root@rabbitmq01 ~]# tar -xf rabbitmq-repo.tar.gz -C /opt/
[root@rabbitmq01 ~]# mkdir /etc/yum.repos.d/repo.bak
[root@rabbitmq01 ~]# mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d/repo.bak/
[root@rabbitmq01 ~]# mkdir /opt/centos
[root@rabbitmq01 ~]# mount /dev/sr0 /opt/centos/
mount: /dev/sr0 is write-protected, mounting read-only
[root@rabbitmq01 ~]# cat /etc/yum.repos.d/rabbitmq.repo
[rabbitmq]
name=rabbitmq
baseurl=file:///opt/rabbitmq-repo
gpgcheck=0
enabled=1
[centos]
name=centos
baseurl=file:///opt/centos
gpgcheck=0
enabled=1
[root@rabbitmq01 ~]# yum clean all;yum repolist
Loaded plugins: fastestmirror
Cleaning repos: centos rabbitmq
Cleaning up list of fastest mirrors
Other repos take up 176 M of disk space (use --verbose for details)
Loaded plugins: fastestmirror
Determining fastest mirrors
centos | 3.6 kB 00:00:00
rabbitmq | 2.9 kB 00:00:00
(1/3): centos/group_gz | 153 kB 00:00:00
(2/3): rabbitmq/primary_db | 35 kB 00:00:00
(3/3): centos/primary_db | 3.3 MB 00:00:00
repo id repo name status
centos centos 4,070
rabbitmq rabbitmq 26
repolist: 4,096
[root@rabbitmq01 ~]#
[root@rabbitmq01 ~]# yum install -y vsftpd
[root@rabbitmq01 ~]# echo "anon_root=/opt" >> /etc/vsftpd/vsftpd.conf
[root@rabbitmq01 ~]# systemctl enable --now vsftpd
Created symlink from /etc/systemd/system/multi-user.target.wants/vsftpd.service to /usr/lib/systemd/system/vsftpd.service.
[root@rabbitmq01 ~]#
[root@rabbitmq02 ~]# mkdir /etc/yum.repos.d/repo.bak
[root@rabbitmq02 ~]# mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d/repo.bak/
[root@rabbitmq02 ~]# cat /etc/yum.repos.d/rabbitmq.repo abbitmq.repo
[rabbitmq]
name=rabbitmq
baseurl=ftp://rabbitmq01/rabbitmq-repo
gpgcheck=0
enabled=1
[centos]
name=centos
baseurl=ftp://rabbitmq01/centos
gpgcheck=0
enabled=1
[root@rabbitmq02 ~]# yum clean all;yum repolist
Loaded plugins: fastestmirror
Cleaning repos: centos rabbitmq
Cleaning up list of fastest mirrors
Other repos take up 176 M of disk space (use --verbose for details)
Loaded plugins: fastestmirror
Determining fastest mirrors
centos | 3.6 kB 00:00:00
rabbitmq | 2.9 kB 00:00:00
(1/3): centos/group_gz | 153 kB 00:00:00
(2/3): rabbitmq/primary_db | 35 kB 00:00:00
(3/3): centos/primary_db | 3.3 MB 00:00:00
repo id repo name status
centos centos 4,070
rabbitmq rabbitmq 26
repolist: 4,096
[root@rabbitmq02 ~]#
[root@rabbitmq03 ~]# mkdir /etc/yum.repos.d/repo.bak
[root@rabbitmq03 ~]# mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d/repo.bak/
[root@rabbitmq03 ~]# cat /etc/yum.repos.d/rabbitmq.repo
[rabbitmq]
name=rabbitmq
baseurl=ftp://rabbitmq01/rabbitmq-repo
gpgcheck=0
enabled=1
[centos]
name=centos
baseurl=ftp://rabbitmq01/centos
gpgcheck=0
enabled=1
[root@rabbitmq03 ~]# yum clean all;yum repolist
Loaded plugins: fastestmirror
Cleaning repos: centos rabbitmq
Cleaning up list of fastest mirrors
Other repos take up 176 M of disk space (use --verbose for details)
Loaded plugins: fastestmirror
Determining fastest mirrors
centos | 3.6 kB 00:00:00
rabbitmq | 2.9 kB 00:00:00
(1/3): rabbitmq/primary_db | 35 kB 00:00:00
(2/3): centos/group_gz | 153 kB 00:00:00
(3/3): centos/primary_db | 3.3 MB 00:00:00
repo id repo name status
centos centos 4,070
rabbitmq rabbitmq 26
repolist: 4,096
[root@rabbitmq03 ~]#
[root@rabbitmq01 ~]# yum install -y rabbitmq-server
[root@rabbitmq02 ~]# yum install -y rabbitmq-server
[root@rabbitmq03 ~]# yum install -y rabbitmq-server
启动并开机自动
[root@rabbitmq01 ~]# systemctl enable --now rabbitmq-server
Created symlink from /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service to /usr/lib/systemd/system/rabbitmq-server.service.
[root@rabbitmq01 ~]#
[root@rabbitmq02 ~]# systemctl enable --now rabbitmq-server
Created symlink from /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service to /usr/lib/systemd/system/rabbitmq-server.service.
[root@rabbitmq02 ~]#
[root@rabbitmq03 ~]# systemctl enable --now rabbitmq-server
Created symlink from /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service to /usr/lib/systemd/system/rabbitmq-server.service.
[root@rabbitmq03 ~]#
配置界面访问
[root@rabbitmq01 ~]# rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
mochiweb
webmachine
rabbitmq_web_dispatch
amqp_client
rabbitmq_management_agent
rabbitmq_management
Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
重启后界面查看
[root@rabbitmq01 ~]# systemctl restart rabbitmq-server
[root@rabbitmq01 ~]# ss -tln | grep 15672
LISTEN 0 128 *:15672 *:*
[root@rabbitmq01 ~]#
可以看到15672端口已开放，打开浏览器，输入rabbitmq1节点的ip加端口15672（http://IP:15672）访问RabbitMQ监控界面，使用用户名：密码guest：guest登录
配置集群节点通信
[root@rabbitmq01 ~]# cat /var/lib/rabbitmq/.erlang.cookie
XQDUOWPJABMYRFCNXQKV
[root@rabbitmq01 ~]# chmod 777 /var/lib/rabbitmq/.erlang.cookie
[root@rabbitmq01 ~]# scp /var/lib/rabbitmq/.erlang.cookie root@rabbitmq02:/var/lib/rabbitmq/
.erlang.cookie 100% 20 12.8KB/s 00:00
[root@rabbitmq01 ~]# scp /var/lib/rabbitmq/.erlang.cookie root@rabbitmq03:/var/lib/rabbitmq/
.erlang.cookie 100% 20 12.5KB/s 00:00
[root@rabbitmq01 ~]#
[root@rabbitmq02 ~]# chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
[root@rabbitmq03 ~]# chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
重启rabbitmq服务
[root@rabbitmq02 ~]#systemctl restart rabbitmq-server
[root@rabbitmq03 ~]#systemctl restart rabbitmq-server
注；经过测试只需要远程把erlang.cookie同步，不做授权和更改文件属性也能成功搭建的，但是做了这些可能后续有服务会用到，就还是做了
配置节点加入集群
默认rabbitmq启动后是磁盘节点，在这个cluster命令下，rabbitmq02和rabbitmq03是内存节点，rabbitmq1是磁盘节点。
如果要使rabbitmq02、rabbitmq03都是磁盘节点，去掉–ram参数即可。
如果想要更改节点类型，可以使用命令rabbitmqctl change_cluster_node_type disc(ram),前提是必须停掉rabbit应用。
[root@rabbitmq02 ~]# rabbitmqctl stop_app
Stopping node rabbit@rabbitmq02 ...
...done.
[root@rabbitmq02 ~]# rabbitmqctl join_cluster --ram rabbit@rabbitmq01
Clustering node rabbit@rabbitmq02 with rabbit@rabbitmq01 ...
...done.
[root@rabbitmq02 ~]# rabbitmqctl start_app
Starting node rabbit@rabbitmq02 ...
...done.
[root@rabbitmq02 ~]#
[root@rabbitmq03 ~]# rabbitmqctl stop_app
Stopping node rabbit@rabbitmq03 ...
...done.
[root@rabbitmq03 ~]# rabbitmqctl join_cluster --ram rabbit@rabbitmq01
Clustering node rabbit@rabbitmq03 with rabbit@rabbitmq01 ...
...done.
[root@rabbitmq03 ~]# rabbitmqctl start_app
Starting node rabbit@rabbitmq03 ...
...done.
[root@rabbitmq03 ~]#
在rabbitmq02和rabbitmq03节点上启用rabbitmq_management
[root@rabbitmq02 ~]# rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
mochiweb
webmachine
rabbitmq_web_dispatch
amqp_client
rabbitmq_management_agent
rabbitmq_management
Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
[root@rabbitmq02 ~]# systemctl restart rabbitmq-server
[root@rabbitmq02 ~]#
[root@rabbitmq03 ~]# rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
mochiweb
webmachine
rabbitmq_web_dispatch
amqp_client
rabbitmq_management_agent
rabbitmq_management
Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
[root@rabbitmq03 ~]# systemctl restart rabbitmq-server
界面验证
常用命令
查看集群状态
[root@rabbitmq01 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq01 ...
[{nodes,[{disc,[rabbit@rabbitmq01]},
{ram,[rabbit@rabbitmq03,rabbit@rabbitmq02]}]},
{running_nodes,[rabbit@rabbitmq02,rabbit@rabbitmq03,rabbit@rabbitmq01]},
{cluster_name,<<"rabbit@rabbitmq01">>},
{partitions,[]}]
...done.
新增用户admin，并设置密码为admin，命令如下：
# rabbitmqctl add_user admin admin
删除admin用户，命令如下：
# rabbitmqctl delete_user admin
修改admin用户的密码为admin123，命令如下：
# rabbitmqctl change_password admin admin123
设置角色命令：
# rabbitmqctl set_user_tags admin administrator monitoring policymaker management
设置用户权限命令：
# rabbitmqctl set_permissions -p VHostPath admin ConfP WriteP ReadP
查询所有权限命令：
# rabbitmqctl list_permissions [-p VHostPath]
指定用户权限命令：
# rabbitmqctl list_user_permissions admin
清除用户权限命令：
# rabbitmqctl clear_permissions [-p VHostPath] admin
集群搭建避坑
1
[root@rabbitmq02 ~]# rabbitmqctl stop_app
Stopping node rabbit@localhost ...
Error: unable to connect to node rabbit@rabbitmq02: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit@rabbitmq02]
rabbit@rabbitmq02:
* connected to epmd (port 4369) on rabbitmq02
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: rabbitmqctl1883@rabbitmq02
- home dir: /var/lib/rabbitmq
- cookie hash: tRBtsDxkfPswwGSJJst8qw==
遇见这个问题，rabbit@localhost，如果在配置了主机名，主机解析，关闭了防火墙、selinux等情况下，而且验证了erlang.cookie三个节点是一致的，就重启服务再试一下，因为有时候你配置完毕后特别是erlang.cookie，是需要重启服务加载一下的，如果确保了这些操作都做完了，再试试最后一招
在使用rabbitmqctl stop_app停止服务时，使用 -n rabbit@rabbitmq02参数，强行指定
rabbitmqctl -n rabbit@rabbitmq02 stop_app
rabbitmqctl -n rabbit@rabbitmq02 join_cluster --ram rabbit@rabbitmq01
rabbitmqctl -n rabbit@rabbitmq02 start_app
2
[root@rabbitmq02 ~]# rabbitmqctl stop_app
Stopping node rabbit@rabbitmq02 ...
Error: unable to connect to node rabbit@rabbitmq02: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit@rabbitmq02]
rabbit@rabbitmq02:
* connected to epmd (port 4369) on rabbitmq02
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: rabbitmqctl1883@rabbitmq02
- home dir: /var/lib/rabbitmq
- cookie hash: tRBtsDxkfPswwGSJJst8qw==
如果报错rabbit@rabbitmq02，这个连接不上，再基于上述操作对erlang.cookie完成后重启一下即可，就能正常使用命令配置集群了