k8s搭建 rabbitmq集群
| 工具 | 版本 |
|---|---|
| k8s | 1.19 |
| rabbitmq | 3.7.8 |
rabbitmq镜像下载-CSDN
mq镜像下载位置
链接：https://pan.baidu.com/s/1tJ1xxfA174Kjj4CYcjFJ8w?pwd=cyr4
提取码：cyr4
1. 创建挂载使用的pv以及pvc
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/bound-by-controller: "yes"
finalizers:
- kubernetes.io/pv-protection
labels:
cattle.io/creator: norman
name: rabbit-cluster-pv
accessModes:
- ReadWriteMany
capacity:
storage: 1Ti
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: rabbit-cluster-pvc
namespace: cloud-test
resourceVersion: "304986615"
nfs:
path: /home/nfs/rabbit-cluster
server: 192.168.1.26
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
finalizers:
- kubernetes.io/pvc-protection
labels:
cattle.io/creator: norman
name: rabbit-cluster-pvc
namespace: cloud-test
resourceVersion: "304986620"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Ti
storageClassName: ""
volumeMode: Filesystem
volumeName: rabbit-cluster-pvc
status:
accessModes:
- ReadWriteMany
capacity:
storage: 1Ti
phase: Bound
2. 创建对应的集群服务和无头服务
kind: Service
apiVersion: v1
metadata:
labels:
app: rabbitmq-cluster
name: rabbitmq-headless
namespace: cloud-test
spec:
clusterIP: None
ports:
- name: rmqport
port: 5672
targetPort: 5672
selector:
app: rabbitmq-cluster
---
kind: Service
apiVersion: v1
metadata:
labels:
app: rabbitmq-cluster
name: rabbitmq-cluster
namespace: cloud-test
spec:
ports:
- name: http
port: 15672
protocol: TCP
targetPort: 15672
- name: rmqport
port: 5672
targetPort: 5672
selector:
app: rabbitmq-cluster
type: ClusterIP
3. 创建rabbitmq需要的RBAC权限
apiVersion: v1
kind: ServiceAccount
metadata:
name: rabbitmq
namespace: cloud-test
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rabbitmq
namespace: cloud-test
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rabbitmq
namespace: cloud-test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rabbitmq
subjects:
- kind: ServiceAccount
name: rabbitmq
namespace: cloud-test
4. 创建rabbitmq的configmap配置文件
apiVersion: v1
kind: ConfigMap
metadata:
name: rabbitmq-cluster-config
namespace: cloud-test
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_delayed_message_exchange].
rabbitmq.conf: |-
default_user = mqadmin ##账户
default_pass = mqadmin ##密码
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s ##开启rabbit集群
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.node_cleanup.interval = 30
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
queue_master_locator=min-masters
loopback_users.guest = false
## cluster_formation.randomized_startup_delay_range.min = 0
## cluster_formation.randomized_startup_delay_range.max = 2
# 必须设置service_name，否则Pod无法正常启动，这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量
#cluster_formation.k8s.service_name = rabbitmq-headless
# default is rabbitmq-cluster's namespace
# hostname_suffix 必须设置hostname_suffix，否则节点不能成为集群,注意修改rabbitmq为ns名称
cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.cloud-test.svc.cluster.local
# 内存上限
vm_memory_high_watermark.absolute = 512MB
# 硬盘上限
disk_free_limit.absolute = 100GB
5. 创建rabbitmq集群
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
generation: 1
labels:
cattle.io/creator: norman
app: rabbitmq-cluster
name: rabbitmq-cluster
namespace: cloud-test
spec:
podManagementPolicy: OrderedReady
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: rabbitmq-cluster
serviceName: rabbitmq-headless
template:
metadata:
labels:
app: rabbitmq-cluster
spec:
containers:
- env:
- name: K8S_SERVICE_NAME ##集群内部的无头服务
value: rabbitmq-headless
- name: MY_POD_NAME ##容器名称
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: MY_POD_NAMESPACE ##命名空间
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: K8S_HOSTNAME_SUFFIX ##集群加入路径
value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
- name: RABBITMQ_ERLANG_COOKIE
value: mycookie
- name: RABBITMQ_NODENAME ##节点名称
value: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
- name: RABBITMQ_USE_LONGNAME
value: "true"
image: 192.168.0.14/project/kubernetes-rabbitmq-autocluster:3.7.8
imagePullPolicy: Always
livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- status
failureThreshold: 3
initialDelaySeconds: 160
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 15
name: rabbitmq-cluster
ports:
- containerPort: 15672
name: 15672tcp00
protocol: TCP
- containerPort: 5672
name: 5672tcp00
protocol: TCP
readinessProbe:
exec:
command:
- rabbitmq-diagnostics
- status
failureThreshold: 3
initialDelaySeconds: 20
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 10
resources: {}
securityContext:
capabilities: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/rabbitmq
name: config-volume
- mountPath: /var/lib/rabbitmq/mnesia
name: rabbitmq-data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: rabbitmq
serviceAccountName: rabbitmq
terminationGracePeriodSeconds: 10
volumes:
- configMap:
defaultMode: 420
items:
- key: rabbitmq.conf
path: rabbitmq.conf ##mq配置
- key: enabled_plugins
path: enabled_plugins ##插件
name: rabbitmq-cluster-config ##configmap
name: config-volume
- name: rabbitmq-data
persistentVolumeClaim:
claimName: rabbit-cluster-pvc ##pvc
updateStrategy:
type: OnDelete
6. 检查rabbitmq集群
注意！！！
如果这个过程会有点长，耐心等待几分钟，如果长时间没有加入集群，请删除后重新部署 rabbitmq的有状态服务
6.1 通过查看日志确认有没有加入集群
kubectl logs rabbitmq-cluster-0 -n cloud-test
I1112 15:00:22.372930 19510 http.go:113] HTTP2 has been explicitly disabled
2022-11-12 06:15:00.345 [info] <0.33.0> Application lager started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.447 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.593 [info] <0.33.0> Application inets started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.593 [info] <0.33.0> Application jsx started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.593 [info] <0.33.0> Application recon started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.615 [info] <0.33.0> Application os_mon started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.615 [info] <0.33.0> Application xmerl started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.615 [info] <0.33.0> Application crypto started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.616 [info] <0.33.0> Application cowlib started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.616 [info] <0.33.0> Application asn1 started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.616 [info] <0.33.0> Application public_key started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.760 [info] <0.33.0> Application ssl started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.780 [info] <0.33.0> Application ranch started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.790 [info] <0.33.0> Application cowboy started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.790 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.790 [info] <0.33.0> Application rabbit_common started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.791 [info] <0.198.0>
Starting RabbitMQ 3.7.8 on Erlang 20.3.8.5
Copyright (C) 2007-2018 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
## ##
## ## RabbitMQ 3.7.8. Copyright (C) 2007-2018 Pivotal Software, Inc.
########## Licensed under the MPL. See http://www.rabbitmq.com/
###### ##
########## Logs: >
database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local
2022-11-12 06:15:06.551 [info] <0.206.0> Memory high watermark set to 488 MiB (512000000 bytes) of 15885 MiB (16657137664 bytes) total
2022-11-12 06:15:06.565 [info] <0.208.0> Enabling free disk space monitoring
2022-11-12 06:15:06.565 [info] <0.208.0> Disk free limit set to 100000MB
2022-11-12 06:15:06.575 [info] <0.211.0> Limiting to approx 1048476 file handles (943626 sockets)
2022-11-12 06:15:06.576 [info] <0.212.0> FHC read buffering: OFF
2022-11-12 06:15:06.576 [info] <0.212.0> FHC write buffering: ON
2022-11-12 06:15:06.578 [info] <0.198.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch...
2022-11-12 06:15:06.578 [info] <0.198.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2022-11-12 06:15:06.578 [info] <0.198.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2022-11-12 06:15:06.578 [info] <0.198.0> Peer discovery backend does not support locking, falling back to randomized delay
2022-11-12 06:15:06.578 [info] <0.198.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
2022-11-12 06:15:06.604 [info] <0.198.0> k8s endpoint listing returned nodes not yet ready: rabbitmq-cluster-0
2022-11-12 06:15:06.604 [info] <0.198.0> All discovered existing cluster peers:
2022-11-12 06:15:06.604 [info] <0.198.0> Discovered no peer nodes to cluster with
2022-11-12 06:15:06.607 [info] <0.33.0> Application mnesia exited with reason: stopped
2022-11-12 06:15:06.643 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:06.812 [info] <0.198.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-11-12 06:15:06.852 [info] <0.198.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-11-12 06:15:06.895 [info] <0.198.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-11-12 06:15:06.895 [info] <0.198.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping registration.
2022-11-12 06:15:06.911 [info] <0.198.0> Priority queues enabled, real BQ is rabbit_variable_queue
2022-11-12 06:15:06.916 [info] <0.395.0> Starting rabbit_node_monitor
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: 1 to apply
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: No durable queues found. Skipping message store migration
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: Removing the old message store data
2022-11-12 06:15:06.994 [info] <0.198.0> message_store upgrades: All upgrades applied successfully
2022-11-12 06:15:07.039 [info] <0.198.0> Management plugin: using rates mode 'basic'
2022-11-12 06:15:07.041 [info] <0.198.0> Adding vhost '/'
2022-11-12 06:15:07.048 [info] <0.437.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2022-11-12 06:15:07.052 [info] <0.437.0> Starting message stores for vhost '/'
2022-11-12 06:15:07.053 [info] <0.441.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-11-12 06:15:07.059 [info] <0.437.0> Started message store of type transient for vhost '/'
2022-11-12 06:15:07.059 [info] <0.444.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-11-12 06:15:07.061 [warning] <0.444.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2022-11-12 06:15:07.065 [info] <0.437.0> Started message store of type persistent for vhost '/'
2022-11-12 06:15:07.067 [info] <0.198.0> Creating user 'mqadmin'
2022-11-12 06:15:07.068 [info] <0.198.0> Setting user tags for user 'mqadmin' to [administrator]
2022-11-12 06:15:07.069 [info] <0.198.0> Setting permissions for 'mqadmin' in '/' to '.*', '.*', '.*'
2022-11-12 06:15:07.072 [info] <0.482.0> started TCP Listener on [::]:5672
2022-11-12 06:15:07.082 [info] <0.198.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.092 [info] <0.198.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.092 [info] <0.33.0> Application rabbit started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.496.0> Peer discovery: enabling node cleanup (will only log warnings). Check interval: 60 seconds.
2022-11-12 06:15:07.093 [info] <0.33.0> Application rabbitmq_peer_discovery_common started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.33.0> Application rabbitmq_web_dispatch started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.33.0> Application rabbitmq_peer_discovery_k8s started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.33.0> Application amqp_client started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.096 [info] <0.33.0> Application rabbitmq_management_agent started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.137 [info] <0.549.0> Management plugin started. Port: 15672
2022-11-12 06:15:07.137 [info] <0.655.0> Statistics database started.
2022-11-12 06:15:07.139 [info] <0.33.0> Application rabbitmq_management started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.139 [info] <0.33.0> Application rabbitmq_delayed_message_exchange started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
completed with 6 plugins.
2022-11-12 06:15:07.428 [info] <0.5.0> Server startup complete; 6 plugins started.
* rabbitmq_delayed_message_exchange
* rabbitmq_management
* rabbitmq_management_agent
* rabbitmq_peer_discovery_k8s
* rabbitmq_web_dispatch
* rabbitmq_peer_discovery_common
2022-11-12 06:16:20.847 [info] <0.395.0> node 'rabbit@rabbitmq-cluster-1.rabbitmq-headless.cloud-test.svc.cluster.local' up
2022-11-12 06:16:21.393 [info] <0.395.0> rabbit on node 'rabbit@rabbitmq-cluster-1.rabbitmq-headless.cloud-test.svc.cluster.local' up
2022-11-12 06:17:41.304 [info] <0.395.0> node 'rabbit@rabbitmq-cluster-2.rabbitmq-headless.cloud-test.svc.cluster.local' up
2022-11-12 06:17:41.851 [info] <0.395.0> rabbit on node 'rabbit@rabbitmq-cluster-2.rabbitmq-headless.cloud-test.svc.cluster.local' up
6.2 通过进入容器确认有没有加入集群
kubectl exec -it rabbitmq-cluster-0 bash -n cloud-test
7. 最终对外暴露出rabbitmq的两个端口，访问rabbit的dashboard页面
8. 开启rabbitmq的镜像集群模式
