Centos7 Kubernetes(K8s) k8s 开发 单服务器部署 rocketmq
这几天对开发服务器中的 k8s和相关模块 进行升级，做个记录
环境，与系统
系统：centos 7.6
内核： 5.1.9-1.el7.elrepo.x86_64
Kubernetes(K8s) 1.14.3
服务器IP：192.168.0.254
服务器IP 说明
如果更改了服务器IP 为其他IP，那么请全局全部替换成你改的IP
fox.风
如何安装 k8s
请看 最近写的一篇文章，
https://blog.csdn.net/fenglailea/article/details/88745642
常用 k8s 命令
https://foxwho.blog.csdn.net/article/details/92137467
服务器资源限制
对CPU,内存都做了限制，请自行针对修改
没有使用持久化存储，这里直接使用宿主机存储，如有需要，请自行增加
目录
请把项目放置到 /www/k8s/foxdev/ 目录下，所有都是基于此目录
或者请看 下面一节的 拉取代码
拉取代码
mkdir -p /www/k8s/
cd /www/k8s/
git clone https://github.com/foxiswho/k8s-nacos-sentinel-rocketmq-zipkin-elasticsearch-redis-mysql.git foxdev
cd foxdev
以下是 独立部署 相关模块
k8s rocketmq 独立部署
rocketmq 采用 server:1主,Broker:1主1从模式
带有 rocketmq-console web 简单管理界面
相关参数说明
端口：
server:宿主机IP:9876
Broker:(master)宿主机IP:10909
Broker:(master)宿主机IP:10911
Broker:(master)宿主机IP:10912
Broker:(slave)宿主机IP:10919
Broker:(slave)宿主机IP:10921
Broker:(slave)宿主机IP:10922
ui:(console)宿主机IP:8180
broker中3个端口说明
listenPort=10911Broker 对外服务的监听端口haListenPort=10912haService中使用 默认值为：listenPort + 1fastListenPort=10909主要用于slave同步master listenPort - 2
XX-rc.yml文件启动参数autoCreateTopicEnable=true，表示 允许 Broker 自动创建Topic
日志相关目录
执行命令时候，会自动创建 以下7个目录，部署时候要映射 pod内部目录
/www/k8s/foxdev/rocketmq/logs/
/www/k8s/foxdev/rocketmq/logs-b/
/www/k8s/foxdev/rocketmq/logs-b-s/
/www/k8s/foxdev/rocketmq/logs-ui/
/www/k8s/foxdev/rocketmq/store/
/www/k8s/foxdev/rocketmq/store-b/
/www/k8s/foxdev/rocketmq/store-b-s/
logs: server 日志目录
logs-b: Broker(master) 日志目录
logs-b-s: Broker(slave) 日志目录
logs-ui: console 日志目录
store: server store 目录
store-b: Broker(master) store 目录
store-b-s: Broker(slave) store 目录
独立部署 rocketmq 操作
直接执行 rocketmq 目录下 rmq.start.sh 即可
cd ./rocketmq/ && ./rmq.start.sh
大约几分钟后，使用rocketmq或者访问浏览器http://192.168.0.254:8180,即可看到 rocketmq-console的管理界面
http://192.168.0.254:8180
使用命令查看相关信息
kubectl get svc,pod,cs,rc
或
kubectl get svc
kubectl get pod
kubectl get cs
kubectl get rc
查看pod
kubectl get pod
输出
NAME READY STATUS RESTARTS AGE
rmqbroker-0 1/1 Running 0 3h5m
rmqbroker-slave-0 1/1 Running 0 3h5m
rmqui-0 1/1 Running 0 3h5m
rocketmq-0 1/1 Running 0 3h5m
查看 service
kubectl get svc
输出
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rmqbroker NodePort 10.1.183.152 <none> 10909:10909/TCP,10911:10911/TCP,10912:10912/TCP 3h59m
rmqbroker-slave NodePort 10.1.128.180 <none> 10919:10919/TCP,10921:10921/TCP,10922:10922/TCP 3h7m
rmqui NodePort 10.1.95.26 <none> 8080:8180/TCP 3h7m
rocketmq NodePort 10.1.128.90 <none> 9876:9876/TCP 3h7m
在开发服务器上部署成功，有问题的，可以联系我
资源编排文件
后缀为-rc.yml 是 StatefulSet/Pod 文件
后缀为-service.yml 是 service 文件
开头rocketmq 是 rocketmq server
开头broker 是 rocketmq broker
开头broker-s 是 rocketmq broker slave ,其他的是 master
开头ui 是 rocketmq console 简单页面管理
rocketmq server 资源编排
rocketmq-rc.yml
# 版本
apiVersion: apps/v1
# 资源对象
kind: StatefulSet
metadata:
#名称
name: rocketmq
#标签
labels:
#标签名和标签值
app: rocketmq
#命名空间
# namespace: default
spec:
#服务名称
serviceName: rocketmq
replicas: 1
selector:
#匹配标签
matchLabels:
#标签名和值
app: rocketmq
template:
metadata:
labels:
app: rocketmq
spec:
containers:
#容器名称
- name: rocketmq
# 镜像
image: foxiswho/rocketmq:server
#Always：不管镜像是否存在都会进行一次拉取
imagePullPolicy: Always
#资源限制
resources:
limits:
cpu: 250m
memory: 1000Mi
requests:
cpu: 100m
memory: 768Mi
#环境变量
env:
#设置时区
- name: TZ
value: Asia/Shanghai
- name: JAVA_OPT
value: -Duser.home=/opt
- name: JAVA_OPT_EXT
value: -server -Xms768m -Xmx768m -Xmn256m
ports:
# 容器端口
- containerPort: 9876
# 加载外部券路径等
volumeMounts:
- name: vol-logs
mountPath: /opt/logs
- name: vol-init-d
mountPath: /opt/store
volumes:
- name: vol-logs
hostPath:
path: /www/k8s/foxdev/rocketmq/logs
type: Directory
- name: vol-init-d
hostPath:
path: /www/k8s/foxdev/rocketmq/store
type: Directory
rocketmq-service.yml
apiVersion: v1
kind: Service
metadata:
# namespace: default
name: rocketmq
labels:
app: rocketmq
spec:
#nodePort 是提供给集群外部客户访问service的入口
type: NodePort
ports:
- port: 9876
targetPort: 9876
# 外部访问端口
nodePort: 9876
selector:
app: rocketmq
rocketmq broker master 资源编排
broker-rc.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rmqbroker
labels:
app: rmqbroker
# namespace: default
spec:
serviceName: rmqbroker
replicas: 1
selector:
matchLabels:
app: rmqbroker
template:
metadata:
labels:
app: rmqbroker
spec:
containers:
- name: rmqbroker
image: foxiswho/rocketmq:broker
imagePullPolicy: Always
resources:
limits:
cpu: 250m
memory: 1000Mi
requests:
cpu: 100m
memory: 512Mi
env:
- name: TZ
value: Asia/Shanghai
- name: NAMESRV_ADDR
value: rocketmq:9876
- name: JAVA_OPT_EXT
value: -Duser.home=/opt -server -Xms512m -Xmx512m -Xmn128m
command: ["mqbroker","-c","/etc/rocketmq/broker.conf","-n","rocketmq:9876","autoCreateTopicEnable=true"]
ports:
- containerPort: 10909
- containerPort: 10911
- containerPort: 10912
volumeMounts:
- name: vol-logs
mountPath: /opt/logs
- name: vol-store
mountPath: /opt/store
- name: vol-broker
mountPath: /etc/rocketmq/broker.conf
volumes:
- name: vol-logs
hostPath:
path: /www/k8s/foxdev/rocketmq/logs-b
type: Directory
- name: vol-store
hostPath:
path: /www/k8s/foxdev/rocketmq/store-b
type: Directory
- name: vol-broker
hostPath:
path: /www/k8s/foxdev/rocketmq/conf/broker.conf
type: File
broker-service.yml
apiVersion: v1
kind: Service
metadata:
# namespace: default
name: rmqbroker
labels:
app: rmqbroker
spec:
type: NodePort
ports:
- port: 10909
targetPort: 10909
nodePort: 10909
name: slave-master
- port: 10911
targetPort: 10911
nodePort: 10911
name: listen
- port: 10912
targetPort: 10912
nodePort: 10912
name: ha-service
selector:
app: rmqbroker
rocketmq broker slave 资源编排
broker-s-rc.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rmqbroker-slave
labels:
app: rmqbroker-slave
# namespace: default
spec:
serviceName: rmqbroker-slave
replicas: 1
selector:
matchLabels:
app: rmqbroker-slave
template:
metadata:
labels:
app: rmqbroker-slave
spec:
containers:
- name: rmqbroker-slave
image: foxiswho/rocketmq:broker
imagePullPolicy: Always
resources:
limits:
cpu: 250m
memory: 1000Mi
requests:
cpu: 100m
memory: 512Mi
env:
- name: TZ
value: Asia/Shanghai
- name: NAMESRV_ADDR
value: rocketmq:9876
- name: JAVA_OPT_EXT
value: -Duser.home=/opt -server -Xms512m -Xmx512m -Xmn128m
command: ["mqbroker","-c","/etc/rocketmq/broker-s.conf","-n","rocketmq:9876","autoCreateTopicEnable=true"]
ports:
- containerPort: 10919
- containerPort: 10921
- containerPort: 10922
volumeMounts:
- name: vol-logs
mountPath: /opt/logs
- name: vol-store
mountPath: /opt/store
- name: vol-broker-s
mountPath: /etc/rocketmq/broker-s.conf
volumes:
- name: vol-logs
hostPath:
path: /www/k8s/foxdev/rocketmq/logs-b-s
type: Directory
- name: vol-store
hostPath:
path: /www/k8s/foxdev/rocketmq/store-b-s
type: Directory
- name: vol-broker-s
hostPath:
path: /www/k8s/foxdev/rocketmq/conf/broker-s.conf
type: File
broker-s-service.yml
apiVersion: v1
kind: Service
metadata:
# namespace: default
name: rmqbroker-slave
labels:
app: rmqbroker-slave
spec:
type: NodePort
ports:
- port: 10919
targetPort: 10919
nodePort: 10919
name: slave-master
- port: 10921
targetPort: 10921
nodePort: 10921
name: broker-listen
- port: 10922
targetPort: 10922
nodePort: 10922
name: ha-service
selector:
app: rmqbroker-slave
rocketmq ui console 资源编排
ui-rc.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rmqui
labels:
app: rmqui
# namespace: default
spec:
serviceName: rmqui
replicas: 1
selector:
matchLabels:
app: rmqui
template:
metadata:
labels:
app: rmqui
spec:
containers:
- name: rmqui
image: styletang/rocketmq-console-ng
imagePullPolicy: Always
resources:
limits:
cpu: 450m
memory: 1768Mi
requests:
cpu: 100m
memory: 768Mi
env:
- name: TZ
value: Asia/Shanghai
- name: JAVA_OPTS
value: -Drocketmq.namesrv.addr=rocketmq:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false -Duser.home=/root -Xms768m -Xmx768m
ports:
- containerPort: 8080
volumeMounts:
- name: vol-logs
mountPath: /root/logs
volumes:
- name: vol-logs
hostPath:
path: /www/k8s/foxdev/rocketmq/logs-ui
type: Directory
ui-service.yml
apiVersion: v1
kind: Service
metadata:
# namespace: default
name: rmqui
labels:
app: rmqui
spec:
type: NodePort
ports:
- port: 8080
targetPort: 8080
nodePort: 8180
selector:
app: rmqui
broker 配置文件
两个文件只有以下配置不一样，其他都是一样的
#0 表示Master，>0 表示Slave
brokerId=1
#Broker 的角色
#- ASYNC_MASTER 异步复制Master
#- SYNC_MASTER 同步双写Master
#- SLAVE
brokerRole=SLAVE
#Broker 对外服务的监听端口
listenPort=10921
broker master 配置文件
broker.conf
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#所属集群名字
brokerClusterName=DefaultCluster
#broker名字，注意此处不同的配置文件填写的不一样，如果在broker-a.properties使用:broker-a,
#在broker-b.properties使用:broker-b
brokerName=broker-a
#0 表示Master，>0 表示Slave
brokerId=0
#nameServer地址，分号分割
#namesrvAddr=rocketmq-0:9876;rocketmq-1:9876
namesrvAddr=rocketmq:9876
#启动IP,如果 docker 报 com.alibaba.rocketmq.remoting.exception.RemotingConnectException: connect to <192.168.0.120:10909> failed
# 解决方式1 加上一句producer.setVipChannelEnabled(false);，解决方式2 brokerIP1 设置宿主机IP，不要使用docker 内部IP
brokerIP1=192.168.0.254
#在发送消息时，自动创建服务器不存在的topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建Topic，建议线下开启，线上关闭 ！！！这里仔细看是false，false，false
#原因下篇博客见~ 哈哈哈哈
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#haService中使用 默认值为：listenPort + 1
#haListenPort=10912
#主要用于slave同步master listenPort - 2
#fastListenPort=10909
#删除文件时间点，默认凌晨4点
deleteWhen=04
#文件保留时间，默认48小时
fileReservedTime=120
#commitLog每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/opt/store
#commitLog 存储路径
#storePathCommitLog=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/commitlog
#消费队列存储
#storePathConsumeQueue=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/consumequeue
#消息索引存储路径
#storePathIndex=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/index
#checkpoint 文件存储路径
#storeCheckpoint=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/checkpoint
#abort 文件存储路径
#abortFile=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制Master
#- SYNC_MASTER 同步双写Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128
broker slave 配置文件
broker.conf
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#所属集群名字
brokerClusterName=DefaultCluster
#broker名字，注意此处不同的配置文件填写的不一样，如果在broker-a.properties使用:broker-a,
#在broker-b.properties使用:broker-b
brokerName=broker-a
#0 表示Master，>0 表示Slave
brokerId=1
#nameServer地址，分号分割
#namesrvAddr=rocketmq-0:9876;rocketmq-1:9876
namesrvAddr=rocketmq:9876
#启动IP,如果 docker 报 com.alibaba.rocketmq.remoting.exception.RemotingConnectException: connect to <192.168.0.120:10909> failed
# 解决方式1 加上一句producer.setVipChannelEnabled(false);，解决方式2 brokerIP1 设置宿主机IP，不要使用docker 内部IP
brokerIP1=192.168.0.254
#Broker 的角色
#- ASYNC_MASTER 异步复制Master
#- SYNC_MASTER 同步双写Master
#- SLAVE
brokerRole=SLAVE
#在发送消息时，自动创建服务器不存在的topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建Topic，建议线下开启，线上关闭 ！！！这里仔细看是false，false，false
#原因下篇博客见~ 哈哈哈哈
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10921
#haService中使用 默认值为：listenPort + 1
#haListenPort=10912
#主要用于slave同步master listenPort - 2
#fastListenPort=10909
#删除文件时间点，默认凌晨4点
deleteWhen=04
#文件保留时间，默认48小时
fileReservedTime=120
#commitLog每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/opt/store
#commitLog 存储路径
#storePathCommitLog=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/commitlog
#消费队列存储
#storePathConsumeQueue=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/consumequeue
#消息索引存储路径
#storePathIndex=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/index
#checkpoint 文件存储路径
#storeCheckpoint=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/checkpoint
#abort 文件存储路径
#abortFile=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128
在开发服务器上部署成功，有问题的，可以联系我
源文件
https://github.com/foxiswho/k8s-nacos-sentinel-rocketmq-zipkin-elasticsearch-redis-mysql/tree/master/rocketmq