kafka+zookeeper伪集群搭建
(
一.zookeeper伪集群搭建
)
1.创建3个zoo.cfg(zoo1.cfg,zoo2.cfg,zoo3.cfg)
zoo1.cfg 设置:
# 数据文件夹
dataDir=/usr/local/zookeeper/data/data1
# 日志文件夹
dataLogDir=/usr/local/zookeeper/logs/logs1
clientPort=2181
server.1=192.168.94.132:2887:3887
server.2=192.168.94.132:2888:3888
server.3=192.168.94.132:2889:3889
zoo2.cfg 设置:
# 数据文件夹
dataDir=/usr/local/zookeeper/data/data2
# 日志文件夹
dataLogDir=/usr/local/zookeeper/logs/logs2
clientPort=2182
server.1=192.168.94.132:2887:3887
server.2=192.168.94.132:2888:3888
server.3=192.168.94.132:2889:3889
zoo3.cfg 设置:
# 数据文件夹
dataDir=/usr/local/zookeeper/data/data3
# 日志文件夹
dataLogDir=/usr/local/zookeeper/logs/logs2
clientPort=2183
server.1=192.168.94.132:2887:3887
server.2=192.168.94.132:2888:3888
server.3=192.168.94.132:2889:3889
2.创建3个myid
在usr/local/zookeeper/data/datax下闯进myid文件作为标识符,对应server.x数字
3.启动伪集群服务
bin/zkServer.sh start conf/zoo1.cfg
bin/zkServer.sh start conf/zoo2.cfg
bin/zkServer.sh start conf/zoo3.cfg
4.查看伪集群服务每个zookeeper状态
[root@bogon zookeeper-3.4.11]# bin/zkServer.sh status conf/zoo1.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo1.cfg
Mode: follower
[root@bogon zookeeper-3.4.11]# bin/zkServer.sh status conf/zoo2.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo2.cfg
Mode: leader
[root@bogon zookeeper-3.4.11]# bin/zkServer.sh status conf/zoo3.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo3.cfg
Mode: follower
(二.kafka
伪集群搭建
)
1.
将配置文件拷贝多份
[root
@master
zookeeper-
3.4
.
11
]
$
cp config/server.properties config/server-1.properties
[root@master zookeeper-3.4.11]$cp config/server.properties config/server-2.properties
[root@master zookeeper-3.4.11]$cp config/server.properties config/server-3.properties
2.修改每个server-x.properties
server-1.properties 设置:
broker.id=1
port=9092
host.name=192.168.94.132
log.dirs=/usr/local/kafka/kafka-logs-1
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
zookeeper.connect=192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
# 可删除topic
delete.topic.enable=true
server-2.properties
设置:
broker.id=2
port=9093
host.name=192.168.94.132
log.dirs=/usr/local/kafka/kafka-logs-2
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
zookeeper.connect=
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
# 可删除topic
delete.topic.enable=true
server-3.properties
设置:
broker.id=3
port=9093
host.name=192.168.94.132
log.dirs=/usr/local/kafka/kafka-logs-3
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
zookeeper.connect=
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
# 可删除topic
delete.topic.enable=true
每个kafka的broker.id不一样,作为唯一标识符
(三)启动集群测试
注：启动时：先启动 zookeeper，后启动 kafka；关闭时：先关闭 kafka，后关闭zookeeper
1.
分别在每个节点上启动
zookeeper
[root
@master
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh start conf/zoo1.cfg
[
root
@worker1
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh start
conf/zoo2.cfg
[
root
@worker2
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh start conf/zoo3.cfg
2.验证zookeeper集群
# 在master节点上
[root
@master
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh status
conf/zoo1.cfg
# 在worker1节点上
[
root
@worker1
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh status
conf/zoo2.cfg
# 在worker2节点上
[
root
@worker2
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh status conf/zoo3.cfg
显示结果为：有一个是
leader
，剩下的都是
follower
3.启动kafka集群
后台启动方式:
[root
@master
kafka_2.12-1.1.0
]
$
bin/kafka-server-start.sh -daemon config/server-1.properties
[root
@master
kafka_2.12-1.1.0
]
$
bin/kafka-server-start.sh config/server.properties config/server-1.properties
[root
@worker1
kafka_2.12-1.1.0
]
$
bin/kafka-server-start.sh config/server.properties config/server-1.properties
[root
@worker2
kafka_2.12-1.1.0
]
$
bin/kafka-server-start.sh config/server.properties config/server-1.properties
4.测试kafka集群
4.1 创建 topic 和 显示 topic 信息
#
创建topic
bin/kafka-topics.sh --create --zookeeper
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
--replication-factor 3 --partitions 3 --topic test
#
显示topic信息
bin/kafka-topics.sh --describe --zookeeper
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
--topic test
# 列出topic
bin/kafka-topics.sh --list --zookeeper 192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
4.2 创建 producer
bin/kafka-console-producer.sh --broker-list
master:
9092
-topic test
4.3 创建 consumer
# 测试消费 也可分别做测试
bin/kafka-console-consumer.sh --zookeeper
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
-topic test --from-beginning
然后在 producer 里输入消息，consumer 中就会显示出同样的内容，表示消费成功
4.3删除节点
bin/kafka-topics.sh --delete --zookeeper
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
--topic test
(四)关闭集群服务
#
删除topic
bin/kafka-topics.sh --delete --zookeeper
192.168.94.132:2181,192.168.94.132:2182,192.168.94.132:2183
--topic test
# 关闭kafka
[
root
@worker2
kafka_2.12-1.1.0
]
$
bin/kafka-server-stop.sh conf/server-1.properties
[
root
@worker2
kafka_2.12-1.1.0
]
$
bin/kafka-server-stop.sh conf/server-2.properties
[
root
@worker2
kafka_2.12-1.1.0
]
$
bin/kafka-server-stop.sh conf/server-3.properties
# 关闭zookeeper
[root
@master
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh stop conf/zoo1.cfg
[
root
@worker1
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh
stop
conf/zoo2.cfg
[
root
@worker2
zookeeper-
3.4
.
11
]
$
bin/zkServer.sh
stop
conf/zoo3.cfg