kafka+zookeeper入门搭建

配置jdk
[root@htapp01 mnt]# java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

解压zookeeper

解压kafka

[root@htapp01 kafka]# vim /etc/hosts  解析本地主机名

[root@htapp01 zookeeper]# cp conf/zoo_sample.cfg conf/zoo_cfg   生成zookeeper配置文件

[root@htapp01 zookeeper]# ./bin/zkServer.sh start    启动zookeeper
JMX enabled by default
Using config: /mnt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@htapp01 kafka]# nohup ./bin/kafka-server-start.sh config/server.properties >> /mnt/kafka.log &   启动kafka     

[root@htapp01 kafka]# ./bin/kafka-topics.sh  --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkatest
[root@htapp01 kafka]# ./bin/kafka-topics.sh  --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
创建2个主题


[root@htapp01 kafka]# bin/kafka-topics.sh --list --zookeeper localhost:2181     查看运行的主题
kafkatest
test
[root@htapp01 kafka]#

在终端运行生产者
[root@htapp01 kafka]# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
wocao
wocao!
我曹


另外开启一个终端运行消费者（卡夫卡有一个命令，是消费者将转储消息到标准输出。）
[root@htapp01 kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
wocao
wocao!
我曹

扩大我们的集群到三个节点（仍然在我们的本地机器）。
复制出另外两个配置文件
[root@htapp01 kafka]# cp config/server.properties config/server_1.properties
[root@htapp01 kafka]# cp config/server.properties config/server_2.properties

修改server_1和server_2配置文件（列出三个配置文件的内容）
server.properties为默认文件
[root@htapp01 kafka]# grep -v "^$" config/server.properties | grep -v "^#"
broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

server_1.properties内容
[root@htapp01 kafka]# grep -v "^$" config/server_1.properties | grep -v "^#"
broker.id=1
listeners=PLAINTEXT://:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs_1
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

server_2.properties内容
[root@htapp01 kafka]# grep -v "^$" config/server_2.properties | grep -v "^#"
broker.id=2
listeners=PLAINTEXT://:9094
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs_2
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

启动另外两个节点
[root@htapp01 kafka]# nohup ./bin/kafka-server-start.sh config/server_1.properties >> /mnt/kafka_1.log &
[1] 4451
[root@htapp01 kafka]# nohup: 忽略输入重定向错误到标准输出端
[root@htapp01 kafka]# nohup ./bin/kafka-server-start.sh config/server_2.properties >> /mnt/kafka_2.log &
[2] 4744
[root@htapp01 kafka]# nohup: 忽略输入重定向错误到标准输出端


现在创建一个新的主题，复制系数为3
[root@htapp01 kafka]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic kafka_3_node

查看主题kafka_3_node
[root@htapp01 kafka]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic kafka_3_node
Topic:kafka_3_node    PartitionCount:1    ReplicationFactor:3    Configs:
    Topic: kafka_3_node    Partition: 0    Leader: 1    Replicas: 1,0,2    Isr: 1,0,2
说明：
Topic（主题）：kafka_3_node
PartitionCount（分区数）：1
Leader（领导）在1节点上
replicas（副本）在1,0,2节点，不管node死活，只是列出信息而已.
isr” 工作中的复制节点的集合. 也就是活的节点的集合.

再看看主题test
[root@htapp01 kafka]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test    PartitionCount:1    ReplicationFactor:1    Configs:
    Topic: test    Partition: 0    Leader: 0    Replicas: 0    Isr: 0

发送一些新的消息kafka_3_node
[root@htapp01 kafka]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafka_3_node
woqu
我去
我去去去去去去去


用另外一个终端接收
[root@htapp01 kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic kafka_3_node
woqu
我去
我去去去去去去去


干掉leader,也就是Broker1
[root@htapp01 kafka]# kill 4451

再看一下kafka_3_node
[root@htapp01 kafka]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic kafka_3_node
Topic:kafka_3_node    PartitionCount:1    ReplicationFactor:3    Configs:
    Topic: kafka_3_node    Partition: 0    Leader: 0    Replicas: 1,0,2    Isr: 0,2

对比一下之前的
[root@htapp01 kafka]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic kafka_3_node
Topic:kafka_3_node    PartitionCount:1    ReplicationFactor:3    Configs:
    Topic: kafka_3_node    Partition: 0    Leader: 1    Replicas: 1,0,2    Isr: 1,0,2

再看一下消费者，消息是没有丢失的
[root@htapp01 kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic kafka_3_node
woqu
我去
我去去去去去去去