1. kafka版本号介绍
kafka版本号介绍，例如：kafka-2.12-2.1.0
前面的2.12 代表scala编译的版本号
后面的2.1.0 代表kafka的版本号，大版本号+小版本号+修订版本号
2.kafka对应jdk版本号
支持 Java 11，最小版本jdk8，低版本会报错。
3.filebeat7.8对应kafka2.1.0
filebeat7.8 input对应的最高的版本是kafka2.1.0
4. kafka server.properties配置
4.1 kafka-1
vi /usr/local/server/kafka-1/config/server.properties
broker.id=1
listeners=PLAINTEXT://localhost:9091
log.dirs=/data/logs/kafka/kafka-1
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
#topic的默认分区数
num.partitions=1
# Kafka接收的数据保存7天,之后会被删除
log.retention.hours=168
# 单独数据块最大为1G，超过需要分块
log.segment.bytes=1073741824
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 允许删除topic
delete.topic.enable=true
4.2. kafka-2
vi /usr/local/server/kafka-2/config/server.properties
broker.id=2
listeners=PLAINTEXT://localhost:9092
log.dirs=/data/logs/kafka/kafka-2
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
#topic的默认分区数
num.partitions=1
# Kafka接收的数据保存7天,之后会被删除
log.retention.hours=168
# 单独数据块最大为1G，超过需要分块
log.segment.bytes=1073741824
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 允许删除topic
delete.topic.enable=true
4.3. kafka-3
vi /usr/local/server/kafka-3/config/server.properties
broker.id=3
listeners=PLAINTEXT://localhost:9093
log.dirs=/data/logs/kafka/kafka-3
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
#topic的默认分区数
num.partitions=1
# Kafka接收的数据保存7天,之后会被删除
log.retention.hours=168
# 单独数据块最大为1G，超过需要分块
log.segment.bytes=1073741824
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 允许删除topic
delete.topic.enable=true
5. kafka zookeeper.properties配置
5.1. kafka-1
vi /usr/local/server/kafka-1/config/zookeeper.properties
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/zk-1
dataLogDir=/data/logs/zookeeper/zk-1
clientPort=2181
server.1=localhost:12888:13888
server.2=localhost:22888:23888
server.3=localhost:32888:33888
5.1. kafka-2
vi /usr/local/server/kafka-2/config/zookeeper.properties
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/zk-2
dataLogDir=/data/logs/zookeeper/zk-2
clientPort=2182
server.1=localhost:12888:13888
server.2=localhost:22888:23888
server.3=localhost:32888:33888
5.3. kafka-3
vi /usr/local/server/kafka-3/config/zookeeper.properties
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/zk-3
dataLogDir=/data/logs/zookeeper/zk-3
clientPort=2183
server.1=localhost:12888:13888
server.2=localhost:22888:23888
server.3=localhost:32888:33888
6. zookeeper myid配置
cat/data/zookeeper/zk-1/myid
1
cat/data/zookeeper/zk-2/myid
2
cat/data/zookeeper/zk-3/myid
3
7. kafka启动、停止
7. 1 kafka后台启动
su - elk -c nohup -c "/usr/local/server/kafka-1/bin/kafka-server-start.sh /usr/local/server/kafka-1/config/server.properties" >/dev/null 2>&1 &
su - elk -c nohup -c "/usr/local/server/kafka-2/bin/kafka-server-start.sh /usr/local/server/kafka-2/config/server.properties" >/dev/null 2>&1 &
su - elk -c nohup -c "/usr/local/server/kafka-3/bin/kafka-server-start.sh /usr/local/server/kafka-3/config/server.properties" >/dev/null 2>&1 &
7. 2 kafka停止
/usr/local/server/kafka-1/bin/kafka-server-stop.sh
/usr/local/server/kafka-2/bin/kafka-server-stop.sh
/usr/local/server/kafka-3/bin/kafka-server-stop.sh
8. zookeeper启动、停止
8.1 zookeepre后台启动
su - elk -c nohup -c "/usr/local/server/kafka-1/bin/zookeeper-server-start.sh /usr/local/server/kafka-1/config/zookeeper.properties" >/dev/null 2>&1 &
su - elk -c nohup -c "/usr/local/server/kafka-2/bin/zookeeper-server-start.sh /usr/local/server/kafka-2/config/zookeeper.properties" >/dev/null 2>&1 &
su - elk -c nohup -c "/usr/local/server/kafka-3/bin/zookeeper-server-start.sh /usr/local/server/kafka-3/config/zookeeper.properties" >/dev/null 2>&1 &
8.2 zookeepre停止
/usr/local/server/kafka-1/bin/zookeeper-server-stop.sh
/usr/local/server/kafka-2/bin/zookeeper-server-stop.sh
/usr/local/server/kafka-3/bin/zookeeper-server-stop.sh
9. kafka使用案例
1.创建主题topics，OK
./kafka-topics.sh --create --topic liuTopic --replication-factor 1 --partitions 1 --zookeeper localhost:2181
2.查看主题topics，OK
./kafka-topics.sh --list --zookeeper localhost:2181
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic liuTopic
3.查看zook配置信息，OK
./zookeeper-shell.sh localhost:2181 <<< "get /brokers/ids/1"
./zookeeper-shell.sh localhost:2181 <<< "get /brokers/ids/myid的值"
4.使用producer生产者发生消息，OK
./kafka-console-producer.sh --broker-list PLAINTEXT://localhost:9091 --topic liuTopic
5.使用consumer消费者查看刚才发送的消息，OK
./kafka-console-consumer.sh --bootstrap-server PLAINTEXT://localhost:9091 --topic liuTopic --from-beginning
6.删除一个Topic
./kafka-topics.sh --delete --topic liuTopic --zookeeper localhost:2181
7.进到zookeeper的客户端，从客户端删掉想要删除的topic。
./zookeeper-shell.sh localhost:2181 # 登录zk客户端
ll # 查看zook支持的命令
ls /brokers/topics #查看现有的topic
ls /brokers/ids #查看brokers的id
ls /brokers/ids/0 #查看id为0的服务器的信息
rmr /brokers/topics/liuTopic # 删除主题liuTopic
10. 增值服务一键部署脚本
快速联系，tel/wx:15000227329
企业级一键部署脚本，增加kafka部署
https://blog.csdn.net/sdcxlgb/article/details/106934668
# 集群版，纯shell脚本编写，都看得懂、学得会，一键部署最多200台
https://item.taobao.com/item.htm?spm=a2oq0.12575281.0.0.50111debY9ELVa&ft=t&id=621966415353
# 单机版，纯shell脚本编写，都看得懂、学得会，一键部署最多9台
https://item.taobao.com/item.htm?spm=a2oq0.12575281.0.0.50111debTWMxWl&ft=t&id=622058094895
