kafka 部署使用

kafka 简单介绍
  • Produce
  • Consumer
  • Broker
  • Topic
    • partition
使用 docker 方式搭建
image.png
  • 搭建zookeeper 环境

    # docker pull zookeeper 
    # docker run -d --restart=always --name zookeeper -p 2181:2181 -v /etc/localtime:/etc/localtime \
                    --log-driver json-file --log-opt max-size=100m --log-opt max-file=2  \
                    zookeeper
    
  • 搭建kafka环境
    192.168.48.134 为我的宿主机ip,下面需要按照自己的环境做对应的修改。

    # docker pull wurstmeister/kafka
    # docker run -d --restart=always --name kafka1 -p 19092:9092 -v /etc/localtime:/etc/localtime \
                    --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
                    -e KAFKA_BROKER_ID=1 \
                    -e KAFKA_ZOOKEEPER_CONNECT=192.168.48.134:2181 \
                    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.48.134:19092 \
                    -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
                    wurstmeister/kafka
    
    # docker run -d --restart=always --name kafka2 -p 19093:9092 -v /etc/localtime:/etc/localtime \
                    --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
                    -e KAFKA_BROKER_ID=2 \
                    -e KAFKA_ZOOKEEPER_CONNECT=192.168.48.134:2181 \
                    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.48.134:19093 \
                    -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
                    wurstmeister/kafka
    
    # docker run -d --restart=always --name kafka3 -p 19094:9092 -v /etc/localtime:/etc/localtime \
                    --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
                    -e KAFKA_BROKER_ID=3 \
                    -e KAFKA_ZOOKEEPER_CONNECT=192.168.48.134:2181 \
                    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.48.134:19094 \
                    -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
                    wurstmeister/kafka
    
  • 验证是否安装成功

    # docker exec -ti kafka1 /bin/bash
    # cd /opt/kafka/bin
    
    // 创建topic:分区数为4,副本因子为3(3个broker)
    # ./kafka-topics.sh --zookeeper 192.168.48.134:2181 --create --topic test --replication-factor 3 --partitions 4 Created topic "test"
    # ./kafka-topics.sh --zookeeper 192.168.48.134:2181 --describe --topic test
      Topic: test TopicId: _A5YjE72QWeECBPEOFCquw PartitionCount: 4   ReplicationFactor: 3    Configs: 
        Topic: test   Partition: 0    Leader: 3   Replicas: 3,2,1 Isr: 3,2,1
        Topic: test   Partition: 1    Leader: 1   Replicas: 1,3,2 Isr: 1,3,2
        Topic: test   Partition: 2    Leader: 2   Replicas: 2,1,3 Isr: 2,1,3
        Topic: test   Partition: 3    Leader: 3   Replicas: 3,1,2 Isr: 3,1,2
    

    可以看出来,目前,3个kafka实例的集群环境已经搭建ok

    进一步,测试消息的发送和消费

    // 在kafka2 上发送消息
    # docker exec -ti kafka2 /bin/bash
    # cd /opt/kafka/bin
    # ./kafka-console-producer.sh --broker-list 192.168.48.134:19092,192.168.48.134:19093,192.168.48.134:19094 --topic test
    >hello, kafka!
    
    // 另启一个ssh 在kafka3 上消费消息
    # docker exec -ti kafka3 /bin/bash
    # cd /opt/kafka/bin
    # ./kafka-console-consumer.sh --bootstrap-server 192.168.48.134:19092,192.168.48.134:19093,192.168.48.134:19094 --topic test --from-beginning
    hello, kafka!
    
物理主机方式搭建

TODO


参考资料
1、https://www.cnblogs.com/answerThe/p/11267129.html#:~:text=%E5%9F%BA%E4%BA%8Edocker%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BAkafka%E9%9B%86%E7%BE%A4%EF%BC%88%E5%8D%95%E6%9C%BA%E7%89%88%EF%BC%89%201%20kafka%E4%BE%9D%E8%B5%96%E4%BA%8Ezookeeper%EF%BC%88%E4%BC%A0%E9%80%81%E9%97%A8https%3A%2F%20...%202%20%E5%90%AF%E5%8A%A8zookeeper%E5%AE%B9%E5%99%A8%EF%BC%88wurstmeister%2Fz%20...%203,4%20%E5%88%86%E5%88%AB%E8%BF%9B%E5%85%A5%E5%AE%B9%E5%99%A8kafka1%E3%80%81kafka2%E3%80%81kafka3%E5%88%9B%E5%BB%BA%20...%205%20%E6%B5%8B%E8%AF%95%E9%9B%86%E7%BE%A4%EF%BC%8C%E5%9C%A8kafka3%E4%B8%8A%E5%90%91test1%E5%8F%91%E9%80%81%E6%B6%88%E6%81%AF%EF%BC%8C%E5%9C%A8kaf%20...%206%20%E4%BE%9D%E6%AC%A1%E5%85%B3%E9%97%ADkafka2%E3%80%81test1%E5%90%8E%E6%9F%A5%E7%9C%8B%E9%9B%86%E7%BE%A4%E7%8A%B6%E6%80%81)

你可能感兴趣的:(kafka 部署使用)