Python连接Kafka问题汇总

1、NoBrokersAvailable: NoBrokersAvailable-Kafka Error

解决方法：连接时指定版本

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],api_version=(0,1,0))

参考链接： https://stackoverflow.com/questions/35689238/kafka-python-producer-is-not-able-to-connect/49212019#49212019

2、AssertionError: Snappy decompression unsupported

解决方法：安装pip install python-snappy

可能需要先安装snappy-dev

# download snappy
http://pkgs.fedoraproject.org/repo/pkgs/snappy/

# install snappy
tar xvfz snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure --prefix=/usr/local/snappy
make
make install

3、python读取kafka指定分区、偏移量、负载均衡（从头读取，数据量大的话，可能没响应）

from kafka import KafkaConsumer
from kafka.structs import TopicPartition

consumer = KafkaConsumer(group_id='test', bootstrap_servers=['192.168.9.1:9002'])
consumer.assign([TopicPartition(topic='test', partition=0), TopicPartition(topic='test', partition=1)])
print(consumer.partitions_for_topic("test"))  # 获取test主题的分区信息
print(consumer.assignment())
print(consumer.beginning_offsets(consumer.assignment()))
consumer.seek(TopicPartition(topic='test', partition=0), 0)
for msg in consumer:
    recv = "%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value)
    print(recv)

参考文献

https://blog.csdn.net/Beyond_F4/article/details/80310340

https://www.cnblogs.com/small-office/p/9399907.html