python连接kafka并消费数据

连接kafka服务端查看topic

在idea里导入模块pykafka

编写代码

# 导入安装包
from pykafka import KafkaClient
# 设置客户端的连接信息
client = KafkaClient(hosts="hadoop01:9092,hadoop02:9092,hadoop03:9092")
# 打印所有的topic
print(client.topics)

查看所有的连接信息

# 设置循环次数
client = KafkaClient(hosts="hadoop01:9092,hadoop02:9092,hadoop03:9092")
for n in client.brokers:
    # 取host
    host = client.brokers[n].host
    # 取端口号
    port = client.brokers[n].port
    # 取broker.id
    id = client.brokers[n].id
    # 打印所有信息
    print("host=%s | port=%s | broker.id=%s " % (host, port, id))

测试生产者和消费者能否使用

在kafka命令行用已有topic开启生产者

bin/kafka-console-producer.sh --broker-list 192.168.147.136:9092,192.168.147.137:9092,192.168.147.138:9092 --topic test02

idea编写消费者代码

client = KafkaClient(hosts="hadoop01:9092,hadoop02:9092,hadoop03:9092")
# 确定使用的topic
topic = client.topics['test02']
consumer = topic.get_simple_consumer(
    # 设置消费者组，没有可不写
    consumer_group="consumer",
    # 设置从头开始消费
    reset_offset_on_start=True
)
# 获取被消费数据的偏移量和消费内容
for message in consumer:
    if message is not None:
    #打印消费信息
        print(message.offset, message.value)

若是消费不成功，这有可能是编码问题