PHP kafka初步测试

1.Mac安装Kafka并启动kafka

brew install kafka

==> Downloading https://homebrew.bintray.com/bottles/kafka-2.2.0.high_sierra.bottle.tar.gz
Already downloaded: /Users/guodong/Library/Caches/Homebrew/downloads/e72f921c1008a377e027235a0736c51d7ab4cfce6875b1f1eb2f19cf05156e95--kafka-2.2.0.high_sierra.bottle.tar.gz
==> Pouring kafka-2.2.0.high_sierra.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
  brew services start kafka
Or, if you don't want/need a background service you can just run:
  zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
  /usr/local/Cellar/kafka/2.2.0: 163 files, 54.4MB

kafka依赖zookeeper，先启动zookeeper，再启动kafka。

 zkServer start
 brew services start kafka
#或者希望在前台运行kafka用下面的命令
 zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

kafka测试

在/usr/local/bin目录下有许多kafka的脚本，可以直接用。

#在本地创建一个名字为topic的主题，生产者向主题里面写入数据
kafka-console-producer --broker-list localhost:9092 --topic topic
##开启另一个终端，打开消费者
kafka-console-consumer --bootstrap-server localhost:9092 --topic topic --from-beginning

image.png

生产者产生数据消费者可以收到了

php操作kafka

使用rdkafka。https://github.com/arnaud-lb/php-rdkafka,这是一个kafka的php扩展。
安装扩展前先安装
https://github.com/edenhill/librdkafka，这是kafka的底层类库。

brew install librdkafka #也可以卸载源代码，自己手动编译安装

安装完成后安装rdkafka
进入rdkafka目录后，执行phpize，对于电脑上有多个php的来说，要看phpize具体是那个文件。
我的电脑如下所示，本地的php被我删掉了，用的是mamp工具带的php

本地.png

下面指定配置路径，然后安装

./configure --with-php-config=/Applications/MAMP/bin/php/php7.1.20/bin/php-config 
make && make install
最后打开php.ini文件
增加extension=kafka.so。

测试

class Kafka
{
    public $brokerList = "localhost:9092";
    public $topic = "topic";
    public $partition = 0;
    public $logFile = "log.txt";

    protected $producer = null;
    protected $consumer = null;

    public function __construct()
    {
        if (empty($this->brokerList)){
            throw new Exception('broker empty',100100);
        }
        $rk = new RdKafka\Producer();
        if (empty($rk)){
            throw new Exception('producer empty',100101);

        }
        $rk->setLogLevel(LOG_DEBUG);
        if (!$rk->addBrokers($this->brokerList)){
            throw new Exception('add broker list error',100101);
        }
        $this->producer = $rk;

    }

    public function send($message)
    {
        $topic = $this->producer->newTopic($this->topic);
        $topic->produce(RD_KAFKA_PARTITION_UA, 0,  json_encode($message));
    }

    public function consumer(callable  $callback)
    {
        $conf = new \RdKafka\Conf();
        $conf->set('group.id',0);
        $conf->set('metadata.broker.list',$this->brokerList);
        $topicConf = new \Rdkafka\TopicConf();
        $topicConf->set('auto.offset.reset','smallest');
        $conf->setDefaultTopicConf($topicConf);
        $consumer = new \Rdkafka\KafkaConsumer($conf);
        $consumer->subscribe([$this->topic]);

        echo  "waiting for messages ....";
        while(true){
            $message = $consumer->consume(120*1000);
            sleep(1);
            $callback($message->payload);
        }
    }
}

一个生产者生成的消息可以被多个消费者消费。