1.Mac安装Kafka并启动kafka
brew install kafka
==> Downloading https://homebrew.bintray.com/bottles/kafka-2.2.0.high_sierra.bottle.tar.gz
Already downloaded: /Users/guodong/Library/Caches/Homebrew/downloads/e72f921c1008a377e027235a0736c51d7ab4cfce6875b1f1eb2f19cf05156e95--kafka-2.2.0.high_sierra.bottle.tar.gz
==> Pouring kafka-2.2.0.high_sierra.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
/usr/local/Cellar/kafka/2.2.0: 163 files, 54.4MB
kafka依赖zookeeper,先启动zookeeper,再启动kafka。
zkServer start
brew services start kafka
#或者希望在前台运行kafka用下面的命令
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
kafka测试
在/usr/local/bin目录下有许多kafka的脚本,可以直接用。
#在本地创建一个名字为topic的主题,生产者向主题里面写入数据
kafka-console-producer --broker-list localhost:9092 --topic topic
##开启另一个终端,打开消费者
kafka-console-consumer --bootstrap-server localhost:9092 --topic topic --from-beginning
生产者产生数据消费者可以收到了
php操作kafka
使用rdkafka。https://github.com/arnaud-lb/php-rdkafka,这是一个kafka的php扩展。
安装扩展前先安装
https://github.com/edenhill/librdkafka,这是kafka的底层类库。
brew install librdkafka #也可以卸载源代码,自己手动编译安装
安装完成后安装rdkafka
进入rdkafka目录后,执行phpize,对于电脑上有多个php的来说,要看phpize具体是那个文件。
我的电脑如下所示,本地的php被我删掉了,用的是mamp工具带的php
下面指定配置路径,然后安装
./configure --with-php-config=/Applications/MAMP/bin/php/php7.1.20/bin/php-config
make && make install
最后打开php.ini文件
增加extension=kafka.so。
测试
class Kafka
{
public $brokerList = "localhost:9092";
public $topic = "topic";
public $partition = 0;
public $logFile = "log.txt";
protected $producer = null;
protected $consumer = null;
public function __construct()
{
if (empty($this->brokerList)){
throw new Exception('broker empty',100100);
}
$rk = new RdKafka\Producer();
if (empty($rk)){
throw new Exception('producer empty',100101);
}
$rk->setLogLevel(LOG_DEBUG);
if (!$rk->addBrokers($this->brokerList)){
throw new Exception('add broker list error',100101);
}
$this->producer = $rk;
}
public function send($message)
{
$topic = $this->producer->newTopic($this->topic);
$topic->produce(RD_KAFKA_PARTITION_UA, 0, json_encode($message));
}
public function consumer(callable $callback)
{
$conf = new \RdKafka\Conf();
$conf->set('group.id',0);
$conf->set('metadata.broker.list',$this->brokerList);
$topicConf = new \Rdkafka\TopicConf();
$topicConf->set('auto.offset.reset','smallest');
$conf->setDefaultTopicConf($topicConf);
$consumer = new \Rdkafka\KafkaConsumer($conf);
$consumer->subscribe([$this->topic]);
echo "waiting for messages ....";
while(true){
$message = $consumer->consume(120*1000);
sleep(1);
$callback($message->payload);
}
}
}
一个生产者生成的消息可以被多个消费者消费。