Chuye.Kafka: 清晰精练的 .Net driver for Apache Kafka

Chuye.Kafka: 清晰精练的 .Net driver for Apache Kafka

Jusfr 原创，转载请注明来自博客园

系列目录

• Chuye.Kafka: 从事件总线和消息队列说起
• Chuye.Kafka: 清晰精练的 .Net driver for Apache Kafka
• 协议部分: TopicMetadata
• [协议部分: Produce/Fetch/MessageSet] planned
• [Chuye.Kafka: BufferWriter/BufferReader] planned
• [协议部分: Offset/OffsetCommit/OffsetFetch] planned

我们的业务一直使用 MSMQ 作为消息中间件，初期确实满足了需求；此后业务增长，我们选择 Kafka 作为替代并基于其 0.9 版本API实现了 .NET 驱动见于 Github Chuye.Kafka。

已经完成的功能

• Core api、Group Membership API、Administrative API；
• 基于配置的 Partition-Consumer 路由；

部分设计点

• 多 Partition 下消费者选取 Partition 的问题，官方的语言是"A simple consumer client can be implemented by simply requiring that the partitions be specified in config, though this will not allow dynamic reassignment of partitions should that consumer fail. We hope to address this gap in the next major release." Chuye.Kafka 实现了一个甚于配置的路由规则。

• Kafka 认为high-level-api consumer 需要处理 Offset 自动推进和 Partition 均衡，但官方是通过 Zookeeper 实现的，本质上不是纯粹的客户端编程。Offset 的自动推进必然有策略问题：每处理一条消息就提交 OffsetCommit 会有性能问题；按条件触发即定时或已处理数量触发而有 Crash 时数据丢失的问题，我们认为官方的 ZookeeperConsumerConnector 掩盖了过多细节，Chuye.Kafka 并没有依样实现，而是交由消费者结合具体业务自主管理；

使用与示例

项目零依赖，你可以使用 NuGet package PM> Install-Package Chuye.Kafka ，也可以选择自行编译；

$/doc 里包含了两个 LINQPad 脚本 Chuye.Kafka-high-level-api.linq、 Chuye.Kafka-low-level-api.linq，修改为自己的服务器节点即可使用。以下是片断，更多示例及 Group Membership API 见于文档。

TopicMetadata

var section = new KafkaConfigurationSection("jusfr.kafka", 9092);
var demoTopics = new String[] { "demoTopic" };
//var demoTopics = new String[] { }; 
var connection = new Router(section); 
connection.TopicMetadata(demoTopics).Dump("Metadata");

Produce

var section = new KafkaConfigurationSection("jusfr.kafka", 9092);
const String demoTopic = "demoTopic";
var producer = new Producer(new Connection(section));
producer.Strategy = AcknowlegeStrategy.Written;
const Int32 count = 10;
for (int i = 0; i < count; i++) {
    var message = String.Concat(Guid.NewGuid().ToString("n"), "#", i);
    producer.Post(demoTopic, message);
}

Fetch

var section = new KafkaConfigurationSection("jusfr.kafka", 9092);
section.Buffer.ResponseBufferSize = 10 * 1024;
const String demoTopic = "demoTopic";
var consumer = new Consumer(new Router(section));
consumer.Fetch(demoTopic, 0).Dump("Fetch");

Offset

var section = new KafkaConfigurationSection("jusfr.kafka", 9092);
const String demoTopic = "demoTopic";
var consumer = new Consumer(new Router(section));
var earliest = consumer.Offset(demoTopic, OffsetTimeOption.Earliest);
var latest = consumer.Offset(demoTopic, OffsetTimeOption.Latest);
new { earliest, latest }.Dump();

OffsetCommit

var section = new KafkaConfigurationSection("jusfr.kafka", 9092);
const String demoTopic = "demoTopic";
const String demoConsumerGroup = "demoConsumerGroup";
var consumer = new Consumer(new Router(section));
consumer.OffsetCommit(demoTopic, demoConsumerGroup, 6); //your value

OffsetFetch

var section = new KafkaConfigurationSection("jusfr.kafka", 9092);
const String demoTopic = "demoTopic";
const String demoConsumerGroup = "demoConsumerGroup";
var consumer = new Consumer(new Router(section));
consumer.OffsetFetch(demoTopic, demoConsumerGroup).Dump("OffsetFetch");

示例配置

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="chuye.kafka" type="Chuye.Kafka.KafkaConfigurationSection, Chuye.Kafka" />
  </configSections>
  <chuye.kafka>
    <broker host="jusfr.kafka" port="9092" />
    <buffer maxBufferPoolSize="1048576" maxBufferSize="65536" requestBufferSize="4096" responseBufferSize="65536" />
    <topicPartitions>
      <add topic="demoTopic1" partition="0" />
      <add topic="demoTopic2" partition="1" />
    </topicPartitions>
  </chuye.kafka>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
  </startup>
</configuration>

根据此配置结合 Router对象的使用，客户端对 demoTopic1 的读写将落入分区 0 所在的 broker，对 demoTopic2 的读写将落入分区 1 所在的 broker。

待完成特性

• 消息压缩
• 更有扩展性的 Partition 路由能力；
• Group Membership API 的完善；
• 符合 Kafka 语义的 high level api consumer

Jusfr 原创，转载请注明来自博客园