转载-zookeeper在kafka中的作用

1)Broker注册

　　Broker在zookeeper中保存为一个临时节点，节点的路径是/brokers/ids/[brokerid],每个节点会保存对应broker的IP以及端口等信息.

　　2)Topic注册

　　在kafka中,一个topic会被分成多个区并被分到多个broker上，分区的信息以及broker的分布情况都保存在zookeeper中，根节点路径为/brokers/topics,每个topic都会在topics下建立独立的子节点，每个topic节点下都会包含分区以及broker的对应信息，例如下图中的状态

　　3)生产者负载均衡

　　当Broker启动时，会注册该Broker的信息，以及可订阅的topic信息。生产者通过注册在Broker以及Topic上的watcher动态的感知Broker以及Topic的分区情况，从而将Topic的分区动态的分配到broker上.

　　4)消费者

　　kafka有消费者分组的概念，每个分组中可以包含多个消费者，每条消息只会发给分组中的一个消费者，且每个分组之间是相互独立互不影响的。

　　5)消费者与分区的对应关系

　　对于每个消费者分组，kafka都会为其分配一个全局唯一的Group ID,分组内的所有消费者会共享该ID,kafka还会为每个消费者分配一个consumer ID,通常采用hostname:uuid的形式。在kafka的设计中规定，对于topic的每个分区，最多只能被一个消费者进行消费，也就是消费者与分区的关系是一对多的关系。消费者与分区的关系也被存储在zookeeper中节点的路劲为 /consumers/[group_id]/owners/[topic]/[broker_id-partition_id],该节点的内容就是消费者的Consumer ID

　　6)消费者负载均衡

　　消费者服务启动时，会创建一个属于消费者节点的临时节点，节点的路径为 /consumers/[group_id]/ids/[consumer_id],该节点的内容是该消费者订阅的Topic信息。每个消费者会对/consumers/[group_id]/ids节点注册Watcher监听器，一旦消费者的数量增加或减少就会触发消费者的负载均衡。消费者还会对/brokers/ids/[brokerid]节点进行监听，如果发现服务器的Broker服务器列表发生变化，也会进行消费者的负载均衡

　　7)消费者的offset

　　在kafka的消费者API分为两种(1)High Level Api：由zookeeper维护消费者的offset (2) Low Level API,自己的代码实现对offset的维护。由于自己维护offset往往比较复杂，所以多数情况下都是使用High Level的APIoffset在zookeeper中的节点路径为/consumers/[group_id]/offsets/[topic]/[broker_id-part_id],该节点的值就是对应的offset

　　二、kakfa在zookeeper中的结构

　　kafka.javaapi.consumer.ZookeeperConsumerConnector.java

　　/**

　　* This class handles the consumers interaction with zookeeper

　　*

　　* Directories:

　　* 1. Consumer id registry:

　　* /consumers/[group_id]/ids[consumer_id] -> topic1,...topicN

　　* A consumer has a unique consumer id within a consumer group. A consumer registers its id as an ephemeral znode

　　* and puts all topics that it subscribes to as the value of the znode. The znode is deleted when the client is gone.

　　* A consumer subscribes to event changes of the consumer id registry within its group.

　　*

　　* The consumer id is picked up from configuration, instead of the sequential id assigned by ZK. Generated sequential

　　* ids are hard to recover during temporary connection loss to ZK, since it's difficult for the client to figure out

　　* whether the creation of a sequential znode has succeeded or not. More details can be found at

　　*

　　* 2. Broker node registry:

　　* /brokers/[0...N] --> { "host" : "host:port",

　　* "topics" : {"topic1": ["partition1" ... "partitionN"], ...,

　　* "topicN": ["partition1" ... "partitionN"] } }

　　* This is a list of all present broker brokers. A unique logical node id is configured on each broker node. A broker

　　* node registers itself on start-up and creates a znode with the logical node id under /brokers. The value of the znode

　　* is a JSON String that contains (1) the host name and the port the broker is listening to, (2) a list of topics that

　　* the broker serves, (3) a list of logical partitions assigned to each topic on the broker.

　　* A consumer subscribes to event changes of the broker node registry.

　　*

　　* 3. Partition owner registry:

　　* /consumers/[group_id]/owner/[topic]/[broker_id-partition_id] --> consumer_node_id

　　* This stores the mapping before broker partitions and consumers. Each partition is owned by a unique consumer

　　* within a consumer group. The mapping is reestablished after each rebalancing.

　　*

　　* 4. Consumer offset tracking:

　　* /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] --> offset_counter_value

　　* Each consumer tracks the offset of the latest message consumed for each partition.

　　*

　　*/