Kafka生产者源码解析之一KafkaProducer

目录

Kafka生产者源码解析之一KafkaProducer
Kafka生产者源码解析之二RecordAccumulator
Kafka生产者源码解析之三NIO
Kafka生产者源码解析之四Sender
Kafka生产者源码解析之五小结

处女作

之前一直作为C端用户，受益良多，因此转型为B端，分享心得，努力做到不误人子弟。当然同时也想得到大神们的指正和解惑。因为最近在学习kafka，故以它开始。

程序开始

kafka版本： 2.1.1
kafka生产者都是以send方法开始的。

@Override
    public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
        // intercept the record, which can be potentially modified; this method does not throw exceptions
        ProducerRecord<K, V> interceptedRecord = this.interceptors.onSend(record);
        return doSend(interceptedRecord, callback);
    }

方法首先会进入拦截器集合ProducerInterceptors，
onSend方法是遍历拦截器的onSend方法。
拦截器的目的是将数据处理加工，kafka本身并没有给出默认的拦截器的实现。
因此，如果需要使用拦截器功能，必须自己实现 ProducerInterceptor 接口。

doSend方法

此方法为kafka生产者的核心方法。

/**
     * Implementation of asynchronously send a record to a topic.
     */
    private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
        // 首先创建一个主题分区类
        TopicPartition tp = null;
        try {
            throwIfProducerClosed();
            // first make sure the metadata for the topic is available
            ClusterAndWaitTime clusterAndWaitTime;
            try {
                clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs);
            } catch (KafkaException e) {
                if (metadata.isClosed())
                    throw new KafkaException("Producer closed while send in progress", e);
                throw e;
            }
            long remainingWaitMs = Math.max(0, maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs);
            Cluster cluster = clusterAndWaitTime.cluster;
            byte[] serializedKey;
            try {
                // 序列化key
                serializedKey = keySerializer.serialize(record.topic(), record.headers(), record.key());
            } catch (ClassCastException cce) {
                throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
                        " to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
                        " specified in key.serializer", cce);
            }
            byte[] serializedValue;
            try {
                // 序列化value
                serializedValue = valueSerializer.serialize(record.topic(), record.headers(), record.value());
            } catch (ClassCastException cce) {
                throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
                        " to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
                        " specified in value.serializer", cce);
            }
            // 获取消息对应的分区号
            int partition = partition(record, serializedKey, serializedValue, cluster);
            // 通过主题和分区信息构造主题分区对象，并赋值给开始创建的引用tp
            tp = new TopicPartition(record.topic(), partition);
			
			// 将消息的标题设置成只读
            setReadOnly(record.headers());
            Header[] headers = record.headers().toArray();

			// 计算消息序列化之后的大小
            int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
                    compressionType, serializedKey, serializedValue, headers);
            // 判断此大小是否大于配置文件设置的最大值，可通过设置 max.request.size 传入配置文件对象中
            ensureValidRecordSize(serializedSize);
            // 获取消息的时间戳，如果为空就取系统时间戳
            long timestamp = record.timestamp() == null ? time.milliseconds() : record.timestamp();
            log.trace("Sending record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
            // producer callback will make sure to call both 'callback' and interceptor callback
            Callback interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);

            if (transactionManager != null && transactionManager.isTransactional())
                transactionManager.maybeAddPartitionToTransaction(tp);
			
			// 消息会首先传到消息累加器中，并返回记录累加器的元数据
            RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
                    serializedValue, headers, interceptCallback, remainingWaitMs);
            // 判断返回结果中的两个标识（存消息的batch缓存满了或者新创建了一个batch）
            if (result.batchIsFull || result.newBatchCreated) {
                log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
                // 开启 NIO 同步非阻塞 模式
                this.sender.wakeup();
            }
            // 返回消息发送之后的结果
            return result.future;
            // handling exceptions and record the errors;
            // for API exceptions return them in the future,
            // for other exceptions throw directly
        } catch (ApiException e) {
            log.debug("Exception occurred during message send:", e);
            if (callback != null)
                callback.onCompletion(null, e);
            this.errors.record();
            this.interceptors.onSendError(record, tp, e);
            return new FutureFailure(e);
        } catch (InterruptedException e) {
            this.errors.record();
            this.interceptors.onSendError(record, tp, e);
            throw new InterruptException(e);
        } catch (BufferExhaustedException e) {
            this.errors.record();
            this.metrics.sensor("buffer-exhausted-records").record();
            this.interceptors.onSendError(record, tp, e);
            throw e;
        } catch (KafkaException e) {
            this.errors.record();
            this.interceptors.onSendError(record, tp, e);
            throw e;
        } catch (Exception e) {
            // we notify interceptor about all exceptions, since onSend is called before anything else in this method
            this.interceptors.onSendError(record, tp, e);
            throw e;
        }
    }

其中里面最重要的有两个方法：
其一是 waitOnMetadata，下面会介绍。
其二是 RecordAccumulator 的 append 方法，这个方法将在下一篇单独分析。

waitOnMetadata方法

 /**
     * Wait for cluster metadata including partitions for the given topic to be available.
     * @param topic The topic we want metadata for
     * @param partition A specific partition expected to exist in metadata, or null if there's no preference
     * @param maxWaitMs The maximum time in ms for waiting on the metadata
     * @return The cluster containing topic metadata and the amount of time we waited in ms
     * @throws KafkaException for all Kafka-related exceptions, including the case where this method is called after producer close
     */
    private ClusterAndWaitTime waitOnMetadata(String topic, Integer partition, long maxWaitMs) throws InterruptedException {
        // add topic to metadata topic list if it is not there already and reset expiry
        // 从元数据中获取节点信息
        Cluster cluster = metadata.fetch();

		// 判断节点中的无效主题是否包含当前主题
        if (cluster.invalidTopics().contains(topic))
            throw new InvalidTopicException(topic);

		// 将主题添加到元数据中，如果之前不存在此主题则会设置需要更新的标记为True
        metadata.add(topic);

		// 统计该节点该主题下当前的分区list大小
        Integer partitionsCount = cluster.partitionCountForTopic(topic);
        // Return cached metadata if we have it, and if the record's partition is either undefined
        // or within the known partition range
        //如果我们有缓存的元数据，并且记录的分区是未定义的或者在已知分区范围内，那么返回缓存的元数据
        if (partitionsCount != null && (partition == null || partition < partitionsCount))
            return new ClusterAndWaitTime(cluster, 0);

		// 记录开始时间戳
        long begin = time.milliseconds();
        long remainingWaitMs = maxWaitMs;
        long elapsed;
        // Issue metadata requests until we have metadata for the topic and the requested partition,
        // or until maxWaitTimeMs is exceeded. This is necessary in case the metadata
        // is stale and the number of partitions for this topic has increased in the meantime.
        do {
            if (partition != null) {
                log.trace("Requesting metadata update for partition {} of topic {}.", partition, topic);
            } else {
                log.trace("Requesting metadata update for topic {}.", topic);
            }
            // 此处再次添加主题，不知为何
            metadata.add(topic);
            // 获取元数据版本号
            int version = metadata.requestUpdate();
            // 开启 NIO 同步非阻塞 模式
            sender.wakeup();
            try {
            	// 等待元数据更新，直到当前版本大于我们所知的最后一个版本
                metadata.awaitUpdate(version, remainingWaitMs);
            } catch (TimeoutException ex) {
                // Rethrow with original maxWaitMs to prevent logging exception with remainingWaitMs
                throw new TimeoutException(
                        String.format("Topic %s not present in metadata after %d ms.",
                                topic, maxWaitMs));
            }
            // 重新获取更新后的元数据的节点信息
            cluster = metadata.fetch();
            // 计算出更新元数据消耗的时间
            elapsed = time.milliseconds() - begin;
            if (elapsed >= maxWaitMs) {
                throw new TimeoutException(partitionsCount == null ?
                        String.format("Topic %s not present in metadata after %d ms.",
                                topic, maxWaitMs) :
                        String.format("Partition %d of topic %s with partition count %d is not present in metadata after %d ms.",
                                partition, topic, partitionsCount, maxWaitMs));
            }
            if (cluster.unauthorizedTopics().contains(topic))
                throw new TopicAuthorizationException(topic);
            if (cluster.invalidTopics().contains(topic))
                throw new InvalidTopicException(topic);
            remainingWaitMs = maxWaitMs - elapsed;
            // 获取节点中该主题的分区数
            partitionsCount = cluster.partitionCountForTopic(topic);
        } while (partitionsCount == null || (partition != null && partition >= partitionsCount));

        return new ClusterAndWaitTime(cluster, elapsed);
    }

以上若有理解不对之处，望不佞赐教。

疑惑

1. waitOnMetadata 此方法开头就调用了 metadata.add(topic)， 可是在判断元数据没有缓存数据之后，又调用了 metadata.add(topic) ？

2. waitOnMetadata 方法里面调用了 sender.wakeup()，跟 doSend方法里面相同，都是NIO模式，channel 是 SocketChannel那为什么要调用两次？