kakfa 2.4.1 java的生产者client发送消息到服务端源码

一、消息添加到缓冲区的主函数

private final RecordAccumulator accumulator;

private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
      	    TopicPartition tp = null;
      	     //.......删除干扰理解的代码行
            int partition = this.partition(record, serializedKey, serializedValue, cluster);
            //这条消息发送到哪个分区这里就确定了
            tp = new TopicPartition(record.topic(), partition);
            //.......删除干扰理解的代码行
            //这里试图把消息追加到this.accumulator，因为是true，所以是追加
            RecordAppendResult result = this.accumulator.append(tp, timestamp, serializedKey, serializedValue, headers, interceptCallback, remainingWaitMs, true);
            //如果需要给缓冲区队列最追加一个新的批次对象，则新增一个，往这个新的批次对象中的集合添加数据
            if (result.abortForNewBatch) {
                int prevPartition = partition;
                this.partitioner.onNewBatch(record.topic(), cluster, partition);
                partition = this.partition(record, serializedKey, serializedValue, cluster);
                tp = new TopicPartition(record.topic(), partition);
                if (this.log.isTraceEnabled()) {
                    this.log.trace("Retrying append due to new batch creation for topic {} partition {}. The old partition was {}", new Object[]{record.topic(), partition, prevPartition});
                }
                interceptCallback = new KafkaProducer.InterceptorCallback(callback, this.interceptors, tp);
                result = this.accumulator.append(tp, timestamp, serializedKey, serializedValue, headers, interceptCallback, remainingWaitMs, false);
            }
           //.......删除干扰理解的代码行
           //如果缓冲区队列中的批次对象有满的了，或者创建新的批次对象了，则直接唤醒sender线程，开始发送数据
            if (result.batchIsFull || result.newBatchCreated) {
                this.log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
                this.sender.wakeup();
            }
            return result.future;
       //.......删除干扰理解的代码行
    }

1、在缓冲区队列最后一个批次对象数据中追加数据

 public RecordAccumulator.RecordAppendResult append(TopicPartition tp, long timestamp, byte[] key, byte[] value, Header[] headers, Callback callback, long maxTimeToBlock, boolean abortOnNewBatch) throws InterruptedException {
        //.......删除干扰理解的代码行
        RecordAccumulator.RecordAppendResult var13;
        try {
            Deque<ProducerBatch> dq = this.getOrCreateDeque(tp);
            synchronized(dq) {
				 //.......删除干扰理解的代码行
                RecordAccumulator.RecordAppendResult appendResult = this.tryAppend(timestamp, key, value, headers, callback, dq);
                //如果追加数据有结果，直接返回，没有返回下面默认的var13
                if (appendResult != null) {
                    RecordAccumulator.RecordAppendResult var15 = appendResult;
                    return var15;
                }
            }
 			//.......删除干扰理解的代码行
 			//RecordAppendResult构造函数的入参是（FutureRecordMetadata future, boolean batchIsFull, boolean newBatchCreated, boolean abortForNewBatch）
            var13 = new RecordAccumulator.RecordAppendResult((FutureRecordMetadata)null, false, false, true);
        } finally {
            //.......删除干扰理解的代码行
        }
        return var13;
    }

1.1 先从缓冲区队列中取出最后一个批次对象

/*
1、首先，代码从队列的末尾获取最后一个生产者批次对象last。
2、如果last不为空，表示队列中已经存在批次，那么代码会调用last.tryAppend()方法，尝试向该批次追加记录。
3、last.tryAppend()方法返回一个FutureRecordMetadata对象，表示追加记录的结果。如果追加成功，代码会创建一个RecordAccumulator.RecordAppendResult对象，并返回。
4、如果追加失败，代码会调用last.closeForRecordAppends()方法，关闭该批次的记录追加。
5、如果队列为空或者追加失败，代码会返回null。

*/
private RecordAccumulator.RecordAppendResult tryAppend(long timestamp, byte[] key, byte[] value, Header[] headers, Callback callback, Deque<ProducerBatch> deque) {
        ProducerBatch last = (ProducerBatch)deque.peekLast();
        if (last != null) {
            FutureRecordMetadata future = last.tryAppend(timestamp, key, value, headers, callback, this.time.milliseconds());
            if (future != null) {
                return new RecordAccumulator.RecordAppendResult(future, deque.size() > 1 || last.isFull(), false, false);
            }

            last.closeForRecordAppends();
        }

        return null;
    }

1.2 判断是否有足够的空间，有足够的空间执行追加逻辑，没有则返回null

//这里判断最后一个批次对象是否有足够的空间
 public FutureRecordMetadata tryAppend(long timestamp, byte[] key, byte[] value, Header[] headers, Callback callback, long now) {
        if (!this.recordsBuilder.hasRoomFor(timestamp, key, value, headers)) {
            return null;
        } else {
        	//说明空间足够，可以往旧的缓冲区队列追加数据
            Long checksum = this.recordsBuilder.append(timestamp, key, value, headers);
            this.maxRecordSize = Math.max(this.maxRecordSize, AbstractRecords.estimateSizeInBytesUpperBound(this.magic(), this.recordsBuilder.compressionType(), key, value, headers));
            this.lastAppendTime = now;
            FutureRecordMetadata future = new FutureRecordMetadata(this.produceFuture, (long)this.recordCount, timestamp, checksum, key == null ? -1 : key.length, value == null ? -1 : value.length, Time.SYSTEM);
            this.thunks.add(new ProducerBatch.Thunk(callback, future));
            ++this.recordCount;
            return future;
        }
    }

2、在缓冲区队列末尾新增一个批次对象(追加失败才执行新增)

仔细看一下上文中的append方法，即返回的是 var13 时，也就是缓冲区队列最后一个批次对象没有足够的空间时，才返回abortForNewBatch=true

private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
        TopicPartition tp = null;
		//.......删除干扰理解的代码行
            int partition = this.partition(record, serializedKey, serializedValue, cluster);
            tp = new TopicPartition(record.topic(), partition);
          //.......删除干扰理解的代码行
            RecordAppendResult result = this.accumulator.append(tp, timestamp, serializedKey, serializedValue, headers, interceptCallback, remainingWaitMs, true);
            //通过目录1知道，当队列最后一个批次对象没有足够的空间才会返回abortForNewBatch=true
            if (result.abortForNewBatch) {
                int prevPartition = partition;
                this.partitioner.onNewBatch(record.topic(), cluster, partition);
                partition = this.partition(record, serializedKey, serializedValue, cluster);
                tp = new TopicPartition(record.topic(), partition);
                if (this.log.isTraceEnabled()) {
                    this.log.trace("Retrying append due to new batch creation for topic {} partition {}. The old partition was {}", new Object[]{record.topic(), partition, prevPartition});
                }
                interceptCallback = new KafkaProducer.InterceptorCallback(callback, this.interceptors, tp);
                result = this.accumulator.append(tp, timestamp, serializedKey, serializedValue, headers, interceptCallback, remainingWaitMs, false);
            }
            if (result.batchIsFull || result.newBatchCreated) {
                this.log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
                this.sender.wakeup();
            }
            return result.future;
        
    }

2.1尝试新增一个批次对象，并且把新增的批次对象放入到缓冲区队列末尾

在新增之前，又走了两遍往队列最后一个批次对象追加的逻辑

 public RecordAccumulator.RecordAppendResult append(TopicPartition tp, long timestamp, byte[] key, byte[] value, Header[] headers, Callback callback, long maxTimeToBlock, boolean abortOnNewBatch) throws InterruptedException {
		//.......删除干扰理解的代码行
        RecordAccumulator.RecordAppendResult var13;
        try {
        	//第一遍追加逻辑
            Deque<ProducerBatch> dq = this.getOrCreateDeque(tp);
            synchronized(dq) {
                if (this.closed) {
                    throw new KafkaException("Producer closed while send in progress");
                }

                RecordAccumulator.RecordAppendResult appendResult = this.tryAppend(timestamp, key, value, headers, callback, dq);
                if (appendResult != null) {
                    RecordAccumulator.RecordAppendResult var15 = appendResult;
                    return var15;
                }
            }
			//新增批次对象，因为入参abortOnNewBatch为flase，取反为true
            if (!abortOnNewBatch) {
                byte maxUsableMagic = this.apiVersions.maxUsableProduceMagic();
                int size = Math.max(this.batchSize, AbstractRecords.estimateSizeInBytesUpperBound(maxUsableMagic, this.compression, key, value, headers));
                this.log.trace("Allocating a new {} byte message buffer for topic {} partition {}", new Object[]{size, tp.topic(), tp.partition()});
                buffer = this.free.allocate(size, maxTimeToBlock);
                synchronized(dq) {
                    if (this.closed) {
                        throw new KafkaException("Producer closed while send in progress");
                    }
					//第二遍追加逻辑
                    RecordAccumulator.RecordAppendResult appendResult = this.tryAppend(timestamp, key, value, headers, callback, dq);
                    if (appendResult != null) {
                        RecordAccumulator.RecordAppendResult var31 = appendResult;
                        return var31;
                    }
                    //确定无法追加，开始新增
                    MemoryRecordsBuilder recordsBuilder = this.recordsBuilder(buffer, maxUsableMagic);
                    ProducerBatch batch = new ProducerBatch(tp, recordsBuilder, this.time.milliseconds());
                    FutureRecordMetadata future = (FutureRecordMetadata)Objects.requireNonNull(batch.tryAppend(timestamp, key, value, headers, callback, this.time.milliseconds()));
                    //往队列末尾添加
                    dq.addLast(batch);
                    this.incomplete.add(batch);
                    buffer = null;
                    RecordAccumulator.RecordAppendResult var20 = new RecordAccumulator.RecordAppendResult(future, dq.size() > 1 || batch.isFull(), true, false);
                    return var20;
                }
            }
            var13 = new RecordAccumulator.RecordAppendResult((FutureRecordMetadata)null, false, false, true);
        } finally {
           //.......删除干扰理解的代码行
        }

        return var13;
    }

二、sender后台线程

通过上文其实也知道，kafka的生产端的消息不是来一条发一条，而是需要唤醒发送一批数据的线程

 private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) {
  			//.......删除干扰理解的代码行
            if (result.batchIsFull || result.newBatchCreated) {
                log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
                this.sender.wakeup();
            }
            return result.future;
            //.......删除干扰理解的代码行
       
    }

1、sender类实现了Runnable接口

public class Sender implements Runnable {
	      //.......删除干扰理解的代码行
}
 

1.1 sender通过run方法里的while循环来保证一直有后台线程执行发送操作

 public void run() {
        log.debug("Starting Kafka producer I/O thread.");
        // main loop, runs until close is called
        //只要程序不停止运行，sender这个线程会一直在while循环里执行runOnce
        while (running) {
            try {
                runOnce();
            } catch (Exception e) {
                log.error("Uncaught error in kafka producer I/O thread: ", e);
            }
        }
 		//.......删除干扰理解的代码行
 		//下面的是kafka客户端停止后会主动关闭连接
        try {
            this.client.close();
        } catch (Exception e) {
            log.error("Failed to close network client", e);
        }
        log.debug("Shutdown of Kafka producer I/O thread has completed.");
    }

1.2 发送完一批数据后，需要等kafka客户端的poll方法(使用Selector对象进行轮询操作)执行完或者被wakeup()唤醒开始下一个循环

void runOnce() {
       //.......删除干扰理解的代码行
        long currentTimeMs = time.milliseconds();
        //sendProducerData这个方法是实际发送的方法实现，这里不探讨，有兴趣的可也单独看看
        long pollTimeout = sendProducerData(currentTimeMs);
        //阻塞当前线程
        client.poll(pollTimeout, currentTimeMs);
    }
    public void wakeup() {
        this.client.wakeup();
    }

  @Override
    public List<ClientResponse> poll(long timeout, long now) {
		//.......删除干扰理解的代码行
        long metadataTimeout = metadataUpdater.maybeUpdate(now);
        try {
        	//this.selector.poll(Utils.min(timeout, metadataTimeout, defaultRequestTimeoutMs))：
        	//使用Selector对象进行轮询操作，等待I/O事件发生。等待时间取决于timeout、metadataTimeout和defaultRequestTimeoutMs的最小值。
            this.selector.poll(Utils.min(timeout, metadataTimeout, defaultRequestTimeoutMs));
        } catch (IOException e) {
            log.error("Unexpected error during I/O", e);
        }
    }

再往下就涉及到netty了，我这里还没有学习，就不往下继续看netty的源码了