Kafka producer客户端源码解析
Kafka producer客户端源码解析
为分析Kafka producer源码，我们先看一下producer发送消息的示例，通过示例跟踪源码，研究producer在整个发送过程中做了那些操作，以及架构设计
发送消息示例
构造属性对象
属性对象设置必要的Kafka producer端配置信息
//创建属性对象
Properties properties = new Properties();
//设置broker的地址，可以支持多个，这样当某些broker挂掉还可以使用其它broker。必须指定
properties.put("bootstrap.servers", "172.23.16.84:9092");
//发送到broker端的任何消息格式都必须是字节数组，因此需要进行序列化，指定key的序列化器，除使用默认提供的
//序列化器，也可以通过实现Serializer接口，创建自定义序列化器
properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//指定value部分的序列化器
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//指定Producer确认请求发送完成之前需要搜集到的反馈信息数量，0将不等待任何服务器的反馈直接认为发送成功
//; 1将等待leader节点写入本地日志，则认为发送成功，不关心follower节点同步情况，leader节点挂掉，但follwer节点又
//没来得及同步，将导致消息丢失;all或-1 leader节点将等待所有副本确认之后在确认这条消息发送成功
properties.put("acks", "-1");
//设置发送消息重试次数
properties.put("retries", 3);
//批次大小
properties.put("batch.size", 323840);
//延迟时间
properties.put("linger.ms", 10);
//缓冲大小
properties.put("buffer.memory", 33554432);
//最大阻塞时间
properties.put("max.block.ms", 3000);
//设置压缩算法,默认不压缩
properties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
构造生产者
使用上步构造的属性对象创建生产者
//构建生产者
Producer producer = new KafkaProducer(properties);
同步发送消息
调用生产者的send方法完成消息发送
// 同步发送消息
try {
producer.send(new ProducerRecord("my-topic",
Integer.toBinaryString(1), Integer.toString(1))).get();
} catch (InterruptedException e) {
throw new RuntimeException(e);
} catch (ExecutionException e) {
throw new RuntimeException(e);
}
关闭生产者
消息发送完毕后，如果不需要继续使用生产者，需要关闭，释放资源
//关闭生产者
producer.close();
发送消息源码解析
上面展示了一个典型的producer发送消息流程。针对上述流程我们来研究一下Kafka producer端发送消息源码。对于第一步的构造配置属性对象，并没有什么可以研究的，无非就是设置属性及对应的值。我们从第二步开始分析
使用属性对象构造生产者
调用KafkaProducer(properties)构造方法初始化生产者，此构造函数内部调用重载构造函数KafkaProducer(Properties properties, Serializer keySerializer, Serializer valueSerializer)完成初始化，其中keySerializer、valueSerializer指定key/value序列化类，会覆盖属性对象中设置的key/value序列化类
public KafkaProducer(Properties properties) {
this(properties, null, null);
}
public KafkaProducer(Properties properties, Serializer keySerializer, Serializer valueSerializer) {
this(Utils.propsToMap(properties), keySerializer, valueSerializer);
}
public KafkaProducer(Map configs, Serializer keySerializer, Serializer valueSerializer) {
this(new ProducerConfig(ProducerConfig.appendSerializerToConfig(configs, keySerializer, valueSerializer)),
keySerializer, valueSerializer, null, null, null, Time.SYSTEM);
}
接着继续调用重载构造函数KafkaProducer(Map
public static Map propsToMap(Properties properties) {
Map map = new HashMap<>(properties.size());
for (Map.Entry entry : properties.entrySet()) {
if (entry.getKey() instanceof String) {
String k = (String) entry.getKey();
map.put(k, properties.get(k));
} else {
throw new ConfigException(entry.getKey().toString(), entry.getValue(), "Key must be a string.");
}
}
return map;
}
接着继续调用重载构造函数KafkaProducer(ProducerConfig config, Serializer keySerializer, Serializer valueSerializer, ProducerMetadata metadata, KafkaClient kafkaClient,ProducerInterceptors
-
首先调用ProducerConfig.appendSerializerToConfig(Map
-
之后调用ProducerConfig(Map
ProducerConfig构造过程
上述第二步中调用ProducerConfig(Map
static {
CONFIG = new ConfigDef().define(BOOTSTRAP_SERVERS_CONFIG, Type.LIST, Collections.emptyList(), new ConfigDef.NonNullValidator(), Importance.HIGH, CommonClientConfigs.BOOTSTRAP_SERVERS_DOC)
.define(CLIENT_DNS_LOOKUP_CONFIG,
Type.STRING,
ClientDnsLookup.USE_ALL_DNS_IPS.toString(),
in(ClientDnsLookup.DEFAULT.toString(),
ClientDnsLookup.USE_ALL_DNS_IPS.toString(),
ClientDnsLookup.RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY.toString()),
Importance.MEDIUM,
CommonClientConfigs.CLIENT_DNS_LOOKUP_DOC)
.define(BUFFER_MEMORY_CONFIG, Type.LONG, 32 * 1024 * 1024L, atLeast(0L), Importance.HIGH, BUFFER_MEMORY_DOC)
.define(RETRIES_CONFIG, Type.INT, Integer.MAX_VALUE, between(0, Integer.MAX_VALUE), Importance.HIGH, RETRIES_DOC)
.define(ACKS_CONFIG,
Type.STRING,
"1",
in("all", "-1", "0", "1"),
Importance.HIGH,
ACKS_DOC)
.define(COMPRESSION_TYPE_CONFIG, Type.STRING, "none", Importance.HIGH, COMPRESSION_TYPE_DOC)
.define(BATCH_SIZE_CONFIG, Type.INT, 16384, atLeast(0), Importance.MEDIUM, BATCH_SIZE_DOC)
.define(LINGER_MS_CONFIG, Type.LONG, 0, atLeast(0), Importance.MEDIUM, LINGER_MS_DOC)
.define(DELIVERY_TIMEOUT_MS_CONFIG, Type.INT, 120 * 1000, atLeast(0), Importance.MEDIUM, DELIVERY_TIMEOUT_MS_DOC)
.define(CLIENT_ID_CONFIG, Type.STRING, "", Importance.MEDIUM, CommonClientConfigs.CLIENT_ID_DOC)
.define(SEND_BUFFER_CONFIG, Type.INT, 128 * 1024, atLeast(CommonClientConfigs.SEND_BUFFER_LOWER_BOUND), Importance.MEDIUM, CommonClientConfigs.SEND_BUFFER_DOC)
.define(RECEIVE_BUFFER_CONFIG, Type.INT, 32 * 1024, atLeast(CommonClientConfigs.RECEIVE_BUFFER_LOWER_BOUND), Importance.MEDIUM, CommonClientConfigs.RECEIVE_BUFFER_DOC)
.define(MAX_REQUEST_SIZE_CONFIG,
Type.INT,
1024 * 1024,
atLeast(0),
Importance.MEDIUM,
MAX_REQUEST_SIZE_DOC)
.define(RECONNECT_BACKOFF_MS_CONFIG, Type.LONG, 50L, atLeast(0L), Importance.LOW, CommonClientConfigs.RECONNECT_BACKOFF_MS_DOC)
.define(RECONNECT_BACKOFF_MAX_MS_CONFIG, Type.LONG, 1000L, atLeast(0L), Importance.LOW, CommonClientConfigs.RECONNECT_BACKOFF_MAX_MS_DOC)
.define(RETRY_BACKOFF_MS_CONFIG, Type.LONG, 100L, atLeast(0L), Importance.LOW, CommonClientConfigs.RETRY_BACKOFF_MS_DOC)
.define(MAX_BLOCK_MS_CONFIG,
Type.LONG,
60 * 1000,
atLeast(0),
Importance.MEDIUM,
MAX_BLOCK_MS_DOC)
.define(REQUEST_TIMEOUT_MS_CONFIG,
Type.INT,
30 * 1000,
atLeast(0),
Importance.MEDIUM,
REQUEST_TIMEOUT_MS_DOC)
.define(METADATA_MAX_AGE_CONFIG, Type.LONG, 5 * 60 * 1000, atLeast(0), Importance.LOW, METADATA_MAX_AGE_DOC)
.define(METADATA_MAX_IDLE_CONFIG,
Type.LONG,
5 * 60 * 1000,
atLeast(5000),
Importance.LOW,
METADATA_MAX_IDLE_DOC)
.define(METRICS_SAMPLE_WINDOW_MS_CONFIG,
Type.LONG,
30000,
atLeast(0),
Importance.LOW,
CommonClientConfigs.METRICS_SAMPLE_WINDOW_MS_DOC)
.define(METRICS_NUM_SAMPLES_CONFIG, Type.INT, 2, atLeast(1), Importance.LOW, CommonClientConfigs.METRICS_NUM_SAMPLES_DOC)
.define(METRICS_RECORDING_LEVEL_CONFIG,
Type.STRING,
Sensor.RecordingLevel.INFO.toString(),
in(Sensor.RecordingLevel.INFO.toString(), Sensor.RecordingLevel.DEBUG.toString(), Sensor.RecordingLevel.TRACE.toString()),
Importance.LOW,
CommonClientConfigs.METRICS_RECORDING_LEVEL_DOC)
.define(METRIC_REPORTER_CLASSES_CONFIG,
Type.LIST,
Collections.emptyList(),
new ConfigDef.NonNullValidator(),
Importance.LOW,
CommonClientConfigs.METRIC_REPORTER_CLASSES_DOC)
.define(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,
Type.INT,
5,
atLeast(1),
Importance.LOW,
MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_DOC)
.define(KEY_SERIALIZER_CLASS_CONFIG,
Type.CLASS,
Importance.HIGH,
KEY_SERIALIZER_CLASS_DOC)
.define(VALUE_SERIALIZER_CLASS_CONFIG,
Type.CLASS,
Importance.HIGH,
VALUE_SERIALIZER_CLASS_DOC)
.define(SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG,
Type.LONG,
CommonClientConfigs.DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT_MS,
Importance.MEDIUM,
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_DOC)
.define(SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG,
Type.LONG,
CommonClientConfigs.DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS,
Importance.MEDIUM,
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_DOC)
/* default is set to be a bit lower than the server default (10 min), to avoid both client and server closing connection at same time */
.define(CONNECTIONS_MAX_IDLE_MS_CONFIG,
Type.LONG,
9 * 60 * 1000,
Importance.MEDIUM,
CommonClientConfigs.CONNECTIONS_MAX_IDLE_MS_DOC)
.define(PARTITIONER_CLASS_CONFIG,
Type.CLASS,
DefaultPartitioner.class,
Importance.MEDIUM, PARTITIONER_CLASS_DOC)
.define(INTERCEPTOR_CLASSES_CONFIG,
Type.LIST,
Collections.emptyList(),
new ConfigDef.NonNullValidator(),
Importance.LOW,
INTERCEPTOR_CLASSES_DOC)
.define(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG,
Type.STRING,
CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
Importance.MEDIUM,
CommonClientConfigs.SECURITY_PROTOCOL_DOC)
.define(SECURITY_PROVIDERS_CONFIG,
Type.STRING,
null,
Importance.LOW,
SECURITY_PROVIDERS_DOC)
.withClientSslSupport()
.withClientSaslSupport()
.define(ENABLE_IDEMPOTENCE_CONFIG,
Type.BOOLEAN,
false,
Importance.LOW,
ENABLE_IDEMPOTENCE_DOC)
.define(TRANSACTION_TIMEOUT_CONFIG,
Type.INT,
60000,
Importance.LOW,
TRANSACTION_TIMEOUT_DOC)
.define(TRANSACTIONAL_ID_CONFIG,
Type.STRING,
null,
new ConfigDef.NonEmptyString(),
Importance.LOW,
TRANSACTIONAL_ID_DOC)
.defineInternal(AUTO_DOWNGRADE_TXN_COMMIT,
Type.BOOLEAN,
false,
Importance.LOW);
}
看似CONFIG初始化很复杂，其实其初始化过程就是在不断定义各种配置的默认值、类型等内容，为用户设置配置属性做类型转换及默认值设置等操作做准备。在AbstractConfig(ConfigDef definition, Map originals)中继续调用其重载构造函数AbstractConfig(ConfigDef definition, Map originals, Map
public AbstractConfig(ConfigDef definition, Map originals, Map configProviderProps, boolean doLog) {
/* check that all the keys are really strings */
for (Map.Entry entry : originals.entrySet())
if (!(entry.getKey() instanceof String))
throw new ConfigException(entry.getKey().toString(), entry.getValue(), "Key must be a string.");
this.originals = resolveConfigVariables(configProviderProps, (Map) originals);
this.values = definition.parse(this.originals);
this.used = Collections.synchronizedSet(new HashSet<>());
Map configUpdates = postProcessParsedConfig(Collections.unmodifiableMap(this.values));
for (Map.Entry update : configUpdates.entrySet()) {
this.values.put(update.getKey(), update.getValue());
}
definition.parse(this.values);
this.definition = definition;
if (doLog)
logAll();
}
其主要做了如下操作
-
判断用户设置参数key是否为字符串类型，如果不是则抛出ConfigException异常
-
调用resolveConfigVariables方法完成ConfigProvider功能支持，实现调用用户设置的provider获取配置并更新用户配置参数值，赋值originals属性
-
之后对上一步处理过后的配置参数originals进行解析，调用ConfigDef的parse方法传入originals，definition中定义了合法的参数及其类型等信息，我们接下来看一下parse方法
public MapundefinedConfigKeys = undefinedDependentConfigs(); if (!undefinedConfigKeys.isEmpty()) { String joined = Utils.join(undefinedConfigKeys, ","); throw new ConfigException("Some configurations in are referred in the dependents, but not defined: " + joined); } // parse all known keys Map -
获取未被定义的配置参数，其实就是看configKeys中key不包含其对应value的dependents属性的值。configKeys是在静态代码块中初始化的
-
如果存在这样的数据则抛出异常
-
循环configKeys的value，从props获取对应参数的值放入values中。我们可以看到values为Map结构，其中key为ConfigKey对象中的name，也即支持的参数，value通过parseValue方法获取，第一个参数为ConfigKey对象key，第二个参数是从props用户参数中获取的支持的参数的值，第三个参数如果用户参数中配置了该参数则为true。在parseValue方法中执行逻辑是，如果用户设置了该参数则调用parseType方法，解析配置参数值转换为指定的类型；如果用户没有设置此参数，则判断是否有默认值，有默认值则获取默认值，没有则抛出异常，如果该参数对于Kafka来说是需要校验的则会调用校验方法校验
-
-
初始化一个线程安全Set赋值给used属性
-
postProcessParsedConfig方法，对参数默认值进行处理。比如用户没有设置reconnect.backoff.max.ms参数值，但设置了reconnect.backoff.ms参数值，则默认将reconnect.backoff.max.ms参数值设置为reconnect.backoff.ms参数值；如果用户设置了transactional.id，但没有设置enable.idempotence参数值, 则默认设置enable.idempotence为true
-
通过循环将上步得到的默认值设置放到配置参数Map中
-
再次调用parse方法
-
将处理过后的definition赋值给当前对象的属性definition
-
打印参数到控制台
总结：上面步骤完成了ProducerConfig的创建工作
其中resolveConfigVariables方法源码如下
private Map resolveConfigVariables(Map configProviderProps, Map originals) {
Map providerConfigString;
Map configProperties;
Map resolvedOriginals = new HashMap<>();
Map indirectVariables = extractPotentialVariables(originals);
resolvedOriginals.putAll(originals);
if (configProviderProps == null || configProviderProps.isEmpty()) {
providerConfigString = indirectVariables;
configProperties = originals;
} else {
providerConfigString = extractPotentialVariables(configProviderProps);
configProperties = configProviderProps;
}
Map providers = instantiateConfigProviders(providerConfigString, configProperties);
if (!providers.isEmpty()) {
ConfigTransformer configTransformer = new ConfigTransformer(providers);
ConfigTransformerResult result = configTransformer.transform(indirectVariables);
if (!result.data().isEmpty()) {
resolvedOriginals.putAll(result.data());
}
}
providers.values().forEach(x -> Utils.closeQuietly(x, "config provider"));
return new ResolvingMap<>(resolvedOriginals, originals);
}
完成如下操作
-
通过extractPotentialVariables方法解析用户设置配置参数中值为String类型的参数并赋值給indirectVariables
-
resolvedOriginals局部变量拷贝用户参数
-
configProviderProps为空，执行将indirectVariables赋值給providerConfigString局部变量，将用户参数赋值configProperties
-
调用instantiateConfigProviders实例化配置提供者，此提供者主要用于获取密钥，比如从环境变量中获取配置但密钥信息，通常Docker、K8s都是通过环境变量指定密钥信息
-
通过上步尝试获取provider，判断如果provider不为空，则先初始化一个ConfigTransformer，之后调用ConfigTransformer的transform方法，通过provider获取配置参数，并替换变量部分
-
使用替换后的参数更新resolvedOriginals
-
循环关闭provider
-
使用resolvedOriginals、originals构建返回对象ResolvingMap
其中instantiateConfigProviders源码如下
private Map instantiateConfigProviders(Map indirectConfigs, Map providerConfigProperties) {
final String configProviders = indirectConfigs.get(CONFIG_PROVIDERS_CONFIG);
if (configProviders == null || configProviders.isEmpty()) {
return Collections.emptyMap();
}
Map providerMap = new HashMap<>();
for (String provider: configProviders.split(",")) {
String providerClass = CONFIG_PROVIDERS_CONFIG + "." + provider + ".class";
if (indirectConfigs.containsKey(providerClass))
providerMap.put(provider, indirectConfigs.get(providerClass));
}
// Instantiate Config Providers
Map configProviderInstances = new HashMap<>();
for (Map.Entry entry : providerMap.entrySet()) {
try {
String prefix = CONFIG_PROVIDERS_CONFIG + "." + entry.getKey() + CONFIG_PROVIDERS_PARAM;
Map configProperties = configProviderProperties(prefix, providerConfigProperties);
ConfigProvider provider = Utils.newInstance(entry.getValue(), ConfigProvider.class);
provider.configure(configProperties);
configProviderInstances.put(entry.getKey(), provider);
} catch (ClassNotFoundException e) {
log.error("ClassNotFoundException exception occurred: " + entry.getValue());
throw new ConfigException("Invalid config:" + entry.getValue() + " ClassNotFoundException exception occurred", e);
}
}
return configProviderInstances;
}
完成如下操作
-
从传入的值为字符串类型的参数中获取用户设置的config.providers参数值
-
如果没有设置此参数则返回空集合
-
如果设置了此参数，则将参数值按逗号分割，循环分割后的每一个provider名称，构造providerClass格式
config.providers.并判断indirectConfigs参数中是否包含key为providerClass的参数，如果包含，则将provider名称、对应真实的provider全路径类名放入providerMap集合中.class -
循环providerMap，构造prefix，格式为
config.providers.。调用configProviderProperties(prefix, providerConfigProperties)方法解析出用户配置参数中以prefix为前缀的key，并截取后面部分，将截取后的key和value放入新的map中赋值给configProperties.param. -
调用Utils.newInstance(entry.getValue(), ConfigProvider.class)方法，根据provider的全路径名实例化provider
-
设置provider的config configProperties
-
将provider名称和对应的实例放入map中返回
ConfigTransformer的transform方法源码如下
public ConfigTransformerResult transform(Map configs) {
Map>> keysByProvider = new HashMap<>();
Map>> lookupsByProvider = new HashMap<>();
// Collect the variables from the given configs that need transformation
for (Map.Entry config : configs.entrySet()) {
if (config.getValue() != null) {
List vars = getVars(config.getValue(), DEFAULT_PATTERN);
for (ConfigVariable var : vars) {
Map> keysByPath = keysByProvider.computeIfAbsent(var.providerName, k -> new HashMap<>());
Set keys = keysByPath.computeIfAbsent(var.path, k -> new HashSet<>());
keys.add(var.variable);
}
}
}
// Retrieve requested variables from the ConfigProviders
Map ttls = new HashMap<>();
for (Map.Entry>> entry : keysByProvider.entrySet()) {
String providerName = entry.getKey();
ConfigProvider provider = configProviders.get(providerName);
Map> keysByPath = entry.getValue();
if (provider != null && keysByPath != null) {
for (Map.Entry> pathWithKeys : keysByPath.entrySet()) {
String path = pathWithKeys.getKey();
Set keys = new HashSet<>(pathWithKeys.getValue());
ConfigData configData = provider.get(path, keys);
Map data = configData.data();
Long ttl = configData.ttl();
if (ttl != null && ttl >= 0) {
ttls.put(path, ttl);
}
Map> keyValuesByPath =
lookupsByProvider.computeIfAbsent(providerName, k -> new HashMap<>());
keyValuesByPath.put(path, data);
}
}
}
// Perform the transformations by performing variable replacements
Map data = new HashMap<>(configs);
for (Map.Entry config : configs.entrySet()) {
data.put(config.getKey(), replace(lookupsByProvider, config.getValue(), DEFAULT_PATTERN));
}
return new ConfigTransformerResult(data, ttls);
}
其完成如下操作
-
循环传入的用户配置Map config，此map为value类型为String类型的配置参数，如果配置参数的value不为空，则尝试获取匹配value值，匹配模式为
\$\{([^}]*?):(([^}]*?):)?([^}]*?)\}即匹配格式为${mycustom:/path/pass/to/get/method:password}的value值，并解析出第一部分mycustom 作为providerName；第二部分/path/pass/to/get/method作为path，path也可能为空；第三部分password作为variable，用于构造ConfigVariable -
循环上步解析到的ConfigVariable集合，将其解析到keysByProvider Map结构中，结构为Map
-
循环上一步解析得到的keysByProvider，获取最外层Map的key即provider名称，从configProviders中获取provider名称对应的provider实例。获取外层Map的value值即path与variable集合的键值对赋值keysByPath，如果两者都不为空则进一步循环keysByPath，获取path，及对应的variable集合赋值keys，调用provider实例的get方法传入path、keys，获取解析的configData对象。provider实例其实是指继承ConfigProvider接口的类的实例化对象，我们可以根据自己需要实现ConfigProvider接口，实现自己获取配置参数的方式，例如通过环境变量获取。此处我以官方的示例FileConfigProvider讲解get(String path, Set keys)方法的实现
public ConfigData get(String path, Setkeys) { Map -
首先判断path是否为空，如果为空直接返回ConfigData实例其data属性为空Map。因为path并不是必须的，如果我们要实现从环境变量中获取配置，其实配不配path没有任何影响，但此处FileConfigProvider就是从path路径加载配置，因此没有直接返回空
-
path不为空继续之后逻辑，通过path文件路径构造Reader，直接通过properties加载reader内容，之后循环keys，从properties获取key对应的配置值，并放入data中，data为Map结构，key为循环的key，value为配置的key对应的配置值。之后通过获取的data构造ConfigData并返回
-
-
获取configData的data属性及ttl属性，如果ttl不为空且大于0，则将ttl与path关联，存储在Map结构中，ttl表示数据的生存时间，单位为秒。初始化lookupsByProvider，将provider名称与path及对应参数列表建立关联存入lookupsByProvider的Map结构中
-
接下来通过上面解析出来的数据替换正则表达式
\\$\\{([^}]*?):(([^}]*?):)?([^}]*?)\\}匹配的部分。替换方法为replace(Mapprivate static String replace(Map- 首先解析出满足正则表达试的部分，然后根据之前provider解析出的配置参数，替换匹配正则表达式的部分，之后便得到替换后的字符串，覆盖之前包含的老值
-
将替换之后的用户配置参数、及上面解析出的ttls用于构造返回对象ConfigTransformerResult
完成生产者创建
在上述步骤中完成ProducerConfig创建后就正式进入创建生产者的流程
KafkaProducer(ProducerConfig config,
Serializer keySerializer,
Serializer valueSerializer,
ProducerMetadata metadata,
KafkaClient kafkaClient,
ProducerInterceptors interceptors,
Time time) {
try {
this.producerConfig = config;
this.time = time;
String transactionalId = config.getString(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
this.clientId = config.getString(ProducerConfig.CLIENT_ID_CONFIG);
LogContext logContext;
if (transactionalId == null)
logContext = new LogContext(String.format("[Producer clientId=%s] ", clientId));
else
logContext = new LogContext(String.format("[Producer clientId=%s, transactionalId=%s] ", clientId, transactionalId));
log = logContext.logger(KafkaProducer.class);
log.trace("Starting the Kafka producer");
Map metricTags = Collections.singletonMap("client-id", clientId);
MetricConfig metricConfig = new MetricConfig().samples(config.getInt(ProducerConfig.METRICS_NUM_SAMPLES_CONFIG))
.timeWindow(config.getLong(ProducerConfig.METRICS_SAMPLE_WINDOW_MS_CONFIG), TimeUnit.MILLISECONDS)
.recordLevel(Sensor.RecordingLevel.forName(config.getString(ProducerConfig.METRICS_RECORDING_LEVEL_CONFIG)))
.tags(metricTags);
List reporters = config.getConfiguredInstances(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG,
MetricsReporter.class,
Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
JmxReporter jmxReporter = new JmxReporter();
jmxReporter.configure(config.originals(Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)));
reporters.add(jmxReporter);
MetricsContext metricsContext = new KafkaMetricsContext(JMX_PREFIX,
config.originalsWithPrefix(CommonClientConfigs.METRICS_CONTEXT_PREFIX));
this.metrics = new Metrics(metricConfig, reporters, time, metricsContext);
this.partitioner = config.getConfiguredInstance(
ProducerConfig.PARTITIONER_CLASS_CONFIG,
Partitioner.class,
Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
long retryBackoffMs = config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG);
if (keySerializer == null) {
this.keySerializer = config.getConfiguredInstance(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
Serializer.class);
this.keySerializer.configure(config.originals(Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)), true);
} else {
config.ignore(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
this.keySerializer = keySerializer;
}
if (valueSerializer == null) {
this.valueSerializer = config.getConfiguredInstance(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
Serializer.class);
this.valueSerializer.configure(config.originals(Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId)), false);
} else {
config.ignore(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
this.valueSerializer = valueSerializer;
}
List> interceptorList = (List) config.getConfiguredInstances(
ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
ProducerInterceptor.class,
Collections.singletonMap(ProducerConfig.CLIENT_ID_CONFIG, clientId));
if (interceptors != null)
this.interceptors = interceptors;
else
this.interceptors = new ProducerInterceptors<>(interceptorList);
ClusterResourceListeners clusterResourceListeners = configureClusterResourceListeners(keySerializer,
valueSerializer, interceptorList, reporters);
this.maxRequestSize = config.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG);
this.totalMemorySize = config.getLong(ProducerConfig.BUFFER_MEMORY_CONFIG);
this.compressionType = CompressionType.forName(config.getString(ProducerConfig.COMPRESSION_TYPE_CONFIG));
this.maxBlockTimeMs = config.getLong(ProducerConfig.MAX_BLOCK_MS_CONFIG);
int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
this.apiVersions = new ApiVersions();
this.transactionManager = configureTransactionState(config, logContext);
this.accumulator = new RecordAccumulator(logContext,
config.getInt(ProducerConfig.BATCH_SIZE_CONFIG),
this.compressionType,
lingerMs(config),
retryBackoffMs,
deliveryTimeoutMs,
metrics,
PRODUCER_METRIC_GROUP_NAME,
time,
apiVersions,
transactionManager,
new BufferPool(this.totalMemorySize, config.getInt(ProducerConfig.BATCH_SIZE_CONFIG), metrics, time, PRODUCER_METRIC_GROUP_NAME));
List addresses = ClientUtils.parseAndValidateAddresses(
config.getList(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG),
config.getString(ProducerConfig.CLIENT_DNS_LOOKUP_CONFIG));
if (metadata != null) {
this.metadata = metadata;
} else {
this.metadata = new ProducerMetadata(retryBackoffMs,
config.getLong(ProducerConfig.METADATA_MAX_AGE_CONFIG),
config.getLong(ProducerConfig.METADATA_MAX_IDLE_CONFIG),
logContext,
clusterResourceListeners,
Time.SYSTEM);
this.metadata.bootstrap(addresses);
}
this.errors = this.metrics.sensor("errors");
this.sender = newSender(logContext, kafkaClient, this.metadata);
String ioThreadName = NETWORK_THREAD_PREFIX + " | " + clientId;
this.ioThread = new KafkaThread(ioThreadName, this.sender, true);
this.ioThread.start();
config.logUnused();
AppInfoParser.registerAppInfo(JMX_PREFIX, clientId, metrics, time.milliseconds());
log.debug("Kafka producer started");
} catch (Throwable t) {
// call close methods if internal objects are already constructed this is to prevent resource leak. see KAFKA-2121
close(Duration.ofMillis(0), true);
// now propagate the exception
throw new KafkaException("Failed to construct kafka producer", t);
}
}
-
对创建的KafkaProducer生产者设置属性值，将传入的config设置给producerConfig属性，将当前系统时间设置给time属性，将client.id参数值设置给clientId属性
-
创建MetricConfig，设置相应属性，包括设置样本数、窗口时间、记录级别、tag，这里的tag是客户端id，创建JmxReporter、MetricsContext，将前面创建的与指标监控相关的对象用于创建Metrics对象并赋值给metrics属性
-
获取分区器并初始化给partitioner属性
-
如果keySerializer参数为空，则从参数中获取配置的key.serializer，并初始化给keySerializer属性。再调用keySerializer的configure方法完成encoding解析设置；如果不为空，则直接将参数keySerializer赋值给属性keySerializer属性
-
同上步一样设置valueSerializer属性值
-
传入的interceptors 拦截器参数不为空，则使用传入的拦截器参数初始化interceptors属性；如果为空，则从config参数中获取配置的拦截器，并用于创建ProducerInterceptors 生产者拦截器
-
configureClusterResourceListeners方法，构造集群资源监听器对象，此处仅仅是构造ClusterResourceListeners对象并设置其属性值，用户可以实现ClusterResourceListener接口，每次集群元数据更新时都将回调用户实现的方法
-
设置各种参数属性
-
configureTransactionState方法，完成事物管理器创建
private TransactionManager configureTransactionState(ProducerConfig config, LogContext logContext) { TransactionManager transactionManager = null; final boolean userConfiguredIdempotence = config.originals().containsKey(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG); final boolean userConfiguredTransactions = config.originals().containsKey(ProducerConfig.TRANSACTIONAL_ID_CONFIG); if (userConfiguredTransactions && !userConfiguredIdempotence) log.info("Overriding the default {} to true since {} is specified.", ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, ProducerConfig.TRANSACTIONAL_ID_CONFIG); if (config.idempotenceEnabled()) { final String transactionalId = config.getString(ProducerConfig.TRANSACTIONAL_ID_CONFIG); final int transactionTimeoutMs = config.getInt(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG); final long retryBackoffMs = config.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG); final boolean autoDowngradeTxnCommit = config.getBoolean(ProducerConfig.AUTO_DOWNGRADE_TXN_COMMIT); transactionManager = new TransactionManager( logContext, transactionalId, transactionTimeoutMs, retryBackoffMs, apiVersions, autoDowngradeTxnCommit); if (transactionManager.isTransactional()) log.info("Instantiated a transactional producer."); else log.info("Instantiated an idempotent producer."); } return transactionManager; }-
从参数中获取是否开启密等性设置、是否设置事物Id，如果设置了事物Id但没有开启幂等性设置，则打印提示信息
-
如果开启幂等设置并且设置事物Id，则进行TransactionManager创建，创建TransactionManager会从配置中获取配置值进行初始化
-
-
创建记录累加器
-
ClientUtils.parseAndValidateAddresses静态方法解析并验证地址，此静态方法需要传入设置的bootstrap.servers服务端地址，以及client.dns.lookup参数值，此值默认为use_all_dns_ips
public static ListparseAndValidateAddresses(List urls, ClientDnsLookup clientDnsLookup) { List addresses = new ArrayList<>(); for (String url : urls) { if (url != null && !url.isEmpty()) { try { String host = getHost(url); Integer port = getPort(url); if (host == null || port == null) throw new ConfigException("Invalid url in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG + ": " + url); if (clientDnsLookup == ClientDnsLookup.RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY) { InetAddress[] inetAddresses = InetAddress.getAllByName(host); for (InetAddress inetAddress : inetAddresses) { String resolvedCanonicalName = inetAddress.getCanonicalHostName(); InetSocketAddress address = new InetSocketAddress(resolvedCanonicalName, port); if (address.isUnresolved()) { log.warn("Couldn't resolve server {} from {} as DNS resolution of the canonical hostname {} failed for {}", url, CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, resolvedCanonicalName, host); } else { addresses.add(address); } } } else { InetSocketAddress address = new InetSocketAddress(host, port); if (address.isUnresolved()) { log.warn("Couldn't resolve server {} from {} as DNS resolution failed for {}", url, CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, host); } else { addresses.add(address); } } } catch (IllegalArgumentException e) { throw new ConfigException("Invalid port in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG + ": " + url); } catch (UnknownHostException e) { throw new ConfigException("Unknown host in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG + ": " + url); } } } if (addresses.isEmpty()) throw new ConfigException("No resolvable bootstrap urls given in " + CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG); return addresses; } -
循环参数bootstrap.servers的服务地址列表，解析出每个地址的host及端口
-
查看client.dns.lookup参数设置值，如果为resolve_canonical_bootstrap_servers_only，则根据上步的host获取对应的IP列表inetAddresses，如果本身host就是IP地址，则只做验证。循环inetAddresses，获取规范的主机名，并使用对应端口，创建InetSocketAddress对象；如果不为resolve_canonical_bootstrap_servers_onl，则直接使用host及端口创建InetSocketAddress。返回解析后的InetSocketAddress列表对象
-
-
创建生产端元数据对象，并使用用户设置的相关元数据的配置进行初始化
-
newSender方法，创建发送器
Sender newSender(LogContext logContext, KafkaClient kafkaClient, ProducerMetadata metadata) { int maxInflightRequests = configureInflightRequests(producerConfig); int requestTimeoutMs = producerConfig.getInt(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG); ChannelBuilder channelBuilder = ClientUtils.createChannelBuilder(producerConfig, time, logContext); ProducerMetrics metricsRegistry = new ProducerMetrics(this.metrics); Sensor throttleTimeSensor = Sender.throttleTimeSensor(metricsRegistry.senderMetrics); KafkaClient client = kafkaClient != null ? kafkaClient : new NetworkClient( new Selector(producerConfig.getLong(ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG), this.metrics, time, "producer", channelBuilder, logContext), metadata, clientId, maxInflightRequests, producerConfig.getLong(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG), producerConfig.getLong(ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG), producerConfig.getInt(ProducerConfig.SEND_BUFFER_CONFIG), producerConfig.getInt(ProducerConfig.RECEIVE_BUFFER_CONFIG), requestTimeoutMs, producerConfig.getLong(ProducerConfig.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG), producerConfig.getLong(ProducerConfig.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG), ClientDnsLookup.forConfig(producerConfig.getString(ProducerConfig.CLIENT_DNS_LOOKUP_CONFIG)), time, true, apiVersions, throttleTimeSensor, logContext); short acks = configureAcks(producerConfig, log); return new Sender(logContext, client, metadata, this.accumulator, maxInflightRequests == 1, producerConfig.getInt(ProducerConfig.MAX_REQUEST_SIZE_CONFIG), acks, producerConfig.getInt(ProducerConfig.RETRIES_CONFIG), metricsRegistry.senderMetrics, time, requestTimeoutMs, producerConfig.getLong(ProducerConfig.RETRY_BACKOFF_MS_CONFIG), this.transactionManager, apiVersions); }-
configureInflightRequests方法，获取配置的max.in.flight.requests.per.connection参数值，如果开启幂等性，则不允许此参数设置值大于5。此参数值控制，在阻塞前在单个连接上允许发送的最大未确定请求数。设置大于1有乱序风险
-
获取请求超时时间
-
ClientUtils.createChannelBuilder静态方法，根据用户配置的参数完成通道构造器创建
-
创建生产者指标参数
-
创建produceThrottleTime传感器
-
创建NetworkClient，NetworkClient包含了各种与连接相关的参数，比如各种缓冲区大小、退避时间、连接超时时间等
-
configureAcks方法，获取并验证acks设置，对于启用事物则必须将acks设置为-1
-
创建Sender对象，将各种解析设置放入Sender属性中，Sender类实现了Runnable接口
-
-
创建KafkaThread io线程，KafkaThread继承了Thread，使用sender对象初始化线程
public KafkaThread(final String name, Runnable runnable, boolean daemon) { super(runnable, name); configureThread(name, daemon); } private void configureThread(final String name, boolean daemon) { setDaemon(daemon); setUncaughtExceptionHandler((t, e) -> log.error("Uncaught exception in thread '{}':", name, e)); }-
其构造方法调用父类Thread的构造方法，使用sender创建线程
-
将此线程设置为守护线程，并设置未捕获异常的处理
-
-
启动io线程
-
AppInfoParser.registerAppInfo方法，记录相关信息
-
自此KafkaProducer生产者创建完成
ClientUtils.createChannelBuilder创建通道构造器解析
此静态方法根据提供的配置信息创建一个新的通道构造器
public static ChannelBuilder createChannelBuilder(AbstractConfig config, Time time, LogContext logContext) {
SecurityProtocol securityProtocol = SecurityProtocol.forName(config.getString(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG));
String clientSaslMechanism = config.getString(SaslConfigs.SASL_MECHANISM);
return ChannelBuilders.clientChannelBuilder(securityProtocol, JaasContext.Type.CLIENT, config, null,
clientSaslMechanism, time, true, logContext);
}
-
首先从配置参数中获取安全协议，默认PLAINTEXT
-
获取用于客户端连接的SASL机制，默认是GSSAPI
-
调用ChannelBuilders的静态方法clientChannelBuilder创建通道构造器
public static ChannelBuilder clientChannelBuilder( SecurityProtocol securityProtocol, JaasContext.Type contextType, AbstractConfig config, ListenerName listenerName, String clientSaslMechanism, Time time, boolean saslHandshakeRequestEnable, LogContext logContext) { if (securityProtocol == SecurityProtocol.SASL_PLAINTEXT || securityProtocol == SecurityProtocol.SASL_SSL) { if (contextType == null) throw new IllegalArgumentException("`contextType` must be non-null if `securityProtocol` is `" + securityProtocol + "`"); if (clientSaslMechanism == null) throw new IllegalArgumentException("`clientSaslMechanism` must be non-null in client mode if `securityProtocol` is `" + securityProtocol + "`"); } return create(securityProtocol, Mode.CLIENT, contextType, config, listenerName, false, clientSaslMechanism, saslHandshakeRequestEnable, null, null, time, logContext, null); }-
首先进行参数值校验，如果安全协议为SASL_PLAINTEXT或SASL_SSL，则contextType、clientSaslMechanism参数必须不能为空
-
调用create方法完成创建
-
create创建ChannelBuilder
完成ChannelBuilder创建
private static ChannelBuilder create(SecurityProtocol securityProtocol,
Mode mode,
JaasContext.Type contextType,
AbstractConfig config,
ListenerName listenerName,
boolean isInterBrokerListener,
String clientSaslMechanism,
boolean saslHandshakeRequestEnable,
CredentialCache credentialCache,
DelegationTokenCache tokenCache,
Time time,
LogContext logContext,
Supplier apiVersionSupplier) {
Map configs = channelBuilderConfigs(config, listenerName);
ChannelBuilder channelBuilder;
switch (securityProtocol) {
case SSL:
requireNonNullMode(mode, securityProtocol);
channelBuilder = new SslChannelBuilder(mode, listenerName, isInterBrokerListener, logContext);
break;
case SASL_SSL:
case SASL_PLAINTEXT:
requireNonNullMode(mode, securityProtocol);
Map jaasContexts;
String sslClientAuthOverride = null;
if (mode == Mode.SERVER) {
@SuppressWarnings("unchecked")
List enabledMechanisms = (List) configs.get(BrokerSecurityConfigs.SASL_ENABLED_MECHANISMS_CONFIG);
jaasContexts = new HashMap<>(enabledMechanisms.size());
for (String mechanism : enabledMechanisms)
jaasContexts.put(mechanism, JaasContext.loadServerContext(listenerName, mechanism, configs));
// SSL client authentication is enabled in brokers for SASL_SSL only if listener-prefixed config is specified.
if (listenerName != null && securityProtocol == SecurityProtocol.SASL_SSL) {
String configuredClientAuth = (String) configs.get(BrokerSecurityConfigs.SSL_CLIENT_AUTH_CONFIG);
String listenerClientAuth = (String) config.originalsWithPrefix(listenerName.configPrefix(), true)
.get(BrokerSecurityConfigs.SSL_CLIENT_AUTH_CONFIG);
// If `ssl.client.auth` is configured at the listener-level, we don't set an override and SslFactory
// uses the value from `configs`. If not, we propagate `sslClientAuthOverride=NONE` to SslFactory and
// it applies the override to the latest configs when it is configured or reconfigured. `Note that
// ssl.client.auth` cannot be dynamically altered.
if (listenerClientAuth == null) {
sslClientAuthOverride = SslClientAuth.NONE.name().toLowerCase(Locale.ROOT);
if (configuredClientAuth != null && !configuredClientAuth.equalsIgnoreCase(SslClientAuth.NONE.name())) {
log.warn("Broker configuration '{}' is applied only to SSL listeners. Listener-prefixed configuration can be used" +
" to enable SSL client authentication for SASL_SSL listeners. In future releases, broker-wide option without" +
" listener prefix may be applied to SASL_SSL listeners as well. All configuration options intended for specific" +
" listeners should be listener-prefixed.", BrokerSecurityConfigs.SSL_CLIENT_AUTH_CONFIG);
}
}
}
} else {
// Use server context for inter-broker client connections and client context for other clients
JaasContext jaasContext = contextType == JaasContext.Type.CLIENT ? JaasContext.loadClientContext(configs) :
JaasContext.loadServerContext(listenerName, clientSaslMechanism, configs);
jaasContexts = Collections.singletonMap(clientSaslMechanism, jaasContext);
}
channelBuilder = new SaslChannelBuilder(mode,
jaasContexts,
securityProtocol,
listenerName,
isInterBrokerListener,
clientSaslMechanism,
saslHandshakeRequestEnable,
credentialCache,
tokenCache,
sslClientAuthOverride,
time,
logContext,
apiVersionSupplier);
break;
case PLAINTEXT:
channelBuilder = new PlaintextChannelBuilder(listenerName);
break;
default:
throw new IllegalArgumentException("Unexpected securityProtocol " + securityProtocol);
}
channelBuilder.configure(configs);
return channelBuilder;
}
-
channelBuilderConfigs方法，主要完成参数处理
-
根据安全策略创建对应安全策略的通道构建器。例如默认创建PlaintextChannelBuilder通道构造器
-
设置第一步返回的参数给通道构建器
发送消息
创建完发送者之后就可以使用发送者发送消息了，但发送的内容还需要构造成发送者需要的对象
构造发送消息需要的对象类型ProducerRecord
发送消息前需要将待发送消息构造成ProducerRecord对象，直接通过ProducerRecord构造方法可以完成创建，我们除需要设置待发送消息Key、Value外，还需要设置需要发送到的topic，我们也可以设置发送的分区等信息
通过生产者send方法完成消息发送
KafkaProducer的send方法用于完成消息的发送，消息发送是异步的，会返回Future的对象。发送消息可以指定错误回调方法也可以不指定,我们下面看一下不指定错误回调方法的消息发送源码
不指定错误回调的发送消息方法
public Future send(ProducerRecord record) {
return send(record, null);
}
public Future send(ProducerRecord record, Callback callback) {
// intercept the record, which can be potentially modified; this method does not throw exceptions
ProducerRecord interceptedRecord = this.interceptors.onSend(record);
return doSend(interceptedRecord, callback);
}
-
调用ProducerInterceptors的onSend方法，循环所有的拦截器，并调用用户注册拦截器的onSend方法，此方法在序列化之前执行。并且上一个拦截器的输出作为下一个拦截器的输入
-
执行doSend真正完成消息发送操作
doSend方法完成真正发送
private Future doSend(ProducerRecord record, Callback callback) {
TopicPartition tp = null;
try {
throwIfProducerClosed();
// first make sure the metadata for the topic is available
long nowMs = time.milliseconds();
ClusterAndWaitTime clusterAndWaitTime;
try {
clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), nowMs, maxBlockTimeMs);
} catch (KafkaException e) {
if (metadata.isClosed())
throw new KafkaException("Producer closed while send in progress", e);
throw e;
}
nowMs += clusterAndWaitTime.waitedOnMetadataMs;
long remainingWaitMs = Math.max(0, maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs);
Cluster cluster = clusterAndWaitTime.cluster;
byte[] serializedKey;
try {
serializedKey = keySerializer.serialize(record.topic(), record.headers(), record.key());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG).getName() +
" specified in key.serializer", cce);
}
byte[] serializedValue;
try {
serializedValue = valueSerializer.serialize(record.topic(), record.headers(), record.value());
} catch (ClassCastException cce) {
throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() +
" to class " + producerConfig.getClass(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG).getName() +
" specified in value.serializer", cce);
}
int partition = partition(record, serializedKey, serializedValue, cluster);
tp = new TopicPartition(record.topic(), partition);
setReadOnly(record.headers());
Header[] headers = record.headers().toArray();
int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(apiVersions.maxUsableProduceMagic(),
compressionType, serializedKey, serializedValue, headers);
ensureValidRecordSize(serializedSize);
long timestamp = record.timestamp() == null ? nowMs : record.timestamp();
if (log.isTraceEnabled()) {
log.trace("Attempting to append record {} with callback {} to topic {} partition {}", record, callback, record.topic(), partition);
}
// producer callback will make sure to call both 'callback' and interceptor callback
Callback interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
if (transactionManager != null && transactionManager.isTransactional()) {
transactionManager.failIfNotReadyForSend();
}
RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
serializedValue, headers, interceptCallback, remainingWaitMs, true, nowMs);
if (result.abortForNewBatch) {
int prevPartition = partition;
partitioner.onNewBatch(record.topic(), cluster, prevPartition);
partition = partition(record, serializedKey, serializedValue, cluster);
tp = new TopicPartition(record.topic(), partition);
if (log.isTraceEnabled()) {
log.trace("Retrying append due to new batch creation for topic {} partition {}. The old partition was {}", record.topic(), partition, prevPartition);
}
// producer callback will make sure to call both 'callback' and interceptor callback
interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
result = accumulator.append(tp, timestamp, serializedKey,
serializedValue, headers, interceptCallback, remainingWaitMs, false, nowMs);
}
if (transactionManager != null && transactionManager.isTransactional())
transactionManager.maybeAddPartitionToTransaction(tp);
if (result.batchIsFull || result.newBatchCreated) {
log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition);
this.sender.wakeup();
}
return result.future;
// handling exceptions and record the errors;
// for API exceptions return them in the future,
// for other exceptions throw directly
} catch (ApiException e) {
log.debug("Exception occurred during message send:", e);
if (callback != null)
callback.onCompletion(null, e);
this.errors.record();
this.interceptors.onSendError(record, tp, e);
return new FutureFailure(e);
} catch (InterruptedException e) {
this.errors.record();
this.interceptors.onSendError(record, tp, e);
throw new InterruptException(e);
} catch (KafkaException e) {
this.errors.record();
this.interceptors.onSendError(record, tp, e);
throw e;
} catch (Exception e) {
// we notify interceptor about all exceptions, since onSend is called before anything else in this method
this.interceptors.onSendError(record, tp, e);
throw e;
}
}
-
throwIfProducerClosed方法，检查sender属性是否为空或者是否已经关闭，如果是则抛出"Cannot perform operation after producer has been closed"异常
-
waitOnMetadata方法，获取集群元数据信息
-
执行waitOnMetadata方法前的时间数+执行完成waitOnMetadata方法时间作为nowMs局部变量值
-
最大阻塞时间 - waitOnMetadata方法获取元数据时间作为remainingWaitMs剩余等待时间值
-
调用keySerializer.serialize方法完成key的序列化
-
调用valueSerializer.serialize方法完成value序列化
-
调用partition方法计算发送消息的目标分区
-
setReadOnly方法，如果发送消息头为RecordHeaders类型对象，则设置只读标识isReadOnly为ture
-
AbstractRecords.estimateSizeInBytesUpperBound方法，获取要发送消息的预估最大值，没有考虑压缩
-
ensureValidRecordSize方法，判断是否超过发送消息大小的上线，主要根据与maxRequestSize、totalMemorySize做对比
-
如果发送数据的时间戳为空则使用上面的nowMs作为时间戳
-
new InterceptorCallback<>创建回调器
-
RecordAccumulator的append方法将要发送的消息记录追加到发送消息的内存缓存池中
-
this.sender.wakeup()唤起发送线程进行消息发送
waitOnMetadata获取集群元数据信息
待补充
partition方法计算发送消息目标分区
待补充
RecordAccumulator的append方法
待补充
唤起发送线程进行消息发送
待补充