Java 连接 启用kerberos的Kafka
前言：
最近换了新工作，在新环境下逐步适应中，来了近三周时间，也未能申请到一套服务器用来搭建CDH集群。一直用的是别人的集群，但是别人的集群各种权限限制，CDH集群还配置了kerberos认证，大大增加了工作量与工作难度。所以能不搞Kerberos认证最好不要弄这玩意，自讨苦吃…
CM为Kafka配置Kerberos
1.实施方案前，假设下面条件满足
- CDH集成Kerberos和Sentry成功
- 具备root权限
2.修改kafka配置
-
进入Kafka服务，修改ssl.client.auth配置为none
-
启用kerberos
-
修改security.inter.broker.protocol
- 最后重启kafka服务
- 完成以上配置Kafka集群已启用Kerberos认证
Xshell 下验证kafka 生产者，消费者
[wms_dev@bi-slave1 kafka_client]$ pwd
/usr/local/kafka_client
[wms_dev@bi-slave1 kafka_client]$ ls
client.properties jaas.conf
- 创建jaas.conf文件
#假设在该目录下建
[root@bi-slave1 kafka_client]# pwd
/usr/local/kafka_client
#创建文件
[root@bi-slave1 kafka_client]# vi jaas.conf
KafkaClient{
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
:wq
- 创建client.properties文件
[root@bi-slave1 kafka_client]# vi client.properties
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
:wq
初始化kerberos账号
[root@bi-slave1 kafka_client]# kinit deng_yb
Password for [email protected]:
[root@bi-slave1 kafka_client]# klist
设置环境变量
#注意这样配置环境变量只针对当前的进程有效
[root@bi-slave1 kafka_client]# export KAFKA_OPTS="-Djava.security.auth.login.config=/usr/local/kafka_client/jaas.conf"
[root@bi-slave1 kafka_client]# echo $KAFKA_OPTS
-Djava.security.auth.login.config=/usr/local/kafka_client/jaas.conf
测试Producer
kafka-console-producer --broker-list 192.xxx.xxx.xx:9092,192.xxx.xxx.xx:9092 --topic kerbero --producer.config /usr/local/kafka_client/client.properties
#输入消息
>Hello World!
>Hello Kerberos!
>
启动producer客户端过程和发消息过程没报错即代表producer客户端配置成功
测试Consumer
kafka-console-consumer --topic kerbero --from-beginning --bootstrap-server 192.xxx.xxx.xx:9092,192.xxx.xxx.xx:9092 --consumer.config client.properties
#以下消息为针对topic kerbero
Hello World!
Hello Kerberos!
>Hello World!
>Hello Kerberos!
启动consumer客户端过程和接受消息过程没报错即代表客户端配置成功
上述为kafka集成kerberos服务端配置和客户端的使用
java代码连接Kafka
- 工程架构目录
新建maven工程，pom.xml添加如下
<dependency>
<groupId>org.apache.kafka</groupId><groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.0.0</version>
</dependency>
添加krb5.conf，可直接从kdc所在服务器中拷贝
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = WONHIGH.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
WONHIGH.COM = {
kdc = bi-master
admin_server = bi-master
default_realm = WONHIGH.COM
kdc = bi-slave1
}
WONHIGH.CN = {
kdc = nn209003:88
admin_server = nn209003:749
default_realm = WONHIGH.CN
}
BELLE.COM = {
kdc = test-gtp-cdh-node01
admin_server = test-gtp-cdh-node01
default_realm = BELLE.COM
}
[domain_realm]
.bi-master = WONHIGH.COM
bi-master = WONHIGH.COM
nn209003 = WONHIGH.CN
test-gtp-cdh-node01 = BELLE.COM
test-gtp-cdh-node04 = BELLE.COM
[capaths]
WONHIGH.COM = {
WONHIGH.CN = .
BELLE.COM = .
}
- 添加keytab文件，身份验证
wms_dev.keytab
这个文件是别人给我的… - jaas.conf 文件
KafkaClient{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="file:D:/keytab/bk/wms_dev.keytab"
principal="[email protected]";
};
Client{
com.sun.security.auth.module.Krb5LoginModule
required
useTicketCache=false
doNotPrompt=true
useKeyTab=true
keyTab="file:D:/keytab/bk/wms_dev.keytab"
principal="[email protected]"
debug=true;
};
kerberos 登录
/**
* 以jass.conf文件认证kerberos
*/
public static void getKerberosJaas(){
System.setProperty("java.security.krb5.conf",
Thread.currentThread().getContextClassLoader().getResource("krb5.conf").getPath());
//加载本地jass.conf文件
System.setProperty("java.security.auth.login.config",
Thread.currentThread().getContextClassLoader().getResource("jaas.conf").getPath());
//加载临时jass.conf
/* File jaasConf = KerberosUtils.configureJAAS(Thread.currentThread().getContextClassLoader()
.getResource("wms_dev.keytab").getPath(), "[email protected]");
System.setProperty("java.security.auth.login.config", jaasConf.getAbsolutePath());*/
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
//System.setProperty("sun.security.krb5.debug","true");
}
上面代码是我根据下面的样例改造出来的
- 工具类，初始化kerberos环境
package deng.yb.kafka_kerberos.utils;
import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;
public class MyProperties extends Properties{
private Properties properties;
private static final String JAAS_TEMPLATE =
"KafkaClient {\n"
+ "com.sun.security.auth.module.Krb5LoginModule required\n" +
"useKeyTab=true\n" +
"keyTab=\"%1$s\"\n" +
"principal=\"%2$s\";\n"
+ "};";
public MyProperties(){
properties = new Properties();
}
public MyProperties self(){
return this;
}
public MyProperties put(String key , String value) {
if (properties == null) {
properties = new Properties();
}
properties.put(key, value);
return self();
}
public static MyProperties initKerberos(){
return new MyProperties()
.put(ConsumerConfig.GROUP_ID_CONFIG, "DemoConsumer")
.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer")
.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer")
.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")
.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000")
.put("security.protocol", "SASL_PLAINTEXT")
.put("sasl.kerberos.service.name", "kafka");
}
public static MyProperties initProducer(){
return new MyProperties()
.put(ProducerConfig.ACKS_CONFIG, "all")
.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
.put("security.protocol", "SASL_PLAINTEXT")
.put("sasl.kerberos.service.name", "kafka");
}
public Properties getProperties() {
return properties;
}
//生成jaas.conf临时文件
public static void configureJAAS(String keyTab, String principal) {
String content = String.format(JAAS_TEMPLATE, keyTab, principal);
File jaasConf = null;
PrintWriter writer = null;
try {
jaasConf = File.createTempFile("jaas", ".conf");
writer = new PrintWriter(jaasConf);
writer.println(content);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (writer != null) {
writer.close();
}
jaasConf.deleteOnExit();
}
System.setProperty("java.security.auth.login.config", jaasConf.getAbsolutePath());
}
}
生产者类
package deng.yb.kafka_kerberos;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import deng.yb.kafka_kerberos.utils.MyProperties;
public class Producer {
//发送topic
public static String TOPIC_NAME = "kafak2hdfs";
public static void main(String[] args) {
System.setProperty("java.security.krb5.conf",
Thread.currentThread().getContextClassLoader().getResource("krb5.conf").getPath());
//初始化jaas.conf文件
MyProperties.configureJAAS(Thread.currentThread().getContextClassLoader().getResource("wms_dev.keytab").getPath(), "[email protected]");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
//System.setProperty("sun.security.krb5.debug","true");
//初始化kerberos环境
MyProperties props = MyProperties.initProducer();
//kafka brokers地址
props.put(
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"bi-slave1:9092,bi-slave2:9092,bi-slave3:9092");
org.apache.kafka.clients.producer.Producer<String, String> producer = new KafkaProducer<String, String>(
props.getProperties());
for (int i = 0; i < 10; i++) {
String key = "key-" + i;
String message = "Message-" + i;
ProducerRecord record = new ProducerRecord<String, String>(
TOPIC_NAME, key, message);
producer.send(record);
System.out.println(key + "----" + message);
}
producer.close();
}
}
消费者
package deng.yb.kafka_kerberos;
import java.util.Arrays;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.ProducerConfig;
import deng.yb.kafka_kerberos.utils.MyProperties;
public class Comsumer {
private static String TOPIC_NAME = "kafak2hdfs";
public static void main(String[] args) {
System.setProperty("java.security.krb5.conf", Thread.currentThread()
.getContextClassLoader().getResource("krb5.conf").getPath());
//初始化jaas.conf文件
MyProperties.configureJAAS(Thread.currentThread().getContextClassLoader().getResource("wms_dev.keytab").getPath(), "[email protected]");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
MyProperties props = MyProperties.initKerberos();
props.put(
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"bi-slave1:9092,bi-slave2:9092,bi-slave3:9092");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(
props.getProperties());
consumer.subscribe(Arrays.asList(TOPIC_NAME));
/*
* TopicPartition partition0= new TopicPartition(TOPIC_NAME, 0);
*
* TopicPartition partition1= new TopicPartition(TOPIC_NAME, 1);
*
* TopicPartition partition2= new TopicPartition(TOPIC_NAME, 2);
*/
// consumer.assign(Arrays.asList(partition0,partition1, partition2));
ConsumerRecords<String, String> records = null;
while (true) {
try {
Thread.sleep(1000);
System.out.println();
records = consumer.poll(Long.MAX_VALUE);
for (ConsumerRecord<String, String> record : records) {
System.out.println("Receivedmessage: (" + record.key()
+ "," + record.value() + ") at offset "
+ record.offset());
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
/*
* while (true){
*
* try {
*
* Thread.sleep(10000l);
*
* System.out.println();
*
* records = consumer.poll(Long.MAX_VALUE);
*
* for (ConsumerRecord record : records) {
*
* System.out.println("Receivedmessage: (" + record.key() + "," +
* record.value() + ") at offset " + record.offset());
*
* }
*
* } **catch** (**InterruptedException** e){
*
* e.printStackTrace();
*
* }
*/
}
}