kafka相关证书配置

一、设置kafka

参考文档:

配置参考文档: https://www.orchome.com/171

https://blog.csdn.net/difffate/article/details/53570344

https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756

https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756/5

文档配置:

https://blog.csdn.net/qq_41926119/article/details/104510481

kafka官方文档:

https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/managing_cluster/ssl_elastic_stack_enable.html

1.  生成证书:

keytool -keystore kafka.server.keystore.jks -alias xxx -validity 3000 -genkey -keyalg RSA

openssl req -new -x509 -keyout ca-key -out ca-cert -days 3000 -passout pass:xxx

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert

keytool -keystore kafka.server.keystore.jks -alias  xxx -certreq -file cert-file

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3000 -CAcreateserial -passin pass:xxx

keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert

keytool -keystore kafka.server.keystore.jks -alias  xx -import -file cert-signed

2.基于jks生成pem证书:

keytool -importkeystore -srckeystore  kafka.server.keystore.jks -destkeystore client.p12 -deststoretype PKCS12

openssl pkcs12 -in client.p12 -nokeys -out client.cer.pem

openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key.pem

keytool -exportcert -alias xxx -keystore kafka.client.truststore.jks -rfc -file serverpub.pem

3. 查看pem证书:

keytool -printcert -file certificate.pem

多个pem证书合并:

cat cert1.pem cert2.pem > bundle.pem

4. 查看jks证书:

keytool -list -v -keystore  server.truststore.jks

5.生成filebeat证书:(需要把ca.pem 加入到 kafka的truststore.jks中,完成服务端授信)

通过ES生成自签的CA证书:./bin/elasticsearch-certutil ca

生成颁发的证书:./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name filebeat  --pem --out filebeat2.zip

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert

6.  kafka的最终配置

listeners=PLAINTEXT://172.28.15.231:9092,SSL://172.28.15.231:9093

ssl.endpoint.identification.algorithm=

ssl.keystore.location=/data/xxx/kafka_ca_cert/kafka.server.keystore.jks

ssl.keystore.password=xxxx

ssl.truststore.location=/data/xxx/kafka_ca_cert/kafka.server.truststore.jks

ssl.keystore.type=JKS

ssl.truststore.type=JKS

ssl.secure.random.implementation=SHA1PRNG

7. 客户端配置:client-ssl.properties

security.protocol=SSL

ssl.truststore.location=/data/xxx/kafka_ca_cert/kafka.client.truststore.jks

ssl.truststore.password=xxxx

ssl.keystore.type=JKS

ssl.truststore.type=JKS

ssl.endpoint.identification.algorithm=

8. kafka 客户端生产:

kafka-console-producer.sh --topic test-log  --broker-list xxxx:9093 --producer.config client-ssl.properties

kafka-console-producer.sh --topic test-log  --broker-list xxxx:9093 --producer.config client-ssl.properties

9. kafka 客户端消费:

kafka-console-consumer.sh --topic test-log --bootstrap-server xxxx:9093 --consumer.config client-ssl.properties


二、filebeat的一些配置


最终filebeat配置:

filebeat.inputs:

- type: log

  enabled: true

  paths:

    - /var/log/nginx/archived/*/*

filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:

  index.number_of_shards: 1

setup.kibana:

output.kafka:

  #指定版本号

  version: "0.9.0"

  enabled: true

  hosts:

    - 172.28.15.231:9093

  topic: "test-log"

  version: "2.0.0"

  ssl:

    enabled: true

    certificate: '/data/xxx/kafka_ca_cert/filebeat/filebeat.crt'

    key: '/data/xxx/kafka_ca_cert/filebeat/filebeat.key'

    certificate_authorities: ['/data/xxx/kafka_ca_cert/serverpub.pem']

    supported_protocols: [TLSv1.1, TLSv1.2, TLSv1.3]

    verification_mode: full

    endpoint:

      identification:

        algorithm:

processors:

  - add_host_metadata: ~

  - add_cloud_metadata: ~

filebeat 的一些配置的官方文档:

https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html

https://www.elastic.co/guide/en/beats/filebeat/master/configuration-ssl.html

https://blog.csdn.net/qq_41926119/article/details/104510481

EFK接入kafka消息队列:

https://www.ucloud.cn/yun/34441.html

需要注意的是,filebeat配置的是pem证书,kafka和logstash的kafka-input插件用的是jks证书~~~因此,证书生成工具最好需要能够同时生成这两种证书。

常见错误集锦:

https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756/11

其他配置:

Filebeat与Logstash配置SSL加密通信:

http://www.manongjc.com/detail/14-jogxwlmgmlwauxs.html

https://www.cnblogs.com/sanduzxcvbnm/p/12055038.html

https://ngx.hk/2017/01/12/%E4%B8%BAfilebeat%E9%85%8D%E7%BD%AE%E6%95%B0%E5%AD%97%E8%AF%81%E4%B9%A6%E4%BB%A5%E4%BD%BF%E7%94%A8ssl%E5%8A%A0%E5%AF%86.html

filebeat常用配置项:https://www.elastic.co/guide/en/beats/filebeat/7.x/kafka-output.html

SSL配置项: https://www.elastic.co/guide/en/beats/filebeat/7.16/configuration-ssl.html#client-verification-mode

你可能感兴趣的:(kafka相关证书配置)