Kafka监控工具Kafka Eagle

Kafka Eagle是一款用于监控Kafka集群的开源工具，兼备Offset、分区等管理能力。

1 环境准备

1.1 安装jdk

yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel

1.2 配置JAVA_HOME 

vi /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ 
export PATH=$PATH:$JAVA_HOME/bin

2 安装Kafka Eagle 

2.1 下载

最新版本：Kafka Eagle 1.3.1 was released on Mar 26, 2019

安装包下载地址：http://download.smartloli.org/

源码包：https://github.com/smartloli/kafka-eagle

2.2 解压

tar -zxvf kafka-eagle-bin-1.3.1.tar.gz
mv kafka-eagle-1.3.1 kafka-eagle

2.3.配置环境变量

vi /etc/profile

export KE_HOME=/kafka-eagle
export PATH=$PATH:$KE_HOME/bin

2.4.配置system-config.properties

cd ${KE_HOME}/conf
vi system-config.properties

# 配置kafka集群的自定义名称和zk信息，如果多个集群参照cluster2配置即可
kafka.eagle.zk.cluster.alias=cluster1,cluster2
cluster1.zk.list=tdn1:2181,tdn2:2181,tdn3:2181
cluster2.zk.list=xdn1:2181,xdn2:2181,xdn3:2181

# 连接zk的线程数，默认即可
kafka.zk.limit.size=25

# kafka-eagle页面访问端口
kafka.eagle.webui.port=8048

# 配置存储偏移量位置，注意1.0.0以前存储的还是zookeeper，之后存储在kafka，不同集群可以分开配置
cluster1.kafka.eagle.offset.storage=kafka
cluster2.kafka.eagle.offset.storage=kafka

# 告警配置，对接钉钉和微信，由于内网没有验证，保持默认
kafka.eagle.im.dingding.enable=true
kafka.eagle.im.dingding.url=https://oapi.dingtalk.com/robot/send?access_token=
kafka.eagle.im.wechat.enable=true
kafka.eagle.im.wechat.token=https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=xxx&corpsecret=xxx
kafka.eagle.im.wechat.url=https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=
kafka.eagle.im.wechat.touser=
kafka.eagle.im.wechat.toparty=
kafka.eagle.im.wechat.totag=
kafka.eagle.im.wechat.agentid=

# 是否启用监控图表
kafka.eagle.metrics.charts=true

# 使用kafka-sql时可以考虑开启，应对offset溢出异常
kafka.eagle.sql.fix.error=false

# 删除topic操作超级管理员token
kafka.eagle.topic.token=keadmin

# 是否启用Kafka SASL协议，conf目录有kafka_client_jaas.conf可配置连接的用户和密码
cluster1.kafka.eagle.sasl.enable=true
cluster1.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster1.kafka.eagle.sasl.mechanism=PLAIN
cluster2.kafka.eagle.sasl.enable=true
cluster2.kafka.eagle.sasl.protocol=SASL_PLAINTEXT
cluster2.kafka.eagle.sasl.mechanism=PLAIN

# 默认使用sqlite存储数据，配置连接串和管理员用户和密码，所配置的数据文件目录必须提前创建好
kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
kafka.eagle.username=root
kafka.eagle.password=smartloli

# 也可选择使用mysql存储数据
#kafka.eagle.driver=com.mysql.jdbc.Driver
#kafka.eagle.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
#kafka.eagle.username=root
#kafka.eagle.password=smartloli

官方配置参数说明：

Base

Name	Default	Type	Describe
kafka.eagle.zk.cluster.alias	cluster1,cluster2	String	Multi-kafka cluser alias.
cluster1.zk.list	tdn1:2181,tdn2:2181,tdn3:2181	String	Kafka cluster1 zookeeper address.
cluster2.zk.list	xdn10:2181,xdn11:2181,xdn12:2181	String	Kafka cluster2 zookeeper address.
kafka.zk.limit.size	25	Int	Kafka Eagle maximum number of connections for the Zookeeper client.
kafka.eagle.webui.port	8048	Int	Kafka Eagle WebUI port.
cluster1.kafka.eagle.offset.storage	kafka	String	Kafka offsets stored in kafka topic.
cluster2.kafka.eagle.offset.storage	zk	String	Kafka offsets stored in zookeeper.
kafka.eagle.metrics.charts	false	Boolean	Kafka Eagle default disable metrics.
kafka.eagle.sql.fix.error	false	Boolean	Kafka Eagle default disable fixed kafka sql query error.
kafka.eagle.sql.topic.records.max	5000	Long	Kafka Eagle SQL query topic max records.
kafka.eagle.topic.token	keadmin	String	Kafka Eagle delete topic token.

Mail

Name	Default	Type	Describe
kafka.eagle.mail.enable	true	Boolean	Kafka Eagle enable alert mail.
kafka.eagle.mail.sa	[email protected]	String	Kafka Eagle alert administrators.
kafka.eagle.mail.username	[email protected]	String	Kafka Eagle alert server username.
kafka.eagle.mail.password	password	String	Kafka Eagle alert server password.
kafka.eagle.mail.server.host	smtp.smartloli.org	String	Kafka Eagle alert server address.
kafka.eagle.mail.server.port	25	String	Kafka Eagle alert server port.

DingDing

Name	Default	Type	Describe
kafka.eagle.im.dingding.enable	true	Boolean	Kafka Eagle enable alert dingding.
kafka.eagle.im.dingding.url	https://oapi.dingtalk.com/robot/send?access_token=	String	Kafka Eagle dingding api.

WeChat

Name	Default	Type	Describe
kafka.eagle.im.wechat.enable	true	Boolean	Kafka Eagle enable alert wechat.
kafka.eagle.im.wechat.token	https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=xxx&corpsecret=xxx	String	Kafka Eagle get wechat token.
kafka.eagle.im.wechat.url	https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=	String	Kafka Eagle wechat api.
kafka.eagle.im.wechat.touser		String	Kafka Eagle wechat touser.
kafka.eagle.im.wechat.toparty		String	Kafka Eagle wechat toparty.
kafka.eagle.im.wechat.totag		String	Kafka Eagle wechat totag.
kafka.eagle.im.wechat.agentid		String	Kafka Eagle wechat agentid.

SASL_PLAINTEXT

Name	Default	Type	Describe
cluster1.kafka.eagle.sasl.enable	false	Boolean	Kafka Eagle cluster1 disable SASL_PLAINTEXT.
cluster1.kafka.eagle.sasl.protocol	SASL_PLAINTEXT	String	Kafka Eagle cluster1 set SASL protocol.
cluster1.kafka.eagle.sasl.mechanism	PLAIN	String	Kafka Eagle cluster1 set SASL mechanism.
cluster2.kafka.eagle.sasl.enable	false	Boolean	Kafka Eagle cluster2 disable SASL_PLAINTEXT.
cluster2.kafka.eagle.sasl.protocol	SASL_PLAINTEXT	String	Kafka Eagle cluster2 set SASL protocol.
cluster2.kafka.eagle.sasl.mechanism	PLAIN	String	Kafka Eagle cluster2 set SASL mechanism.

Database property

MySQL

Name	Default	Type	Describe
kafka.eagle.driver	com.mysql.jdbc.Driver	String	Kafka Eagle store database driver.
kafka.eagle.url	jdbc:mysql://127.0.0.1:3306/ke3?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull	String	Kafka Eagle store database url.
kafka.eagle.username	root	String	Kafka Eagle store database username.
kafka.eagle.password	123456	String	Kafka Eagle store database password.

SQLite

Name	Default	Type	Describe
kafka.eagle.driver	org.sqlite.JDBC	String	Kafka Eagle store database driver.
kafka.eagle.url	jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db	String	Kafka Eagle store database url.
kafka.eagle.username	root	String	Kafka Eagle store database username.
kafka.eagle.password	123456	String	Kafka Eagle store database password.

2.5 启动

cd ${KE_HOME}/bin
chmod +x ke.sh
./ke.sh start

3 访问

 页面访问地址：http://host:port/ke

4 附录：

官方文档：https://ke.smartloli.org/