Grafana+Prometheus监控Flume
搭建和部署Grafana
Grafana
安装Grafana之前，需要安装Prometheus
安装
下载地址
Download Grafana | Grafana Labs
选择版本
https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.2-1.x86_64.rpm
上传 /opt/package
安装Grafana
yum install -y /opt/package/grafana-enterprise-8.3.2-1.x86_64.rpm
修改配置
vi /etc/grafana/grafana.ini
修改以下内容
# 页面访问端口
http_port = 23000
启动Grafana
systemctl start grafana-server
配置
登录网页
http://ip:23000（默认账号密码为admin）
设置密码
# 密码
admin
添加数据源
URL：http://localhost:9090
点击左下角的save & Test按钮，如果提示success，就代表配置成功，然后点击Back返回
部署flume_exporter
flume_exporter需要go环境编译
配置go环境
下载链接
解压、添加环境变量
tar -zxvf go*.tar.gz
vi /etc/profile
#在最后一行添加
export GOROOT=/你解压的路径/go
export PATH=$PATH:$GOROOT/bin
#保存退出后source一下（vim 的使用方法可以自己搜索一下）
source /etc/profile
生效后用go version命令查看一下是否成功
下载Flume_exporter
下载地址
Download
选择版本
上传 到 /opt/package
解压 flume_exporter-0.0.2.tar.gz
# 解压
tar -zxvf /opt/package/node_exporter-1.3.1.linux-amd64.tar.gz -C /opt/software
# 进入文件目录
cd /opt/software/flume_exporter-0.0.2
执行go install 命令
把结果中含有go get …的都复制下来，用&&连接多个下载命令
如果下载过程中出现网络错误，go添加国内镜像源
go env -w GO111MODULE=on
go env -w GOPROXY=https://goproxy.cn,direct
下载完成后，再次执行go install，如果没有go get输出，就会开始编译了
[prometheus@host1 flume_exporter-0.0.2]$ export GOPATH=/home/prometheus/exporter/flume_exporter-0.0.2
[prometheus@host1 flume_exporter-0.0.2]$ go install
go: downloading github.com/Sirupsen/logrus v1.0.6
go: downloading github.com/prometheus/client_golang v1.12.2
go: downloading github.com/prometheus/common v0.34.0
go: downloading github.com/woozhijun/flume_exporter v0.0.2
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading golang.org/x/sys v0.0.0-20220114195835-da31bd327af9
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/prometheus/procfs v0.7.3
go: downloading google.golang.org/protobuf v1.26.0
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/bitly/go-simplejson v0.5.0
go: downloading gopkg.in/yaml.v2 v2.4.0
成功后，将生成的文件复制出来
mkdir ../flume_exporter
cp bin/flume_exporter-0.0.2 ../flume_exporter
cp *.yml ../flume_exporter
flume启动方式添加JSON Reporting
nohup /opt/software/flume/bin/flume-ng agent --name fence --conf /opt/software/flume/conf/ --conf-file /opt/software/flume/conf/flume-conf-fence.properties -Dflume.monitoring.type=http -Dflume.monitoring.port=36001 >/dev/null 2>&1 &
其中-Dflume.monitoring.type=http -Dflume.monitoring.port=36001 就是开启的被监听端口，给flume_exporter监控
flume_exporter配置
修改flume_exporter下的config.yml文件，端口号改成自己设置的，如上36001，可监控多个flume，但也需在prometheus增加对应的配置参数
prometheus.yml配置
在scrape_configs: 下面添加flume_exporter的配置
#添加flume监控
- job_name: "flume-agents"
metrics_path: "/metrics"
static_configs:
- targets: ["localhost:9360"]
因为flume_exporter的默认端口就是9360，所以prometheus要监控这个端口
配置服务方式启动
vim /etc/systemd/system/flume_exporter.service
[Unit]
Description=flume_exporter service
After=network.target
Wants=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/opt/software/exporter/flume_exporter/flume_exporter-0.0.2 --config-file=/opt/software/exporter/flume_exporter/config.yml --metric-file=/opt/software/exporter/flume_exporter/metrics.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target
启动flume_exporter
# 重新加载服务的配置文件
systemctl daemon-reload
# 设置开机自启
systemctl enable flume_exporter
# 启动服务
systemctl start flume_exporter
验证flume数据
curl -get http://localhost:9360/metrics
重启 prometheus
若开启了热更新配置则直接命令更新
热更新参数，在prometheus的启动命令上添加这个
--web.enable-lifecycle
curl -X POST http://localhost:9090/-/reload
添加模板
导入Grafana Dashboard ID为10736的模板
在线模板id：10736
然后就可以看到监控到画面了
参数定义：
| 字段名称 | 含义 |
|---|---|
| SOURCE.OpenConnectionCount | 打开的连接数 |
| SOURCE.TYPE | 组件类型 |
| SOURCE.AppendBatchAcceptedCount | 追加到channel中的批数量 |
| SOURCE.AppendBatchAcceptedCount | source端刚刚追加的批数量 |
| SOURCE.EventAcceptedCount | 成功放入channel的event数量 |
| SOURCE.AppendReceivedCount | source追加目前收到的数量 |
| SOURCE.StartTime(StopTIme) | 组件开始时间、结束时间 |
| SOURCE.EventReceivedCount | source端成功收到的event数量 |
| SOURCE.AppendAcceptedCount | source追加目前放入channel的数量 |
| CHANNEL.EventPutSuccessCount | 成功放入channel的event数量 |
| CHANNEL.ChannelFillPercentage | 通道使用比例 |
| CHANNEL.EventPutAttemptCount | 尝试放入将event放入channel的次数 |
| CHANNEL.ChannelSize | 目前在channel中的event数量 |
| CHANNEL.EventTakeSuccessCount | 从channel中成功取走的event数量 |
| CHANNEL.ChannelCapacity | 通道容量 |
| CHANNEL.EventTakeAttemptCount | 尝试从channel中取走event的次数 |
| SINK.BatchCompleteCount | 完成的批数量 |
| SINK.ConnectionFailedCount | 连接失败数 |
| SINK.EventDrainAttemptCount | 尝试提交的event数量 |
| SINK.ConnectionCreatedCount | 创建连接数 |
| SINK.Type | 组件类型 |
| SINK.BatchEmptyCount | 批量取空的数量 |
| SINK.ConnectionClosedCount | 关闭连接数量 |
| SINK.EventDrainSuccessCount | 成功发送event的数量 |
| SINK.BatchUnderflowCount | 正处于批量处理的batch数 |
参考文章
https://blog.csdn.net/qq_27710549/article/details/100745914
https://blog.csdn.net/qq_36434219/article/details/124781815
https://blog.csdn.net/qq_41142053/article/details/124862598