Telegraf+TDengine+Grafana快速搭建简易实时监控系统
Telegraf+TDengine+Grafana快速搭建简易实时监控系统
-
- 前期规划：
- 1.安装Telegraf
- 2.安装TDengine
- 3.安装Golang
- 4.安装blm_telegraf
- 5.配置Telegraf
- 6.安装Grafana
- 7.安装TDengine for Grafana插件
- 8.配置Grafana
TDengine官方文档-grafana
TDengine官方文档-telegraf
前期规划：
操作系统: CentOS Linux release 7.6 【安装开发包避免缺失依赖关系】
关闭SeLinux，关闭防火墙
IP规划：
192.168.1.80 ms10.dyq.com ms10
相关组件介绍：
Telegraf：收集监控信息并进行投递。
TDengine：时序数据库，用于存储监控信息
Golang：Go语言编译工具
blm_telegraf：TDengine针对Telegraf开发的插件
Grafana：可视化工具
1.安装Telegraf
下载地址https://portal.influxdata.com/downloads/
wget https://dl.influxdata.com/telegraf/releases/telegraf-1.18.0-0.rc0.x86_64.rpm
[root@ms10 soft]# rpm -ivh telegraf-1.18.0-0.rc0.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:telegraf-1.18.0-0.0 ################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/telegraf.service to /usr/lib/systemd/system/telegraf.service.
2.安装TDengine
tar xvf TDengine-server-2.0.16.0-Linux-x64.tar.gz
cd TDengine-server-2.0.16.0
./install.sh
详细安装步骤见TDengine学习笔记-安装
创建监控用数据库和用户
taos> create database teledb;
Query OK, 0 row(s) affected (0.001972s)
taos> create user tele pass 'tele';
Query OK, 0 row(s) affected (0.001945s)
taos> show databases\G;
*************************** 1.row ***************************
name: log
created_time: 2021-03-06 18:19:41.289
ntables: 4
vgroups: 1
replica: 1
quorum: 1
days: 10
keep0,keep1,keep(D): 30,30,30
cache(MB): 1
blocks: 3
minrows: 100
maxrows: 4096
wallevel: 1
fsync: 3000
comp: 2
cachelast: 0
precision: us
update: 0
status: ready
*************************** 2.row ***************************
name: teledb
created_time: 2021-03-06 18:20:10.575
ntables: 0
vgroups: 0
replica: 1
quorum: 1
days: 10
keep0,keep1,keep(D): 3650,3650,3650
cache(MB): 16
blocks: 6
minrows: 100
maxrows: 4096
wallevel: 1
fsync: 3000
comp: 2
cachelast: 0
precision: ms
update: 0
status: ready
Query OK, 2 row(s) in set (0.001838s)
taos> show users;
name | privilege | create_time | account |
============================================================================================
_root | writable | 2021-03-06 18:19:40.285 | root |
tele | writable | 2021-03-06 18:20:21.406 | root |
monitor | writable | 2021-03-06 18:19:40.285 | root |
root | super | 2021-03-06 18:19:40.285 | root |
Query OK, 4 row(s) in set (0.001340s)
3.安装Golang
下载地址：https://golang.google.cn/dl/
因为blm_telegraf是源代码，需要编译，所以必须安装go编译工具。
###解压文件到安装目录
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.14.3.linux-amd64.tar.gz
###编辑环境变量
vi .bash_profile
export PATH=$PATH:/usr/local/go/bin
###导入环境变量
source .bash_profile
###验证安装结果
go version
[root@ms10 ~]# go version
go version go1.16 linux/amd64
4.安装blm_telegraf
感谢龙！！！同学提醒，这个工具可以不安装，直接使用telegraf连接TDengine的6041端口即可。
TDengine提供一个小工具Bailongma，可将Telegraf采集的数据直接写入TDengine，并按规则在TDengine自动创建库和相关表项。下载地址：https://github.com/taosdata/Bailongma.git
tar xzvf Bailongma-master.zip
cd Bailongma-master
go mod init bailongma/v2
cd blm_telegraf/
go build
编译过程中可能会提示要求下载驱动包，执行以下命令即可
go get github.com/taosdata/TDengine/src/connector/go/src/taosSql
但是github上起来并不容易，笔者在安装中服务器就一直无法连到github。因此采用以下方法解决：
先用浏览器下载源代码包，地址：
https://github.com/taosdata/driver-go/
解压代码包，并复制到go目录下
tar xzvf driver-go-master.zip
cp -r driver-go-master/taosSql /usr/local/go/src
到blm_telegraf目录下修改server.go 文件，将
_ "github.com/taosdata/driver-go/taosSql"
修改为
_"taosSql"
然后编译
cd blm_telegraf/
go build
编译成功后，即生成 blm_telegraf可执行文件
[root@ms10 blm_telegraf]# go build
[root@ms10 blm_telegraf]# ll
total 6652
-rwxr-xr-x 1 root root 6753184 Mar 6 20:32 blm_telegraf
-rw-r--r-- 1 root root 621 Feb 24 10:49 Dockerfile
-rw-r--r-- 1 root root 6308 Feb 24 10:49 README.md
-rw-r--r-- 1 root root 12529 Mar 6 20:32 server.go
-rw-r--r-- 1 root root 12559 Mar 6 20:29 server.go.bak
-rw-r--r-- 1 root root 10323 Feb 24 10:49 Telegraf写入TDengine的API.md
将文件复制到bin目录，并写入开机启动。
[root@ms10 blm_telegraf]# cp blm_telegraf /usr/bin/
vi /etc/rc.local
nohup blm_telegraf -host ms10.dyq.com -port 9090 -dbname teledb -dbuser tele -dbpassowrd tele 1>/var/log/blm_telegraf.log 2>/var/log/blm_telegraf.log &
5.配置Telegraf
生成配置文件并修改如下：
telegraf --input-filter cpu:mem:disk:diskio:net --output-filter http config >telegraf.conf
[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "ms"
hostname = "ms10.dyq.com"
omit_hostname = false
[[outputs.http]]
url = "http://ms10.dyq.com:9090/telegraf"
###9090端口即blm_telegraf中配置的监听端口
data_format = "json"
json_timestamp_units = "1ms"
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
interfaces = ["eth0"]
如果不安装blm_telegraf，则配置如下
[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "ms"
hostname = "ms10.dyq.com"
omit_hostname = false
[[outputs.http]]
url = "http://ms10.dyq.com:6041/telegraf/test"
##为了不与上面的方法冲突，新创建了数据库test
method = "POST"
username = "tele"
password = "tele"
data_format = "json"
json_timestamp_units = "1ms"
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
interfaces = ["eth0"]
两种方式产生的超级表结构不同
启动Telegraf
systemctl start telegraf
查看数据库是否接收到数据
###使用blm_telegraf方式接受到的数据
taos> show tables;
table_name | created_time | columns | stable_name | uid | tid | vgId |
==========================================================================================================================================================
md5_7d385967ed7ba600ceab7dd... | 2021-03-06 20:42:51.298 | 2 | net | 844426624662707 | 101 | 3 |
md5_1d7d4093da9c57b2838d969... | 2021-03-06 20:43:01.169 | 2 | cpu | 844428503741116 | 213 | 3 |
md5_97c6490fba785a102cfb71b... | 2021-03-06 20:42:51.264 | 2 | net | 844426205225894 | 76 | 3 |
md5_b800884657c3407adeedff6... | 2021-03-06 20:42:51.165 | 2 | mem | 844425114689612 | 11 | 3 |
md5_a914583ac0b19695cd2b5b0... | 2021-03-06 20:42:51.325 | 2 | disk | 844426926657675 | 119 | 3 |
md5_c9b098e78fa785ffa1a7b21... | 2021-03-06 20:42:51.242 | 2 | net | 844425953563777 | 61 | 3 |
taos> show stables;
name | created_time | columns | tags | tables |
============================================================================================
mem | 2021-03-06 20:42:51.066 | 2 | 3 | 34 |
cpu_str | 2021-03-06 20:43:01.076 | 2 | 4 | 0 |
disk | 2021-03-06 20:42:51.312 | 2 | 7 | 14 |
diskio_str | 2021-03-06 20:42:51.335 | 2 | 4 | 0 |
net_str | 2021-03-06 20:42:51.203 | 2 | 4 | 0 |
mem_str | 2021-03-06 20:42:51.068 | 2 | 3 | 0 |
cpu | 2021-03-06 20:43:01.074 | 2 | 4 | 50 |
diskio | 2021-03-06 20:42:51.334 | 2 | 4 | 55 |
disk_str | 2021-03-06 20:42:51.313 | 2 | 7 | 0 |
net | 2021-03-06 20:42:51.202 | 2 | 4 | 76 |
Query OK, 10 row(s) in set (0.001245s)
###直接连接6041端口产生的数据
taos> show tables;
table_name | created_time | columns | stable_name | uid | tid | vgId |
==========================================================================================================================================================
cpu_usage_ms10_dyq_com_cpu2 | 2021-03-10 23:22:49.323 | 11 | cpu_usage | 1125900108240390 | 12 | 4 |
mem_ms10_dyq_com | 2021-03-10 23:22:39.401 | 35 | mem | 1125900024352609 | 7 | 4 |
diskio_ms10_dyq_com_sr0 | 2021-03-10 23:22:39.385 | 12 | diskio | 1125899940464724 | 2 | 4 |
disk_ms10_dyq_com_sda3_xfs_... | 2021-03-10 23:22:39.404 | 8 | disk | 1125900041130337 | 8 | 4 |
cpu_usage_ms10_dyq_com_cpu_... | 2021-03-10 23:22:49.329 | 11 | cpu_usage | 1125900141795334 | 14 | 4 |
diskio_ms10_dyq_com_sda3 | 2021-03-10 23:22:39.326 | 12 | diskio | 1125899923687232 | 1 | 4 |
diskio_ms10_dyq_com_sda2 | 2021-03-10 23:22:39.397 | 12 | diskio | 1125900007574868 | 6 | 4 |
cpu_usage_ms10_dyq_com_cpu1 | 2021-03-10 23:22:49.321 | 11 | cpu_usage | 1125900091463150 | 11 | 4 |
diskio_ms10_dyq_com_sda | 2021-03-10 23:22:39.388 | 12 | diskio | 1125899957242196 | 3 | 4 |
diskio_ms10_dyq_com_sda1 | 2021-03-10 23:22:39.395 | 12 | diskio | 1125899990797396 | 5 | 4 |
disk_ms10_dyq_com_sda1_xfs_... | 2021-03-10 23:22:39.406 | 8 | disk | 1125900057907796 | 9 | 4 |
cpu_usage_ms10_dyq_com_cpu3 | 2021-03-10 23:22:49.327 | 11 | cpu_usage | 1125900125018094 | 13 | 4 |
net_ms10_dyq_com_all | 2021-03-10 23:22:39.392 | 76 | net | 1125899974019937 | 4 | 4 |
cpu_usage_ms10_dyq_com_cpu0 | 2021-03-10 23:22:49.318 | 11 | cpu_usage | 1125900074685548 | 10 | 4 |
Query OK, 14 row(s) in set (0.003337s)
taos> show stables;
name | created_time | columns | tags | tables |
============================================================================================
disk | 2021-03-10 23:22:39.403 | 8 | 5 | 2 |
net | 2021-03-10 23:22:39.391 | 76 | 2 | 1 |
diskio | 2021-03-10 23:22:39.319 | 12 | 2 | 5 |
mem | 2021-03-10 23:22:39.399 | 35 | 1 | 1 |
cpu_usage | 2021-03-10 23:22:49.316 | 11 | 2 | 5 |
Query OK, 5 row(s) in set (0.001261s)
6.安装Grafana
下载地址：https://dl.grafana.com/oss/release/grafana-7.4.3-1.x86_64.rpm
安装依赖包：
yum install fontconfig
yum install urw-fonts
安装grafana
rpm -ivh grafana-7.4.3-1.x86_64.rpm
按照提示加载grafana-server服务并启动
[root@ms10 soft]# systemctl daemon-reload
[root@ms10 soft]# systemctl start grafana-server.service
7.安装TDengine for Grafana插件
###复制插件到grafana插件目录
cp -rf /usr/local/taos/connector/grafanaplugin /var/lib/grafana/plugins/tdengine
###修改相关权限
chown -R grafana:grafana /var/lib/grafana/plugins/tdengine
8.配置Grafana
打开Grafana 页面 http://192.168.1.0:3000，用户名：admin，密码：admin
进入后点击添加数据源
选择TDengine 数据源
填入url以及用户名密码
点击保存进行测试
返回首页，点击添加数据仪表盘
选择刚才添加的数据源，并填入SQL查询语句，点击【Generate SQL】生成图表
再次感谢龙！！！同学