Flume监控之Ganglia部署与使用

部署环境

       CentOS 7.6 

       Flume 1.6.0-cdh5.15.1

前置准备

      1)安装httpd服务与php

[js@bigdata001 flume]$ sudo yum -y install httpd php

      2)安装其他依赖

[js@bigdata001 flume]$ sudo yum -y install rrdtool perl-rrdtool rrdtool-devel
[js@bigdata001 flume]$ sudo yum -y install apr-devel

     3)   配置hostname。本文为bigdata001

安装Ganglia

     1)下载

       下载地址:http://ganglia.info/?page_id=66

        这里通过yum安装:        

   

//升级rpm源
[js@bigdata001 flume]$ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
//安装
[js@bigdata001 flume]$ sudo yum -y install ganglia-gmetad 
[js@bigdata001 flume]$ sudo yum -y install ganglia-web
[js@bigdata001 flume]$ sudo yum install -y ganglia-gmond

     2)修改配置文件ganglia.conf

[js@bigdata001 flume]$ sudo vi /etc/httpd/conf.d/ganglia.conf
# Ganglia monitoring system php web frontend
Alias /ganglia /usr/share/ganglia

  Require all granted
  # Allow from 127.0.0.1
  # Allow from ::1
  # Allow from .example.com

    3) 修改配置文件gmetad.conf

[js@bigdata001 flume]$ sudo vi /etc/ganglia/gmetad.conf

    将data_source "my cluster" localhost 修改为对应的Hostname.

    这里改为 : data_source "bigdata001" bigdata001

    4)修改配置文件gmond.conf 

[js@bigdata001 flume]$ sudo vi /etc/ganglia/gmond.conf 

修改红字部分:

cluster {

  name = "bigdata001" //注:此名要和2)配置的名字相同

  owner = "unspecified"

  latlong = "unspecified"

  url = "unspecified"

}

udp_send_channel {

  #bind_hostname = yes # Highly recommended, soon to be default.

                       # This option tells gmond to use a source address

                       # that resolves to the machine's hostname.  Without

                       # this, the metrics may appear to come from any

                       # interface and the DNS names associated with

                       # those IPs will be used to create the RRDs.

  # mcast_join = 239.2.11.71

  host = bigdata001

  port = 8649

  ttl = 1

}

udp_recv_channel {

  # mcast_join = 239.2.11.71

  port = 8649

  bind = bigdata001

  retry_bind = true

  # Size of the UDP buffer. If you are handling lots of metrics you really

  # should bump it up to e.g. 10MB or even higher.

  # buffer = 10485760

}

 

   5)关闭selinux

[js@bigdata001 flume]$ sudo vi /etc/selinux/config

    设置SELINUX=disabled。设置完成后, 重启或者运行sudo setenforce 0 。

 

   6)创建ganglia的web UI目录的软连接

[js@bigdata001 ~]$ sudo ln -s /usr/share/ganglia/ /var/www/html/ganglia

   7)启动Ganglia

[js@bigdata001 flume]$ sudo systemctl start httpd.service
[js@bigdata001 flume]$ sudo systemctl start gmetad.service
[js@bigdata001 flume]$ sudo systemctl start gmond.service

访问 http://bigdata001/ganglia

若出现以下提示,则更改/var/lib/ganglia的权限为777:

Flume监控之Ganglia部署与使用_第1张图片

若出现以下界面,配置成功:

Flume监控之Ganglia部署与使用_第2张图片

 

使用Ganglia监控Flume

          1)修改flume-env.sh

JAVA_OPTS="-Dflume.monitoring.type=ganglia

-Dflume.monitoring.hosts=bigdata001:8649

-Xms100m

-Xmx200m"

          2) 启动Flume任务

添加

-Dflume.monitoring.type=ganglia \
-Dflume.monitoring.hosts=bigdata001:8649

如下 :

[js@bigdata001 flume]$ bin/flume-ng agent \
--conf conf/ \
--name a1 \
--conf-file job/netcat-memory-logger.conf \
-Dflume.root.logger==INFO,console \
-Dflume.monitoring.type=ganglia \
-Dflume.monitoring.hosts=bigdata001:8649

         3)登陆界面查看

在Mobile中 选择配置的名称(bigdata001),找到Flume, 就可以了。或直接登陆http://bigdata001/ganglia/mobile_helper.php?show_host_metrics=1&h=bigdata001&c=bigdata001&r=hour&cs=&ce=

Flume监控之Ganglia部署与使用_第3张图片

Flume监控之Ganglia部署与使用_第4张图片

字段(图表名称)

字段含义

EventPutAttemptCount

source尝试写入channel的事件总数量

EventPutSuccessCount

成功写入channel且提交的事件总数量

EventTakeAttemptCount

sink尝试从channel拉取事件的总数量。这不意味着每次事件都被返回,因为sink拉取的时候channel可能没有任何数据。

EventTakeSuccessCount

sink成功读取的事件的总数量

StartTime

channel启动的时间(毫秒)

StopTime

channel停止的时间(毫秒)

ChannelSize

目前channel中事件的总数量

ChannelFillPercentage

channel占用百分比

ChannelCapacity

channel的容量

 

你可能感兴趣的:(Flume)