ELK(ElasticSearch、Logstash和Kiabana)

=============基础环境安装(需要epel源)================

[on master 10.0.0.20 slave 10.0.0.21]
1)下载并安装GPG Key
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2)添加yum仓库
vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3)安装elasticsearch
        yum install -y elasticsearch
4)  安装Redis nginx java
        yum install -y redis nginx  java
安装完java后,检测
        java -version

=============[on master 10.0.0.20]====================

systemctl stop firewalld && setenforce 0
hostnamectl set-hostname master
cat >> /etc/hosts <

=====================[on slave 10.0.0.21]===============

systemctl stop firewalld && setenforce 0
hostnamectl set-hostname slave
cat >> /etc/hosts <

=================访问两个插件:================

http://10.0.0.20:9200/_plugin/head/
http://10.0.0.20:9200/_plugin/kopf/#!/cluster

http://10.0.0.21:9200/_plugin/head/
http://10.0.0.21:9200/_plugin/kopf/#!/cluster

====Logstash安装配置(这个在客户机上是要安装的。[on master 10.0.0.20 slave 10.0.0.21])===

基础环境安装(客户端安装logstash,收集到的数据写入到elasticsearch里,就可以登陆logstash界面查看到了)
1)下载并安装GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2)添加yum仓库
[root@hadoop-node1 ~]# vim /etc/yum.repos.d/logstash.repo
[logstash-2.1]
name=Logstash repository for 2.1.x packages
baseurl=http://packages.elastic.co/logstash/2.1/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3)安装logstash
[root@elk-node1 ~]# yum install -y logstash
4)logstash启动
[root@elk-node1 ~]# systemctl start elasticsearch
[root@elk-node1 ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-11-07 18:33:28 CST; 3 days ago
     Docs: http://www.elastic.co
 Main PID: 8275 (java)
   CGroup: /system.slice/elasticsearch.service
            └─8275 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC  -XX:CMSInitiatingOccupancyFrac...

===================Kibana安装配置=====================

1)kibana的安装:
[root@master ~]# cd /usr/local/src
[root@master src]# wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
[root@master src]# tar zxf kibana-4.3.1-linux-x64.tar.gz
[root@master src]# mv kibana-4.3.1-linux-x64 /usr/local/kibana
2)修改配置文件:
[root@slave config]# pwd
/usr/local/kibana/config
[root@master config]# cp kibana.yml kibana.yml.bak
[root@master config]# vim kibana.yml 
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.0.0.20:9200"
kibana.index: ".kibana"
[root@master ~]# yum -y install screen
[root@master ~]# screen  -S "zz"                        #这样就另开启了一个终端窗口
[root@master config]# /usr/local/kibana/bin/kibana    [ctrl+Z 运行到后台]
  log   [04:44:50.393] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready
  log   [04:44:50.428] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [04:44:50.448] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready
  log   [04:44:50.458] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready
在起一个终端
[root@maste终端2 ~]# screen -ls
There is a screen on:
        15041.zz pts-0.elk-node1   (Detached)
1 Socket in /var/run/screen/S-root.
[root@tivf18 root]# screen -r zz
3)收集java日志,其中包含上面讲到的日志收集
[root@maste终端2 ~]# vim  file.conf
input {
    file {
      path => "/var/log/messages"
      type => "system"
      start_position => "beginning"
    }
}
input {
    file {
       path => "/var/log/elasticsearch/huanqiu.log"
       type => "es-error"
       start_position => "beginning"  \\死的
    }
}
output {
 
    if [type] == "system"{
        elasticsearch {
           hosts => ["10.0.0.20:9200"]
           index => "system-%{+YYYY.MM.dd}"
        }
    }
    if [type] == "es-error"{
            elasticsearch {
               hosts => ["10.0.0.20:9200"]
               index => "es-error-%{+YYYY.MM.dd}"
            }
        }
}
[root@maste终端2 ~]# /opt/logstash/bin/logstash -f file.conf &  \\放后台
  访问kibana:http://10.0.0.20:5601/
  在settings中填写 system-*  (file.conf里定义的system-%{+YYYY.MM.dd})--》create
a)查看日志登陆,需要点击“Discover”-->"message",点击它后面的“add”
注意:
需要右边查看日志内容时带什么属性,就在左边点击相应属性后面的“add”
b)添加新的日志采集项,点击Settings->+Add New,比如添加
system系统日志。注意后面的*不要忘了。
c)如果打开kibana查看日志,发现没有日志内容,出现“No results found”,如下图所示,这说明要查看的日志在当前时间没有日志信息
输出,可以点击右上角的时间钟来调试日志信息的查看。

日志配置实例

修改nginx的配置文件,分别在nginx.conf的http和server配置区域添加下面内容:
http里追加下面:
log_format  json '{"@timestamp":"$time_iso8601",'
                           '"@version":"1",'
                           '"client":"$remote_addr",'
                           '"url":"$uri",'
                           '"status":"$status",'
                           '"domain":"$host",'
                           '"host":"$server_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"referer": "$http_referer",'
                           '"ua": "$http_user_agent"'
               '}';

server追加下面:
access_log  /var/log/nginx/access_json.log  json;

启动nginx服务:
[root@master ~]# systemctl start nginx
[root@master ~]# systemctl status nginx
[root@master ~]# cat file.conf  (基于端口要编辑 /etc/rsyslog.conf)
input {
    file {
      path => "/var/log/messages"
      type => "system"
      start_position => "beginning"
    }
 
    file {
       path => "/var/log/elasticsearch/huanqiu.log"
       type => "es-error"
       start_position => "beginning"
       codec => multiline {
           pattern => "^\["
           negate => true
           what => "previous"
       }
    }
    file {
       path => "/var/log/nginx/access_json.log"
       codec => json
       start_position => "beginning"
       type => "nginx-log"
    }
    syslog {
        type => "system-syslog"
        host => "10.0.0.20"
        port => "514"
    }
}
 
 
output {
 
    if [type] == "system"{
        elasticsearch {
           hosts => ["10.0.0.20:9200"]
           index => "system-%{+YYYY.MM.dd}"
        }
    }
 
    if [type] == "es-error"{
        elasticsearch {
           hosts => ["10.0.0.20:9200"]
           index => "es-error-%{+YYYY.MM.dd}"
        }
    }
    if [type] == "nginx-log"{
        elasticsearch {
           hosts => ["10.0.0.20:9200"]
           index => "nginx-log-%{+YYYY.MM.dd}"
        }
    }
     if [type] == "system-syslog"{
        elasticsearch {
           hosts => ["10.0.0.21:9200"]
           index => "system-syslog-%{+YYYY.MM.dd}"
        }
    }
}
[root@master ~]# /opt/logstash/bin/logstash -f file.conf --configtest
在开一个终端(screen 也可)
[root@master ~]# /usr/local/kibana/bin/kibana


基于端口收集系统日志

格式
[root@elk-node1 ~]# cat syslog.conf
input {
    syslog {
        type => "system-syslog"
        host => "10.0.0.21"
        port => "514"
    }
}
 
output {
    stdout {
        codec => "rubydebug"
    }
}
对上面的采集文件进行执行:
[root@elk-node1 ~]# /opt/logstash/bin/logstash -f syslog.conf
重新开启一个窗口,查看服务是否启动:
[root@master ~]# netstat -ntlp|grep 514
tcp6       0      0 192.168.1.160:514       :::*     LISTEN      17842/java          
[root@master ~]# vim /etc/rsyslog.conf
#*.* @@remote-host:514                   【在此行下面添加如下内容】
*.* @192.168.1.160    客户端上的配置
[root@master ~]# systemctl  restart rsyslog
回到原来的窗口(即上面采集文件的执行终端),就会出现数据:

你可能感兴趣的:(ELK(ElasticSearch、Logstash和Kiabana))