elk实验

实验拓扑图

lab

1、jdk环境部署好（java-1.8.0-openjdk）
2、ELK软件版
redis 2.8 epel
logstash 1.5 rpm
es 1.7 rpm
kibana 4.1 rpm
3、安装部署
elk-node3:(logstash,nginx 192.168.9.120)
# ~]# yum install /data/pkg/logstash-1.5.4-1.noarch.rpm nginx -y
# systemctl start nginx.service
# systemctl enable nginx.service
# vim /etc/logstash/conf.d/nginx-redis.conf
# input {
# file {
# path => ["/var/log/nginx/access.log"]
# type => "nginxlog"
# }
# }
# output {
# redis {
# host => "192.168.9.119"
# port => "6379"
# data_type => "list"
# key => "logstash-nginxlog"
# }
# }
# ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-redis.conf --configtest
# ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-redis.conf

elk-node2:(redis 192.168.9.119)
# vim /etc/redis.conf
# bind 0.0.0.0
# systemctl start redis.service
# systemctl enable redis.service

elk-node1:(logstash-server 192.168.9.118)
# ~]# yum install /data/pkg/logstash-1.5.4-1.noarch.rpm -y
# vi /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/nginx
# NGUSERNAME [a-zA-Z.@-+_%]+
# NGUSER %{NGUSERNAME}
# NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}
# ========
# vim /etc/logstash/conf.d/redis-grok-es.conf
# input {
# redis {
# host => "192.168.9.119"
# port => "6379"
# data_type => "list"
# key => "logstash-nginxlog"
# }
# }
# filter {
# grok {
# match => {"message" => "%{NGINXACCESS}"}
# }
# }
# output {
# elasticsearch {
# cluster => "loges"
# index => "logstash-%{+YYYY.MM.dd}"
# }
# }
elk:(elasticsearch,kibana 192.168.9.77)
# yum install -y elasticsearch-1.7.2.noarch.rpm
# vi /etc/elasticsearch/elasticsearch.yml
# cluster.name: loges
# node.name: "elk"
# 安装head插件，上传到plugins目录解压就可以用了
# cd /usr/share/elasticsearch/plugins/
# unzip elasticsearch-head-latest.zip
# mv elasticsearch-head-master/ head
# systemctl enable elasticsearch.service
# systemctl start elasticsearch.service
注意点：(简单部署测试)
1、elasticsearch是最终的数据分布式存储。
2、logstash-server实时去redis拉取数据
3、logstash-agent实时向redis推送数据
4、当运气起来后，redis中是看不到数据了，查看keys是显示没有一个keys，其实数据已经被推送到了ES。
5、web日志数据结构化(grok)是在logstash-server上实施(当然也可以在agent，在大规模环境下，数据统一由logstash-server处理会比较好，减少前段web负载，日志时间)
ES查看数据信息：

head

# kibana
# tar xf kibana-4.1.2-linux-x64.tar.gz -C /usr/local/
# chown -R root.root kibana-4.1.2-linux-x64/
# ln -s kibana-4.1.2-linux-x64/ kibana
# vim kibana/config/kibana.yml
# elasticsearch_url: " http://localhost:9200"
# /usr/local/kibana/bin/kibana &
kibana展示：

注意了：
做这个实验的时候，我系统时间没有同步ntp,时区也不对
ntpdate cn.ntp.org.cn
timedatectl set-timezone Asia/Shanghai
导致在kibana discovery数据的时候，没有一笔数据，最好是先设置时间，时区，
在kibana上查看最近几天的数据才可以发现数据。