2019独角兽企业重金招聘Python工程师标准>>> 
ELK实时日志分析系统搭建
相关文章：searchguard插件安装：http://my.oschina.net/lengchuan/blog/729745
ElasticSearch安装及其配置
安装
- ELK下载：https://www.elastic.co/downloads/
- 解压安装：
tar -zxvf elasticsearch-2.3.3.tar.gz
cd elasticsearch-2.3.3
插件安装
- 安装head插件
./bin/plugin install mobz/elasticsearch-head
ES配置
编辑配置文件elasticsearch.yml
vi config/elasticsearch.yml
修改以下配置项：
cluster.name=es_cluster #集群名称，同一集群名称必须相同
node.name=node0 #集群节点的名称
path.data=/tmp/elasticsearch/data #数据存储路径
path.logs=/tmp/elasticsearch/logs #日志存储路径
network.host=192.168.1.101 #当前hostname或IP
network.port=9200 #端口号，默认9200
其它配置按需配置。
启动ES
ES跟其他的节点的传输端口为9300，接受HTTP请求的端口为9200。
./bin/elasticsearch
或者后台进程启动
nohup ./bin/elasticsearch &
浏览器打开http://ip:9200/就可以获取到ES相关信息 head插件页面：http://ip:9200/_plugin/head,即可查看集群页面信息
Logstash安装配置
安装
- ELK下载：https://www.elastic.co/downloads/
- 解压安装：
tar -zxvf logstash-2.3.3.tar.gz
cd logstash-2.3.3
配置
编写配置文件(名字和位置可以随意，这里放在config目录下，取名为log4j_to_es.conf，配置文件可以有多个，可以同时启动多个logstash实例) 更多配置请参考logstash的官方文档:https://www.elastic.co/guide/en/logstash/index.html
关于logstash收集log4j的日志信息，请大家参考另一篇博文：http://my.oschina.net/lengchuan/blog/725358
mkdir config vi config/log4j_to_es.conf 具体的配置请参考logstash的官方文档:https://www.elastic.co/guide/en/logstash/index.html 输入以下内容:
#For detail structure of this file #Set:https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html input { #For detail config for log4j as input, #See:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html log4j { mode => "client" host => "localhost" port => 4560 type => "cms_log" } log4j { mode => "client" host => "localhost" port => 4561 type => "front_log" } } filter { #Only matched data are send to output. } output { #For detail config for elasticsearch as output, #See:https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html elasticsearch { action => "index" #The operation on ES hosts => ["ip地址:9200"] #ElasticSearch host, can be array. index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" #The index to write data to. } }
启动logstash
logstash命令只有2个参数，我们使用agent来启动,使用-f来启动：
nohup ./bin/logstash agent -f config/log4j_to_es.conf &
kibana4安装及配置
安装
- ELK下载：https://www.elastic.co/downloads/
- 解压安装 :
tar -zxvf kibana-4.5.0-linux-x86.tar.gz
cd kibana-4.5.0-linux-x86
vi config/kibana.yml
配置
修改以下几项:
server.port: 5601
server.host: "localhost"
elasticsearch.url: http://localohost:9200
kibana.index: ".kibana"
启动
nohup ./bin/kibana &
在生产环境部署
Nginx 代理配置
upstream kibana4 {
server 127.0.0.1:5601 fail_timeout=0;
}
server {
listen *:80;
server_name kibana_server;
access_log /var/log/nginx/kibana.srv-log-dev.log;
error_log /var/log/nginx/kibana.srv-log-dev.error.log;
ssl on;
ssl_certificate /etc/nginx/ssl/all.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
location / {
root /var/www/kibana;
index index.html index.htm;
}
location ~ ^/kibana4/.* {
proxy_pass http://kibana4;
rewrite ^/kibana4/(.*) /$1 break;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
}
}
如果用户够多，可以单独跑一个 kibana4 集群，然后在 upstream 配置段中添加多个代理地址做负载均衡。