单机部署ELK日志收集、分析系统
最近做日志分析，发现logstash较符合自己的需求，
Logstash:做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助。一般使用shipper作为log收集、indexer作为log转载。
Logstash shipper收集log 并将log转发给redis 存储
Logstash indexer从redis中读取数据并转发给elasticsearch
redis:是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。
Elasticsearch:elasticsearch是基于lucene的开源搜索引擎,用来做索引。
Kibana: 开源web展现,界面很漂亮,是一个功能强大的elasticsearch数据显示客户端，logstash已经内置了kibana，你也可以单独部署kibana，最新版的kibana3是纯html+js客户端。
软件下载目录
http://www.elasticsearch.org/downloads/
我的环境如下
IP:192.168.81.44 t44 OS:CentOS6.5 x86_64 openjdk version "1.8.0_31" nginx-1.0.15 redis-2.4.10 elasticsearch-1.5.0 logstash-1.4.2 kibana-3.1.0
一、配置epel YUM源
yum -y localinstall http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
二、安装jdk环境
yum -y install java-1.8.0-openjdk
三、安装redis、启动redis
yum -y install redis ; /etc/init.d/redis restart
四、安装配置Elasticsearch、启动Elasticsearch
wget -c https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.0.tar.gz -O /root/elasticsearch-1.5.0.tar.gz tar -xvf /root/elasticsearch-1.5.0.tar.gz -C /usr/local/
添加2行配置:
tail -n2 /usr/local/elasticsearch-1.5.0/config/elasticsearch.yml http.cors.allow-origin: "/.*/" http.cors.enabled: true
启动Elasticsearch
/usr/local/elasticsearch-1.5.0/bin/elasticsearch -d
查看Elasticsearch日志
tail -f /usr/local/elasticsearch-1.5.0/logs/elasticsearch.log
五、安装配置Logstash、启动Logstash
wget -N https://download.elastic.co/logstash/logstash/logstash-1.4.2.tar.gz -O /root/logstash-1.4.2.tar.gz tar -xvf /root/logstash-1.4.2.tar.gz -C /usr/local/
配置index.conf
cat /usr/local/logstash-1.4.2/bin/index.conf
input {
redis {
host => "127.0.0.1"
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use the 'json' codec here because we expect to read
# json events from redis.
codec =>json
}
file {
type =>"t44message"
path =>["/var/log/messages"]
}
file {
type =>"t44secure"
path =>["/var/log/secure"]
}
file {
type =>"t44nginx"
path =>["/var/log/nginx/*.log"]
}
}
output {
# stdout { debug => true debug_format => "json"}
stdout { codec =>rubydebug }
elasticsearch {
host => "127.0.0.1"
}
}
配置filter.conf
cat /usr/local/logstash-1.4.2/bin/filter.conf
input {
redis {
host => "127.0.0.1"
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use the 'json' codec here because we expect to read
# json events from redis.
codec =>json
}
file {
type =>"t44message"
path =>["/var/log/nginx/*.log"]
}
}
filter {
grok {
match => { "message" => "No such file or directory" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
if "No such file or directory" in [t44message]
{ exec {
command => "echo '%{@timestamp} %{@message}' | mail -s nginx_no_such_file_or_directory -u alarm [email protected]"
} }
}
配置syslog.conf
cat /usr/local/logstash-1.4.2/bin/syslog.conf
input {
tcp {
port => 514
type => syslog
}
udp {
port => 514
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
启动Logstash
/usr/local/logstash-1.4.2/bin/logstash -f /usr/local/logstash-1.4.2/bin/index.conf /usr/local/logstash-1.4.2/bin/logstash -f /usr/local/logstash-1.4.2/bin/filter.conf /usr/local/logstash-1.4.2/bin/logstash -f /usr/local/logstash-1.4.2/bin/syslog.conf
六、安装nginx、kibana
yum -y install nginx wget -c https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz -O /root/kibana-3.1.0.tar.gz tar -xvf /root/kibana-3.1.0.tar.gz -C /usr/share/nginx/html/
配置nginx:
egrep -v '^#| #' /etc/nginx/conf.d/default.conf |grep -v '^$'
server {
listen 80 default_server;
server_name _;
include /etc/nginx/default.d/*.conf;
location / {
root /usr/share/nginx/html;
index index.html index.htm index.php;
}
error_page 404 /404.html;
location = /404.html {
root /usr/share/nginx/html;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location ~ \.php$ {
root /usr/share/nginx/html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_buffer_size 32k;
fastcgi_buffers 8 32k;
include fastcgi_params;
}
}
配置kibana:
grep 'elasticsearch:' /usr/share/nginx/html/kibana/config.js |grep -v '\*'
elasticsearch: "http://192.168.81.44:9200",
\cp /usr/share/nginx/html/kibana/app/dashboards/default.json{,.bak}
\cp /usr/share/nginx/html/kibana/app/dashboards/logstash.json /usr/share/nginx/html/kibana/app/dashboards/default.json
访问kibana:
http://192.168.81.44/kibana/
本文出自 “让一切随风” 博客，谢绝转载！