elk+redis centos6.6安装与配置
本人安装配置ELK走了不少弯路，初学者按以下步骤安装及可使用； 建议安装配置之前先对阅读相关架构原理。
logstash,elasticsearch,kibana 怎么进行nginx的日志分析呢？首先，架构方面，nginx是有日志文件的，它的每个请求的状态等都有日志文件进行记录。其次，需要有个队 列，redis的list结构正好可以作为队列使用。然后分析使用elasticsearch就可以进行分析和查询了。
我们需要的是一个分布式的，日志收集和分析系统。logstash有agent和indexer两个角色。对于agent角色，放在单独的web机 器上面，然后这个agent不断地读取nginx的日志文件，每当它读到新的日志信息以后，就将日志传送到网络上的一台redis队列上。对于队列上的这 些未处理的日志，有不同的几台logstash indexer进行接收和分析。分析之后存储到elasticsearch进行搜索分析。再由统一的kibana进行日志web界面的展示。
相关环境：Centos_x64_6.6 redis2.8.23 Elasticsearch 2.0.0, Logstash 2.0.0, Kibana4.2.0
软件下载地址：
redis： http://redis.io/download Elasticsearch: wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.0.0/elasticsearch-2.0.0.rpm Logstash: wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.0.0-1.noarch.rpmKibana: wget https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz Kibana: wget https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz jdk： wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpmyum localinstall jdk-8u65-linux-x64.rpm
Install Java 8
[root@localhost src]# rpm -ivh jdk-8u65-linux-x64.rpm Preparing... ########################################### [100%] 1:jdk1.8.0_65 ########################################### [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... jfxrt.jar... [root@localhost src]# ln -s /usr/java/jdk1.8.0_65/jre/bin/java /usr/bin/java
Install redis
先安装依赖tcl [root@localhost src]# yum install tcl 安装redis [root@localhost src]#tar zxvf redis-2.8.23.tar.gz [root@localhost src]# cd redis-2.8.23 [root@localhost redis-2.8.23]# [root@localhost redis-2.8.23]# make [root@localhost redis-2.8.23]# make install redis安装后配置
[root@localhost redis-2.8.23]# cd utils/ [root@localhost utils]# ./install_server.sh
Welcome to the redis service installer This script will help you easily set up a running redis server Please select the redis port for this instance: [6379] Selecting default: 6379 Please select the redis config file name [/etc/redis/6379.conf] Selected default - /etc/redis/6379.conf Please select the redis log file name [/var/log/redis_6379.log] Selected default - /var/log/redis_6379.log Please select the data directory for this instance [/var/lib/redis/6379] Selected default - /var/lib/redis/6379 Please select the redis executable path [/usr/local/bin/redis-server] Selected config: Port : 6379 Config file : /etc/redis/6379.conf Log file : /var/log/redis_6379.log Data dir : /var/lib/redis/6379 Executable : /usr/local/bin/redis-server Cli Executable : /usr/local/bin/redis-cli Is this ok? Then press ENTER to go on or Ctrl-C to abort. Copied /tmp/6379.conf => /etc/init.d/redis_6379 Installing service... Successfully added to chkconfig! Successfully added to runlevels 345! Starting Redis server... Installation successful!
Install Elasticsearch
[root@localhost src]#rpm -ivh elasticsearch-2.0.0.rpm warning: elasticsearch-2.0.0.rpm: Header V4 RSA/SHA1 Signature, key ID d88e42b4: NOKEY Preparing... ########################################### [100%] Creating elasticsearch group... OK Creating elasticsearch user... OK 1:elasticsearch ########################################### [100%] ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig sudo chkconfig --add elasticsearch ### You can start elasticsearch service by executing sudo service elasticsearch start [root@localhost src]# /etc/init.d/elasticsearch start 正在启动 elasticsearch： [确定] [root@localhost src]# chkconfig elasticsearch on
Install Kibana
[root@localhost opt]#groupadd -g 1005 kibana [root@localhost opt]#useradd -u 1005 -g 1005 kibana [root@localhost opt]# tar zxvf kibana-4.2.0-linux-x64.tar.gz -C /opt/ [root@localhost ~]# cd /opt[root@localhost opt]# mv kibana-4.2.0-linux-x64/ kibana [root@localhost opt]#chown -R kibana: /opt/kibana [root@localhost opt]#cd /etc/init.d && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init [root@localhost opt]#cd /etc/init.d && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-initcd /etc/default && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default [root@localhost opt]#service kibana start [root@localhost opt]#chkconfig kibana on 如果启动不成功请手动启动 nohup /opt/kibana/bin/kibana -l /var/log/kibana.log & 检查5601端口是否启动 netstat -ntlup |grep ":5601"
Install Logstash
[root@localhost src]rpm -ivh logstash-2.0.0-1.noarch.rpm
Preparing... ########################################### [100%]
1:logstash ########################################### [100%]
server端配置
[root@localhost conf.d]# cat /etc/logstash/conf.d/index.conf
input {
redis {
host => "127.0.0.1"
port => 6379
data_type => "list"
key => "logstash:redis"
type => "redis-input"
}
}
filter {
grok {
match => [ "message", "%{WORD:http_host} %{URIHOST:api_domain} %{IP:inner_ip} %{IP:lvs_ip} \[%{HTTPDATE:timestamp}\] \"%{WORD:http_verb} %{URIPATH:baseurl}(?:\?%{NOTSPACE:request}|) HTTP/%{NUMBER:http_version}\" (?:-|%{NOTSPACE:request}) %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float} (?:%{NUMBER:time_backend_response:float}|-)"]
kv {
prefix => "request."
field_split => "&"
source => "request"
}
urldecode {
all_fields => true
}
date { type => "log-date"
match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}
本机日志配置
[root@localhost conf.d]# cat /etc/logstash/conf.d/agent.conf
input {
file { path => "日志路径" } #logstash用户需要对文件有读权限
}
output {
redis {
host => "127.0.0.1" #redis IP地址
data_type => "list"
key => "logstash:redis"
}
}
[root@localhost src]# /etc/init.d/logstash start
[root@localhost src]# chkconfig logstash on
客房端安装配置
在nginx.conf 中设置日志格式：logstash
log_format logstash '$http_host $server_addr $remote_addr [$time_local] "$request" ' '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" ' '$request_time $upstream_response_time';
[root@localhost src]rpm -ivh logstash-2.0.0-1.noarch.rpm Preparing... ########################################### [100%] 1:logstash ########################################### [100%]
[root@localhost conf.d]# cat /etc/logstash/conf.d/agent.conf
input {
file { path => "日志路径" } #logstash用户需要对文件有读权限
}
output {
redis {
host => "x.x.x.x" #redis IP地址
data_type => "list"
key => "logstash:redis"
}
}