ELK(Elasticsearch + Logstash + Kibana) 是一套开源的日志管理方案
Elasticsearch：负责日志检索和分析
Logstash：负责日志的收集，处理和储存
Kibana：负责日志的可视化
Logstash: The server component of Logstash that processes incoming logs
Elasticsearch: Stores all of the logs
Kibana 4: Web interface for searching and visualizing logs, which will be proxied through Nginx
Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash
Reference：
JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch - https://www.elastic.co/downloads/elasticsearch
Logstash - https://www.elastic.co/downloads/logstash
Kibana - https://www.elastic.co/downloads/kibana
redis - http://redis.io/download
数据流流向如下
Logstash-forwarder--->Logstash--->Elasticsearch--->kibana--->nginx--->客户浏览器
其中Logstash-forwarder是客户端的日志收集工具将日志发送给服务端Logstash后
Logstash通过使用grok匹配规则对日志进行匹配切割
然后保存在Elasticsearch中
最后通过kibana从Elasticsearch中读取数据并转交给nginx来处理后返回给客户。
好了下面就是ELK系统的安装过程了。
首先安装JAVA环境
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm" rpm -Uvh jdk-8u65-linux-x64.rpm
或者直接yum安装jdk也行不过要保证安装好对应的版本。
安装好jdk环境之后需要安装Elasticsearch
rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch rpm -ivh https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm
修改配置文件如下
grep -v "^.*#\|^$" /etc/elasticsearch/elasticsearch.yml network.host: localhost path.data: /data/elasticsearch chown -R elasticsearch:elasticsearch /data/elasticsearch
安装Elasticsearch插件
cd /usr/share/elasticsearch/ && ./bin/plugin -install mobz/elasticsearch-head && ./bin/plugin -install lukas-vlcek/bigdesk
启动Elasticsearch
service elasticsearch start chkconfig elasticsearch on
测试Elasticsearch
curl http://localhost:9200
{
"status" : 200,
"name" : "Black Goliath",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.2",
"build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
"build_timestamp" : "2015-09-14T09:49:53Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
然后开始安装kibana
去https://www.elastic.co/downloads/kibana 找合适的版本
每个版本下面有这么一行内容一定要注意这些内容Compatible with Elasticsearch x.x -- x.x
这里选择的是kibana-4.1.3-linux-x64.tar.gz
wget https://download.elastic.co/kibana/kibana/kibana-4.1.3-linux-x64.tar.gz tar xf kibana-4.1.3-linux-x64.tar.gz mv kibana-4.1.3-linux-x64 /usr/local/kibana cd !$ grep -v "^.*#\|^$" /usr/local/kibana/config/kibana.yml port: 5601 host: "localhost" elasticsearch_url: "http://localhost:9200" elasticsearch_preserve_host: true kibana_index: ".kibana" default_app_id: "discover" request_timeout: 300000 shard_timeout: 0 verify_ssl: true bundled_plugin_ids: - plugins/dashboard/index - plugins/discover/index - plugins/doc/index - plugins/kibana/index - plugins/markdown_vis/index - plugins/metric_vis/index - plugins/settings/index - plugins/table_vis/index - plugins/vis_types/index - plugins/visualize/index
配置文件中指明kibana侦听5601端口并且通过9200端口从elasticsearch里面获取数据
启动 Kibana
nohup /usr/local/kibana/bin/kibana -l /var/log/kibana.log &
或者也可以看看下面两个脚本
cd /etc/init.d && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init cd /etc/default && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default ln -s /usr/local/kibana /opt/kibana groupadd -g 1005 kibana useradd -u 1005 -g 1005 kibana chown -R kibana:kibana /usr/local/kibana service kibana start chkconfig kibana on
安装nginx
yum -y install epel-release
yum -y install nginx httpd-tools
grep -v "^.*#\|^$" /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $upstream_response_time $request_time $body_bytes_sent '
'"$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_body" '
'$scheme $upstream_addr';
# 修改日志格式是为了匹配后面的Logstash的grok匹配规则
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
grep -v "^.*#\|^$" /etc/nginx/conf.d/kibana.conf
server {
listen 80;
server_name ocean-lab.ocean.org;
location / {
proxy_pass http://localhost:5601;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
启动nginx
service nginx start chkconfig nginx on
之后就需要安装Logstash了
rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch vi /etc/yum.repos.d/logstash.repo [logstash-1.5] name=Logstash repository for 1.5.x packages baseurl=http://packages.elasticsearch.org/logstash/1.5/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 yum -y install logstash
Logstash 安装成功了 但是仍需要配置
生成 SSL 证书
logstash和logstash-forwarder通信需要使用tls证书认证。
Logstash Forwarder上面只需公钥logstash需要配置公钥、私钥。
在logstash服务器上生成ssl证书。
创建ssl证书有两种方式一种指定IP地址一种指定fqdn(dns)。
1、指定IP地址方式 [本实验使用此种方式]
vi /etc/pki/tls/openssl.cnf
在[ v3_ca ]下面配置
subjectAltName = IP:172.16.7.11
# 切记这条很重要因为logstash-forwarder.conf 还需要 如果配置错误 就会一直无法实现认证
cd /etc/pki/tls
openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
# 注意将-days设置大点以免证书过期。
*** 如果logstash服务端的IP地址变换了证书不可用了 ***
like that:
2015/11/29 16:23:48.274974 Failed to tls handshake with 127.0.0.1 x509: certificate is valid for 172.16.7.11, not 127.0.0.1
2、使用 FQDN 方式
不需要修改openssl.cnf文件。
cd /etc/pki/tls
** CN=FQDN **
openssl req -subj '/CN=elk.suzf.net/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
# 将 elk.suzf.net 换成你自己的域名。同时到域名解析那添加 elk.suzf.net 的A记录。
配置logstash
logstash配置文件是以json格式设置参数的
配置文件位于/etc/logstash/conf.d目录下配置包括三个部分 输入/输出和过滤器
首先创建一个01-lumberjack-input.conf文件
设置lumberjack输入Logstash-Forwarder使用的协议。
cat /etc/logstash/conf.d/01-lumberjack-input.conf
input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
再来创建一个02-nginx.conf用于过滤nginx日志
cat /etc/logstash/conf.d/02-nginx.conf
filter {
if [type] == "nginx" {
grok {
match => { "message" => "%{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: %{URIPROTO:proto}/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:upstime}|-) %{NUMBER:reqtime} (?:%{NUMBER:size}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{QS:reqbody} %{WORD:scheme} (?:%{IPV4:upstream}(:%{POSINT:port})?|-)" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
add_tag => [ "geoip" ]
fields => ["country_name", "country_code2","region_name", "city_name", "real_region_name", "latitude", "longitude"]
remove_field => [ "[geoip][longitude]", "[geoip][latitude]" ]
}
}
}
这个过滤器会寻找被标记为“nginx”类型Logstash-forwarder定义的的日志尝试使用“grok”来分析传入的nginx日志使之结构化和可查询。
type要与logstash-forwarder相匹配。
同时注意将nginx日志格式设置成上面的。
日志格式不对grok匹配规则要重写。
可以通过http://grokdebug.herokuapp.com/ 在线工具进行调试。多半ELK没数据错误在此处。
grok 匹配日志不成功不要往下看了。搞对为止先。
同时多看看http://grokdebug.herokuapp.com/patterns# grok匹配模式对后面写规则匹配很受益的。
最后创建一文件来定义输出。
cat /etc/logstash/conf.d/30-lumberjack-output.conf
output {
if "_grokparsefailure" in [tags] {
file { path => "/var/log/logstash/grokparsefailure-%{type}-%{+YYYY.MM.dd}.log" }
}
elasticsearch {
host => "127.0.0.1"
protocol => "http"
index => "logstash-%{type}-%{+YYYY.MM.dd}"
document_type => "%{type}"
workers => 5
template_overwrite => true
}
#stdout { codec =>rubydebug }
}
定义结构化的日志存储到elasticsearch对于不匹配grok的日志写入到文件。
注意后面添加的过滤器文件名要位于01-99之间。因为logstash配置文件有顺序的。
在调试时候先不将日志存入到elasticsearch而是标准输出以便排错。
同时多看看日志很多错误在日志里有体现也容易定位错误在哪。
在启动logstash服务之前最好先进行配置文件检测如下
/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/* Configuration OK
也可指定文件名检测直到OK才行。不然logstash服务器起不起来。
最后就是启动logstash服务了。
service logstash start chkconfig logstash on
然后就是配置Logstash-forwarder客户端了。
安装logstash-forwarder
wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm rpm -ivh logstash-forwarder-0.4.0-1.x86_64.rpm
需要将在安装logstash时候创建的ssl证书的公钥拷贝到每台logstash-forwarder服务器上。
scp 172.16.7.11:/etc/pki/tls/certs/logstash-forwarder.crt /etc/pki/tls/certs/
配置logstash-forwarder
grep -v "^.*#\|^$" /etc/logstash-forwarder.conf
{
"network": {
"servers": [ "172.16.7.11:5043" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/var/log/messages",
"/var/log/secure"
],
"fields": { "type": "syslog" }
},{
"paths": [
"/var/log/nginx/access.log"
],
"fields": { "type": "nginx" }
}
]
}
这也是个json个是的配置文件
json格式不对logstash-forwarder服务是启动不起来的
service logstash-forwarder start
连接到 Kibana
创建index
当上面的所有都配置正确的话就可以访问kibana来查看数据了。
访问效果如下所示
参考文档：
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7
https://www.digitalocean.com/community/tutorials/how-to-troubleshoot-common-elk-stack-issues
http://xianglinhu.blog.51cto.com/5787032/1716274
http://www.wklken.me/posts/2015/04/26/elk-for-nginx-log.html