ELK集群部署及收集nginx日志
一、ELK说明
二、架构图
三、规划说明
四、安装部署nginx+logstash
五、安装部署redis
六、安装部署logstash server
七、安装部署elasticsearch集群
八、安装kibana
一、ELK说明
ELK Stack 是 Elasticsearch、Logstash、Kibana 三个开源软件的组合。在实时数据检索和分析场合，三者通常是配合共用，而且又都先后归于 Elastic.co 公司名下，故有此简称。
ELK Stack 在最近两年迅速崛起，成为机器数据分析，或者说实时日志处理领域，开源界的第一选择。和传统的日志处理方案相比，ELK Stack 具有如下几个优点：
处理方式灵活。Elasticsearch 是实时全文索引，不需要像 storm 那样预先编程才能使用；
配置简易上手。Elasticsearch 全部采用 JSON 接口，Logstash 是 Ruby DSL 设计，都是目前业界最通用的配置语法设计；
检索性能高效。虽然每次查询都是实时计算，但是优秀的设计和实现基本可以达到全天数据查询的秒级响应；
集群线性扩展。不管是 Elasticsearch 集群还是 Logstash 集群都是可以线性扩展的；
前端操作炫丽。Kibana 界面上，只需要点击鼠标，就可以完成搜索、聚合功能，生成炫丽的仪表板。
二、架构图
三、规划说明
logstash收集数据:
192.168.1.90(nginx+logstash)
192.168.1.93(nginx+logstash)
redis提供队列:
192.168.1.89(redis)
logstash-server:
192.168.1.87(logstash[server])
elasticsearch集群:
192.168.1.88(elasticsearch)
192.168.1.91(elasticsearch)
192.168.1.92(elasticsearch+kibana)
所有服务器时间同步：
# ntpdate 133.100.11.8
软件版本：
jdk-8u91-linux-x64.tar.gz elasticsearch-2.3.2.rpm logstash-2.3.2-1.noarch.rpm kibana-4.5.0-linux-x64.tar.gz
四、安装部署nginx+logstash
192.168.1.90 192.168.1.93
以上两个安装配置一样：
4.1.安装zlib
tar -xf zlib-1.2.8.tar.gz cd zlib-1.2.8 ./configure make make insatll
4.2.安装pcre
tar -xf pcre-8.37.tar.gz ./configure cd pcre-8.37 make make install
4.3.安装openssl
tar -xf openssl-1.0.2.tar.gz ./config --prefix=/usr/local --openssldir=/usr/local/openssl make make insatll
4.4.安装nginx
tar -xf nginx-1.8.1.tar.gz ./configure --prefix=/app/local/nginx \ --pid-path=/app/local/nginx \ --user=nginx \ --group=nginx \ --with-threads \ --with-file-aio \ --with-http_ssl_module \ --with-pcre=/tmp/soft/pcre-8.37 \ --with-openssl=/tmp/soft/openssl-1.0.2 make make install
4.5.查看配置文件
vim /app/local/nginx/conf/nginx.conf
user nginx;
worker_processes 1;
error_log logs/error.log notice;
pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for $request_length $msec $connection_requests $request_time';
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
access_log logs/access.log main;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
4.6.启动nginx
/app/local/nginx/sbin/nginx -c /app/local/nginx/conf/nginx.conf
4.7.部署java环境
下载oracle jdk
http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html
4.8.安装jdk
tar -xf jdk-8u91-linux-x64.tar.gz -C /usr/local/ vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8.0_91 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
4.9.安装logstsh
下载：https://www.elastic.co/
#yum install logstash-2.3.2-1.noarch.rpm
4.10.配置logstash
vim /etc/logstash/conf.d/logstash_cli.conf
input {
file {
path => ["/app/local/nginx/logs/access.log"]
type => "nginx_log"
start_position => "beginning"
}
}
output {
stdout {
codec => rubydebug
}
}
检查语法：
/opt/logstash/bin/logstash -f ./logstash_cli.conf --configtest Configuration OK
启动查看收集nginx状况
# /opt/logstash/bin/logstash -f ./logstash_cli.conf
Settings: Default pipeline workers: 4
Pipeline main started
{
"message" => "192.168.1.105 - - [11/May/2016:14:09:59 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36\" - 476 1462946999.744 9 0.000",
"@version" => "1",
"@timestamp" => "2016-05-11T06:10:00.546Z",
"path" => "/app/local/nginx/logs/access.log",
"host" => "qmcent7.com",
"type" => "nginx_log"
}
{
"message" => "192.168.1.105 - - [11/May/2016:14:09:59 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36\" - 476 1462946999.952 10 0.000",
"@version" => "1",
"@timestamp" => "2016-05-11T06:10:00.547Z",
"path" => "/app/local/nginx/logs/access.log",
"host" => "qmcent7.com",
"type" => "nginx_log"
}
........................
修改配置文件，将数据输出到redis：
vim /etc/logstash/conf.d/logstash_cli.conf
input {
file {
path => ["/app/local/nginx/logs/access.log"]
type => "nginx_log"
start_position => "beginning"
}
}
output {
redis {
host => "192.168.1.89"
key => 'logstash-redis'
data_type => 'list'
}
}
检查语法：
/opt/logstash/bin/logstash -f ./logstash_cli.conf --configtest Configuration OK
启动服务：
/etc/init.d/logstash start
查看进程：
# ps -ef | grep logstash logstash 4818 1 0 May10 pts/2 00:03:42 /usr/local/jdk1.8.0_91/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath :.:/usr/local/jdk1.8.0_91/lib/dt.jar:/usr/local/jdk1.8.0_91/lib/tools.jar -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log root 6097 3733 0 15:07 pts/2 00:00:00 grep logstash
五、安装部署redis
192.168.1.89(redis)
5.1.下载redis官网下载
tar -xf redis-3.0.7.tar.gz
cd redis-3.0.7
make
mkdir -p /usr/local/redis/{conf,bin}
cp *.conf /usr/local/redis/conf/
cp runtest* /usr/local/redis/
cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-dump redis-cli redis-sentinel redis-server redis-trib.rb /usr/local/redis/bin/
mkdir -pv /data/redis/db
mkdir /data/log/redis -pv
5.2.启动redis:
/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf
六、安装部署logstash server
192.168.1.87(logstash[server])
6.1.安装jdk
tar -xf jdk-8u91-linux-x64.tar.gz -C /usr/local/ vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8.0_91 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
6.2.安装logstsh
下载：https://www.elastic.co/
#yum install logstash-2.3.2-1.noarch.rpm
6.3.配置logstash server
vim /etc/logstash/conf.d/logstash_server.conf
input {
redis {
port => "6379"
host => "192.168.1.89"
data_type => "list"
key => "logstash-redis"
type => "redis-input"
}
}
output {
stdout {
codec => rubydebug
}
}
检查语法：
/opt/logstash/bin/logstash -f ./logstash_server.conf --configtest Configuration OK
检查收集数据：
# /opt/logstash/bin/logstash -f ./logstash_server.conf
Settings: Default pipeline workers: 4
Pipeline main started
{
"message" => "192.168.1.105 - - [11/May/2016:22:00:53 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36\" - 369 1462975253.311 1 0.000",
"@version" => "1",
"@timestamp" => "2016-05-11T14:08:15.895Z",
"path" => "/app/local/nginx/logs/access.log",
"host" => "qmcent7.com",
"type" => "nginx_log"
}
{
"message" => "192.168.1.105 - - [11/May/2016:22:00:53 +0800] \"GET /favicon.ico HTTP/1.1\" 404 570 \"http://192.168.1.93/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36\" - 310 1462975253.373 2 0.000",
"@version" => "1",
"@timestamp" => "2016-05-11T14:08:16.024Z",
"path" => "/app/local/nginx/logs/access.log",
"host" => "qmcent7.com",
"type" => "nginx_log"
.....
修改logstash配置文件：
将收集的数据输出到ES集群中
input {
redis {
port => "6379"
host => "192.168.1.89"
data_type => "list"
key => "logstash-redis"
type => "redis-input"
}
}
output {
elasticsearch {
hosts => "192.168.1.91"
index => "logstash-%{+YYYY.MM.dd}"
}
}
启动服务：
/etc/init.d/logstash start
七、安装部署elasticsearch集群
192.168.1.88(elasticsearch) 192.168.1.91(elasticsearch) 192.168.1.92(elasticsearch)
部署java环境，全部一致：
tar -xf jdk-8u91-linux-x64.tar.gz -C /usr/local/
vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8.0_91 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
安装ES、全部一致：
yum install elasticsearch-2.3.2.rpm
192.168.1.88(elasticsearch)
修改ES配置文件：
vim /etc/elasticsearch/elasticsearch.yml cluster.name: qm.ela.com node.name: n1.qm.com path.logs: /data/log/ network.host: 192.168.1.88 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.1.91","192.168.1.92"]
[88]启动服务：
/etc/init.d/elasticsearch start
192.168.1.91(elasticsearch)
修改ES配置文件：
vim /etc/elasticsearch/elasticsearch.yml cluster.name: qm.ela.com node.name: n2.qm.com path.logs: /data/log network.host: 192.168.1.91 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.1.88","192.168.1.92"]
[91]启动服务：
/etc/init.d/elasticsearch start
192.168.1.92(elasticsearch)
vim /etc/elasticsearch/elasticsearch.yml cluster.name: qm.ela.com node.name: n3.qm.com path.logs: /data/log network.host: 192.168.1.92 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.1.88","192.168.1.91"]
[92]启动服务：
/etc/init.d/elasticsearch start
查看端口：
# netstat -ntpl Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:45955 0.0.0.0:* LISTEN 1573/rpc.statd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1517/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1847/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1610/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1926/master tcp 0 0 :::44937 :::* LISTEN 1573/rpc.statd tcp 0 0 :::111 :::* LISTEN 1517/rpcbind tcp 0 0 ::ffff:192.168.1.88:9200 :::* LISTEN 1089/java tcp 0 0 ::ffff:192.168.1.88:9300 :::* LISTEN 1089/java tcp 0 0 :::22 :::* LISTEN 1847/sshd tcp 0 0 ::1:631 :::* LISTEN 1610/cupsd tcp 0 0 ::1:25 :::* LISTEN 1926/master
启动两个端口
9200：集群之间事务通信
9300：集群选举等
信息查看
1.查看健康信息：
# curl -XGET 'http://192.168.1.88:9200/_cluster/health?pretty'
{
"cluster_name" : "qm.ela.com",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 14,
"active_shards" : 28,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
2.查看节点数：
# curl -XGET '192.168.1.88:9200/_cat/nodes?v' host ip heap.percent ram.percent load node.role master name 192.168.1.91 192.168.1.91 10 87 0.00 d m n2.qm.com 192.168.1.92 192.168.1.92 7 93 0.00 d m n3.qm.com 192.168.1.88 192.168.1.88 17 88 0.00 d * n1.qm.com
由此可见，192.168.1.88已经成为主节点
3.查看节点分片信息：
# curl -XGET '192.168.1.88:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size green open .kibana 1 1 3 0 49.8kb 24.9kb green open .marvel-es-data-1 1 1 14 274 131.5kb 63.2kb green open logstash-2016.05.10 5 1 168 0 543.3kb 271.6kb green open logstash-2016.05.11 5 1 118 0 315.6kb 161kb green open .marvel-es-1-2016.05.11 1 1 38433 56 35.6mb 17.8mb green open .marvel-es-1-2016.05.10 1 1 42622 16 24.5mb 12.2mb
安装插件：
1.安装插件marvel，三个节点安装一致：
/usr/share/elasticsearch/bin/plugin install license /usr/share/elasticsearch/bin/plugin install marvel-agent
2.安装head插件，三个节点安装一致：
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
3.安装bigdesk 插件：
cd /usr/share/elasticsearch/plugins mkdir bigdesk cd bigdesk/ git clone https://github.com/lukas-vlcek/bigdesk _site sed -i '142s/==/>=/' _site/js/store/BigdeskStore.js cat >plugin-descriptor.properties<<EOF description=bigdesk - Live charts and statistics for Elasticsearch cluster. version=2.5.1 site=true name=bigdesk EOF
4.安装kopf插件：
/usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
查看安装的插件：
/usr/share/elasticsearch/bin/plugin list Installed plugins in /usr/share/elasticsearch/plugins: - head - bigdesk - analysis-kuromoji - kopf - license - marvel-agent
访问head，查看集群是否一致
访问bigdesk，查看信息：
访问kopf，查看信息：
八、安装kibana
192.168.1.92(kibana)
8.1.下载包：
https://www.elastic.co/downloads/kibana
8.2.安装：
tar -xf kibana-4.5.0-linux-x64.tar.gz -C /usr/local
安装插件：
cd /usr/local/kibana bin/kibana plugin --install elasticsearch/marvel/latest
启动服务：
bin/kibana log [12:47:34.251] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready log [12:47:34.272] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [12:47:34.275] [info][status][plugin:marvel] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [12:47:34.312] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready log [12:47:34.316] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready log [12:47:34.319] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready log [12:47:34.328] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready log [12:47:34.331] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready log [12:47:34.337] [info][status][plugin:elasticsearch] Status changed from yellow to green - Kibana index ready log [12:47:34.338] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready log [12:47:35.599] [info][status][plugin:marvel] Status changed from yellow to green - Marvel index ready log [12:47:35.638] [info][listening] Server running at http://0.0.0.0:5601
查看数据：
访问marvel：
错误说明：
Stopping elasticsearch: [ OK ] Starting elasticsearch: Exception in thread "main" java.lang.IllegalStateException: marvel plugin requires the license plugin to be installed at org.elasticsearch.marvel.license.LicenseModule.verifyLicensePlugin(LicenseModule.java:37) at org.elasticsearch.marvel.license.LicenseModule.<init>(LicenseModule.java:25) at org.elasticsearch.marvel.MarvelPlugin.nodeModules(MarvelPlugin.java:87) at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263) at org.elasticsearch.node.Node.<init>(Node.java:179) at org.elasticsearch.node.Node.<init>(Node.java:140) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) Refer to the log for complete error details. [FAILED]
需要安装license
/usr/share/elasticsearch/bin/plugin install license