使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践--转载

原文地址：https://wsgzao.github.io/post/elk/

另外可以参考：https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

前言

Elasticsearch + Logstash + Kibana（ELK）是一套开源的日志管理方案，分析网站的访问情况时我们一般会借助Google/百度/CNZZ等方式嵌入JS做数据统计，但是当网站访问异常或者被攻击时我们需要在后台分析如Nginx的具体日志，而Nginx日志分割/GoAccess/Awstats都是相对简单的单节点解决方案，针对分布式集群或者数据量级较大时会显得心有余而力不足，而ELK的出现可以使我们从容面对新的挑战。

• Logstash：负责日志的收集，处理和储存
• Elasticsearch：负责日志检索和分析
• Kibana：负责日志的可视化

ELK(Elasticsearch + Logstash + Kibana)

---

更新记录

2015年08月31日 - 初稿

阅读原文 - http://wsgzao.github.io/post/elk/

扩展阅读

CentOS 7.x安装ELK(Elasticsearch+Logstash+Kibana) - http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/
Centos 6.5 安装nginx日志分析系统 elasticsearch + logstash + redis + kibana - http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582
logstash-forwarder and grok examples - https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/
三斗室 - http://chenlinux.com/
elastic - https://www.elastic.co/guide
LTMP索引 - http://wsgzao.github.io/index/#LTMP

---

组件预览

JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch - https://www.elastic.co/downloads/elasticsearch
Logstash - https://www.elastic.co/downloads/logstash
Kibana - https://www.elastic.co/downloads/kibana
redis - http://redis.io/download

设置FQDN

创建SSL证书的时候需要配置FQDN

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

#修改hostname cat /etc/hostname elk #修改hosts cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 10-10-87-19 10.10.87.19 elk.ooxx.com elk #刷新环境 hostname -F /etc/hostname #复查结果 hostname -f elk.ooxx.com hostname elk

服务端

Java

1 2 3 4 5 6 7 8 9

cat /etc/redhat-release CentOS release 6.5 (Final) yum install java-1.7.0-openjdk java -version java version "1.7.0_85" OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01) OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)

Elasticsearch

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

#下载安装 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm yum localinstall elasticsearch-1.7.1.noarch.rpm #启动相关服务 service elasticsearch start service elasticsearch status #查看Elasticsearch的配置文件 rpm -qc elasticsearch /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf #查看端口使用情况 netstat -nltp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 1765/java tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 1765/java tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1509/sshd tcp 0 0 :::22 :::* LISTEN 1509/sshd #测试访问 curl -X GET http://localhost:9200/

Kibana

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

#下载tar包 wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz #解压 tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/ cd /usr/local/ mv kibana-4.1.1-linux-x64 kibana #创建kibana服务 vi /etc/rc.d/init.d/kibana #!/bin/bash ### BEGIN INIT INFO # Provides: kibana # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Runs kibana daemon # Description: Runs the kibana daemon as a non-root user ### END INIT INFO # Process name NAME=kibana DESC="Kibana4" PROG="/etc/init.d/kibana" # Configure location of Kibana bin KIBANA_BIN=/usr/local/kibana/bin # PID Info PID_FOLDER=/var/run/kibana/ PID_FILE=/var/run/kibana/$NAME.pid LOCK_FILE=/var/lock/subsys/$NAME PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN DAEMON=$KIBANA_BIN/$NAME # Configure User to run daemon process DAEMON_USER=root # Configure logging location KIBANA_LOG=/var/log/kibana.log # Begin Script RETVAL=0 if [ `id -u` -ne 0 ]; then echo "You need root privileges to run this script" exit 1 fi # Function library . /etc/init.d/functions start() { echo -n "Starting $DESC : " pid=`pidofproc -p $PID_FILE kibana` if [ -n "$pid" ] ; then echo "Already running." exit 0 else # Start Daemon if [ ! -d "$PID_FOLDER" ] ; then mkdir $PID_FOLDER fi daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 & sleep 2 pidofproc node > $PID_FILE RETVAL=$? [[ $? -eq 0 ]] && success || failure echo [ $RETVAL = 0 ] && touch $LOCK_FILE return $RETVAL fi } reload() { echo "Reload command is not implemented for this service." return $RETVAL } stop() { echo -n "Stopping $DESC : " killproc -p $PID_FILE $DAEMON RETVAL=$? echo [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE } case "$1" in start) start ;; stop) stop ;; status) status -p $PID_FILE $DAEMON RETVAL=$? ;; restart) stop start ;; reload) reload ;; *) # Invalid Arguments, print the following message. echo "Usage: $0 {start|stop|status|restart}" >&2 exit 2 ;; esac #修改启动权限 chmod +x /etc/rc.d/init.d/kibana #启动kibana服务 service kibana start service kibana status #查看端口

转载于:https://www.cnblogs.com/davidwang456/p/5238973.html