Logstash+ElasticSearch+Kibana3架构日志系统
一，分析系统简介
日志监控和分析在保障业务稳定运行时，起到了很重要的作用，不过一般情况下日志都分散在各个生产服务器，且开发人员无法登陆生产服务器，这时候就需要一个集中式的日志收集装置，对日志中的关键字进行监控，触发异常时进行报警，并且开发人员能够查看相关日志。logstash+elasticsearch+kibana3就是实现这样功能的一套系统，并且功能更强大。
logstash是一个管理日志和事件的工具，你可以收集它们，解析它们，并存储它们以供以后使用(例如日志搜索)，logstash有一个内置的web界面，用来搜索你的所有日志。logstash在部署时有两种运行模式：standalone和centralized：
* standalone：standalone的意思是所有的事情都在一台服务器上运行，包括日志收集、日志索引、前端WEB界面都部署在一台机器上。
* centralized：就是多服务器模式，从很多服务器运输(ship)日志到一台总的日志(collector)服务器上用来索引和查找。
需要注意的是logstash本身并没有什么shipper和collector这种说法，因为不论是运输日志的进程还是汇集总的日志的进程运行的都是同一个程序，只是使用的配置文件不同而已。
elasticsearch:
基于lucene的开源搜索引擎，是一个分布式的搜索分析系统，主要特点有：realtime data、real time analytics、distributed、high availability、multi-tenancy、fulltext search、document oriented、conflict management、schema free、restful api等等。
kibana3:
可视化日志和数据系统，作为WEB前端可以很容易的和elasticsearch系统结合。kibana有版本2和版本3的区分，版本2采用ruby编写，部署起来很麻烦，需要安装很多ruby依赖包(目前网上多是这个版本的部署)，版本3采用纯html+css编写，因此部署起来很方便，解压即用。
二，测试步骤
1，测试架构两台机器试验，CentOS6.5
192.168.6.140：logstash agent，redis，jdk
192.168.6.135：logstash index， elasticsearch ， kibana，jdk
其中redis上安装agent端，agent和index的logstash安装是一样的，唯一不同点是conf文件不同。
2，redis和logstash agent安装（192.168.6.140）
1），安装java环境
http://www.java.com/en/download/manual.jsp
rpm -ivh jdk-7u67-linux-x64.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
jfxrt.jar...
#java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build24.65-b04, mixed mode)
2），安装redis
wget
http://download.redis.io/releases/redis-2.8.17.tar.gz
tar -zxvf redis-2.8.17.tar.gz
cd /redis-2.8.17.tar.gz
先安装tcl,否则下面会报错
yum install tcl -y
make MALLOC=libc
make test
make install
# pwd
/soft/redis-2.8.17/utils
./install_server.sh
Welcome to the redis service installer
This script will help you easily set up a runningredis server
Please select the redis port for this instance: [6379]
Selecting default: 6379
Please select the redis config file name[/etc/redis/6379.conf]
Selected default - /etc/redis/6379.conf
Please select the redis log file name[/var/log/redis_6379.log]
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance[/var/lib/redis/6379]
Selected default - /var/lib/redis/6379
Please select the redis executable path[/usr/local/bin/redis-server]
Selected config:
Port :6379
Config file :/etc/redis/6379.conf
Log file :/var/log/redis_6379.log
Data dir :/var/lib/redis/6379
Executable :/usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
# pwd
/soft/redis-2.8.17/src
[root@logserver src]# ./redis-cli -h 127.0.0.1 -p 6379
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> set name foo
OK
127.0.0.1:6379> get name
"foo"
127.0.0.1:6379> bye
(error) ERR unknown command 'bye'
127.0.0.1:6379> quit
3），安装logstashagent
# rpm -ivhlogstash-1.4.2-1_2c0f5a1.noarch.rpm
Preparing... ########################################### [100%]
1:logstash ########################################### [100%]
[root@logserver soft]# rpm -ivhlogstash-contrib-1.4.2-1_efd53ef.noarch.rpm
Preparing... ########################################### [100%]
1:logstash-contrib ########################################### [100%]
修改配置文件
cat /etc/logstash/conf.d/logstash.agent.conf
input {
file {
path =>"/var/log/messages"
type => "syslog"
}
file {
path => "/var/log/httpd/access_log"
type => "apache"
}
}
output {
redis {
host => "192.168.6.140"
data_type =>"list"
key => "logstash"
}
}
启动logstash agent
service logstash start
root 14203 0.7 32.6 1397132 158811pxs/0 SNl 14:23 1:02 /usr/java/jdk1.7.0_67/bin/java -Djava.io.tmpdir=/var/lib/logstash-Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib/opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l/var/log/logstash/logstash.log
2，logstashindex 安装（192.168.6.135）
1），安装java环境
http://www.java.com/en/download/manual.jsp
rpm -ivh jdk-7u67-linux-x64.rpm
Preparing... ###########################################[100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
jfxrt.jar...
#java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build24.65-b04, mixed mode)
2），安装logstash
# rpm -ivhlogstash-1.4.2-1_2c0f5a1.noarch.rpm
Preparing... ###########################################[100%]
1:logstash ########################################### [100%]
[root@logserver soft]# rpm -ivhlogstash-contrib-1.4.2-1_efd53ef.noarch.rpm
Preparing... ###########################################[100%]
1:logstash-contrib ########################################### [100%]
修改配置文件
cat/etc/logstash/conf.d/logstash.index.conf
input {
redis
{
host => "192.168.6.140"
data_type =>"list"
port => "6379"
key => "logstash"
type => "redis-input"
}
}
output
{
elasticsearch {
host => "192.168.6.135"
port => "9300"
}
}
启动logstash index
service logstash start
ps aux |grep logstash
root 6365 1.1 38.5 1319172 140912 ? SNl 14:22 1:27 /usr/bin/java-Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC-XX:+UseConcMarkSweepGC -Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib/opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l/var/log/logstash/logstash.log
3)，安装elasicsearch
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.noarch.rpm
rpm -ivh elasticsearch-1.3.4.noarch.rpm
Preparing... ###########################################[100%]
1:elasticsearch ########################################### [100%]
### NOT starting on installation, pleaseexecute the following statements to configure elasticsearch to start automaticallyusing chkconfig
sudo /sbin/chkconfig --add elasticsearch
### You can start elasticsearch byexecuting
sudo service elasticsearch start
chkconfig --add elasticsearch
chkconfig elasticsearch on
service elasticsearch start
4),安装httpd和kibana3
yum -y install httpd
wget
https://download.elasticsearch.org/kibana/kibana/kibana-3.1.1.tar.gz
tar kibana-3.1.1.tar.gz –C/var/www/html
mv /var/www/html/kibana-3.1.1 /var/www/html/kibana
修改默认面板为logstash.json
grep default_route/var/www/html/kibana/config.js
default_route :'/dashboard/file/logstash.json',
启动httpd服务器，在web浏览器上访问
http://192.168.6.135/kibana/#/dashboard/file/default.json
