ELK日志收集部署

时间同步：

ntpdate pool.ntp.org
echo ‘*/5 * * * * ntpdate pool.ntp.org’>> /var/spool/cron/root

关闭防火墙和selinux

/etc/init.d/iptables  stop
chkconfig iptables off
Sed –I ‘s/SELINUX=enforcing/SELINUX=disabled/g’  /etc/selinux/config

安装java

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
tar zxf jdk-7u45-linux-x64.tar.gz
mv jdk1.8.0_92/ /usr/local/jdk
设置jdk环境变量
vi /etc/profile
-------------------------------------------------------
JAVA_HOME=/usr/local/jdk
PATH=$PATH:$JAVA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export JAVA_HOME PATHCLASSPATH
-------------------------------------------------------
source /etc/profile
java -version


下载redis

wget 
http://download.redis.io/releases/redis-3.2.3.tar.gz
tar zxf redis-3.2.3.tar.gz
cd redis-3.2.3
make 
make PREFIX=/usr/local/redis install
mkdir /usr/local/redis/conf
cp  redis.conf /usr/local/redis/conf/redis.conf.bak
cd /usr/local/redis/conf
cp redis.conf.bak redis.conf

添加环境变量

echo 'PATH=$PATH:/usr/local/redis/bin/' >>/etc/profilesource /etc/profile
启动redis：
/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf &


下载logstash elasticsearch  kibana

https://www.elastic.co/downloads
elasticsearch-5.0.0.tar.gz        
logstash-5.0.0.tar.gz
kibana-5.0.0-linux-x86_64.tar.gz

解压文件：

tar zxf logstash-5.0.0.tar.gz 
tar zxf elasticsearch-5.0.0.tar.gz 
tar zxf kibana-5.0.0-linux-x86_64.tar.gz

移动到统一管理目录：

mv elasticsearch-5.0.0 /usr/local/elasticsearch
mv logstash-5.0.0 /usr/local/logstash
mv kibana-5.0.0-linux-x86_64 /usr/local/kibana


备份配置文件：

cp /usr/local/logstash/config/logstash.yml /usr/local/logstash/config/logstash.yml.bak.$(date+%F)      
cp /usr/local/elasticsearch/config/elasticsearch.yml/usr/local/elasticsearch/config/elasticsearch.yml.bak.$(date +%F)
cp /usr/local/kibana/config/kibana.yml /usr/local/kibana/config/kibana.yml.bak.$(date+%F)


配置Elasticsearch


创建用户

默认elasticsearch不支持root用户启动，所以需要先创建一个普通用户
groupadd elastic 
useradd elastic –g  elastic  –M
chown -R elastic.elastic   /usr/local/elasticsearch/

修改配置文件：

network.host: 192.168.0.248
http.port: 9200
su elastic
/usr/local/elasticsearch/bin/elasticsearch –d

验证启动：

curl http://localhost:9200/

添加开机启动：

echo ‘/usr/local/elasticsearch/bin/elasticsearch-d’ >>/etc/rc.local

注意错误：

ERROR: bootstrap checks failed
问题：max file descriptors [65535] for elasticsearch process likelytoo low, increase to at least [65536]
解决：vi /etc/security/limits.conf
*             -       nofile          65536
或者
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
问题：max number of threads [1024] for user [elasticsearch] likely toolow, increase to at least [2048]
解决：vi/etc/security/limits.d/90-nproc.conf
* soft nproc 2048
需重启生效。
问题：max virtual memory areas vm.max_map_count [65530] likely toolow, increase to at least [262144]
解决：vi /etc/sysctl.conf
vm.max_map_count=655360
sysctl -p


配置kibana：

修改配置文件：
vi /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.0.248"
elasticsearch.url: http://192.168.0.248:9200


测试logstash：

cd logstash-5.0.0
bin/logstash -e 'input { stdin { } } output {stdout {} }'
hello world
2013-11-21T01:22:14.405+0000 0.0.0.0 helloworld


配置logstash服务端：

vi /usr/local/logstash/config/logstash.yml
input {
         redis {
                   host => “192.168.0.248”
                   port => 6379
                   type =>”redis-input”
                   data_type =>”list”
                   key =>”logstash:redis”
}
}
output{
elasticsearch {
        hosts=> ["192.168.0.248:9200"]
           }        
}


客户端：

vi /usr/local/logstash/config/logstash.yml
input {
         file{
                   type=>”nginx_access”
                   path=>[”/usr/local/nginx/logs/access.log”]
}
}
output {
         redis{
                   host  => “192.168.0.248”
                   data_type=> “list”
                   key=> “logstash:redis”
}
}


启动客户端：

/usr/local/logstash/bin/logstash –f /usr/local/logstash/conf/logstash.conf
或yum安装：
/usr/share/logstash/bin/logstash -f/etc/logstash/conf.d/logstash.conf

客户端yum安装：

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <