ELK+filebeat

安装filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-linux-x86_64.tar.gz
tar -xvf filebeat-6.0.0-linux-x86_64.tar.gz
配置文件
vim /usr/local/filebeat/filebeat.yml

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /usr/local/tomcat/logs/catalina.out
    #- c:\programdata\elasticsearch\logs\*

  output.logstash:
  # The Logstash host (or  innernet ip)
     hosts: ["localhost:5044"]

启动filebeat
./filebeat
后台运行filebeat
nohup ./filebeat 然后关闭窗口

安装logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.tar.gz
tar -zxvf logstash-6.0.0.tar.gz
logstash收集filebeat收集的日志
vim /etc/logstash/conf.d/tomcatlog.conf

input {
  beats {
    port => 5044
 type =>"log"
    }
}
filter {
   grok {
      patterns_dir => ["/usr/local/logstash-6.0.0/config/user.patterns"]
      match => { "message" => ["%{JAVA_TIMESTAMP_ISO8601:log_time}[\s]+%{LOGLEVEL:log_level}[\s]+%{NOTSPACE:class_method}[\s]+\-[\s]+%{ALL_DATA:log_info}"]}
        remove_field => ["message"]
   }
}
output {
  elasticsearch {
    hosts => ["192.168.149.20:9200"]
    index => "tomcat-access-%{+YYYY.MM.dd}" #索引名
  }
}

上面的message内容为log4j设置的日志格式，具体如下，可自定义

2018-07-24 17:27:51.515 DEBUG BlackWhiteDao.getWhiteAppList - ==>  Preparing: SELECT room_id AS roomId

其中user.patterns内容为

JAVA_TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}\.%{INT:msec}
ALL_DATA [\s\S]*

启动logstash
cd /usr/local
./logstash -f /etc/logstash/conf.d/tomcatlog.conf
后台启动logstash
nohup ./logstash -f /etc/logstash/conf.d/tomcatlog.conf（可直接关闭窗口）

安装elasticsearch

cd /usr/local
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.tar.gz
tar -xvf elasticsearch-6.0.0.tar.gz
新增用户
useradd elk
passwd elk

给elk权限
chown -R elk /usr/local/elasticsearch-6.0.0

• 一定要切换到elk启动

su elk
启动elasticsearch服务
cd /usr/local/elasticsearch-6.0.0/bin
./elasticsearch （可以加-d后台启动）

elasticsearch检查是否健康
curl '127.0.0.1:9200/_cluster/health?pretty'

问题：
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决办法：
切换到root用户修改配置sysctl.conf
vi /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
并执行命令：
sysctl -p
然后，重新启动elasticsearch，即可启动成功。
问题：
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
解决：
vim /etc/security/limits.conf 在最后面追加

* hard nofile 65536
* soft nofile 65536

安装kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-linux-x86_64.tar.gz
tar -xvf kibana-6.0.0-linux-x86_64.tar.gz

mv kibana-6.0.0-linux-x86_64 kibana-6.0.0
kibana进行配置
vim kibana-6.0.0/config/kibana.yml(默认可以不需要配置)

server.port: 5601  # 配置kibana的端口
server.host: 192.168.149.20  # 配置监听ip(本地内网)
elasticsearch.url: "http://192.168.149.20:9200"  # 配置es服务器的ip，如果是集群则配置该集群中主节点的ip
logging.dest: /var/log/kibana.log  # 配置kibana的日志文件路径，不然默认是messages里记录日志

后台启动kibana
bin/kibana &
然后输入exit;回车退出