ELK日志采集系统搭建

ELK日志采集系统搭建

ELK5.0 依赖于jdk1.8

1. 下载所需的软件包。 ELK官网下载地址
   
   • kibana-5.6.3-linux-x86_64.tar.gz
   • logstash-5.6.3.tar.gz
   • elasticsearch-5.6.3.tar.gz
   • filebeat-5.6.3-linux-x86_64.tar.gz

2. 解压并 创建ELK用户 （elk启动不能为root账号）

   mkdir -p /opt/software/elk
   tar -zxvf kibana-5.6.3-linux-x86_64.tar.gz -C /opt/software/elk
   tar -zxvf logstash-5.6.3.tar.gz -C /opt/software/elk
   tar -zxvf  elasticsearch-5.6.3.tar.gz -C /opt/software/elk
   tar -zxvf  filebeat-5.6.3-linux-x86_64.tar.gz -C /opt/software/elk
   
   
   #创建ELK用户和组，并给elk安装目录赋予权限
   
   groupadd elk 
   useradd -g elk elk 
   passwd elk 
   chown -R elk /opt/software/elk

3. elasticsearch配置

       #修改 $ELASTICSEARCH_HOME/config/elasticsearch.yml
       bootstrap.memory_lock: false
       bootstrap.system_call_filter: false
       network.host: 0.0.0.0
       http.cors.enabled: true
       http.cors.allow-origin: "*"
       # 启动
       ./bin/elasticsearch
   
       # 启动出现相关的异常如下：
       # 1.Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)  ==> ES5.0 默认分配的是2g内存 修改config/jvm.options里面分配的内存大小
       # 2.Java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMPandCONFIG_SECCOMP_FILTER ==> Linux内核版本过低 只是警告 可以忽略。
       # 3.ERROR: bootstrap checks failed
   max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536] ==> 无法创建本地文件问题,用户最大可创建文件数太小  切换到root目录 修改 limits.conf配置文件 如下：
   vim /etc/security/limits.conf 添加如下内容:
   * soft nofile 65536
   * hard nofile 131072
   * soft nproc 2048
   * hard nproc 4096
     # 4.max number of threads [1024] for user [es] likely too low, increase to at least [2048] ==> 无法创建本地线程问题,用户最大可创建线程数太 切换到root用户，进入limits.d目录下，修改90-nproc.conf 配置文件 vim /etc/security/limits.d/90-nproc.conf
   找到如下内容：
   * soft nproc 1024
   
   #修改为
   
   * soft nproc 2048
    # 5.max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144] ==> 最大虚拟内存太小 切换到root用户下，修改配置文件sysctl.conf  vim /etc/sysctl.conf
   添加下面配置：
   vm.max_map_count=655360
   并执行命令：
   sysctl -p
    # 6.ERROR: bootstrap checks failed
   system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk ==> Centos6 不支持SecComp，而ES5.6.4默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动 修改如下：
   在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
   bootstrap.memory_lock: false
   bootstrap.system_call_filter: false

4. logstash配置

   cd /opt/software/elk/logstash-5.6.3
   vim config/startup.options
   JAVACMD=/usr/local/common/jdk/bin/java # 本机环境的jdk
   LS_HOME=/opt/software/elk/logstash-5.6.3
   
   vim first-pipeline.conf
   
   # 添加如下内容：
   
   input {
       beats {
           port => "5044"
       }
   }
   
   # The filter part of this file is commented out to indicate that it is
   
   
   # optional.
   
   filter {
       grok {
           match => { "message" => "%{COMBINEDAPACHELOG}"}
       }
       geoip {
           source => "clientip"
       }
   }
   output {
      #stdout { codec => rubydebug }
      elasticsearch {
           hosts => [ "localhost:9200" ]
       }
   }
   
   # 可以使用 bin/logstash -f first-pipeline.conf --config.test_and_exit 测试配置文件是否正确
   
   
   # bin/logstash -f first-pipeline.conf --config.reload.automatic  修改文件后自动更新
   
   
   # 启动logstash
   
   ./bin/logstash -f first-pipeline.conf 

5. filebeat配置

   cd /opt/software/elk/filebeat-5.6.3-linux-x86_64
   vim filebeat.yml 
   
   #修改如下 
   
   paths:
       - /opt/elk/logs/test.log
   output.logstash:
     # The Logstash hosts
     hosts: ["localhost:5044"]
   
   #启动filebeat
   
   sudo ./filebeat -e -c filebeat.yml -d "publish"

6. kibana配置

cd /opt/software/elk/kibana-5.6.3-linux-x86_64
# 修改文件
server.host: "0.0.0.0"
vim config/kibana.yml
# 启动
./bin/kibana

# 然后往 /opt/elk/logs/test.log中写入数据 格式如下(官网的例子中的数据)：
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png
HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

elasticsearch-head插件的安装 git地址

# elasticsearch-head 编译需要npm支持 所以需要安装nodejs
#解压nodejs
xz -d node-v8.9.1-linux-x64.tar.xz
tar -xvf node-v8.9.1-linux-x64.tar
#建立软连接
ln -s /opt/tools/node-v8.9.1-linux-x64/bin/npm /usr/local/bin/npm
ln -s /opt/tools/node-v8.9.1-linux-x64/bin/node /usr/local/bin/node
#添加sudo权限 npm编译的时候需要sudo权限
sudo ln -s /usr/local/bin/node /usr/bin/node
sudo ln -s /usr/local/lib/node /usr/lib/node
sudo ln -s /usr/local/bin/npm /usr/bin/npm

#指定npm国内镜像
npm config set registry "http://registry.npm.taobao.org/"
sudo npm install #编译
npm run start #启动
# 插件默认端口为9100，打开后如果无法连接到Elasticsearch，在Elasticsearch配置文件中加如下参数即可：
http.cors.enabled: true
http.cors.allow-origin: "*"