CentOs7 搭建ELK+fileBeat+nginx及部分问题解决

注意防火墙

 

JDK  配置

export JAVA_HOME=/usr/java/jdk8

export PATH=$JAVA_HOME/bin:$PATH

 

（都要放在要安装的目录下）elasticsearch在线安装 curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.2.tar.gz

ELK

  配置防火墙

-A INPUT -p tcp -m state --state new -m tcp --dport 9200 -j ACCEPT

-A INPUT -p tcp -m state --state new -m tcp --dport 9300 -j ACCEPT

-A INPUT -p tcp -m state --state new -m tcp --dport 5601 -j ACCEPT  

云服务器 ECS Linux CentOS 7 下重启服务不再通过 service 操作，而是通过 systemctl 操作。

systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动

查看：firewall-cmd    --state

启动：systemctl  start   firewalld.service

重启：systemctl  restart  iptables.service

自启：systemctl enable sshd.service

关闭：systemctl  stop   firewalld.service

永久禁止端口访问firewall-cmd --remove-port=9200/tcp --zone=public –permanent

永久开启

firewall-cmd --add-port=9200/tcp --zone=public –permanent

重新载入

firewall-cmd --reload

创建elsearch用户组及elsearch用户

groupadd elsearch

useradd elsearch -gelsearch -p elasticsearch

·       1

·       2

更改elasticsearch文件夹及内部文件的所属用户及组为elsearch:elsearch

chown -Relsearch:elsearch  elasticsearch

·       1

elasticsearch为你elasticsearch的目录名称

切换到elsearch用户再启动

su elsearch #切换账户

cd elasticsearch/bin #进入你的elasticsearch目录下的bin目录

./elasticsearch

安装分词器

（elasticsearch目录执行）

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.2.2/elasticsearch-analysis-ik-6.2.2.zip

ElasticSearch 5.0.0 安装部署常见错误或问题

问题一：

 

[2016-11-06T16:27:21,712][WARN][o.e.b.JNANatives ] unable to install syscall filter:

 

Java.lang.UnsupportedOperationException:seccomp unavailable: requires kernel 3.5+ withCONFIG_SECCOMPandCONFIG_SECCOMP_FILTERcompiledinatorg.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:349)~[elasticsearch-5.0.0.jar:5.0.0]

 

atorg.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630)~[elasticsearch-5.0.0.jar:5.0.0]

 

原因：报了一大串错误，大家不必惊慌，其实只是一个警告，主要是因为你Linux版本过低造成的。

 

解决方案：

1、重新安装新版本的Linux系统

2、警告不影响使用，可以忽略

 

问题二：

ERROR: bootstrapchecks failed

 

max filedescriptors [4096] for elasticsearch process likely too low, increase to atleast [65536]

 

原因：无法创建本地文件问题,用户最大可创建文件数太小

 

解决方案：

切换到root用户，编辑limits.conf配置文件， 添加类似如下内容：

 

vi/etc/security/limits.conf

 

添加如下内容:

*  soft nofile65536

 

* hard nofile131072

 

* soft nproc 2048

 

* hard nproc 4096

备注：* 代表Linux所有用户名称（比如hadoop）

 

保存、退出、重新登录才可生效

 

问题三：

max number ofthreads [1024] for user [es] likely too low, increase to at least [2048]

原因：无法创建本地线程问题,用户最大可创建线程数太小

解决方案：切换到root用户，进入limits.d目录下，修改90-nproc.conf 配置文件。

 

vi /etc/security/limits.d/90-nproc.conf

 

找到如下内容：

 

* soft nproc 1024

 

#修改为

 

* soft nproc 2048

 

问题四：

max virtual memoryareas vm.max_map_count [65530] likely too low, increase to at least [262144]

原因：最大虚拟内存太小

解决方案：切换到root用户下，修改配置文件sysctl.conf

 

vi /etc/sysctl.conf

 

添加下面配置：

 

vm.max_map_count=655360

 

并执行命令：

 

sysctl -p

 

然后重新启动elasticsearch，即可启动成功。

 

问题五：

ElasticSearch启动找不到主机或路由

原因：ElasticSearch 单播配置有问题

解决方案：

检查ElasticSearch中的配置文件

vi config/elasticsearch.yml

找到如下配置：

 

discovery.zen.ping.unicast.hosts:["192.168.**.**:9300","192.168.**.**:9300"]

一般情况下，是这里配置有问题，注意书写格式

 

问题六：

org.elasticsearch.transport.RemoteTransportException:Failed to deserialize exception response from stream

 

原因:ElasticSearch节点之间的jdk版本不一致

 

解决方案：ElasticSearch集群统一jdk环境

 

问题七：

Unsupportedmajor.minor version 52.0

 

原因：jdk版本问题太低

解决方案：更换jdk版本，ElasticSearch5.0.0支持jdk1.8.0

 

问题八：

bin/elasticsearch-plugininstall license

ERROR: Unknownplugin license

 

原因：ElasticSearch5.0.0以后插件命令已经改变

解决方案：使用最新命令安装所有插件

bin/elasticsearch-plugin install x-pack

 

补：启动异常：ERROR: bootstrapchecks failed
system call filters failed to install; check the logs and fix yourconfiguration or disable system call filters at your own risk

 

问题原因：因为Centos6不支持SecComp，而ES5.2.1默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。详见：https://github.com/elastic/elasticsearch/issues/22899

 

解决方法：在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
bootstrap.memory_lock: false

bootstrap.system_call_filter: false

Nginx反向代理权限不足

执行

setsebool -P httpd_can_network_connect 1

 

 

 

nginx对elasticsearch加密

 

http {

    include       /etc/nginx/mime.types;

   default_type application/octet-stream;

 

 

  upstreamelasticsearch {

    server192.168.37.128:9200;

  }

 

  server {

    listen 8080;

    auth_basic"Protected Elasticsearch";

   auth_basic_user_file passwords;

 

    location / {

      proxy_passhttp://elasticsearch;

     proxy_redirect off;

    }

  }  log_format main  '$remote_addr - $remote_user[$time_local] "$request" '

                     '$status $body_bytes_sent "$http_referer" '

                     '"$http_user_agent" "$http_x_forwarded_for"';

 

 

Nginx持久化http+加密+反向代理

 

 

error_log  /var/log/nginx/error.logwarn;

pid       /var/run/nginx.pid;

 

 

events {

   worker_connections  1024;

}

 

 

http {

    include       /etc/nginx/mime.types;

   default_type application/octet-stream;

     upstreamelasticsearch {

    server192.168.37.128:9200;

 

    keepalive15;

  }

    server {

    listen 8080;

    auth_basic"Protected Elasticsearch";

   auth_basic_user_file passwords;

    location / {

 

      proxy_passhttp://elasticsearch;

    proxy_redirect off;

       proxy_http_version 1.1;

        proxy_set_header Connection "Keep-Alive";

     proxy_set_header Proxy-Connection "Keep-Alive";

    }

 

 

 

  }

   log_format  main  '$remote_addr - $remote_user [$time_local]"$request" '

                     '$status $body_bytes_sent "$http_referer" '

                     '"$http_user_agent" "$http_x_forwarded_for"';

 

   #access_log /var/log/nginx/access.log  main;

 

   sendfile        on;

   #tcp_nopush     on;

 

   keepalive_timeout  65;

 

FileBeat.yml配置

 

 

 

=========================== Filebeat prospectors=============================

 

filebeat.prospectors:

 

# Each - is a prospector. Most options can be set atthe prospector level, so

# you can use different prospectors for variousconfigurations.

# Below are the prospector specific configurations.

 

- type: log

 

  # Change totrue to enable this prospector configuration.

  enabled: true

 

  # Paths thatshould be crawled and fetched. Glob based paths.

  paths:

    -/var/logs/a.log

 

  fields:

    app: haixing

 fields_under_root: true

- type: log

 

  # Change totrue to enable this prospector configuration.

  enabled: true

 

  # Paths thatshould be crawled and fetched. Glob based paths.

  paths:

    -/var/logs/b.log

  fields:

    app: shenda

 fields_under_root: true

#-c:\programdata\elasticsearch\logs\*

 

 

 

=================================================================

Logstash.conf配置

 

input {

  beats{codec => json

        port => "5044"

  }

}

 

 

filter{

        mutate{

        remove_field => "offset"

        remove_field => "source"

        remove_field => "@version"

        remove_field => "host"

        remove_field => "beat"

        remove_field => "tags"

        remove_field =>"prospector"

        remove_field => "_score"

         remove_field =>"_version"

        }

      }

 

 

output {

 

        elasticsearch {

                       hosts =>["localhost:9200"]

                       index=>"%{app}-%{+YYYY.MM.dd}"

         }

 

  stdout{ codec => rubydebug }

}

需要注意的是：

   yml格式  空格复制（中文情况）的识别不了