centos7搭建日志服务elasticsearch6.3.2

centOS7 搭建日志服务

版本

elasticsearch-6.3.2、kibana-6.3.2、filebeat-6.3.2

参考链接

elasticsearch6.3.1 启用x-pack

安装Kibana6.3.2与Elasticsearch6.3.2集成

x-pack6.3破解

filebeat 5.2.2详细配置说明

logstash 启动缓慢问题-haveged安装

elasticsearch start more than one nodes on the same data folder

elasticsearch6.3.2集群安装

一、elasticsearch

1、安装 rz sz

sudo yum install lrzsz -y

2、安装 jdk

/usr/local ------rpm -ivh jdk-8u152-linux-x64.rpm

3、下载 elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
tar -zxvf /usr/local/elasticsearch-6.3.2.tar.gz -C /usr/local/

4、elasticsearch配置

elasticsearch不能用root用户启动

改变elasticsearch文件夹所有者到当前用户

sudo chown -R noroot:noroot /usr/local/elasticsearch-6.3.2

设置配置文件elasticsearch.yml

$ grep -Ev "^$|^[#;]" elasticsearch.yml 
cluster.name: es
node.name: node-1
node.master: true
node.data: true
path.data: /usr/local/elasticsearch-6.3.2/data
path.logs: /usr/local/elasticsearch-6.3.2/logs
network.host: 0.0.0.0
http.port: 9200
#xpack.security.enabled: true
#xpack.security.transport.ssl.enabled: true
#http.cors.enabled: true
#http.cors.allow-origin: "*"
#http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

5、启动

nohup ./elasticsearch >/dev/null 2>&1 &

6、 x-pack

6.1、启用 x-pack

elasticsearch6.3.2 安装包中已经集成了 x-pack 插件，无需使用 elasticsearch-plugin install 来安装 x-pack ，我们需要做的是使 x-pack 工作

curl -H "Content-Type:application/json" -XPOST http://localhost:9200/_xpack/license/start_trial?acknowledge=true
[2018-08-14T19:41:05,208][INFO ][o.e.l.LicenseService     ] [node-1] license [1fe2f382-c8cc-41d6-8d3a-d11d71ad2004] mode [basic] - valid
[2018-08-14T19:41:05,220][INFO ][o.e.g.GatewayService     ] [node-1] recovered [0] indices into cluster_state
[2018-08-14T19:43:24,005][INFO ][o.e.l.LicenseService     ] [node-1] license [37b55178-574f-47a8-aedf-dd20c6f710a6] mode [trial] - valid

可以看到elasticsearch控制台显示license 已变为trial

6.2、设置用户名密码

$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,kibana,logstash_system,beats_system.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [elastic]

6.3、elasticsearch.yml 中开启安全验证配置

此时访问 http://localhost:9200 并不需要输入账号密码，需 elasticsearch.yml 中添加如下配置：

xpack.security.enabled: true
#xpack.security.transport.ssl.enabled: true

重启 elasticsearch 之后 http://localhost:9200 就需要使用账号验证了

6.4、查看license

curl -u elastic:123456 "http://localhost:9200/_license"

6.5、更新license

curl -XPUT -u elastic 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json

6.6、x-pack 破解

elk6.3版本之后，x-pack都是默认安装，无需install

分别创建两个文件

vim LicenseVerifier.java

package org.elasticsearch.license;
import java.nio.*; import java.util.*;
import java.security.*;
import org.elasticsearch.common.xcontent.*;
import org.apache.lucene.util.*;
import org.elasticsearch.common.io.*;
import java.io.*;

public class LicenseVerifier {
    public static boolean verifyLicense(final License license, final byte[] encryptedPublicKeyData) {
        return true;
    }

    public static boolean verifyLicense(final License license)     {
        return true;
    }
}

vim XPackBuild.java

package org.elasticsearch.xpack.core;
import org.elasticsearch.common.io.*;
import java.net.*;
import org.elasticsearch.common.*;
import java.nio.file.*;
import java.io.*;
import java.util.jar.*;
public class XPackBuild {
    public static final XPackBuild CURRENT;
    private String shortHash;
    private String date;
    @SuppressForbidden(reason = "looks up path of xpack.jar directly") static Path getElasticsearchCodebase() {
        final URL url = XPackBuild.class.getProtectionDomain().getCodeSource().getLocation();
        try { return PathUtils.get(url.toURI()); }
        catch (URISyntaxException bogus) {
            throw new RuntimeException(bogus); }
        }

    XPackBuild(final String shortHash, final String date) {
            this.shortHash = shortHash;
            this.date = date;
            }

    public String shortHash() {
        return this.shortHash;
        }
    public String date(){
        return this.date;
        }

    static {
        final Path path = getElasticsearchCodebase();
        String shortHash = null;
        String date = null;
        Label_0157: { shortHash = "Unknown"; date = "Unknown";
    }

    CURRENT = new XPackBuild(shortHash, date);
    }
}

将刚创建的两个java包打包成class文件，我们需要做的就是替换这两个class文件（因里面需要引用到其他的jar，故需要用到javac -cp命令）

#编译成class文件
javac -cp "/usr/local/elasticsearch-6.3.2/lib/elasticsearch-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/lucene-core-7.3.1.jar:/usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/elasticsearch-core-6.3.2.jar" XPackBuild.java

#编译成class文件
sudo javac -cp "/usr/local/elasticsearch-6.3.2/lib/elasticsearch-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/lucene-core-7.3.1.jar:/usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar:/usr/local/elasticsearch-6.3.2/lib/elasticsearch-core-6.3.2.jar" XPackBuild.java

#创建解压目录
sudo mkdir x-pack-core-6.3.2
#解压到目标目录
sudo unzip x-pack-core-6.3.2.jar -d x-pack-core-6.3.2
#将生成的class文件 copy 到 解压后的文件夹
sudo cp -a LicenseVerifier.class x-pack-core-6.3.2/org/elasticsearch/license/
sudo cp -a XPackBuild.class x-pack-core-6.3.2/org/elasticsearch/xpack/core/
#进入到解压目录 压缩成 jar文件
sudo jar -cvf x-pack-core-6.3.2.jar *
#旧jar包备份
sudo cp /usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar /usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar.bak
#新生成 jar 包替换
sudo cp x-pack-core-6.3.2.jar /usr/local/elasticsearch-6.3.2/modules/x-pack/x-pack-core/x-pack-core-6.3.2.jar

破解后的文件下载地址 x-pack-core-6.3.2.jar破解版

启动elasticsearch

替换 license

官网注册 license https://register.elastic.co/registration

替换其中的 type 为: platinum 白金版 和expiry_date_in_millis 为 1565740799999 过期时间

{
  "license": {
    "uid": "e684c8fa-7281-40df-8764-8cf1fd00b406",
    "type": "basic",
    "issue_date_in_millis": 1534118400000,
    "expiry_date_in_millis": 1565740799999,
    "max_nodes": 100,
    "issued_to": "zhang kai (hc)",
    "issuer": "Web Form",
    "signature": "AAAAAwAAAA2TCeprkZ2aX7p+rKBcAAABmC9ZN0hjZDBGYnVyRXpCOW5Bb3FjZDAxOWpSbTVoMVZwUzRxVk1PSmkxaktJRVl5MUYvUWh3bHZVUTllbXNPbzBUemtnbWpBbmlWRmRZb25KNFlBR2x0TXc2K2p1Y1VtMG1UQU9TRGZVSGRwaEJGUjE3bXd3LzRqZ05iLzRteWFNekdxRGpIYlFwYkJiNUs0U1hTVlJKNVlXekMrSlVUdFIvV0FNeWdOYnlESDc3MWhlY3hSQmdKSjJ2ZTcvYlBFOHhPQlV3ZHdDQ0tHcG5uOElCaDJ4K1hob29xSG85N0kvTWV3THhlQk9NL01VMFRjNDZpZEVXeUtUMXIyMlIveFpJUkk2WUdveEZaME9XWitGUi9WNTZVQW1FMG1DenhZU0ZmeXlZakVEMjZFT2NvOWxpZGlqVmlHNC8rWVVUYzMwRGVySHpIdURzKzFiRDl4TmM1TUp2VTBOUlJZUlAyV0ZVL2kvVk10L0NsbXNFYVZwT3NSU082dFNNa2prQ0ZsclZ4NTltbU1CVE5lR09Bck93V2J1Y3c9PQAAAQANFdbpeTp95neGTKTT5R3ovic2yqJJaBHOGSIo3YvVDBCpXEU9C6Er1n5FX64GgpMZKivb/HCHvXY/c4ucArTwKi6pKVamjso1O7KGl0PioQCtUXZN4Dz22NT/2Rleql7zuRHcgU2LA6KOPIjcsmpkHZD3eGz0lQowwOrOPyurZFdAqi43423xtPgojfo3KfLo2Hhhl0aVzSyQ9n/pFHnAFucC9igAJaXm/bG9zIdFejKSAfJVckjAbr55rhmAtP4NwFWAB5xDQUdABKEQUeqZnYN3y7cBekvHwswznWOHawZkDz1rbMhbbvGAeNK2WUqyq/G+RO0MFmCCNSymTZxu",
    "start_date_in_millis": 1534118400000
  }
}

更新 license

curl -XPUT -u elastic 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json

报错

{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"Cannot install a [PLATINUM] license unless TLS is configured or security is disabled"}],"type":"illegal_state_exception","reason":"Cannot install a [PLATINUM] license unless TLS is configured or security is disabled"},"status":500}

在 elasticsearch.yml 文件中配置

xpack.security.transport.ssl.enabled: true

查看是否为白金会员
　　
curl -XGET -u elastic:123456 localhost:9200/_license

也可以在 kibana 的 Management→ElasticSearch→License Management 看到过期时间

7、集群配置

集群ip : 192.168.99.101,192.168.99.102,192.168.99.103,192.168.99.104,192.168.99.105

拷贝 elasticsearch-6.3.2.tar.gz 到所有服务器，解压，替换破解之后的 x-pack-core-6.3.2.jar

elasticsearch.yml配置文件如下

cluster.name: es
node.name: node-1
node.master: true
node.data: true
node.max_local_storage_nodes: 256  #replace 256 with another num that greater than 1 
path.data: /usr/local/elasticsearch-6.3.2/data
path.logs: /usr/local/elasticsearch-6.3.2/logs
network.host: 192.168.99.101
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["192.168.99.101","192.168.99.102","192.168.99.103","192.168.99.104","192.168.99.105"]
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: 6s

不同节点修改对应的 node.name 和 network.host 属性

集群启动，x-pack服务开启

更改 elasticsearch.yml 配置文件

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

TLS/SSL

生成认证文件 elastic-certificates.p12

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

添加刚才输入的密码到 lasticsearch keystore ，会在 config 文件夹下生成 elasticseaerch.keystore 文件

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

复制 elastic-certificates.p12 到 config 文件夹下

真正起作用的是 config 文件夹下的 elasticseaerch.keystore 文件

拷贝 elastic-certificates.p12 和 elasticseaerch.keystore 到所有节点机器

启动集群，更新 license

查看集群健康状态

http://192.168.99.101:9200/_cat/health?v

二、kibana

1、下载官网kibana-6.3.2-linux-x86_64.tar.gz文件

#下载
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
#解压
tar xzvf kibana-6.3.2-linux-x86_64.tar.gz

2、修改config/kibana.yml配置文件

$ grep -Ev "^$|^[#;]" kibana.yml 
#端口
server.port: 5601
#服务器IP
server.host: "192.168.1.210"
#elasticsearch服务器
elasticsearch.url: "http://192.168.1.210:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "123456"

注意：默认server.host=localhost，这样在自己本机启动可以，如果是想让网络上其他人使用，就需要改成本机IP地址。否则，访问时会提示：

dial tcp 192.168.1.210:5601: connectex: No connection could be made because the target machine actively refused it.

3、启动

改变kibana文件夹所有者到当前用户

sudo chown -R noroot:noroot /usr/local/kibana-6.3.2

启动

nohup ./kibana >/dev/null 2>&1 &

4、停止

查看端口占用命令

#查看运行的后台进程
jobs -l  或者  ps -ef | grep node 或者 lsof -i :5601
#杀掉进程
kill -9 xxxx

5、浏览器访问默认kibana端口5601

使用 kibana/123456 可以进行登录访问

使用 elastic/123456 在菜单 Management→Security→Users、Roles可进行用户管理

6、kibana 登陆用户管理

创建角色

Management → Security → Roles → Create role

填写

name：app
Run As Privileges ： 选择用户
Index Privileges.Indices : *
Index Privileges.Privileges : read,read_cross_cluster
Index Privileges.Granted Fields : *

创建用户

Management → Security → Users → Create user

Username :
Password :
Password Again, Please :
Full Name :
Email :
Roles : 

之后该用户可以进行登陆，并可以查询 Discover 里的日志

三、filebeat

1、下载

#下载
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz
#解压
tar xzvf filebeat-6.3.2-linux-x86_64.tar.gz

2、修改配置文件

可以配置多个配置文件，在启动时选择

直接输出日志到 elasticsearch

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
  #log文件路径
    - /opt/logs/app/api/*.log
  #多行日志合并
  multiline.pattern: ^\[
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500
  multiline.timeout: 5s 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
#index修改后下面两个配置需要添加
setup.template.name: "appname"
setup.template.pattern: "appname-*"

setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "192.168.99.100:5601"
  username: "kibana"  
  password: "123456"
output.elasticsearch:
  hosts: ["192.168.99.11:9200"]
  index: appname
  protocol: "http"
  username: "elastic"
  password: "123456"

3、启动

nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

4、停止 filebeat

ps -ef|grep filebeat

四、logstash

1、下载

#下载
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
#解压
tar -xvf logstash-6.3.2

2、配置logstash.conf

直接输出日志到 elasticsearch

可以配置多个logstash.conf 启动指定不同文件，端口和数据路径

input{
    file{
        path =>"/opt/logs/appname/api/main.log"
        start_position=>"beginning"
    }
    stdin {
        codec => multiline {
            pattern => "^\["
            negate => true
            what => "previous"
        }
    }
}

filter{
    grok{
        match=>{
           "message"=>"%{DATA:clientIp} - - \[%{HTTPDATE:accessTime}\] \"%{DATA:method} %{DATA:requestPath} %{DATA:httpversion}\" %{DATA:retcode} %{DATA:size} \"%{DATA:fromHtml}\" \"%{DATA:useragent}\""
        }
        remove_field=>"message"
   }
   date{
        match=>["accessTime","dd/MMM/yyyy:HH:mm:ss Z"]
   }

   if ([message] =~ "^debug") {    
      drop {}                   
   }
}

output {
    elasticsearch {
        hosts => "192.168.99.100"
        user => "elastic"
        password => "123456"
        index => "appname"
    }
    stdout { 
            codec => rubydebug 
    }
}

3、启动

nohup ./logstash -f ../config/logstash.conf >/dev/null 2>&1 &

4、停止

ps -ef|grep logstash
kill -9 pid

5、logstash启动特别慢

测试是否需要安装haveged

$ cat /proc/sys/kernel/random/entropy_avail
186

如果值比较低(<1000),建议安装 haveged. 否则加密程序会等待系统有足够的熵。

安装haveged

加入epel源(实际就是添加yum源配置)

# yum install epel-release -y

安装haveged

#yum install haveged -y

启动haveged并开机启动,并查看haveged状态(对centos7不熟悉的童鞋需要补补哦_)

# systemctl start haveged                    #启动
# systemctl enable haveged                   #开机启动
# systemctl status haveged                   #查看启动状态
Created symlink from /etc/systemd/system/multi-user.target.wants/haveged.service to /usr/lib/systemd/system/haveged.service.
# systemctl list-unit-files | grep haveged    #查看开机启动状态
haveged.service                             enabled

装了还是很慢

五、安装中问题

1、elasticsearch start more than one nodes on the same data folder

aused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/elasticsearch-5.4.0/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

当我在开启多个elasticsearch 实例时，遇到上述异常，解决方法是：在 config/elasticsearch.yml文件中新增一个配置变量

node.max_local_storage_nodes: 256  #replace 256 with another num that greater than 1

2、javax.net.ssl.SSLException: Received fatal alert

没有配置 ssl 认证，参考集群 ssl 配置

[2018-08-16T20:03:05,110][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-1] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:47968, remoteAddress=/192.168.99.101:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure

3、keystore password was incorrect

如果没有配置 config/elasticsearch.keystore , 集群 ssl 开启情况下会报如下错误，将第一台自己上生成的认证文件拷贝一份即可

[2018-08-17T16:21:50,332][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-5] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
    ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
    ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
    at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:60) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_152]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:79) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:134) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
    ... 6 more
Caused by: java.io.IOException: keystore password was incorrect
    at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
    at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_152]
    at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:265) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:256) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_152]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:79) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:134) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
    ... 6 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded
    at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
    at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_152]
    at org.elasticsearch.xpack.core.ssl.CertUtils.readKeyStore(CertUtils.java:265) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.CertUtils.trustManager(CertUtils.java:256) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:407) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_152]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:79) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.(XPackPlugin.java:134) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.plugins.PluginsService.(PluginsService.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:311) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
    ... 6 more