ElasticSearch 开发总结—— failed to send join request to master[...] reason RemoteTransportException
ElasticSearch版本6.6.1
本地搭建了一个包含三个节点的ELasticsearch集群，作为测试环境，三个节点都是这样的错：failed to send join request to master。具体错误日志如下：
[2019-03-11T18:00:19,521][INFO ][o.e.d.z.ZenDiscovery ] [slave] failed to send join request to master [{master}{5Uge7dikTTGMSsqMJewVYQ}{ZWmmJ06mQ4GqrnrdtjAZfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17054658560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[master][127.0.0.1:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{master}{5Uge7dikTTGMSsqMJewVYQ}{ZWmmJ06mQ4GqrnrdtjAZfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17054658560, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
[2019-03-11T18:00:25,641][INFO ][o.e.d.z.ZenDiscovery ] [slave] failed to send join request to master [{master}{5Uge7dikTTGMSsqMJewVYQ}{ZWmmJ06mQ4GqrnrdtjAZfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17054658560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[master][127.0.0.1:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{master}{5Uge7dikTTGMSsqMJewVYQ}{ZWmmJ06mQ4GqrnrdtjAZfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17054658560, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
[2019-03-11T18:00:31,871][INFO ][o.e.d.z.ZenDiscovery ] [slave] failed to send join request to master [{master}{5Uge7dikTTGMSsqMJewVYQ}{ZWmmJ06mQ4GqrnrdtjAZfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17054658560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[master][127.0.0.1:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{master}{5Uge7dikTTGMSsqMJewVYQ}{ZWmmJ06mQ4GqrnrdtjAZfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17054658560, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] not master for join request]; ], tried [3] times
没有master node ,但是配置文件明明配置了，从官网看java引入jar包版本和Elasticsearch版本一致，无奈只能慢慢排除；
步骤一：
排查每个节点的elasticsearch.yml文件
# 换个集群的名字，免得跟别人的集群混在一起
cluster.name: elastic
# 换个节点名字
node.name: master
node.master: true
node.data: true
# host
network.host: 127.0.0.1
# port
http.port: 9200
transport.tcp.port: 9300
# discovery settings
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "127.0.0.1:9301"]
discovery.zen.minimum_master_nodes: 1经过排查，没发现问题
步骤二
各种搜索
failed to send join request to master connect_timeout 127.0.0.1:9350
然而，并没有解决
步骤三：
一个一个节点分别启动，启动一个节点，选为master，然后分别启动另外两个节点加入集群
启动一个节点后，自动选为master，启动第二个节点，发现日志异常：
not enough master nodes discovered during pinging
应该是该异常导致的博客提到的异常信息，具体日志如下：
[2019-03-11T18:00:33,670][WARN ][o.e.d.z.ZenDiscovery ] [slave] not enough master nodes discovered during pinging (found [[Candidate{node={slave}{5Uge7dikTTGMSsqMJewVYQ}{7gglOGteQm-lbwZb7g37Yw}{127.0.0.1}{127.0.0.1:9301}{ml.machine_memory=17054658560, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
解决方案：
逻辑上讲是，一个节点去ping master节点，结果却没有得到回应，多次重试依旧失败。应该重点观察：
1.网络，master启动后，ip有没有开通防火墙的权限；
2.查看该端口下，如9300是哪个服务在运行；
3.数据不一致，elasticsearch目录下data存放数据的目录，是否和本节点的数据不一致；
个人的问题正好是第三个，data目录数据不一致。因为是复制别人的elasticsearch，原来的节点是有data的，复制过程中，data也存放了原来的信息，因为是本地测试环境，所以data目录删除后，一切恢复正常！其实，日志提示的错误很明显，每个节点单独启动，但是读取的数据却是一直的，导致多个节点不能形成集群。