elasticsearch集群报错解决

http://stackoverflow.com/questions/37451247/elasticsearch-cluster-cant-join-new-node
问题一：
[es-node2] failed to connect to master [{es-node1}

解决办法：
node2
91222.elk.node2.com<2016-12-14 11:46:45> /opt/elasticsearch-node2/bin
elasticsearch>$ ./plugin install license
elasticsearch>$ ./plugin install marvel-agent
-> Installing marvel-agent...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.4.1/marvel-agent-2.4.1.zip ...
Downloading ..........DONE
Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.4.1/marvel-agent-2.4.1.zip checksums if available ...
Downloading .DONE
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

• java.lang.RuntimePermission setFactory
• javax.net.ssl.SSLPermission setHostnameVerifier
  See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
  for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed marvel-agent into /opt/elasticsearch-node2/plugins/marvel-agent

node3
91223.elk.node3.com<2016-12-14 11:47:22> /opt/elasticsearch-node3/bin
elasticsearch>$ ./plugin install license
elasticsearch>$ ./plugin install marvel-agent
-> Installing marvel-agent...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.4.1/marvel-agent-2.4.1.zip ...
Downloading ..........DONE
Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.4.1/marvel-agent-2.4.1.zip checksums if available ...
Downloading .DONE
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

• java.lang.RuntimePermission setFactory
• javax.net.ssl.SSLPermission setHostnameVerifier
  See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
  for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed marvel-agent into /opt/elasticsearch-node3/plugins/marvel-agent

问题二：
[2016-12-14 12:03:19,126][ERROR][marvel.agent.exporter.http] index failure (index:[.marvel-es-data-1] type: [cluster_info]): {reason=[.marvel-es-data-1][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [.marvel-es-data-1] containing [4] requests], type=unavailable_shards_exception}
解决办法：
查看索引
91221.elk.node1.com<2016-12-14 12:02:17> ~
root># curl -XGET http://192.168.91.221:9200/_cat/shards
.marvel-es-1-2016.12.14 0 p STARTED 433 259.6kb 192.168.91.221 es-node1
.marvel-es-1-2016.12.14 0 r STARTED 432 276.3kb 192.168.91.222 es-node2
.kibana 0 p STARTED 1 3.1kb 192.168.91.222 es-node2
.kibana 0 r STARTED 1 3.1kb 192.168.91.223 node-3
.marvel-es-data-1 0 p UNASSIGNED
.marvel-es-data-1 0 r UNASSIGNED

删除UNASSIGNED状态的索引
91221.elk.node1.com<2016-12-14 12:06:22> ~
root># curl -XDELETE 'http://192.168.91.221:9200/.marvel-es-data-1*'
{"acknowledged":true}
91221.elk.node1.com<2016-12-14 12:06:36> ~
root># curl -XGET http://192.168.91.221:9200/_cat/shards
.marvel-es-1-2016.12.14 0 p STARTED 514 583.7kb 192.168.91.221 es-node1
.marvel-es-1-2016.12.14 0 r STARTED 514 567kb 192.168.91.222 es-node2
.kibana 0 p STARTED 1 3.1kb 192.168.91.222 es-node2
.kibana 0 r STARTED 1 3.1kb 192.168.91.223 node-3

elasticseachr日志发现集群状态已经恢复了
[2016-12-14 12:06:46,980][INFO ][cluster.routing.allocation] [es-node1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.marvel-es-data-1][0]] ...]).
[2016-12-14 12:06:47,995][INFO ][cluster.routing.allocation] [es-node1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.marvel-es-data-1][0]] ...]).
如图所示，从红色变成了绿色

11.png