All data nodes of One ES Cluster have read-only issue

Diagnose

1. There's one ES Cluster combined with VMS with volumes attached to it as below:

Node	Role
es5	data
es6	data
es7	data
es8	master
es9	master
es10	master

2. All the 3 data nodes are attached with one cinder volume "/data1" as one of their data folder.
3. However due to one Network issue happened last Friday, all of the 3 volumes became "Read-Only File System", which means they're not able to write new documents in.
4. Then it impacts the whole ES Cluster to do operation on shards (distribute/replica). So the whole ES Cluster is in red status since some indices having shard issues.
5. Then restart the Elasticsearch service on es5, and see exceptions as below:

Caused by: java.io.FileNotFoundException: /data1/instance00/stackstash-elasticsearch/nodes/0/indices/alert_08052014/3/index/_8h5j.fdx (Read-only file system)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.(FileOutputStream.java:212)
    at java.io.FileOutputStream.(FileOutputStream.java:165)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:384)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:277)
    at org.apache.lucene.store.FileSwitchDirectory.createOutput(FileSwit

6. The attached volume is still in "Read-Only" mode.
7. Then reboot the data node, and "Read-Only" issue is resolved.
8. However Elasticsearch service can’t start on es5 node because "failed to read local state". There may be some broken state files due to read-only issue.
9. The same thing happened on es6 and es7.
10. So till now, the ES Cluster only has 3 master nodes working normally. While all of the 3 data nodes can not start up Elasticsearch service.

How to recover

1. The replica count for each index was set to 2 which means there was a backup of all the data on the remaining 2 data node anyways.
2. Shutdown the data node with the corrupt Lucene shards, which is es5.
3. Start up Elasticsearch service on es6 and es7 by disable shard-allocation (can do it on any master node):

curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.enable": "none"}}'

4. Above steps bring up the data nodes, and can see all is ok. So enable shards allocation as below, and the cluster took a bit of time to turned green:

curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.enable": "all"}}'

5. Change the replica count all for indices to 1 (original is 2) in order to speed up recovery and rebalancing.
6. Wait for shards to rebalance (we’re still red because there are unassigned primaries).
7. Purge the data filesystem on the corrupt data node.
8. Restart the corrupt data node (it’s fine now) and allow rebalancing to start.
9. Identify remaining primaries that are not automatically being initialised by ES.
10. Manually assign remaining primaries to the newly recovered data node.

for index in $(curl -s -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $1}' | sort -u); do
 for shard in $(curl -s -XGET http://localhost:9200/_cat/shards/$index | grep UNASSIGNED | awk '{print $2}' | sort -u); do
   curl -s -XPOST 'localhost:9200/_cluster/reroute' -d "{
       \"commands\" : [ {
             \"allocate\" : {
                 \"index\" : \"$index\",
                 \"shard\" : $shard,
                 \"node\" : \"es5\",
                 \"allow_primary\" : true
             }
           }
       ]
   }"
   sleep 5
 done

11. Wait for cluster to turn green.
12. Change the replica count for all indices back to 2 (cluster goes yellow).
13. Wait for cluster to turn green.

What we get

1. Move away from volumes as local disk is plentiful on most VM flavors and instead rely on replicating shards across data nodes at some point.
2. During recover, we can set replica to "1" to speed up rebalancing. Then set replica back after recovering.

Reference

1. Shard allocation disable/enable