ELK从5.5.0升级到6.1.3升级方案

写在最前，ES6的重大变化

1. 安装6.1.3版本的软件

1.1 安装JAVA1.8（不用操作，沿用老的）

• 卸载1.8以下的版本

# rpm -–nodeps –e java-*

• 安装jdk1.8

# mkdir /usr/local/elk
# tar xf jdk-8u141-linux-x64.tar -C /usr/local/elk/

• 设置JAVA环境变量

# vi /etc/profile
JAVA_HOME=/usr/local/elk/jdk1.8.0_144
JRE_HOME=/usr/local/elk/jdk1.8.0_144/jre
CLASSPATH=.:$JAVA_HOME/lib:/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME
export JRE_HOME

• JAVA环境变量生效

# source /etc/profile

• JAVA安装验证

#java –version

1.2 安装ElasticSearch 6.1.3

• 解压文件，并放置到对应的目录

# unzip elasticsearch-6.1.3.zip
# mv elasticsearch-6.1.3 /usr/local/elk/

• 添加用户，设置目录权限

# useradd elk
# chown –R elk:elk /usr/local/elk/elasticsearch-6.1.3

• 修改配置文件

# vi /usr/local/elk/elasticsearch-6.1.3/config/elasticsearch.yml

cluster.name: es-test
node.name: node-122
#注意此处设置为一个公共的位置，不放置到某一个版本里面
path.data: /usr/local/elk/elasticsearch/data
path.logs: /usr/local/elk/elasticsearch/logs
network.host: 192.168.237.122
http.port: 9200
#如果集群内有多台ES，否则按默认
discovery.zen.ping.unicast.hosts: ["192.168.237.119","192.168.237.109","192.168.237.122"]
discovery.zen.minimum_master_nodes: 2
#CentOS7.0版本以下需要做这个设置
bootstrap.system_call_filter: false
#X-PACK插件需要这个配置
xpack.security.transport.ssl.enabled: true

#如果配置cross-cluster
search:
    remote:
        es-test1:
            seeds: 192.168.237.109:9300
        es-test2:
            seeds: 192.168.237.119:9300
#集群节点（master/data）设置(安装x-pack后)：
master:
node.master: true 
node.data: false 
node.ingest: false 
node.ml: false 
xpack.ml.enabled: true 

data:
node.master: false 
node.data: true 
node.ingest: false 
node.ml: false

• 修改系统参数

# vi /etc/security/limits.conf

* soft nofile 65536
* hard nofile 131072
#注意这个值在新版本中必须设置为最少是4096
* soft nproc 4096
* hard nproc 8192

# vi /etc/sysctl.conf

vm.max_map_count=655360

# sysctl -p
# vi /etc/security/limits.d/90-nproc.conf

* soft nproc unlimited

1.3 为ElasticSearch安装X-PACK插件，并破解

• 本地安装

# cd /usr/local/elk/elasticsearch-6.1.3/bin
# ./elasticsearch-plugin install file:///data/src/x-pack-6.1.3.zip
# chown -R elk:elk /usr/elk/local/elasticsearch-6.1.3/

• 生成kibana、logstash_system、elastic用户的密码

$bin/x-pack/setup-passwords auto
Initiating the setup of passwords for reserved users elastic,kibana,logstash_system.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y

Changed password for user kibana
PASSWORD kibana = !2&ffs6&W&$WOEft9tcX

Changed password for user logstash_system
PASSWORD logstash_system = ev##5$n0H$4AYe75KJL8

Changed password for user elastic
PASSWORD elastic = BdcjL76vK*l%^Y5G#CZI

• 验证

# curl –XGET –u user:passwd ‘http://192.168.237.122:9200/’

• 破解X-PACK的License
  我是破解的方法，点我！点我！点我！
  本文假设已经生成了破解后的x-pack-6.1.3.jar ,并获得了license.json文件。进行如下操作进行破解。

# mv /usr/elk/local/elasticsearch-6.1.3/plugins/x-pack/x-pack-6.1.3.jar /usr/elk/local/elasticsearch-6.1.3/plugins/x-pack/x-pack-6.1.3.jar.bak
# mv /root/src/License/x-pack-6.1.3.jar /usr/elk/local/elasticsearch-6.1.3/plugins/x-pack/
# curl -XPUT -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://127.0.0.1:9200/_xpack/license?acknowledge=true'   -H "Content-Type: application/json" -d @/data/src/License/license.json
# chown -R elk:elk /usr/elk/local/elasticsearch-6.1.3/

1.4 安装Logstash

• 解压缩到对应的目录

#tar xzf logstash-6.1.3.tar.gz -C /usr/local/elk/

• 性能调优
  修改jvm内存为物理机内存的一半

# vi /usr/local/elk/logstash-6.1.3/config/jvm.options

-Xms4g
-Xmx4g

pipeline.workers为cpu逻辑核数的1-1.5倍

#vi /usr/local/elk/logstash-6.1.3/config/logstash.yml

node.name: test122
pipeline.workers: 12
pipeline.batch.size: 8192
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http://192.168.237.109:9200","http://192.168.237.119:9200"]
xpack.monitoring.elasticsearch.username: "user"
xpack.monitoring.elasticsearch.password: "passwd"

• 安装X-PACK

#cd /usr/local/elk/logstash-6.1.3/bin
#logstash-plugin install file:///root/src/x-pack-6.1.3.zip

• 清空logstatch的template

curl -XDELETE 'http://localhost:9200/_template/logstash

1.5 安装Kibana安装

• 解压到对应的位置

# tar xzf kibana-6.1.3-linux-x86_64.tar.gz –C /usr/local/elk/

• 配置文件修改

# vi /usr/local/elk/kibana-6.1.3-linux-x86_64/config/kibana.yml

server.port: 5601
elasticsearch.url: http://192.168.237.122:9200
#此处注意使用自动生成的kibana用户
elasticsearch.username: "user"
elasticsearch.password: "passwd"
#如果多台kibana连接到同一ES，需要设置不同的kibana.index
kibana.index: ".kibana"
#如果一台机器开启多个kibana应用，需要设置不同的pid
pid.file: /var/run/kibana.pid

• 安装X-PACK

#cd /usr/local/elk/kibana-6.1.3-linux-x86_64/bin
#lkibana-plugin install file:///root/src/x-pack-6.1.3.zip

2. 升级（full cluster restart upgrade 方式升级）

2.1 关闭Logstash，停止继续采集日志

2.2 停止旧版本elasticsearch服务并拷贝data目录

• 关闭shard Allocation：避免不必要的I/O

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}
'

• 停止索引，并且进行一次同步的flush

curl -XPOST 'localhost:9200/_flush/synced?pretty'

• 关闭elasticsearch-5.5.0
• 将旧版本的data目录和log目录拷贝到一个公共的目录，例如：/usr/local/elasticsearch/data
• 配置新的elasticsearch-6.1.3中的data目录和log目录指向这个/usr/local/elasticsearch/data和/usr/local/elasticsearch/log
• 开启elasticsearch-6.1.3
• X-PACK重置密码

bin/x-pack/setup-passwords auto

• 检查需要升级的索引（注意用新新生成的密码）

curl -XGET -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://192.168.237.119:9200/_xpack/migration/assistance?pretty'

• 升级这些要升级的索引

curl -XPOST -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://192.168.237.119:9200/_xpack/migration/upgrade/要升级索引的名字?pretty'

• 此时连接kibana会报错，需要升级和kibana6的对接

curl -XPUT -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://192.168.237.119:9200/.kibana/_settings?pretty' -H 'Content-Type: application/json' -d'
{
  "index.blocks.write": true
}

curl -XPUT -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://192.168.237.119:9200/.kibana-6?pretty' -H 'Content-Type: application/json' -d'
{
  "settings" : {
    "number_of_shards" : 1,
    "index.mapper.dynamic": false
  },
  "mappings" : {
    "doc": {
      "properties": {
        "type": {
          "type": "keyword"
        },
        "updated_at": {
          "type": "date"
        },
        "config": {
          "properties": {
            "buildNum": {
              "type": "keyword"
            }
          }
        },
        "index-pattern": {
          "properties": {
            "fieldFormatMap": {
              "type": "text"
            },
            "fields": {
              "type": "text"
            },
            "intervalName": {
              "type": "keyword"
            },
            "notExpandable": {
              "type": "boolean"
            },
            "sourceFilters": {
              "type": "text"
            },
            "timeFieldName": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            }
          }
        },
        "visualization": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "savedSearchId": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "visState": {
              "type": "text"
            }
          }
        },
        "search": {
          "properties": {
            "columns": {
              "type": "keyword"
            },
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "sort": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "dashboard": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "optionsJSON": {
              "type": "text"
            },
            "panelsJSON": {
              "type": "text"
            },
            "refreshInterval": {
              "properties": {
                "display": {
                  "type": "keyword"
                },
                "pause": {
                  "type": "boolean"
                },
                "section": {
                  "type": "integer"
                },
                "value": {
                  "type": "integer"
                }
              }
            },
            "timeFrom": {
              "type": "keyword"
            },
            "timeRestore": {
              "type": "boolean"
            },
            "timeTo": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "url": {
          "properties": {
            "accessCount": {
              "type": "long"
            },
            "accessDate": {
              "type": "date"
            },
            "createDate": {
              "type": "date"
            },
            "url": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 2048
                }
              }
            }
          }
        },
        "server": {
          "properties": {
            "uuid": {
              "type": "keyword"
            }
          }
        },
        "timelion-sheet": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "timelion_chart_height": {
              "type": "integer"
            },
            "timelion_columns": {
              "type": "integer"
            },
            "timelion_interval": {
              "type": "keyword"
            },
            "timelion_other_interval": {
              "type": "keyword"
            },
            "timelion_rows": {
              "type": "integer"
            },
            "timelion_sheet": {
              "type": "text"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "graph-workspace": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "numLinks": {
              "type": "integer"
            },
            "numVertices": {
              "type": "integer"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "wsState": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}
'

curl -XPOST -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://192.168.237.119:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": ".kibana"
  },
  "dest": {
    "index": ".kibana-6"
  },
  "script": {
    "inline": "ctx._source = [ ctx._type : ctx._source ]; ctx._source.type = ctx._type; ctx._id = ctx._type + \":\" + ctx._id; ctx._type = \"doc\"; ",
    "lang": "painless"
  }
}
'

curl -XPOST -uelastic:BdcjL76vK*l%^Y5G#CZI 'http://192.168.237.119:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
  "actions" : [
    { "add":  { "index": ".kibana-6", "alias": ".kibana" } },
    { "remove_index": { "index": ".kibana" } }
  ]
}
'

• 重新开启allocation

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
 "transient": {
   "cluster.routing.allocation.enable": "all"
 }
}
'

• 检查ES的状态

curl -XGET 'localhost:9200/_cat/health?pretty'
curl -XGET 'localhost:9200/_cat/recovery?pretty'

3. 配置集群间的TLS

不加密在集群间传输所有的数据将不允许安装一个带X-PACK安全组件的license。

2.1 为每一个es节点生成一个私钥和X.509证书。

• 为整个cluster生成一个CA。

bin/x-pack/certutil ca

• 将这个CA拷贝到cluster的每台服务器上，并且生成证书和私钥（证书和私钥都在同一个文件），可以不配置密码。

bin/x-pack/certutil cert --ca elastic-stack-ca.p12

• 如果配置密码的话还要设置keystore

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

• 将这个证书

2.2 将证书配置到es节点的配置文件上

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

2.3