Upgrade Elasticsearch7.5.0

1. Elasticsearch7.5.0安装

   切换到root用户

   sudo su -

   配置elasticsearch用户

   usermod -s /bin/bash elasticsearch

   创建数据和日志目录

   mkdir /var/log/elasticsearch7.5.0
   mkdir /data/elasticsearch7.5.0

   分配给elasticsearch用户

   chown -R elasticsearch:elasticsearch /var/log/elasticsearch7.5.0
   chown -R elasticsearch:elasticsearch /data/elasticsearch7.5.0

   下载安装

   cd /opt
   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.0-linux-x86_64.tar.gz
   tar zxvf elasticsearch-7.5.0-linux-x86_64.tar.gz
   chown -R elasticsearch elasticsearch-7.5.0

2. 修改配置文件

   1. GIS单节点 /opt/elasticsearch-7.5.0/config/elasticsearch.yml 供参考

      cluster.name: gis_es
      node.name: gis-es-1.novalocal
      node.attr.rack: rc-1
      path.data: /data/elasticsearch7.5.0
      path.logs: /var/log/elasticsearch7.5.0
      bootstrap.memory_lock: true
      network.host: ...
      http.port: 9201
      transport.tcp.port: 9301
      cluster.initial_master_nodes: ["gis_es"]
      reindex.remote.whitelist: ...:9200

   2. Trip 集群es-traffic-mgmt-1节点 /opt/elasticsearch-7.5.0/config/elasticsearch.yml 供参考

      cluster.name: trip-traffic_es
      node.name: es-traffic-mgmt-1.novalocal
      node.attr.rack: rc-1
      path.data: /data/elasticsearch7.5.0
      path.logs: /var/log/elasticsearch7.5.0
      bootstrap.memory_lock: true
      network.host: ...
      http.port: 9201
      transport.tcp.port: 9301
      discovery.seed_hosts: ["...:9301", "...:9301"]
      cluster.initial_master_nodes: ["es-traffic-mgmt-1", "es-trip-mgmt-1"]
      reindex.remote.whitelist: ...:9200

   3. Trip 集群es-trip-mgmt-1节点 /opt/elasticsearch-7.5.0/config/elasticsearch.yml 供参考

      cluster.name: trip-traffic_es
      node.name: es-trip-mgmt-1.novalocal
      node.attr.rack: rc-1
      path.data: /data/elasticsearch7.5.0
      path.logs: /var/log/elasticsearch7.5.0
      bootstrap.memory_lock: true
      network.host: ...
      http.port: 9201
      transport.tcp.port: 9301
      discovery.seed_hosts: ["...:9301", "...:9301"]
      cluster.initial_master_nodes: ["es-traffic-mgmt-1", "es-trip-mgmt-1"]
      reindex.remote.whitelist: ...:9200

3. 启动

   1. 切换到elasticsearch启动
      su - elasticsearch
      /opt/elasticsearch-7.5.0/bin/elasticsearch -d

      如果报错：
      ERROR: [2] bootstrap checks failed
      [1]: max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]
      cp /etc/security/limits.d/elsearch-limit.conf /etc/security/limits.d/elsearch-limit.conf.bak
      则修改配置文件：vi /etc/security/limits.d/elsearch-limit.conf
      elasticsearch soft nproc 2048 改为 4096
      elasticsearch hard nproc 2048改为 4096

   2. 重启命令
      找到进程kill ps -ef |grep elasticsearch-7.5.0
      启动 /opt/elasticsearch-7.5.0/bin/elasticsearch -d

4. 数据迁移

   1. 主要使用官方自带的Reindex from a remote cluster
      参考:https://www.elastic.co/guide/en/elasticsearch/reference/7.5/reindex-upgrade-remote.html

   2. 但是迁移数据前需要手工在es7.5创建好索引，所以新建python3脚本来创建索引和迁移数据

   3. 运行脚本时候，很有可能报错”maximum shards open“则修改max_shards_per_node

   4. |

      curl -X PUT ``"127.0.0.1:9201/_cluster/settings" -H ``'Content-Type: application/json' -d'

      {

      "persistent" : {

      "cluster.max_shards_per_node" : ``"2000"

      }

      }

      '

      |

   5. 修改脚本中的3行，替换脚本中的127.0.0.1 为具体的ip

      reindex_py3.py 展开源码

      es_old ip port 必须与 /opt/elasticsearch-7.5.0/config/elasticsearch.yml 中的 reindex.remote.whitelist: 
      : 保持一致，如必须同为 127.0.0.1:9200 或者localhost:9200， 不能一个用localhost 另一个用127.0.0.1

      ES7.5.0 默认内存是1G， 导入可能报错, 需要修改 /opt/elasticsearch-7.5.0/config/jvm.options, 根据vm 内存进行调整，内存过大可能导致机器无法使用

      |

      -Xms3g

      -Xmx3g

      |

      Error log

      |

      elasticsearch.exceptions.TransportError: TransportError(``429``, ``'circuit_breaking_exception'``, ``'[parent] Data too large, data for [] would be [1023885146/976.4mb], which is larger than the limit of [1003493785/957mb], real usage: [1023865048/976.4mb], new bytes reserved: [20098/19.6kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=20098/19.6kb, accounting=20440594/19.4mb]'``)

      |

5. 验证结果

   1. 打开es7.5.0 查看索引“.tasks“数据，记录了Reindex 的结果。

6. 配置为系统服务

   1. 待补充

7. 停止旧版本ES服务

   1. root用户 systemctl stop elasticsearch

8. 多类型索引更新
   因为旧版本支持一个索引包含多个type，而新版本一个索引建议只包含一个type，所以旧版本包含多个type的所以需要拆分多个新索引再同步数据