2019-07-03 使用 elasticsearch curator 管理索引 --原创

什么是elasticsearch curator

curator 是管理elasticsearch 索引生命周期的工具, 有如下功能

• 对alias增加删除索引
• open/close索引
• forcemerge 索引
• reindex索引
• backup

installation

• es 版本5.6 curator版本5.7.6 注意版本兼容性
• 使用virtualenv创建一个隔离环境

~ cd /opt/app
~ virtualenv  --no-site-packages --python=python2.7 elasticsearch-curator
~ source elasticsearch-curator/bin/activate
~ pip install -U elasticsearch-curator==5.7.6 
~ deactivate

config

• 增加config.yml
  • 需要配置es节点的ip及端口, log等信息
  • vim config.yml

 client:
  hosts:
    - XXX.XXX.XXX.XX1:9200
    - XXX.XXX.XXX.XX2:9200
    - XXX.XXX.XXX.XX3:9200
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['urllib3']

• 示例增加一个force_merge的任务配置 (当然还支持其他类型的任务请参考这里)

  • force_merge 索引的范围
    以XXXXXXX开头的索引, 并且其中单个shard的segment个数大于2的索引进行force merge

  • vim formerge.yml

actions:
  1:
    action: forcemerge
    description: >-
      forceMerge XXXXX- prefixed indices and over
      to 2 segments per shard.  Delay 60 seconds between each
      forceMerge operation to allow the cluster to quiesce. Skip indices that
      have already been forcemerged to the minimum number of segments to avoid
      reprocessing.
    options:
      max_num_segments: 1
      delay: 10
      timeout_override: 21600
      continue_if_exception:
      disable_action:
    filters:
    - filtertype: pattern
      kind: prefix
      value: XXXXXXX
      exclude:
    - filtertype: forcemerged
      max_num_segments: 2
      exclude: True

• 增加一个shell 去执行force merge任务

  • 需要注意我们的任务是在python的虚拟环境中执行的,所以在执行任务前需要激活python虚拟环境. 如下代码中的activate

  • vim force_merge.sh

#!/bin/bash
set -e
. "/etc/profile"
. "/home/$USER/.bashrc"
. "/opt/app/elasticsearch-curator/bin/activate"
curDir=$(cd `dirname $0`; pwd)
cd $curDir
curator --config config.yml formerge.yml

上调度定时执行

• 运行结果

  
  
  

  红框中有3个索引被force merge

总结

使用curator比自己写shell去维护索引方便很多 , 而且功能支持的支持的比较全面.

renfrence

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about-features.html