elasticsearch-curator 随笔
elasticsearch-curator 随笔
- 安装
- 配置
安装
安装方式有pip及yum，本人采用pip安装。
首先确认系统是否已经有pip，没有的话按以下方式安装
[root@tmaster3 src]# wget https://bootstrap.pypa.io/get-pip.py
--2020-09-11 10:26:58-- https://bootstrap.pypa.io/get-pip.py
正在解析主机 bootstrap.pypa.io (bootstrap.pypa.io)... 151.101.228.175, 2a04:4e42:1a::175
正在连接 bootstrap.pypa.io (bootstrap.pypa.io)|151.101.228.175|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：1885433 (1.8M) [text/x-python]
正在保存至: “get-pip.py”
100%[===================================================================================================================================================================================================>] 1,885,433 1.57MB/s 用时 1.1s
2020-09-11 10:27:00 (1.57 MB/s) - 已保存 “get-pip.py” [1885433/1885433])
##安装
[root@tmaster3 src]# python get-pip.py
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pip
Downloading pip-20.2.3-py2.py3-none-any.whl (1.5 MB)
|████████████████████████████████| 1.5 MB 489 kB/s
Collecting setuptools
Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
|████████████████████████████████| 583 kB 2.0 MB/s
Collecting wheel
Downloading wheel-0.35.1-py2.py3-none-any.whl (33 kB)
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-20.2.3 setuptools-44.1.1 wheel-0.35.1
确认安装情况
[root@tmaster3 src]# pip --version
pip 20.2.3 from /usr/lib/python2.7/site-packages/pip (python 2.7)
安装elasticsearch-curator
[root@tmaster3 ~]# pip install elasticsearch-curator -i https://mirrors.aliyun.com/pypi/simple/
-i https://mirrors.aliyun.com/pypi/simple/ 使用国内阿里仓库提高速度
ERROR: No matching distribution found for botocore<1.18.0,>=1.17.57 (from boto3>=1.9.142->elasticsearch-curator)
安装过程出现以上错误。没有仔细研究，猜测是elasticsearch-curator需要botocore更改版本在pip仓库里面没找到，因为本机的Python是用到老的2.7版本。本人解决办法，先独自安装botocore
[root@tmaster3 ~]# pip install botocore -i https://mirrors.aliyun.com/pypi/simple/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting botocore
Downloading https://mirrors.aliyun.com/pypi/packages/b1/82/499909b818bddde1a4fc1228389d9d29cc2ede766a2a7370aed033dd07f9/botocore-1.17.56-py2.py3-none-any.whl (6.6 MB)
|████████████████████████████████| 6.6 MB 1.7 MB/s
Collecting urllib3<1.26,>=1.20; python_version != "3.4"
Downloading https://mirrors.aliyun.com/pypi/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
|████████████████████████████████| 127 kB 2.1 MB/s
Collecting python-dateutil<3.0.0,>=2.1
Downloading https://mirrors.aliyun.com/pypi/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
|████████████████████████████████| 227 kB 2.1 MB/s
Collecting docutils<0.16,>=0.10
Downloading https://mirrors.aliyun.com/pypi/packages/3a/dc/bf2b15d1fa15a6f7a9e77a61b74ecbbae7258558fcda8ffc9a6638a6b327/docutils-0.15.2-py2-none-any.whl (548 kB)
|████████████████████████████████| 548 kB 2.1 MB/s
Collecting jmespath<1.0.0,>=0.7.1
Downloading https://mirrors.aliyun.com/pypi/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting six>=1.5
Using cached https://mirrors.aliyun.com/pypi/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: urllib3, six, python-dateutil, docutils, jmespath, botocore
Successfully installed botocore-1.17.56 docutils-0.15.2 jmespath-0.10.0 python-dateutil-2.8.1 six-1.15.0 urllib3-1.25.10
然后再重新安装elasticsearch-curator 使用–ignore-installed botocore 跳过这个依赖
[root@tmaster3 ~]# pip install elasticsearch-curator --ignore-installed botocore -i https://mirrors.aliyun.com/pypi/simple/
[root@tmaster3 ~]# curator --version
curator, version 5.8.1
至此安装完毕。
配置
1、删除索引
这里以下面日志索引为例。索引信息如下
配置按日期规则删除保留三天有效索引
创建配置文件目录
[bravo@tmaster3 ~]$ mkdir -p /apps/conf/elasticsearch-curator
编辑主配置文件
[bravo@tmaster3 ~]$ vi /apps/conf/elasticsearch-curator/curator.yml
加入以下配置信息
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- 172.16.96.27
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth: elastic:9CQOR******5DS9X
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile: /apps/logs/elasticsearch-curator/curator.log
logformat: default
blacklist: ['elasticsearch', 'urllib3']
编辑action配置文件，该文件主要是对索引的具体操作动作
[bravo@tmaster3 elasticsearch-curator]$ vi action.yml
---
actions:
1:
action: delete_indices #删除索引动作
description: "delete index as chatserver-log" #描述
options:
ignore_empty_list: True #根据您的索引及其过滤方式，可能会向操作显示一个空列表。这导致错误状态。当ignore_empty_list选项设置为时True，该操作将退出，并显示一条指示此操作的INFO级别日志消息，此设置的默认值为 False
filters: #过滤条件
- filtertype: pattern
kind: prefix
value: chatserver-log-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 3
验证配置文件
[bravo@tmaster3 elasticsearch-curator]$ curator --dry-run --config /apps/conf/elasticsearch-curator/curator.yml /apps/conf/elasticsearch-curator/action.yml
改参数是模拟执行过程，但不真正执行 --dry-run Do not perform any changes.
[bravo@tmaster3 elasticsearch-curator]$ tail /apps/logs/elasticsearch-curator/curator.log
2020-09-11 14:48:27,086 INFO Instantiating client object
2020-09-11 14:48:27,087 INFO Testing client connectivity
2020-09-11 14:48:27,090 INFO Successfully created Elasticsearch client object with provided settings
2020-09-11 14:48:27,091 INFO Trying Action ID: 1, "delete_indices": delete index as chatserver-log
2020-09-11 14:48:27,188 INFO DRY-RUN MODE. No changes will be made.
2020-09-11 14:48:27,188 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2020-09-11 14:48:27,188 INFO DRY-RUN: delete_indices: chatserver-log-2020.09.07 with arguments: {}
2020-09-11 14:48:27,189 INFO DRY-RUN: delete_indices: chatserver-log-2020.09.08 with arguments: {}
2020-09-11 14:48:27,189 INFO Action ID: 1, "delete_indices" completed.
2020-09-11 14:48:27,189 INFO Job completed.
查看日志可以发现正常的过滤出两个符合条件的索引并进行删除动作
2、强合并索引
在action配置文件新增force merge 动作
[bravo@tmaster3 elasticsearch-curator]$ vi action.yml
新增以下内容
2:
action: forcemerge
description: "forceMerge chatserver-log prefixed indices yesterday"
options:
max_num_segments: 1
delay: 120
timeout_override:
continue_if_exception: False
# disable_action: True
filters:
- filtertype: pattern
kind: prefix
value: chatserver-log-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 1
- filtertype: forcemerged
max_num_segments: 1
exclude: true
验证配置文件
2020-09-15 09:43:11,668 INFO Preparing Action ID: 2, "forcemerge"
2020-09-15 09:43:11,668 INFO Creating client object and testing connection
2020-09-15 09:43:11,669 INFO Instantiating client object
2020-09-15 09:43:11,670 INFO Testing client connectivity
2020-09-15 09:43:11,674 INFO Successfully created Elasticsearch client object with provided settings
2020-09-15 09:43:11,676 INFO Trying Action ID: 2, "forcemerge": forceMerge chatserver-log prefixed indices yesterday
2020-09-15 09:43:11,766 INFO DRY-RUN MODE. No changes will be made.
2020-09-15 09:43:11,767 INFO (CLOSED) indices may be shown that may not be acted on by action "forcemerge".
2020-09-15 09:43:11,767 INFO DRY-RUN: forcemerge: chatserver-log-2020.09.14 with arguments: {'max_num_segments': 1, 'delay': 120.0}
2020-09-15 09:43:11,767 INFO Action ID: 2, "forcemerge" completed.
最后将命令执行加入的crontab任务计划中。