微信公众号:运维开发故事,作者:wanger
esrally是elastic search官方用于对ES集群进行压力测试的工具,使用esrally可以为我们构建不同版本集群,构造不同的参数和数据来进行压力测试,并且可以对产生的压测结果进行比较,rally顾名思义是拉力赛的意思,esrally的一些名词也都与拉力赛有关。
github地址:https://github.com/elastic/rally
python3.8 pip3
jdk8
git 1.9+
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make libffi-devel
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz
tar -xvJf Python-3.8.2.tar.xz
mkdir /usr/local/python3
cd Python-3.8.2/
./configure --prefix=/usr/local/python3
make && make install
ln -s /usr/local/python3/bin/python3 /usr/local/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/local/bin/pip3
由于yum安装的git版本默认为1.8,因此我需要编译安装较新版本的git,如果已通过yum安装过git,可以使用yum remove git卸载
yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel gcc perl-ExtUtils-MakeMaker
cd /tmp
wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.22.0.tar.gz
tar xzf git-2.22.0.tar.gz
cd git-2.22.0
make prefix=/usr/local/git all
make prefix=/usr/local/git install
echo "export PATH=$PATH:/usr/local/git/bin" >> /etc/bashrc
source /etc/bashrc
jdk安装
rpm -ivh jdk-8u221-linux-x64.rpm
esrally安装
python3 -m pip install esrally
vim /etc/profile
JAVA_HOME=/usr/java/jdk1.8.0_221-amd64/jre
export PATH=$PATH:/usr/local/python3/bin/:/usr/local/git/bin JAVA_HOME
source /etc/profile
配置ESrally
用于定义一些数据的配置,可以选择把结果存储到已有的ES中进行分析
esrally configure
配置完成后,将会覆写ESrally的配置文件/root/.rally/rally.ini
ESrally名词解释
track
赛道的意思,用于构建不同的数据和策略进行压测,track.json定义压测的数据集,以geonames/track.json为例
{% import "rally.helpers" as rally with context %}
{
"version": 2,
"description": "POIs from Geonames",
"data-url": "http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames",
"indices": [
{
"name": "geonames",
"body": "index.json"
}
],
"corpora": [
{
"name": "geonames",
"base-url": "http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames",
"documents": [
{
"source-file": "documents-2.json.bz2",
"document-count": 11396503,
"compressed-bytes": 265208777,
"uncompressed-bytes": 3547613828
}
]
}
],
"operations": [
{{ rally.collect(parts="operations/*.json") }}
],
"challenges": [
{{ rally.collect(parts="challenges/*.json") }}
]
}
当我们开始用esrally开始比赛的时候,会先根据base-url和source-files构建链接从亚马逊下载数据,下载测试数据的速度很慢,我们可以提前下好离线数据来进行测试,测试数据存放在/home/elk/.rally/benchmarks/data/geonames目录下,indices定义了索引名和索引的具体设置,通过index.json文件可以查看
{
"settings": {
"index.number_of_shards": {{number_of_shards | default(5)}},
"index.number_of_replicas": {{number_of_replicas | default(0)}},
"index.store.type": "{{store_type | default('fs')}}",
"index.requests.cache.enable": false
},
"mappings": {
"dynamic": "strict",
"_source": {
"enabled": {{ source_enabled | default(true) | tojson }}
},
"properties": {
"elevation": {
"type": "integer"
},
"name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"geonameid": {
"type": "long"
},
"feature_class": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"location": {
"type": "geo_point"
},
"cc2": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"timezone": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"dem": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"country_code": {
"type": "text",
"fielddata": true,
"fields": {
"raw": {
"type": "keyword"
}
}
},
"admin1_code": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"admin2_code": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"admin3_code": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"admin4_code": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"feature_code": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"alternatenames": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"asciiname": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"population": {
"type": "long"
}
}
}
}
operations目录下定义了压测的具体操作,包括索引的写入、更新、段合并、各种查询,每个操作都可以构建不同的参数,
{
"name": "index-append",
"operation-type": "bulk",
"bulk-size": {{bulk_size | default(5000)}},
"ingest-percentage": {{ingest_percentage | default(100)}}
},
{
"name": "index-update",
"operation-type": "bulk",
"bulk-size": {{bulk_size | default(5000)}},
"ingest-percentage": {{ingest_percentage | default(100)}},
"conflicts": "{{conflicts | default('random')}}",
"on-conflict": "{{on_conflict | default('index')}}",
"conflict-probability": {{conflict_probability | default(25)}},
"recency": {{recency | default(0)}}
},
{
"name": "default",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
}
}
},
{
"name": "term",
"operation-type": "search",
"body": {
"query": {
"term": {
"country_code.raw": "AT"
}
}
}
},
{
"name": "phrase",
"operation-type": "search",
"body": {
"query": {
"match_phrase": {
"name": "Sankt Georgen"
}
}
}
},
{
"name": "country_agg_uncached",
"operation-type": "search",
"body": {
"size": 0,
"aggs": {
"country_population": {
"terms": {
"field": "country_code.raw"
},
"aggs": {
"sum_population": {
"sum": {
"field": "population"
}
}
}
}
}
}
},
{
"name": "country_agg_cached",
"operation-type": "search",
"cache": true,
"body": {
"size": 0,
"aggs": {
"country_population": {
"terms": {
"field": "country_code.raw"
},
"aggs": {
"sum_population": {
"sum": {
"field": "population"
}
}
}
}
}
}
},
{
"name": "scroll",
"operation-type": "search",
"pages": 25,
"results-per-page": 1000,
"body": {
"query": {
"match_all": {}
}
}
},
{
"name": "expression",
"operation-type": "search",
"body": {
"query": {
"function_score": {
"query": {
"match_all": {}
},
"functions": [
{
"script_score": {
"script": {
"source": "abs(ln(abs(doc['population']) + 1) + doc['location'].lon + doc['location'].lat) * _score",
"lang": "expression"
}
}
}
]
}
}
}
},
{
"name": "painless_dynamic",
"operation-type": "search",
"body": {
"query": {
"function_score": {
"query": {
"match_all": {}
},
"functions": [
{
"script_score": {
"script": {
"source": "Math.abs(Math.log(Math.abs(doc['population'].value) + 1) + doc['location'].lon * doc['location'].lat)/_score",
"lang": "painless"
}
}
}
]
}
}
}
},
{
"name": "decay_geo_gauss_function_score",
"operation-type": "search",
"body": {
"query": {
"function_score": {
"query": {
"match_all": {}
},
"gauss": {
"location": {
"origin": "52.37, 4.8951",
"scale": "500km",
"offset": "0km",
"decay" : 0.1
}
}
}
}
}
},
{
"name": "decay_geo_gauss_script_score",
"operation-type": "search",
"body": {
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "decayGeoGauss(params.origin, params.scale, params.offset, params.decay, doc['location'].value)",
"params": {
"origin": "52.37, 4.8951",
"scale": "500km",
"offset": "0km",
"decay" : 0.1
}
}
}
}
}
},
{
"name": "random_script_score",
"operation-type": "search",
"body": {
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "randomScore(100, '_seq_no')"
}
}
}
}
},
{
"name": "large_terms",
"operation-type": "search",
"param-source": "pure-terms-query-source"
},
{
"name": "desc_sort_population",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
},
"sort" : [
{"population" : "desc"}
]
}
},
{
"name": "asc_sort_population",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
},
"sort" : [
{"population" : "asc"}
]
}
},
{
"name": "desc_sort_geonameid",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
},
"sort" : [
{"geonameid" : "desc"}
]
}
},
{
"name": "asc_sort_geonameid",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
},
"sort" : [
{"geonameid" : "asc"}
]
}
}
challenge指定了压测时执行的task,不同的track包含一个或多个challenge,“name”: "append-no-conflicts"表示默认的challenge为append-no-conflicts,执行esrally list tracks可以查看不同track支持的challenge,default为true时,为默认执行的challenge,schedule定义了执行的任务列表
{
"name": "append-no-conflicts",
"description": "Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only. After that a couple of queries are run.",
"default": true,
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default({}) | tojson}}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
}
}
},
{
"operation": "index-append",
"warmup-time-period": 120,
"clients": {{bulk_indexing_clients | default(8)}}
},
{
"name": "refresh-after-index",
"operation": "refresh"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "refresh-after-force-merge",
"operation": "refresh"
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
},
{
"operation": "index-stats",
"warmup-iterations": 500,
"iterations": 1000,
"target-throughput": 90
},
{
"operation": "node-stats",
"warmup-iterations": 100,
"iterations": 1000,
"target-throughput": 90
},
{
"operation": "default",
"warmup-iterations": 500,
"iterations": 1000,
"target-throughput": 50
},
{
"operation": "term",
"warmup-iterations": 500,
"iterations": 1000,
"target-throughput": 150
},
{
"operation": "country_agg_uncached",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 3.6
},
{
"operation": "country_agg_cached",
"warmup-iterations": 1000,
"iterations": 1000,
"target-throughput": 100
},
{
"operation": "scroll",
"warmup-iterations": 200,
"iterations": 100,
"#COMMENT": "Throughput is considered per request. So we issue one scroll request per second which will retrieve 25 pages",
"target-throughput": 0.8
},
{
"operation": "expression",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 2
},
{
"operation": "painless_static",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.5
},
{
"operation": "painless_dynamic",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.5
},
{
"operation": "decay_geo_gauss_function_score",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1
},
{
"operation": "decay_geo_gauss_script_score",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1
},
{
"operation": "field_value_function_score",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.5
},
{
"operation": "field_value_script_score",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.5
},
{
"operation": "random_function_score",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.5
},
{
"operation": "large_filtered_terms",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.1
},
{
"operation": "large_prohibited_terms",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 1.1
},
{
"operation": "desc_sort_geonameid",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 6
},
{
"operation": "asc_sort_geonameid",
"warmup-iterations": 200,
"iterations": 100,
"target-throughput": 6
}
]
},
{
"name": "append-fast-with-conflicts",
"description": "Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings. Rally will produce duplicate ids in 25% of all documents (not configurable) so we can simulate a scenario with appends most of the time and some updates in between.",
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {%- if index_settings is defined %} {{index_settings | tojson}} {%- else %} {
"index.refresh_interval": "30s",
"index.number_of_shards": {{number_of_shards | default(6)}},
"index.translog.flush_threshold_size": "4g"
}{%- endif %}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
}
}
},
{
"operation": "index-update",
"warmup-time-period": 45,
"clients": {{bulk_indexing_clients | default(8)}}
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
}
]
}
car
用于定义不同配置的ES实例,我们可以定义堆内存大小,垃圾回收器,既然是赛车,那就可以改装,我们可以自定义不同配置的car,执行命令esrally list car可查看所有的赛车,配置在/home/elk/.rally/benchmarks/teams/default/cars/v1目录下
race
表示某一次的压测,并且可以指定赛道和赛车配置,不指定则使用默认配置,race结果存储在/home/elk/.rally/benchmarks/races目录下,执行命令esrally list races可查看以往比赛结果
并且可以对不同的race进行比较
esrally compare --baseline=d1be27e5-c459-4a40-ab7b-f3576cda7144 --contender=694f5c89-2ed1-4a82-8524-98a8e59778d4
就是定义以何种方式来构建这个集群,通过esrally list pipelines可查看所有的pipeline
from-sources-complete :表示从源代码构建ES, --revision参数可以指定一个ES的修订版本,默认为最新版本、
from-sources-skip-build :表示如果之前已经通过源码构建过一次相同版本的ES了,那么本次将不再进行构建,这样可以节省测试的时间
from-distribution:esrally将下载正式的ES发行版来进行测试, --distribution-version 参数可以指定ES版本,最低支持1.7.5
benchmark-only :将对已有的ES集群进行压测,–target-hosts参数可以指定ES集群地址
测试5.4.3、6.4.3与7.8.1版本之间的写入性能差异
使用非root用户运行,并且保证内存够用,这里我是用geonames赛道进行测试,使用默认的car为1gb堆内存,为了快速测试,可以将测试数据和ES二进制包提前下到指定的目录下,ES二进制包存储在/home/elk/.rally/benchmarks/distributions目录下,测试数据存放在/home/elk/.rally/benchmarks/data/geonames目录下
esrally race --distribution-version=5.4.3 --track=geonames --user-tag="version:5.4.3" --include-tasks="type:bulk"
esrally race --distribution-version=6.4.3 --track=geonames --user-tag="version:6.4.3" --include-tasks="type:bulk"
esrally race --distribution-version=7.8.1 --track=geonames --user-tag="version:7.8.1" --include-tasks="type:bulk"
压测过程可以通过/home/elk/.rally/logs.log查看
对压测结果进行比较
比较7.8.1与5.4.3的写入差异
esrally compare --baseline=27265e6e-566a-4a47-a0d9-1fd2f8830041 --contender=66086ef0-5834-4743-a870-fd9c0bb41688
可以看到5.4版本的写入性能差别还是很大的
再看一下6.4.3与7.8.1的差异,写入性能差别不是很大
esrally compare --baseline=27265e6e-566a-4a47-a0d9-1fd2f8830041 --contender=b02fb6fe-824f-48ee-beb5-ab2ca2ad4bbc
环境
阿里云2v4g 志强Platinum 8269CY CPU 2.50GHz x86_64
华为云2v4g 鲲鹏920 CPU 2.6GHz arm平台
两台云主机的ES配置均相同,2G堆内存,系统参数相同
还是要提前下好测试数据,这个测试周日跑了一天,结果截图没保存,可能配置太低了,结果差别不是很大,而且不是在同一台机器上跑的,没法使用esrally compare进行比较
esrally race --pipeline=benchmark-only --target-hosts=172.16.0.95:9200 --track=http_logs --offline
esrally race --pipeline=benchmark-only --target-hosts=172.26.214.32:9200 --track=http_logs --offline
中间遇到的问题
开始压测之后,报不能进入比赛,pid文件不可用
之后查看了相关race的日志,发现内存不够用了,因为之前已经跑了一个ES实例了,导致了我内存不足,把之前的ES实例关掉即可解决
参考链接:https://esrally.readthedocs.io/en/stable/quickstart.html
- END -
公众号:运维开发故事
github:https://github.com/orgs/sunsharing-note/dashboard
爱生活,爱运维
如果你觉得文章还不错,就请点击右上角选择发送给朋友或者转发到朋友圈。您的支持和鼓励是我最大的动力。喜欢就请关注我吧~
扫码二维码
关注我,不定期维护优质内容
温馨提示
如果你喜欢本文,请分享到朋友圈,想要获得更多信息,请关注我。
........................