esRally国内安装使用避坑指南-全网最新

  按照官网的步骤来,一步一个坑,浪费了我一天的时间!多数是因为国内的网络,下载和拉取速度太慢,导致安装失败。所以写了这个文章,来记录我成功安装的案例,希望可以帮到想用esrally做基准测试的人!

  网上也有文章不过都过时了!官网呢,又不适合中国网络情况的使用。我甚至严重怀疑,官网的步骤,他们有没有执行一遍试试。因为用哪些命令肯定会遇到问题。比如:

sudo yum install git

  我不相信这条命令,能帮我们下载新版本的git!

  注意目前esrally最新版本是 2.5.0!

  官网地址(英文的最新版):Quickstart — Rally 2.5.0 documentation

  中文文档(0.9版本的):安装 — Rally 0.9.0 文档

准备环境-需要的环境介绍

  我在看文档的时候,其实蛮想知道我为什么要安装这些东西,以及不安装行不行,我需要安装什么版本的。我在看着官网折腾这个的时候浪费了挺多时间,所以特地写一篇博客来带大家避坑!

  操作系统 centos7,我是在centos7下安装的,其它版本,操作系统不适用,其它版本我没有验证能否成功。esrally只能运行在linux下!

  安装过程需要网络环境。即使有网络,在装一些东西的时候依旧有很多问题,所以我在安装一些东西的时候,也会用安装包安装,来减少问题的出现。

  另外说一句,esrally文档是写给外国人的,因为一些网络环境我们不具备,所以根据文档操作,一大堆问题。

想要运行esrally必备的工具

  1. python3.8以上版本,我看官网的例子是用python3.8.10的版本。为了减少问题,我也用python3.8.10。说一下为什么要用它,因为esrally是用python写的,安装也用到了pip3。这些都属于必备的!
  2. git,版本1.9及以上版本。用git是因为es用到的数据都放在了网上,用到的测试用例也放在git上,所以需要我们在用的时候拉取。这个也是必须的,没有不行!
  3. JDK。用这个是因为,esrally支持自动构建es集群,就是帮你搭建集群。es低版本(7版本之前)并没有内嵌JDK,所以这些版本需要JDK,并配置好环境。如果你是自建集群,对已有的集群进行测试,那就不需要JDK了。如果需要它帮你搭建集群,你就需要将JDK环境配置好。
  4. esrally,es基准测试的主角。

如何在linux centos下安装python环境

  这篇文章是我在用esrally的时候写的,我跟着官网做的时候遇到很多坑,主要是浪费了时间,实在没必要。我的这个方案更简单!请看我这篇文章:

centos下编译安装各个版本的python_水的精神的博客-CSDN博客

如何升级git,或者说安装git

  centos7默认安装了git,遗憾的是它的版本是1.8的。也就是低于官网要求的1.9版本。所以需要升级。在网上找文档折腾的半天,都不靠谱。所以建议看我这篇文章(特简单,三条命令):

Linux centos下三条命令安装升级git_水的精神的博客-CSDN博客

如何安装JDK

下载安装jdk13

https://www.oracle.com/java/technologies/javase-jdk13-downloads.html

复制

esRally国内安装使用避坑指南-全网最新_第1张图片

下载,上传到服务器上。

然后rpm安装

rpm -ivh jdk-13.0.2_linux-x64_bin.rpm

配置环境变量,在 /etc/profile文件的末尾加入以下内容

export JAVA_HOME=/usr/java/jdk-13.0.2
export PATH=$JAVA_HOME/bin:$PATH 
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

并执行下列命令使环境变量生效

source /etc/profile

安装esrally

  官网给的操作是使用一条命令安装,开起来很简单:

pip3 install esrally

  但是我执行不成功。可能我对python不是很熟悉的原因把。我用的其它方式来安装的(离线安装 esrally 是最靠谱的,解决拉取问题!我在尝试上边的命令的时候,遇到了一堆问题。不想解决了!所以用离线的方式来安装它):

  1. 安装上面文档提到的所有依赖包
  2. 下载 最新发布的离线包 ,并将它拷贝到目标机器。
  3. 解压安装包 tar -xzf esrally-dist-*.tar.gz 。
  4. 执行安装脚本 sudo ./esrally-dist-*/install.sh 。

  在执行第四条的时候,还是遇到了问题(我把遇到几个问题贴出来,希望可以帮到你们):

执行 ./install.sh 第一个问题:

ERROR: Could not find a version that satisfies the requirement py-cpuinfo==7.0.0 (from esrally) (from versions: 7.0.0)
ERROR: No matching distribution found for py-cpuinfo==7.0.0

esRally国内安装使用避坑指南-全网最新_第2张图片

解决命令:

pip3 install py-cpuinfo -i https://pypi.tuna.tsinghua.edu.cn/simple/ --trusted-host pypi.tuna.tsinghua.edu.cn 

执行 ./install.sh 第二个问题(python pip相关的问题):

问题

WARNING: You are using pip version 21.1.1; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/local/python38/bin/python3.8 -m pip install --upgrade pip' command.

解决命令:

python3 -m pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple/ --trusted-host pypi.tuna.tsinghua.edu.cn

第三个问题:

esRally国内安装使用避坑指南-全网最新_第3张图片

解决问题:

pip3 install --upgrade setuptools  -i https://pypi.tuna.tsinghua.edu.cn/simple/ --trusted-host pypi.tuna.tsinghua.edu.cn

以上三个问题解决完以后,esrally已经可以成功安装了!

尝试运行一下,因为我已经安装过了es集群。所以已经可以使用了!

命令如下:

esrally race --track=percolator --target-hosts=10.10.10.10:9200,10.10.10.10:9201,10.10.10.10:9202 --pipeline=benchmark-only         

如果你没有建好的集群,尝试使用这个命令来运行(这个会帮你创建一个es集群,耐心等待,时间比较长):

esrally --distribution-version=6.0.0

我的案例运行结果如下 :

[root@estest esrally-track]# esrally race --track=percolator --target-hosts=10.10.10.10:9200,10.10.10.10:9201,10.10.10.10:9202 --pipeline=benchmark-only

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [7f280f60-6b48-41db-9ab3-f6977170c417]
[WARNING] Could not update tracks. Continuing with your locally available state.
[WARNING] Could not update tracks. Continuing with your locally available state.
[INFO] Downloading track data (121.1 kB total size)                               [100.0%]
[INFO] Decompressing track data from [/root/.rally/benchmarks/data/percolator/queries-2.json.bz2] to [/root/.rally/benchmarks/data/percolator/queries-2.json] (resulting size: [0.1... [OK]
[INFO] Preparing file offset table for [/root/.rally/benchmarks/data/percolator/queries-2.json] ... [OK]
[INFO] Racing on track [percolator], challenge [append-no-conflicts] and car ['external'] with version [6.8.0].

[WARNING] merges_total_time is 8107257 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 1274256 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 79966971 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 3806308 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 522750 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running percolator_with_content_president_bush                                 [100% done]
Running percolator_with_content_saddam_hussein                                 [100% done]
Running percolator_with_content_hurricane_katrina                              [100% done]
Running percolator_with_content_google                                         [100% done]
Running percolator_no_score_with_content_google                                [100% done]
Running percolator_with_highlighting                                           [100% done]
Running percolator_with_content_ignore_me                                      [100% done]
Running percolator_no_score_with_content_ignore_me                             [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                         Metric |                                       Task |         Value |   Unit |
|---------------------------------------------------------------:|-------------------------------------------:|--------------:|-------:|
|                     Cumulative indexing time of primary shards |                                            |  1334.87      |    min |
|             Min cumulative indexing time across primary shards |                                            |     0.39835   |    min |
|          Median cumulative indexing time across primary shards |                                            |    15.4366    |    min |
|             Max cumulative indexing time across primary shards |                                            |    26.0545    |    min |
|            Cumulative indexing throttle time of primary shards |                                            |     0         |    min |
|    Min cumulative indexing throttle time across primary shards |                                            |     0         |    min |
| Median cumulative indexing throttle time across primary shards |                                            |     0         |    min |
|    Max cumulative indexing throttle time across primary shards |                                            |     0         |    min |
|                        Cumulative merge time of primary shards |                                            |   135.302     |    min |
|                       Cumulative merge count of primary shards |                                            |   805         |        |
|                Min cumulative merge time across primary shards |                                            |     0.03545   |    min |
|             Median cumulative merge time across primary shards |                                            |     1.25905   |    min |
|                Max cumulative merge time across primary shards |                                            |     8.2117    |    min |
|               Cumulative merge throttle time of primary shards |                                            |    21.2376    |    min |
|       Min cumulative merge throttle time across primary shards |                                            |     0         |    min |
|    Median cumulative merge throttle time across primary shards |                                            |     0         |    min |
|       Max cumulative merge throttle time across primary shards |                                            |     3.46082   |    min |
|                      Cumulative refresh time of primary shards |                                            |    63.4783    |    min |
|                     Cumulative refresh count of primary shards |                                            |  4067         |        |
|              Min cumulative refresh time across primary shards |                                            |     0.00725   |    min |
|           Median cumulative refresh time across primary shards |                                            |     0.758333  |    min |
|              Max cumulative refresh time across primary shards |                                            |     1.46298   |    min |
|                        Cumulative flush time of primary shards |                                            |     8.7125    |    min |
|                       Cumulative flush count of primary shards |                                            |   233         |        |
|                Min cumulative flush time across primary shards |                                            |     0         |    min |
|             Median cumulative flush time across primary shards |                                            |     0.0939167 |    min |
|                Max cumulative flush time across primary shards |                                            |     0.46015   |    min |
|                                        Total Young Gen GC time |                                            |    11.337     |      s |
|                                       Total Young Gen GC count |                                            |   777         |        |
|                                          Total Old Gen GC time |                                            |     0         |      s |
|                                         Total Old Gen GC count |                                            |     0         |        |
|                                                     Store size |                                            |    84.4625    |     GB |
|                                                  Translog size |                                            |     2.95658   |     GB |
|                                         Heap used for segments |                                            |   172.674     |     MB |
|                                       Heap used for doc values |                                            |     0.014431  |     MB |
|                                            Heap used for terms |                                            |   141.275     |     MB |
|                                            Heap used for norms |                                            |     0.038147  |     MB |
|                                           Heap used for points |                                            |     7.70514   |     MB |
|                                    Heap used for stored fields |                                            |    23.6414    |     MB |
|                                                  Segment count |                                            |    85         |        |
|                                    Total Ingest Pipeline count |                                            |     0         |        |
|                                     Total Ingest Pipeline time |                                            |     0         |      s |
|                                   Total Ingest Pipeline failed |                                            |     0         |        |
|                                                     error rate |                                      index |     0         |      % |
|                                                 Min Throughput |     percolator_with_content_president_bush |    47.9       |  ops/s |
|                                                Mean Throughput |     percolator_with_content_president_bush |    48.16      |  ops/s |
|                                              Median Throughput |     percolator_with_content_president_bush |    48.16      |  ops/s |
|                                                 Max Throughput |     percolator_with_content_president_bush |    48.43      |  ops/s |
|                                        50th percentile latency |     percolator_with_content_president_bush |    10.2912    |     ms |
|                                        90th percentile latency |     percolator_with_content_president_bush |    13.3935    |     ms |
|                                        99th percentile latency |     percolator_with_content_president_bush |    16.1495    |     ms |
|                                       100th percentile latency |     percolator_with_content_president_bush |    16.4619    |     ms |
|                                   50th percentile service time |     percolator_with_content_president_bush |     9.44676   |     ms |
|                                   90th percentile service time |     percolator_with_content_president_bush |    12.2333    |     ms |
|                                   99th percentile service time |     percolator_with_content_president_bush |    15.5729    |     ms |
|                                  100th percentile service time |     percolator_with_content_president_bush |    15.6444    |     ms |
|                                                     error rate |     percolator_with_content_president_bush |     0         |      % |
|                                                 Min Throughput |     percolator_with_content_saddam_hussein |    50.07      |  ops/s |
|                                                Mean Throughput |     percolator_with_content_saddam_hussein |    50.07      |  ops/s |
|                                              Median Throughput |     percolator_with_content_saddam_hussein |    50.07      |  ops/s |
|                                                 Max Throughput |     percolator_with_content_saddam_hussein |    50.08      |  ops/s |
|                                        50th percentile latency |     percolator_with_content_saddam_hussein |     3.68269   |     ms |
|                                        90th percentile latency |     percolator_with_content_saddam_hussein |     4.41383   |     ms |
|                                        99th percentile latency |     percolator_with_content_saddam_hussein |     8.45466   |     ms |
|                                       100th percentile latency |     percolator_with_content_saddam_hussein |     9.41373   |     ms |
|                                   50th percentile service time |     percolator_with_content_saddam_hussein |     2.81123   |     ms |
|                                   90th percentile service time |     percolator_with_content_saddam_hussein |     3.50136   |     ms |
|                                   99th percentile service time |     percolator_with_content_saddam_hussein |     7.73842   |     ms |
|                                  100th percentile service time |     percolator_with_content_saddam_hussein |     8.28083   |     ms |
|                                                     error rate |     percolator_with_content_saddam_hussein |     0         |      % |
|                                                 Min Throughput |  percolator_with_content_hurricane_katrina |    50.13      |  ops/s |
|                                                Mean Throughput |  percolator_with_content_hurricane_katrina |    50.15      |  ops/s |
|                                              Median Throughput |  percolator_with_content_hurricane_katrina |    50.15      |  ops/s |
|                                                 Max Throughput |  percolator_with_content_hurricane_katrina |    50.17      |  ops/s |
|                                        50th percentile latency |  percolator_with_content_hurricane_katrina |     8.31714   |     ms |
|                                        90th percentile latency |  percolator_with_content_hurricane_katrina |     9.77985   |     ms |
|                                        99th percentile latency |  percolator_with_content_hurricane_katrina |    11.9031    |     ms |
|                                       100th percentile latency |  percolator_with_content_hurricane_katrina |    16.0777    |     ms |
|                                   50th percentile service time |  percolator_with_content_hurricane_katrina |     6.9336    |     ms |
|                                   90th percentile service time |  percolator_with_content_hurricane_katrina |     8.12209   |     ms |
|                                   99th percentile service time |  percolator_with_content_hurricane_katrina |    11.4533    |     ms |
|                                  100th percentile service time |  percolator_with_content_hurricane_katrina |    14.9582    |     ms |
|                                                     error rate |  percolator_with_content_hurricane_katrina |     0         |      % |
|                                                 Min Throughput |             percolator_with_content_google |    26.57      |  ops/s |
|                                                Mean Throughput |             percolator_with_content_google |    26.77      |  ops/s |
|                                              Median Throughput |             percolator_with_content_google |    26.76      |  ops/s |
|                                                 Max Throughput |             percolator_with_content_google |    26.97      |  ops/s |
|                                        50th percentile latency |             percolator_with_content_google |    83.5241    |     ms |
|                                        90th percentile latency |             percolator_with_content_google |   104.656     |     ms |
|                                        99th percentile latency |             percolator_with_content_google |   115.239     |     ms |
|                                       100th percentile latency |             percolator_with_content_google |   119.399     |     ms |
|                                   50th percentile service time |             percolator_with_content_google |    34.601     |     ms |
|                                   90th percentile service time |             percolator_with_content_google |    44.8482    |     ms |
|                                   99th percentile service time |             percolator_with_content_google |    52.8977    |     ms |
|                                  100th percentile service time |             percolator_with_content_google |    53.925     |     ms |
|                                                     error rate |             percolator_with_content_google |     0         |      % |
|                                                 Min Throughput |    percolator_no_score_with_content_google |    99.79      |  ops/s |
|                                                Mean Throughput |    percolator_no_score_with_content_google |    99.79      |  ops/s |
|                                              Median Throughput |    percolator_no_score_with_content_google |    99.79      |  ops/s |
|                                                 Max Throughput |    percolator_no_score_with_content_google |    99.79      |  ops/s |
|                                        50th percentile latency |    percolator_no_score_with_content_google |     3.85806   |     ms |
|                                        90th percentile latency |    percolator_no_score_with_content_google |     4.32838   |     ms |
|                                        99th percentile latency |    percolator_no_score_with_content_google |     4.84981   |     ms |
|                                       100th percentile latency |    percolator_no_score_with_content_google |     5.66612   |     ms |
|                                   50th percentile service time |    percolator_no_score_with_content_google |     2.98046   |     ms |
|                                   90th percentile service time |    percolator_no_score_with_content_google |     3.37479   |     ms |
|                                   99th percentile service time |    percolator_no_score_with_content_google |     4.04749   |     ms |
|                                  100th percentile service time |    percolator_no_score_with_content_google |     5.13068   |     ms |
|                                                     error rate |    percolator_no_score_with_content_google |     0         |      % |
|                                                 Min Throughput |               percolator_with_highlighting |    48.79      |  ops/s |
|                                                Mean Throughput |               percolator_with_highlighting |    48.94      |  ops/s |
|                                              Median Throughput |               percolator_with_highlighting |    48.94      |  ops/s |
|                                                 Max Throughput |               percolator_with_highlighting |    49.09      |  ops/s |
|                                        50th percentile latency |               percolator_with_highlighting |     5.55708   |     ms |
|                                        90th percentile latency |               percolator_with_highlighting |     6.13534   |     ms |
|                                        99th percentile latency |               percolator_with_highlighting |    10.0739    |     ms |
|                                       100th percentile latency |               percolator_with_highlighting |    21.7479    |     ms |
|                                   50th percentile service time |               percolator_with_highlighting |     4.72898   |     ms |
|                                   90th percentile service time |               percolator_with_highlighting |     5.13776   |     ms |
|                                   99th percentile service time |               percolator_with_highlighting |     9.4044    |     ms |
|                                  100th percentile service time |               percolator_with_highlighting |    21.1273    |     ms |
|                                                     error rate |               percolator_with_highlighting |     0         |      % |
|                                                 Min Throughput |          percolator_with_content_ignore_me |     0.08      |  ops/s |
|                                                Mean Throughput |          percolator_with_content_ignore_me |     0.08      |  ops/s |
|                                              Median Throughput |          percolator_with_content_ignore_me |     0.08      |  ops/s |
|                                                 Max Throughput |          percolator_with_content_ignore_me |     0.08      |  ops/s |
|                                        50th percentile latency |          percolator_with_content_ignore_me | 10721.8       |     ms |
|                                        90th percentile latency |          percolator_with_content_ignore_me | 11339.3       |     ms |
|                                        99th percentile latency |          percolator_with_content_ignore_me | 11920.1       |     ms |
|                                       100th percentile latency |          percolator_with_content_ignore_me | 12114.7       |     ms |
|                                   50th percentile service time |          percolator_with_content_ignore_me | 10719.2       |     ms |
|                                   90th percentile service time |          percolator_with_content_ignore_me | 11337.3       |     ms |
|                                   99th percentile service time |          percolator_with_content_ignore_me | 11916.3       |     ms |
|                                  100th percentile service time |          percolator_with_content_ignore_me | 12112         |     ms |
|                                                     error rate |          percolator_with_content_ignore_me |     0         |      % |
|                                                 Min Throughput | percolator_no_score_with_content_ignore_me |    15.03      |  ops/s |
|                                                Mean Throughput | percolator_no_score_with_content_ignore_me |    15.05      |  ops/s |
|                                              Median Throughput | percolator_no_score_with_content_ignore_me |    15.05      |  ops/s |
|                                                 Max Throughput | percolator_no_score_with_content_ignore_me |    15.06      |  ops/s |
|                                        50th percentile latency | percolator_no_score_with_content_ignore_me |    31.3156    |     ms |
|                                        90th percentile latency | percolator_no_score_with_content_ignore_me |    50.9371    |     ms |
|                                        99th percentile latency | percolator_no_score_with_content_ignore_me |    58.0967    |     ms |
|                                       100th percentile latency | percolator_no_score_with_content_ignore_me |    58.1975    |     ms |
|                                   50th percentile service time | percolator_no_score_with_content_ignore_me |    30.048     |     ms |
|                                   90th percentile service time | percolator_no_score_with_content_ignore_me |    49.9852    |     ms |
|                                   99th percentile service time | percolator_no_score_with_content_ignore_me |    57.0299    |     ms |
|                                  100th percentile service time | percolator_no_score_with_content_ignore_me |    57.3463    |     ms |
|                                                     error rate | percolator_no_score_with_content_ignore_me |     0         |      % |

[WARNING] No throughput metrics available for [index]. Likely cause: The benchmark ended already during warmup.

----------------------------------
[INFO] SUCCESS (took 1691 seconds)
----------------------------------

你可能感兴趣的:(Elasticsearch,大数据)