【openGauss2.1.0实现X-Tuner进行参数调优】

openGauss2.1.0作为一款全新的开源关系型数据库，在性能方面具有很大的优势。通过使用X-Tuner工具，可以对openGauss2.1.0的参数进行优化，以提高其性能。

一、环境说明

1. 华为云ECS 鲲鹏架构 8vCPU 32G
2. 操作系统：openEuler20.03TLS
3. 数据库版本：openGauss2.1.0

二、参数调优步骤

以下是openGauss2.1.0利用X-Tuner进行参数优化的步骤：

1. 安装X-Tuner工具

   如果不是源码安装openGauss，而是使用编译好的openGauss安装包安装，则X-Tuner是openGauss2.1.0自带的一款工具。在安装openGauss2.1.0时，X-Tuner也会被一同安装。具体安装可以参考：【基于openEuler上安装单机openGauss2.1.0企业版】

   如果是源码安装，则需要单独安装X-Tuner，可参考：【基于openEuler 20.03 和openGauss2.1.0环境中安装X-Tuner】
   本文是基于这种方式进行测试

2. 参数调优前，使用【omm用户】先执行TPCH脚本queries.sql（该文件是在【openGauss2.1.0 TPC-C数据导入】产生的），记录下脚本执行时间，这个过程可能需要花费十多分钟的时间，请耐心等待。

   [omm@sjmopengauss01 queries]$  gsql -d tpch -p 5432 -r -f /opt/software/tpch-kit/dbgen/queries/queries.sql > /opt/software/tpch-kit/dbgen/queries/queries01.log
   

   执行结束后，查看queries01.log：

   tail -10 /opt/software/tpch-kit/dbgen/queries/queries01.log
   

   结果如下：

   ……
   total time: 1190160  ms
   

3. 设置X-Tuner工具
   切换到【root用户】，利用X-Tuner获得参数优化建议，执行如下命令，需要输入正确的omm数据库用户密码及omm操作系统用户密码（密码为关卡一中创建omm时设置的用户密码）。

   gs_xtuner recommend --db-name tpch --db-user omm --port 5432 --host 127.0.0.1 --host-user omm
   

   注意：上述命令执行时需要输入数据库密码和当前数据库所在主机的密码。
   执行结果如下：

   [root@sjmopengauss01 ~]# gs_xtuner recommend --db-name tpch --db-user omm --port 5432 --host 127.0.0.1 --host-user omm
   Please input the password of database:  #此次输入数据库密码
   Please input the password of host:  # 此处输入host-user后面的用户的登录密码
   Start to recommend knobs. Just a moment, please.
   ***************************************************************** Knob Recommendation Report *****************************************************************
   INFO:
   +---------------------------------------+--------------------------+
   |                 Metric                |          Value           |
   +---------------------------------------+--------------------------+
   |             workload_type             |            ap            |
   |         dirty_background_bytes        |            0             |
   |          current_locks_count          |           0.0            |
   |      current_prepared_xacts_count     |           0.0            |
   |         fetched_returned_ratio        |  0.0005785625290161765   |
   |         average_connection_age        |         0.015156         |
   | checkpoint_proactive_triggering_ratio |    0.0330097087378641    |
   |             cache_hit_rate            |    0.1942159297710038    |
   |        checkpoint_avg_sync_time       |    0.566990291262136     |
   |               nb_gaussdb              |            1             |
   |          current_connections          |           1.0            |
   |              min_free_mem             |          112640          |
   |      shared_buffer_heap_hit_rate      |    12.503635086409243    |
   |           all_database_size           |     1511078.8203125      |
   |             temp_file_size            |     1617.81002869898     |
   |             max_processes             |           203            |
   |          track_activity_size          |          203.0           |
   |                used_mem               |       209986048.0        |
   |            read_write_ratio           |    28.662076606117495    |
   |         rollback_commit_ratio         |  3.400469829314437e-05   |
   |            current_free_mem           |         30957760         |
   |            write_tup_speed            |     320.757764805979     |
   |              os_mem_total             |         32116096         |
   |          search_modify_ratio          |    363442.1821084965     |
   |  checkpoint_dirty_writing_time_window |          450.0           |
   |      shared_buffer_toast_hit_rate     |    57.31707317073171     |
   |      shared_buffer_tidx_hit_rate      |     79.4392523364486     |
   |              os_cpu_count             |            8             |
   |               block_size              |           8.0            |
   |                 uptime                |     8.32598471666667     |
   |                ap_index               |           8.5            |
   |             read_tup_speed            |     9194.14576082865     |
   |       shared_buffer_idx_hit_rate      |    99.66851198714228     |
   |                 is_hdd                |          False           |
   |           enable_autovacuum           |           True           |
   |                is_64bit               |           True           |
   |              load_average             | [0.02, 0.01375, 0.04375] |
   +---------------------------------------+--------------------------+
   p.s: The unit of storage is kB.
   WARN:
   [0]. The value of wal_buffers is a bit high. Generally, an excessively large value does not bring better performance. You can also set this parameter to -1. The database automatically performs adaptation.
   [1]. Detect that your appointed workload does not seem to a TP workload, hence disable enable_mergejoin is better.
   [2]. The number of CPU cores is a little small. Please do not run too high concurrency. You are recommended to set max_connections based on the number of CPU cores. If your job/workload does not consume much CPU, you can also increase it.
   [3]. Detect that your appointed workload does not seem to a TP workload, hence disable enable_nestloop is better.
   [4]. The database runs for a short period of time, and the database description may not be accumulated. The recommendation result may be inaccurate.
   [5]. The total size of all databases is less than the memory size. Therefore, it is unnecessary to set shared_buffers to a large value.
   BAD:
   [0]. We only found 1 gaussdb process(es). In this case, your 'max_process_memory' setting is heavily small.
   [1]. Most applications do not use XA prepared transactions, so should set the max_prepared_transactions to 0. If you do require prepared transactions, you should set this equal to max_connections to avoid blocking. May require increasing kernel memory parameters.
   ****************************************************************** Recommended Knob Settings ******************************************************************
   +---------------------------+-----------+----------+----------+---------+
   |            name           | recommend |   min    |   max    | restart |
   +---------------------------+-----------+----------+----------+---------+
   | default_statistics_target |    1000   |   100    |   1000   |  False  |
   |    effective_cache_size   |  21602334 |  188888  | 21602334 |  False  |
   |  effective_io_concurrency |    200    |   150    |   250    |  False  |
   |      enable_mergejoin     |    off    |    0     |    1     |  False  |
   |      enable_nestloop      |    off    |    0     |    1     |  False  |
   |      max_connections      |    370    |    50    |   741    |   True  |
   | max_prepared_transactions |    370    |    50    |   741    |   True  |
   |     max_process_memory    |  28803112 | 22402420 | 28803112 |   True  |
   |      random_page_cost     |    1.0    |   1.0    |   2.0    |  False  |
   |       shared_buffers      |   188888  |  188892  |  217224  |   True  |
   |        wal_buffers        |    5902   |   2048   |   5902   |   True  |
   +---------------------------+-----------+----------+----------+---------+
   [root@sjmopengauss01 ~]#
   

   再次切换到【omm用户】，根据以上【Recommended Knob Settings 】内容，进行参数优化：

   [omm@sjmopengauss01 ~]$ gs_guc set -D /opt/software/openGauss/data/ -c "shared_buffers = 188888" -c "max_connections = 370" -c "effective_cache_size = 21602334" -c "effective_io_concurrency = 200" -c "wal_buffers = 5902" -c "random_page_cost = 1.0" -c "default_statistics_target = 1000"
   

   执行结果如下：

   [omm@sjmopengauss01 queries]$ gs_guc set -D /opt/software/openGauss/data/ -c "shared_buffers = 188888" -c "max_connections = 370" -c "effective_cache_size = 21602334" -c "effective_io_concurrency = 200" -c "wal_buffers = 5902" -c "random_page_cost = 1.0" -c "default_statistics_target = 1000"
   The gs_guc run with the following arguments: [gs_guc -D /opt/software/openGauss/data/ -c shared_buffers = 188888 -c max_connections = 370 -c effective_cache_size = 21602334 -c effective_io_concurrency = 200 -c wal_buffers = 5902 -c random_page_cost = 1.0 -c default_statistics_target = 1000 set ].
   NOTICE: This parameter has no effect on GaussDB Kernel allocated shared memory size, it does not use the kernel disk buffer, it is only used to estimate. The values are used to calculate the disk page, each page is usually 8192 bytes. Higher than the default value may result in the use of index scans, lower values may result in the selection order of scan.
   NOTICE: Every time a transaction is committed, the contents of WAL buffers are written to disk, it is set to a large value will not bring significant performance gains. If you set it to hundreds of megabytes, you may have written to the disk to improve performance on the server a lot of real-time transaction commits. According to experience, the default value is sufficient for most situations.
   expected instance path: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: shared_buffers=188888: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: max_connections=370: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: effective_cache_size=21602334: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: effective_io_concurrency=200: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: wal_buffers=5902: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: random_page_cost=1.0: [/opt/software/openGauss/data/postgresql.conf]
   gs_guc set: default_statistics_target=1000: [/opt/software/openGauss/data/postgresql.conf]
   
   Total instances: 1. Failed instances: 0.
   Success to perform gs_guc!
   

   使用omm用户，重启数据库:

   [omm@sjmopengauss01 queries]$ gs_ctl restart -D /opt/software/openGauss/data -Z single_node -l logfile
   

   查看调优后的参数，执行如下命令：

   [omm@sjmopengauss01 ~]$ cd /opt/software/openGauss/data
   [omm@sjmopengauss01 data]$ cat postgresql.conf|grep -E 'shared_buffers|max_connections|effective_cache_size|effective_io_concurrency|wal_buffers|random_page_cost|default_statistics_target'
   

   结果如下：

   [omm@sjmopengauss01 data]$ cat postgresql.conf|grep -E 'shared_buffers|max_connections|effective_cache_size|effective_io_concurrency|wal_buffers|random_page_cost|default_statistics_target'
   max_connections = 370                   # (change requires restart)
   # Note:  Increasing max_connections costs ~400 bytes of shared memory per
   shared_buffers = 188888                 # min 128kB
   bulk_write_ring_size = 2GB              # for bulkload, max shared_buffers
   #standby_shared_buffers_fraction = 0.3 #control shared buffers use in standby, 0.1-1.0
   effective_io_concurrency = 200          # 1-1000; 0 disables prefetching
   wal_buffers = 5902                      # min 32kB
   random_page_cost = 1.0                  # same scale as above
   effective_cache_size = 21602334
   default_statistics_target = 1000        # range 1-10000
   # max_locks_per_transaction * (max_connections + max_prepared_transactions)
   [omm@sjmopengauss01 data]$
   

4. 参数调优后，再次使用【omm用户】先执行TPCH脚本queries.sql，查看最终的所消耗的时间，结果如下：

   [omm@sjmopengauss01 queries]$  gsql -d tpch -p 5432 -r -f /opt/software/tpch-kit/dbgen/queries/queries.sql > /opt/software/tpch-kit/dbgen/queries/queries02.log
   

   等待执行完毕，查看queries02.log：

   tail -10 /opt/software/tpch-kit/dbgen/queries/queries02.log
   

   结果如下：

   ……
   total time: 1153799 ms

   比未进行参数调优时耗时少了36361 ms，差不多半分钟时间。如果系统要求性能较高，在高并发和大业务量的场景下，虽缩短了半分钟时间，但从整体上看，性能还是有不少的提升。
   由此，可以看出，总时长比调优前少许多，说明本次的参数调优建议是可取的。
   总之，openGauss2.1.0利用X-Tuner进行参数优化可以有效提高其性能，使其更加适合企业级数据管理。