MySQL5.7优化测试方案
文章目录
- MySQL5.7优化测试方案
- 系统相关信息:
- 系统环境:
- 硬件信息:
- 系统参数调整
- 目前磁盘使用情况
- 系统磁盘IO测试
- 使用脚本
- 观测:
- 运行结果
- 脚本运行结果
- iostat观测结果(部分)
- iopp观测结果
- MySQL 优化
- Mysql插入数据优化
- mysql配置文件(修改后)
- 插入:每秒平均插入数据条数
- is_norm_data_his表结构
- 插入测试用sql语句
- 50并发, 5220条/s
- 100并发, 4652条/s
- 200并发, 4457条/s
- mysql查询优化
- 目前配置信息
- 目前性能
- 50并发, 9960条/s
- Mysql实际运行情况
- 100并发, 1w条/s
- 200并发, 1w条/s
- 读写性能
- 写:50并发,2488条/s; 读: 50并发,4447条/s
MySQL5.7优化测试方案
系统相关信息:
系统环境:
[root@ts ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
硬件信息:
cpu信息:4 Intel® Xeon® CPU E5-2630 0 @ 2.30GHz
内存: 8G
磁盘:500G
系统参数调整
目前磁盘使用情况
系统磁盘IO测试
参考文档
使用脚本
sync;/usr/bin/time -p bash -c "(dd if=/dev/zero of=test.dd bs=1000K
count=200000;sync)"
脚本大概会生成200G的数据,用来测试IO的性能
观测:
iostat -x 5
iopp -i -k 5 (安装可以参考:https://www.laofuxi.com/332.html)
运行结果
脚本运行结果
iostat观测结果(部分)
iopp观测结果
MySQL 优化
Mysql插入数据优化
mysql配置文件(修改后)
参考文件:
https://blog.csdn.net/NLOneDay/article/details/84637317
https://www.cnblogs.com/glon/p/6497377.html
https://www.cnblogs.com/glon/p/6484912.html
https://blog.51cto.com/xiaozhagn/2073900
插入:每秒平均插入数据条数
is_norm_data_his表结构
CREATE TABLE `is_norm_data_his` (
`norm_data_id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键id',
`dtu_id` varchar(50) NOT NULL COMMENT 'DTU id',
`sb_id` varchar(50) DEFAULT NULL COMMENT '设备id',
`manufacture_id` varchar(50) DEFAULT NULL COMMENT '厂商id',
`device_protocol_id` varchar(50) NOT NULL COMMENT '协议id',
`fault_code` varchar(255) DEFAULT NULL COMMENT '故障码',
`is_warn` tinyint(4) DEFAULT '0' COMMENT '数据警告:0:不警告;1:警告',
`on_off` decimal(12,4) DEFAULT NULL COMMENT '开关机',
`model` decimal(12,4) DEFAULT NULL COMMENT '模式',
`water_inlet_temperature` decimal(12,4) DEFAULT NULL COMMENT '进水温度',
`water_outlet_temperature` decimal(12,4) DEFAULT NULL COMMENT '出水温度',
`discharge_temperature` decimal(12,4) DEFAULT NULL COMMENT '排气温度',
`discharge_pressure` decimal(12,4) DEFAULT NULL COMMENT '排气压力',
`suction_temperature` decimal(12,4) DEFAULT NULL COMMENT '吸气温度',
`suction_pressure` decimal(12,4) DEFAULT NULL COMMENT '吸气压力',
`coil_temperature` decimal(12,4) DEFAULT NULL COMMENT '盘管温度',
`fin_tube_temperature` decimal(12,4) DEFAULT NULL COMMENT '翅片温度',
`tank_temperature` decimal(12,4) DEFAULT NULL COMMENT '水箱温度',
`evaporating_temperature` decimal(12,4) DEFAULT NULL COMMENT '蒸发温度',
`set_temperature` decimal(12,4) DEFAULT NULL COMMENT '设置温度',
`heating_set_temperature` decimal(12,4) DEFAULT NULL COMMENT '制热设定温度',
`refrigerating_set_temperature` decimal(12,4) DEFAULT NULL COMMENT '制冷设定温度',
`heating_water_set_temperature` decimal(12,4) DEFAULT NULL COMMENT '制热水设定温度',
`outdoor_temperature` decimal(12,4) DEFAULT NULL COMMENT '室外温度',
`indoor_temperature` decimal(12,4) DEFAULT NULL COMMENT '室内温度',
`condenser_temperature` decimal(12,4) DEFAULT NULL COMMENT '冷凝器温度',
`defrost_entry_temperature` decimal(12,4) DEFAULT NULL COMMENT '除霜进入温度',
`defrost_exit_temperature` decimal(12,4) DEFAULT NULL COMMENT '除霜退出温度',
`defrost_operation_time` decimal(12,4) DEFAULT NULL COMMENT '除霜运行时间',
`defrost_intervals` decimal(12,4) DEFAULT NULL COMMENT '除霜间隔',
`compressor_state` decimal(12,4) DEFAULT NULL COMMENT '压缩机状态',
`compressor_current` decimal(12,4) DEFAULT NULL COMMENT '压缩机电流',
`compressor_frequency` decimal(12,4) DEFAULT NULL COMMENT '压缩机频率',
`electronic_expansion_valve_opening` decimal(12,4) DEFAULT NULL COMMENT '电子膨胀阀开度',
`enthalpy_valve_opening` decimal(12,4) DEFAULT NULL COMMENT '増焓阀开度',
`throttle_opening` decimal(12,4) DEFAULT NULL COMMENT '节流阀开度',
`triple_valves` decimal(12,4) DEFAULT NULL COMMENT '三通阀',
`solenoid_value` decimal(12,4) DEFAULT NULL COMMENT '电磁阀',
`four_way_valve` decimal(12,4) DEFAULT NULL COMMENT '四通阀',
`electrical_heating` decimal(12,4) DEFAULT NULL COMMENT '电加热',
`temperature_control_methods` decimal(12,4) DEFAULT NULL COMMENT '温控方式',
`winterizing` decimal(12,4) DEFAULT NULL COMMENT '防冻',
`stable_run_time` varchar(50) DEFAULT NULL COMMENT '稳定运行时间',
`frequency` decimal(12,4) DEFAULT NULL COMMENT '频率',
`instantaneous_delivery` decimal(12,4) DEFAULT NULL COMMENT '瞬时流量',
`instantaneous_velocity` decimal(12,4) DEFAULT NULL COMMENT '瞬时流速',
`fluid_conductance_ratio` decimal(12,4) DEFAULT NULL COMMENT '流体电导比',
`flow_accumulation_value` decimal(12,4) DEFAULT NULL COMMENT '流量累积数值',
`instantaneous_cold_unit` decimal(12,4) DEFAULT NULL COMMENT '瞬时冷量单位',
`total_cooling_capacity_unit` decimal(12,4) DEFAULT NULL COMMENT '冷量总量单位',
`instantaneous_heat_unit` decimal(12,4) DEFAULT NULL COMMENT '瞬时热量单位',
`cumulative_total_flow_unit` decimal(12,4) DEFAULT NULL COMMENT '流量累计总量单位',
`pressure_limit` decimal(12,4) DEFAULT NULL COMMENT '压力范围',
`total_heat_unit` decimal(12,4) DEFAULT NULL COMMENT '热量总量单位',
`air_tube_alarm` decimal(12,4) DEFAULT NULL COMMENT '空管报警',
`system_alarm` decimal(12,4) DEFAULT NULL COMMENT '系统报警',
`instantaneous_heat_flow_rate` decimal(12,4) DEFAULT NULL COMMENT '瞬时热流量',
`total_cumulative_value_of_heat` decimal(12,4) DEFAULT NULL COMMENT '热量总累计值',
`total_cold_accumulation_value` decimal(12,4) DEFAULT NULL COMMENT '冷量总累积值',
`instantaneous_cooling_capacity` decimal(12,4) DEFAULT NULL COMMENT '瞬时冷量',
`current_total_electric_energy` decimal(12,4) DEFAULT NULL COMMENT '当前总电能',
`voltage_a` decimal(12,4) DEFAULT NULL COMMENT '电压A',
`voltage_b` decimal(12,4) DEFAULT NULL COMMENT '电压B',
`voltage_c` decimal(12,4) DEFAULT NULL COMMENT '电压C',
`current_a` decimal(12,4) DEFAULT NULL COMMENT '电流A',
`current_b` decimal(12,4) DEFAULT NULL COMMENT '电流B',
`current_c` decimal(12,4) DEFAULT NULL COMMENT '电流C',
`active_power_t` decimal(12,4) DEFAULT NULL COMMENT '有功功率T',
`power_factor_t` decimal(12,4) DEFAULT NULL COMMENT '功率因数T',
`recharge_valve_working_mode` decimal(12,4) DEFAULT NULL COMMENT '补水阀工作模式',
`working_state_of_water_supply_valve` decimal(12,4) DEFAULT NULL COMMENT '补水阀工作状态',
`recharge_valve_manual_switch` decimal(12,4) DEFAULT NULL COMMENT '补水阀手动开关',
`real_time_liquid_level_of_water_tank` decimal(12,4) DEFAULT NULL COMMENT '水箱实时液位',
`solenoid_valve_water_supply_level_setting` decimal(12,4) DEFAULT NULL COMMENT '电磁阀补水液位设定',
`solenoid_valve_closing_level_setting` decimal(12,4) DEFAULT NULL COMMENT '电磁阀关闭液位设定',
`high_liquid_level_protection_value` decimal(12,4) DEFAULT NULL COMMENT '电磁阀高液位保护值',
`outlet_pump_working_mode` decimal(12,4) DEFAULT NULL COMMENT '出水泵工作模式',
`working_state_of_outlet_pump` decimal(12,4) DEFAULT NULL COMMENT '出水泵工作状态',
`outlet_pump_manual_switch` decimal(12,4) DEFAULT NULL COMMENT '出水泵手动开关',
`low_water_temperature_of_water_tank` decimal(12,4) DEFAULT NULL COMMENT '水箱低位水温',
`high_water_temperature_of_water_tank` decimal(12,4) DEFAULT NULL COMMENT '水箱高位水温',
`water_temperature_of_water_tank_action` decimal(12,4) DEFAULT NULL COMMENT '水箱动作水温',
`outlet_water_temperature_setting_outlet_pump` decimal(12,4) DEFAULT NULL COMMENT '出水泵出水温度设定',
`outlet_pump_stop_temperature_setting` decimal(12,4) DEFAULT NULL COMMENT '出水泵停止温度设定',
`low_liquid_level_protection_value_outlet_pump` decimal(12,4) DEFAULT NULL COMMENT '出水泵低液位保护值',
`other_data` text COMMENT '每个协议的所有数据项字段,数据格式json',
`is_deleted` tinyint(4) NOT NULL DEFAULT '0' COMMENT '逻辑删除 1 表示删除，0 表示未删除',
`gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`version` bigint(20) NOT NULL DEFAULT '0' COMMENT '数据版本号 当前时间戳',
`sjly` varchar(255) NOT NULL DEFAULT 'ope' COMMENT '数据来源',
`fault_codes` varchar(255) DEFAULT NULL COMMENT '所有故障码',
`warn_data_items` varchar(255) DEFAULT NULL COMMENT '数据跳变的数据项id,值(json格式)',
PRIMARY KEY (`norm_data_id`) USING BTREE,
KEY `index_sbid` (`sb_id`) USING BTREE COMMENT '设备id索引',
KEY `index_manufactureid_gmtcreate_sjly` (`manufacture_id`,`gmt_create`,`sjly`) USING BTREE COMMENT '厂商id,创建时间,数据来源联合索引',
KEY `index_deviceprotocolid` (`device_protocol_id`) USING BTREE COMMENT '协议id索引',
KEY `index_dtuid` (`dtu_id`) USING BTREE COMMENT 'dtu id索引'
) ENGINE=InnoDB AUTO_INCREMENT=226080 DEFAULT CHARSET=utf8mb4 COMMENT='标准数据历史表';
插入测试用sql语句
INSERT INTO is_norm_data_his(gmt_create,current_total_electric_energy,fault_codes,total_cumulative_value_of_heat,voltage_a,compressor_current,dtu_id,voltage_b,sb_id,voltage_c,heating_set_temperature,is_deleted,model,device_protocol_id,other_data,current_a,water_inlet_temperature,on_off,water_outlet_temperature,version,current_b,current_c,stable_run_time,total_heat_unit,manufacture_id,fault_code)VALUES(now(),"0.0","71,72","1756392","0.0","0.0","861390030217502","0.0","sb_2007259560374156150523901440000000","0.0","30.0",0,"1.0","dp_1907686864628569150523909027841592","[{\"dataItemId\":\"dpdi_2007258026928453150523901443287436\",\"fieldName\":\"total_heat_unit\",\"value\":\"KWh\"},{\"dataItemId\":\"dpdi_1923833608357059150523901446177317\",\"fieldName\":\"f274\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2010581506333511150523901525212414\",\"fieldName\":\"voltage_c\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923833478952310150523901444962968\",\"fieldName\":\"f273\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923833772059726150523901446514053\",\"fieldName\":\"f276\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2010581994799914150523901521479136\",\"fieldName\":\"stable_run_time\",\"value\":\"3.495821312E9\"},{\"dataItemId\":\"dpdi_2010581340549546150523901521038841\",\"fieldName\":\"voltage_a\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923833585275156150523901447360753\",\"fieldName\":\"compressor_current\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923833700257792150523901448919163\",\"fieldName\":\"f275\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011939242335304150523901442846792\",\"fieldName\":\"f417\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923834414298522150523901442673528\",\"fieldName\":\"water_inlet_temperature\",\"value\":\"24.8\"},{\"dataItemId\":\"dpdi_2011939064992150150523901442692142\",\"fieldName\":\"f415\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011939423959619150523901448581795\",\"fieldName\":\"f419\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011939153711016150523901443712293\",\"fieldName\":\"f416\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011939604454318150523901445445782\",\"fieldName\":\"f421\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2010581669818758150523901522023650\",\"fieldName\":\"current_b\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1907688892346094150523909024183571\",\"fieldName\":\"heating_set_temperature\",\"value\":\"30.0\"},{\"dataItemId\":\"dpdi_2011938451391341150523901441923474\",\"fieldName\":\"f412\",\"value\":\"1.0\"},{\"dataItemId\":\"dpdi_2010581750366081150523901524665708\",\"fieldName\":\"current_c\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2010581588191253150523901525224206\",\"fieldName\":\"current_a\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923834399151389150523901447482614\",\"fieldName\":\"on_off\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2010581173235127150523901522630696\",\"fieldName\":\"current_total_electric_energy\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011938975494000150523901449072221\",\"fieldName\":\"f414\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011938884732353150523901446452632\",\"fieldName\":\"f413\",\"value\":\"1.0\"},{\"dataItemId\":\"dpdi_1907689058114090150523909029134119\",\"fieldName\":\"model\",\"value\":\"1.0\"},{\"dataItemId\":\"dpdi_2007258358551716150523901449111378\",\"fieldName\":\"total_cumulative_value_of_heat\",\"value\":\"1.756392E7\"},{\"dataItemId\":\"dpdi_2011939333703087150523901446122310\",\"fieldName\":\"f418\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2010581423702302150523901523257742\",\"fieldName\":\"voltage_b\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_2011939513281455150523901444573319\",\"fieldName\":\"f420\",\"value\":\"0.0\"},{\"dataItemId\":\"dpdi_1923834806992586150523901441758504\",\"fieldName\":\"water_outlet_temperature\",\"value\":\"24.9\"}]","0.0","24.8","0.0","24.9",1561619292404,"0.0","0.0","3495821312","2.0","565229777","72");
50并发, 5220条/s
Mysql服务器 iostat监测
Mysql服务器 free -m 监测
Mysql服务器top 监测
Mysql测试服务器, Iftop监测
100并发, 4652条/s
Mysql服务器 iostat监测
Mysql服务器 free -m 监测
Mysql服务器top 监测
Mysql测试服务器, Iftop监测
200并发, 4457条/s
Mysql服务器 iostat监测
Mysql服务器 free -m 监测
Mysql服务器top 监测
Mysql测试服务器, Iftop监测
mysql查询优化
目前配置信息
[root@ts ~]# cat /etc/my.cnf
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# open_files_limit = 1048576
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction_isolation=READ-COMMITTED
innodb_buffer_pool_size=5G
innodb_buffer_pool_instances=5
innodb_log_file_size=3G
innodb_thread_concurrency=16
innodb_flush_log_at_trx_commit=0
innodb_file_per_table=ON
innodb_autoextend_increment=256M
innodb_log_buffer_size=256M
innodb_flush_method = O_DIRECT
query_cache_size=0
query_cache_type = 0
max_connections=2000
thread_cache_size = 256
sync_binlog=0
# Disabling symbolic-links is recommended to prevent assorted security risks
# skip-grant-tables
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
secure_file_priv="/"
default-storage-engine=INNODB
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
max_allowed_packet=50M
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
#log_bin=OFF
#sql_log_bin=OFF
#innodb_doublewrite=OFF
[mysql]
default-character-set=utf8mb4
[client]
default-character-set=utf8mb4
目前性能
50并发, 9960条/s
Mysql服务器iostat监测
Mysql服务器free监测
Mysql服务器 top监测
压测服务器 iftop监测
压测服务器 top监测
压测服务器 free -m监测
Mysql实际运行情况
查看总共分配了多少内存
mysql> select * from memory_global_total;
+-----------------+
| total_allocated |
+-----------------+
| 144.24 MiB |
+-----------------+
1 row in set (0.00 sec)
哪个文件产生了最多的IO，读多，还是写的多？
哪个表占用了最多的buffer pool？
每个库（database）占用多少buffer pool？
MySQL索引使用情况统计
100并发, 1w条/s
200并发, 1w条/s
读写性能
写:50并发,2488条/s; 读: 50并发,4447条/s
Mysql服务器iostat监测
Mysql服务器free监测
Mysql服务器 top监测
压测服务器 iftop监测
压测服务器 top监测
压测服务器 free -m监测
