测试机
Master
Slave
数据库
本说明手册是在RHEL 6.3 (64 Bit),jdk-7u79环境下编译过的,如果直接使用该文件进行测试,请直接跳到“benchmarksql5.0测试步骤”。因为该文件中已增加对mysql的支持与优化。
注: 针对mysql5.5的优化没有反应到其中。如果测试mysql5.5,请参考后面的章节,加入mysql5.5的优化。
下载地址:https://sourceforge.net/projects/benchmarksql/
安装:直接解压,假设解压路径 /data/benchmarksql-5.0
yum install jdk-7u79-linux-x64.rpm
用于编译benchmarkSQL, 5.0版本是需要编译的。
下载地址:https://www.baidu.com/#ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&tn=baidu&wd=txkbps&oq=rxpktsps&rsv_pq=fda53d7e00006c93&rsv_t=95daqeDHwjUCPg%2F53yzOfJJcalfg0MQo0jTBW3kJTzUaRK2hjG0RqTOhbpI&rqlang=cn&rsv_enter=0&inputT=1380&rsv_n=2&rsv_sug3=12&rsv_sug4=1380&rsv_sug=2
安装:直接解压apache-ant-1.9.1-bin.tar.gz包即可。编译benchmarksql时在环境变量PATH中添加上解压包的bin路径即可。
注:编译ant的jdk版本不能高于上面安装的JDK版本(本次为jdk7),ant.jar包中的MANIFEST.MF文件中记载了jdk版本。如“Created-By: 1.8.0_121-b13 (Oracle Corporation)”表示jdk8。
下载地址:https://jdbc.postgresql.org/download.html
本次下载postgresql-42.0.0.jre7.jar
将jar包放入benchmarksql的lib目录下的postgres目录下:
#cp postgresql-42.0.0.jre7.jar /data/benchmarksql-5.0/lib/postgres/
下载地址:https://dev.mysql.com/downloads/connector/j/
本次下载的是mysql-connector-java-5.1.42.tar.gz
解压后在解压目录会有mysql-connector-java-5.1.42-bin.jar包。
由于benchmarksql5.0不支持mysql,所以需要在benchmarksql的lib目录下创建mysql目录,并将jar包放入该目录:
# mkdir -p /data/benchmarksql-5.0/lib/mysql
# cp mysql-connector-java-5.1.42-bin.jar /data/benchmarksql-5.0/lib/mysql/
跑测试脚本时,由于benchmarksql与OS不兼容,导致如下错误:
Traceback (most recent call last):
File "", line 299, in
File "", line 62, in main
File "", line 166, in initSystemUsage
Exception: cpu line in /proc/stat too short
10:45:36,966 [Thread-0] ERROR OSCollector$CollectData : OSCollector, unexpected EOF while reading from external helper process
解决方法:
根据测试机器中 /proc/stat文件第一行的列数(假设为N),修改benchmarksql-5.0/run/misc/os_collector_linux.py脚本。
# vi /proc/stat
cpu 2924404 2878 881591 4758076392 305979 20 28284 74916 0
......
// 共10列,所以N为10
# vi run/misc/os_collector_linux.py
......
if len(lastStatData) != 10: //此处的10修改为N-1的值,即9
raise Exception("cpu line in /proc/stat too short");
......
return [
'cpu_user', 'cpu_nice', 'cpu_system',
'cpu_idle', 'cpu_iowait', 'cpu_irq',
'cpu_softirq', 'cpu_steal',
'cpu_guest', 'cpu_guest_nice',
'vm_nr_dirty',
]
//这里多返回1列,根据/proc/stat第一行各列的含义,此处多了'cpu_guest_nice',删除'cpu_guest_nice'即可。
# vim run/funcs.sh
...
function setCP()
{
...
postgres)
cp="../lib/postgres/*:../lib/*"
;;
mysql)
cp="../lib/mysql/*:../lib/*"
;;
...
}
...
case "$(getProp db)" in
firebird|oracle|postgres|mysql)
...
# vim src/client/jTPCC.java
...
if (iDB.equals("firebird"))
dbType = DB_FIREBIRD;
else if (iDB.equals("oracle"))
dbType = DB_ORACLE;
else if (iDB.equals("postgres"))
dbType = DB_POSTGRES;
else if (iDB.equals("mysql"))
dbType = DB_UNKNOWN;
else
{
log.error("unknown database type '" + iDB + "'");
return;
}
# vim src/client/jTPCCConnection.java
...
// PreparedStatements for STOCK_LEVEL
switch (dbType)
{
case jTPCCConfig.DB_POSTGRES:
stmtStockLevelSelectLow = dbConn.prepareStatement(
"SELECT count(*) AS low_stock FROM (" +
" SELECT s_w_id, s_i_id, s_quantity " +
" FROM bmsql_stock " +
" WHERE s_w_id = ? AND s_quantity < ? AND s_i_id IN (" +
" SELECT ol_i_id " +
" FROM bmsql_district " +
" JOIN bmsql_order_line ON ol_w_id = d_w_id " +
" AND ol_d_id = d_id " +
" AND ol_o_id >= d_next_o_id - 20 " +
" AND ol_o_id < d_next_o_id " +
" WHERE d_w_id = ? AND d_id = ? " +
" ) " +
" ) AS L");
break;
default:
stmtStockLevelSelectLow = dbConn.prepareStatement(
"SELECT count(*) AS low_stock FROM (" +
" SELECT s_w_id, s_i_id, s_quantity " +
" FROM bmsql_stock " +
" WHERE s_w_id = ? AND s_quantity < ? AND s_i_id IN (" +
" SELECT ol_i_id " +
" FROM bmsql_district " +
" JOIN bmsql_order_line ON ol_w_id = d_w_id " +
" AND ol_d_id = d_id " +
" AND ol_o_id >= d_next_o_id - 20 " +
" AND ol_o_id < d_next_o_id " +
" WHERE d_w_id = ? AND d_id = ? " +
" ) " +
" ) AS aliasA"); /* mysql的子查询必须添加一个别名 */
break;
}
...
为mysql创建sql的目录:
# mkdir -p benchmarksql-5.0\run\sql.mysql
将要修改的.sql文件先拷贝到sql.mysql目录,然后再进行修改。
# cp run\sql.common\buildFinish.sql run\sql.mysql\buildFinish.sql
# cp run\sql.common\tableCreates.sql run\sql.mysql\tableCreates.sql
# cp run\sql.common\tableDrops.sql run\sql.mysql\tableDrops.sql
修改上面的3个.sql文件:
1) 修改run\sql.mysql\buildFinish.sql,在文件末尾追加以下内容:
ANALYZE TABLE bmsql_config;
ANALYZE TABLE bmsql_warehouse;
ANALYZE TABLE bmsql_district;
ANALYZE TABLE bmsql_customer;
ANALYZE TABLE bmsql_history;
ANALYZE TABLE bmsql_oorder;
ANALYZE TABLE bmsql_new_order;
ANALYZE TABLE bmsql_order_line;
ANALYZE TABLE bmsql_stock;
ANALYZE TABLE bmsql_item;
2) 修改run\sql.mysql\tableCreates.sql,删除以下内容:
create sequence bmsql_hist_id_seq;
3) 修改run\sql.mysql\tableDrops.sql,删除以下内容:
drop sequence bmsql_hist_id_seq;
为mysql复制建库脚本
cp benchmarksql-5.0/run/runDatabaseBuild.sh benchmarksql-5.0/run/runDatabaseBuild_mysql.sh
优化mysql建库脚本步骤(先建索引再插数据),修改内容如下:
# vi benchmarksql-5.0/run/runDatabaseBuild_mysql.sh
...
BEFORE_LOAD="tableCreates indexCreates foreignKeys"
AFTER_LOAD="buildFinish"
...
测试mysql5.5时,总会遇到死锁或者锁超时的错误,而且测试的性能(tpm)很低。通过调查发现,有一条测试SQL语句执行时间太长,导致了死锁或锁超时的发生。而在mysql5.6中,该语句的执行时间很短(mysql5.6对查询有性能优化)。因此,为了解决该问题,在测试mysql5.5时,将对该查询语句进行优化修改,使测试正常进行。修改方法如下:
修改benchmarksql-5.0/src/client/jTPCCConnection.java
修改前:
...
default:
stmtStockLevelSelectLow = dbConn.prepareStatement(
"SELECT count(*) AS low_stock FROM (" +
" SELECT s_w_id, s_i_id, s_quantity " +
" FROM bmsql_stock " +
" WHERE s_w_id = ? AND s_quantity < ? AND s_i_id IN (" +
" SELECT ol_i_id " +
" FROM bmsql_district " +
" JOIN bmsql_order_line ON ol_w_id = d_w_id " +
" AND ol_d_id = d_id " +
" AND ol_o_id >= d_next_o_id - 20 " +
" AND ol_o_id < d_next_o_id " +
" WHERE d_w_id = ? AND d_id = ? " +
" ) " +
" ) AS aliasA");
...
修改后:
...
default:
stmtStockLevelSelectLow = dbConn.prepareStatement(
"SELECT count(*) AS low_stock FROM (" +
" SELECT distinct( s_i_id),s_w_id, s_quantity " +
" FROM bmsql_stock,bmsql_district " +
" JOIN bmsql_order_line ON ol_w_id = d_w_id " +
" AND ol_d_id = d_id AND ol_o_id >= d_next_o_id - 20" +
" WHERE s_w_id = ? " +
" AND s_quantity < ? " +
" AND s_i_id = ol_i_id " +
" AND d_w_id = ? " +
" AND d_id = ?" +
" ) AS aliasA");
...
# cd benchmarksql5.0
# ant
以下全是使用root用户,若以其他用户登录(如postgres用户),则先切换到该用户环境下在执行以下步骤。
1)测试机上
cd /root/.ssh/
rm -f ./*
ssh-keygen //随后按3次ENTER键
此时在.ssh目录下生成了id_rsa和id_rsa.pub。其中id_rsa是私钥,本机要用;id_rsa.pub是公钥,给其他的机器用。
2)将公钥上传到master上
# scp ./id_rsa.pub [email protected]:/root/.ssh/
3)将公匙写入master中的/root/.ssh/authorized_keys文件
# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
以下步骤以postgreSQL数据库为例,mysql的操作步骤与之类似。
1)在master上创建benchmarksql的测试用户和数据库
[postgres#localhost ~] $ psql postgres
psql (9.5.2)
Type "help" for help.
postgres=# CREATE USER benchmarksql WITH ENCRYPTED PASSWORD 'changeme';
postgres=# CREATE DATABASE benchmarksql OWNER benchmarksql;
postgres=# \q
2)回到测试机的benchmarksql目录,修改配置文件prop.pg(可以先备份配置文件)
# cp props.pg props.pg.bak
# vi props.pg
db=postgres
driver=org.postgresql.Driver
conn=jdbc:postgresql://localhost:5432/benchmarksql_100
user=benchmarksql
password=benchmarksql12345
warehouses=100 //测试因子,本次取值100,500
loadWorkers=8
terminals=1 //测试因子,本次取值1,10,30,100,300,500,1000
//To run specified transactions per terminal- runMins must equal zero
runTxnsPerTerminal=0
//To run for specified minutes- runTxnsPerTerminal must equal zero
runMins=20
//Number of total transactions per minute
limitTxnsPerMin=0
//Set to true to run in 4.x compatible mode. Set to false to use the
//entire configured database evenly.
terminalWarehouseFixed=false
//The following five values must add up to 100
//The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec
newOrderWeight=45
paymentWeight=43
orderStatusWeight=4
deliveryWeight=4
stockLevelWeight=4
// Directory name to create for collecting detailed result data.
// Comment this out to suppress.
resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS
osCollectorScript=./misc/os_collector_linux.py
osCollectorInterval=1
[email protected] //使用ssh访问master,前面必须配置免密码登录
osCollectorDevices=net_eth0 blk_vda blk_vdb //这里的块设备(如vda、vdb)应该在 /sys/block/ 目录下存在,需要改成本地数据目录所在的块名称
建立测试库,并加载数据
$ ./runDatabaseBuild.sh props.pg
注:数据库为mysql时,请使用runDatabaseBuild_mysql.sh脚本。
4)备份数据目录,这样以后测试时就不需要删除表再重新加载数据了。
5)跑测试,并生成结果
$ ./runBenchmark.sh props.pg
6)多次测试时,为了测试的准确性,最好删除当前数据目录,然后从4)中备份的数据目录中拷贝一份用于新的测试。或者执行以下步骤,清空数据库,再重建数据库(这样比较耗时间,500个warehouses的库可能需要4个小时以上,因此不推荐)。
$ ./runDatabaseDestroy.sh props.pg
$ ./runDatabaseBuild.sh props.pg
1) 安装R语言
下载安装R需要依赖的rpm包,本次缺少2个
ftp://fr2.rpmfind.net/linux/centos/6.8/os/x86_64/Packages/texinfo-tex-4.13a-8.el6.x86_64.rpm
http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/Packages/libjpeg-turbo-1.2.1-3.el6_5.x86_64.rpm
安装
yum install texinfo-tex-4.13a-8.el6.x86_64.rpm
yum install libjpeg-turbo-1.2.1-3.el6_5.x86_64.rpm
yum install R
2) 生成html汇总结果
$ ./generateReport.sh 结果路径(runBenchmark.sh测试结果的路径)