mycat 性能采集工具

创建一个 person表，主键为Id，hash方式分片，主键自增（采用数据库方式），当自增的step分别为10,100,1万的三种情况下，对此表做性能测试。
person表结构如下
Id，主键，Mycat自增主键
name,字符串，16字节最长
school,毕业学校，数字，1-1000范围，是学校编号
age，年龄，18-60
addr,地址，32字节，建议为 gz-tianhe（城市-地区两级 枚举的仿真数据）
zcode,邮编，
birth，生日，为日期类型， 1980到2010年之间随机的日期
score，得分，0-100分

测试环境采用标准的3个机器（或者虚机）
一个运行测试工具
一个运行Myat Server
一个运行MySQL

安装性能采集工具，按照PPT进程测试，给出step步长不同情况下，性能的区别，并做解释

1，机器准备：

192.168.136.130：mycat服务器
192.168.136.128：mysql服务器
192.168.136.129：测试服务器

2 准备全局序列号

在mycat命令窗口上面， 创建MYCAT_SEQUENCE表：

DROP TABLE IF EXISTS MYCAT_SEQUENCE;
CREATE TABLE MYCAT_SEQUENCE (NAME VARCHAR(50) NOT NULL,current_value INT NOT NULL,increment INT NOT NULL DEFAULT 100, PRIMARY KEY(NAME)) ENGINE=INNODB;

INSERT INTO MYCAT_SEQUENCE(NAME,current_value,increment) VALUES ('GLOBAL', 100000, 100);

2.1 在 dn21节点上，创建相关function

DROP FUNCTION IF EXISTS `mycat_seq_currval`;
DELIMITER ;;
CREATE DEFINER=`root`@`192.168.%` FUNCTION `mycat_seq_currval`(seq_name VARCHAR(50)) 
RETURNS VARCHAR(64) CHARSET latin1
    DETERMINISTIC
BEGIN 
        DECLARE retval VARCHAR(64);
        SET retval="-999999999,null";  
        SELECT CONCAT(CAST(current_value AS CHAR),",",CAST(increment AS CHAR) ) INTO retval 
          FROM MYCAT_SEQUENCE  WHERE NAME = seq_name;  
        RETURN retval ; 
END
;;
DELIMITER ;

mysql> USE db3;
DATABASE CHANGED
mysql> DROP FUNCTION IF EXISTS `mycat_seq_currval`;
QUERY OK, 0 ROWS affected, 1 warning (0.05 sec)

mysql> DELIMITER ;;
mysql> CREATE DEFINER=`root`@`192.168.%` FUNCTION `mycat_seq_currval`(seq_name VARCHAR(50)) 
    -> RETURNS VARCHAR(64) CHARSET latin1
    -> DETERMINISTIC
    -> BEGIN 
    -> DECLARE retval VARCHAR(64);
    -> SET retval="-999999999,null";  
    -> SELECT CONCAT(CAST(current_value AS CHAR),",",CAST(increment AS CHAR) ) INTO retval 
    -> FROM MYCAT_SEQUENCE  WHERE NAME = seq_name;  
    -> RETURN retval ; 
    -> END
    -> ;;
QUERY OK, 0 ROWS affected, 1 warning (0.08 sec)

mysql> DELIMITER ;
mysql> 

2.2 设置sequence值

DROP FUNCTION IF EXISTS `mycat_seq_nextval`;
DELIMITER ;;
CREATE DEFINER=`root`@`192.168.%` FUNCTION `mycat_seq_nextval`(seq_name VARCHAR(50)) RETURNS VARCHAR(64)
 CHARSET latin1
    DETERMINISTIC
BEGIN 
         UPDATE MYCAT_SEQUENCE  
                 SET current_value = current_value + increment 
                  WHERE NAME = seq_name;  
         RETURN mycat_seq_currval(seq_name);  
END
;;
DELIMITER ;

mysql> DROP FUNCTION IF EXISTS `mycat_seq_nextval`;
QUERY OK, 0 ROWS affected, 1 warning (0.00 sec)

mysql> DELIMITER ;;
mysql> CREATE DEFINER=`root`@`192.168.%` FUNCTION `mycat_seq_nextval`(seq_name VARCHAR(50)) RETURNS VARCHAR(64)
    -> CHARSET latin1
    -> DETERMINISTIC
    -> BEGIN 
    -> UPDATE MYCAT_SEQUENCE  
    -> SET current_value = current_value + increment 
    -> WHERE NAME = seq_name;  
    -> RETURN mycat_seq_currval(seq_name);  
    -> END
    -> ;;
QUERY OK, 0 ROWS affected, 1 warning (0.01 sec)

mysql> DELIMITER ;
mysql> 

2.3 获取下一个sequence值

DROP FUNCTION IF EXISTS `mycat_seq_setval`;
DELIMITER ;;
CREATE DEFINER=`root`@`192.168.%` FUNCTION `mycat_seq_setval`(seq_name VARCHAR(50), VALUE INTEGER) RETURNS VARCHAR(64) CHARSET latin1 DETERMINISTIC BEGIN UPDATE MYCAT_SEQUENCE SET current_value = VALUE WHERE NAME = seq_name;  
         RETURN mycat_seq_currval(seq_name);  
END ;;
DELIMITER ;

2.4 配置

sequence_db_conf.properties相关配置,指定sequence相关配置在哪个节点上：

PERSON=dn21

注意：MYCAT_SEQUENCE表和以上的3个function，需要放在同一个节点上。function请直接在具体节点的数据库上执行，如果执行的时候报：

you might want TO USE the LESS safe log_bin_trust_function_creators variable

需要对数据库做如下设置：

my.cnf中[mysqld]加上log_bin_trust_function_creators=1

修改完后，即可在mysql数据库中执行上面的函数.

使用示例：

INSERT INTO t1(id) VALUES(NEXT VALUE FOR MYCATSEQ_GLOBAL); mysql> create table T_VOTE(id int,name varchar(255));
Query OK, 0 rows affected (0.22 sec)

mysql> 
mysql> INSERT INTO t1(id) VALUES(NEXT VALUE FOR MYCATSEQ_GLOBAL); Query OK, 1 row affected (0.01 sec) mysql> select * from t1; +---------+
| id | +---------+
| 100202 | +---------+
4 rows in set (0.08 sec)

mysql> 

3，设置hash分片

schema.xml配置文件：

    <table name="PERSON" primaryKey="id_" autoIncrement="true"  dataNode="dn21,dn22" rule="sharding-by-murmur6" />

rule.xml配置文件：

    <tableRule name="sharding-by-murmur6">
             <rule>
                    <columns>PROVINCE</columns>
                    <algorithm>murmur</algorithm>
            </rule>
    </tableRule>

创建表，录入数据测试：

CREATE TABLE person(id INT AUTO_INCREMENT PRIMARY KEY,NAME VARCHAR(16),school INT, age INT, addr VARCHAR(32),zcode VARCHAR(6),birth DATETIME,score INT);

 insert into person(id,name, school, age, addr, zcode, birth, score) values( 1169314970333220864,'zhang', 99, 20,'heibei-baoding','072450','1989-02-01',90);
 mysql> insert into person(id,name, school, age, addr, zcode, birth, score) values( 1169314970333220865,'zhang', 99, 20,'heibei-baoding','072450','1989-02-01',90);
Query OK, 1 row affected (0.03 sec)

mysql> select * from person; +--------+-------+--------+------+----------------+--------+---------------------+-------+
| id | NAME | school | age | addr | zcode | birth | score | +--------+-------+--------+------+----------------+--------+---------------------+-------+
| 100300 | NULL  |   NULL | NULL | NULL           | NULL   | NULL                |  NULL |
| 100304 | zhang | 99 | 20 | heibei-baoding | 072450 | 1989-02-01 00:00:00 | 90 | +--------+-------+--------+------+----------------+--------+---------------------+-------+
2 rows in set (0.02 sec)

mysql> 

4，准备安装性能监控工具nmon

4.1，去下载 http://nmon.sourceforge.net/pmwiki.php?n=Site.Download中的nmon16d_x86.tar.gz 9MB

4.2，解压缩
mkdir nmon;
tar -xvf nmon16d_x86.tar.gz -C /soft/nmon
cd /soft/nmon
[root@oracle_mycat nmon]# ll nmon*centos
-rwx—— 1 210 201 343097 Feb 4 08:54 nmon_x86_64_centos6
-rwx—— 1 210 201 397618 Feb 4 08:54 nmon_x86_64_centos7
-rwx—— 1 210 201 302830 Feb 4 08:54 nmon_x86_centos6
[root@oracle_mycat nmon]#

4.3，执行监控命令
./nmon_x86_64_centos7 -f -t -s 1 -c 180
-f：按标准格式输出文件：_YYYYMMDD_HHMM.nmon；
-t：输出中包括占用率较高的进程
-s 5：每 5 秒进行一次数据采集
-c 360 ：一共采集 360 次，即总共采集半小时数据

回车后，将自动在当前目录生成一个hostname_timeSeries.nmon的文件，如果hosname为bogon，生产的文件为：bogon_160306_1606.nmon

采集结束之后，通过sort命令可以将nmon结果文件转换为csv文件：
sort bogon_160306_1606.nmon > bogon_160306_1606.csv
执行完sort命令后即可在当前目录生产 bogon_160306_1606.csv 文件

4.4，excel分析器
下载http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmonanalyser

4.5 下载基准测试工具
TestTools工具包，下载地址：https://github.com/MyCATApache/Mycat-download/tree/master/1.5-GA
，解压即可使用（需要JDK1.7环境），下载Mycat-server-1.5-GA-20160225120029-testtool.tar.gz这个包。

5，开始基准测试

分两个片录入数据：0-500M是第一分片，500M-1000M是第二分片
./test_stand_insert_perf.sh jdbc:mysql://192.168.136.130:8066/TESTDB test test 10 “0-500M,501M-1000M”

准备sql：vim mydata-create.sql

total=10000000
insert into person(id,name, school, age, addr, zcode, birth, score) values( 1169314970333220866,’li’, 99, 20, char([a−f,0−9]8:8),′072450′, {date(yyyyMMddHHmmssSSS-[2014-2016]y)},${int(50,100)});

执行报错如下：
[root@oracle_standby bin]# ./test_stand_insert_perf.sh jdbc:mysql://192.168.136.130:8066/TESTDB test test 10 file=mydata-create.sql
check JAVA_HOME & java
———set HOME_DIR————
create jobs …
total record 10000000 batch size:100 autocomit false
Exception in thread “main” java.lang.NullPointerException
at org.opencloudb.performance.RandomDataValueUtil.parselRandVarTemplateString(RandomDataValueUtil.java:46)
at org.opencloudb.performance.AbstractMultiTreadBatchTester.createSQLTemplateJobs(AbstractMultiTreadBatchTester.java:146)
at org.opencloudb.performance.AbstractMultiTreadBatchTester.createAllJobs(AbstractMultiTreadBatchTester.java:128)
at org.opencloudb.performance.AbstractMultiTreadBatchTester.startTest(AbstractMultiTreadBatchTester.java:182)
at org.opencloudb.performance.AbstractMultiTreadBatchTester.run(AbstractMultiTreadBatchTester.java:80)
at org.opencloudb.performance.TestInsertPerf.main(TestInsertPerf.java:35)
[root@oracle_standby bin]#

，看到报错了，google没有有用的信息，一看时间已经2016/3/7 1:45了，得先睡觉了，明天还要早起上班，后续明天晚上继续测试。

刚经过victor提醒，说是sql文件里面第二行要加上sql=

vim mydata-create.sql

total=10000000
sql=insert into person(id,name, school, age, addr, zcode, birth, score) values( 1169314970337415168,’li’, 99, 20, char([a−f,0−9]8:8),′072450′, {date(yyyyMMddHHmmss-[2014-2016]y)},${int(50,100)});

再次执行正常，结果如下：

[root@oracle_standby bin]# time ./test_stand_insert_perf.sh jdbc:mysql://192.168.136.130:8066/TESTDB test test 10 file=mydata-create.sql check JAVA_HOME & java ---------set HOME_DIR------------ create jobs ... total record 10000000 batch size:100 autocomit false success ful created connections ,total :10 create jobs finished ,begin run test... success create job count: 10 teset threads: 10 06 19:59:21 finished records :0 failed:0 speed:0.0 06 19:59:22 finished records :0 failed:0 speed:0.0 06 19:59:23 finished records :0 failed:0 speed:0.0 06 19:59:24 finished records :1000 failed:0 speed:328.94736842105266 06 19:59:25 finished records :1000 failed:0 speed:247.27992087042531 06 19:59:26 finished records :1700 failed:0 speed:337.0340999206979 06 19:59:27 finished records :2000 failed:0 speed:330.90668431502314 06 19:59:28 finished records :2300 failed:0 speed:326.47267565649395 06 19:59:29 finished records :3000 failed:0 speed:372.85607755406414 06 19:59:30 finished records :3100 failed:0 speed:342.6929029405262 06 19:59:31 finished records :4000 failed:0 speed:397.81203381402287 06 19:59:32 finished records :4000 failed:0 speed:361.794500723589 06 19:59:33 finished records :4700 failed:0 speed:389.8150452019574 06 19:59:34 finished records :5000 failed:0 speed:382.93635597763654 06 19:59:35 finished records :5400 failed:0 speed:384.1229193341869 06 19:59:36 finished records :6000 failed:0 speed:398.4592907424625 06 19:59:37 finished records :6200 failed:0 speed:386.07634348340497 06 19:59:38 finished records :6800 failed:0 speed:398.61656603552376 06 19:59:39 finished records :7300 failed:0 speed:404.20819490586933 06 19:59:40 finished records :7700 failed:0 speed:403.60624803438515 06 19:59:41 finished records :8000 failed:0 speed:398.1486089682974 06 19:59:42 finished records :8700 failed:0 speed:412.45910965723226 06 19:59:43 finished records :8900 failed:0 speed:402.82429618901057 ...... 07 01:50:35 finished records :9998900 failed:0 speed:474.47918045773486 07 01:50:36 finished records :9999300 failed:0 speed:474.4756463546189 07 01:50:37 finished records :9999700 failed:0 speed:474.47209007382673 finishend:10000000 failed:0 used time total:21076seconds tps:474.47108335982466 real 351m17.558s user 12m15.684s sys 51m39.295s You have new mail in /var/spool/mail/root [root@oracle_standby bin]# -- 去mycat命令里面查看录入的真实数据： mysql> select count(1) from person;
+----------+
| COUNT0   |
+----------+
| 10000000 |
+----------+
1 row in set (2.65 sec) mysql> 

看到跑了压测10000000，花费时间是351分钟，finishend:10000000 failed:0，real 351m17.558s。

6，10000000有些多了，改成100W试试看，开始步长为1的测试，这里需要重置下序列数据

1是修改序列号
mysql> truncate table PERSON;
Query OK, 0 rows affected (0.75 sec)

mysql> truncate table MYCAT_SEQUENCE;
Query OK, 0 rows affected (0.20 sec)

mysql> INSERT INTO MYCAT_SEQUENCE(NAME,current_value,increment) VALUES (‘GLOBAL’, 100000, 100);
Query OK, 2 rows affected (0.05 sec)

mysql>
之后重启mycat生效

2是设置分片范围：
[root@oracle_mycat conf]# vim autopartition-long.txt

0-50M=0
50M-100M=1
100M-1000M=0

然后开启数据录入测试：time ./test_stand_insert_perf.sh jdbc:mysql://192.168.136.130:8066/TESTDB test test 10 file=mydata-create.sql
再在3台测试机器上开启性能监控：./nmon_x86_64_centos6 -f -t -s 1 -c 3600，性能图如下所示：

6.1 mysql 分片1服务器性能图：

6.2 mysql分片2服务器性能图

6.3 mycat服务器性能图

7 步长为10的性能测试

参考文章：http://www.songwie.com/articlelist/68