【MySQL】TokuDB引擎初探（MySQL升级为Percona，MySQL升级为MariaDB）
参考：http://blog.sina.com.cn/s/blog_4673e6030102v46l.html
参考：http://hcymysql.blog.51cto.com/5223301/1431698
参考：http://blog.csdn.net/yueliangdao0608/article/details/21230627
参考：http://code.openark.org/blog/mysql/tokudb-configuration-variables-of-interest
参考：http://blog.itpub.net/22418990/viewspace-1251497/
官网说明：http://docs.tokutek.com/tokudb/tokudb-index-variables.html
官网说明：https://mariadb.com/kb/en/mariadb/tokudb-system-variables/#tokudb_pk_insert_mode
因为MariaDB 10版本需要单独编译TokuDB引擎，编译需要GCC >= 4.7和CMake >=2.8.8版本，暂没有进行单独编译，后续再测试。
官网TokuDB源码下面的说明：
Before you start, make sure you have a C++11-compatible compiler (GCC >= 4.7 is recommended), as well as CMake >=2.8.8, and the libraries and header files for valgrind,zlib, and Berkeley DB. We are using the gcc 4.7 in devtoolset-1.1.
测试版本：Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101.tar.gz
测试版本：Percona-Server-5.6.24-rel72.2-TokuDB.Linux.x86_64.ssl101.tar.gz
OS:Centos 6.2
# - path of TokuDB file - # > tokudb_data_dir > tokudb_log_dir > tokudb_tmp_dir # - variables of TokuDB engine - # > tokudb_cache_size (Dynamic: NO,Default Value: Half of the total system memory) This variable configures the size in bytes of the TokuDB cache table. The default cache table size is 1/2 of physical memory. Tokutek highly recommends using the default setting if using buffered IO, if using direct IO then consider setting this parameter to 80% of available memory. > tokudb_directio (Dynamic: NO,Default Value: OFF) When set to ON, TokuDB employs Direct IO rather than Buffered IO for writes. When using Direct IO consider increasing tokudb cache size from its default of 1/2 physical memory. The default values is OFF. > tokudb_commit_sync (Dynamic: Yes,Default Value: ON) Session variable tokudb_commit_sync controls whether or not the transaction log is flushed when a transaction commits. The default behavior is that the transaction log is flushed by the commit. Setting this parameter to off may make the system run faster. However, transactions committed since the last checkpoint are not guaranteed to survive a crash. > tokudb_fsync_log_period (Dynamic: Yes,Default Value: 0) Controls the frequency, in milliseconds, for fsync() operations. If set to 0 then the fsync() behavior is only controlled by the tokudb commit sync, which is on or off. The default values is 0. > tokudb_row_format (Dynamic: Yes,Default Value: tokudb_zlib) tokudb_default, tokudb_zlib: Use the zlib library, tokudb_fast, tokudb_quicklz: Use the quicklz library, the lightest compression with low CPU usage, tokudb_small, tokudb_lzma: Use the lzma library. the highest compression and highest CPU usage tokudb_uncompressed: No compression is used. http://docs.tokutek.com/tokudb/tokudb-index-using-tokudb.html#tokudb-compression-details > tokudb_read_buf_size > tokudb_loader_memory_size
上面两个官网的说明没有读懂...求高人指点
二进制包解压后把toku包中的文件放到server包中的路径下
注意配置要添加
[mysqld_safe] malloc_lib = /usr/local/Percona-Mysql/lib/mysql/libjemalloc.so [mysqld] plugin_dir = /usr/local/Percona-Mysql/lib/mysql/plugin/ plugin_load=ha_tokudb.so
修改系统Huge pages配置
[root@test percona]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled [root@test percona]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defarg
否则会报错
#### 系统Huge pages配置需要修改，Centos我的是redhat_transparent_hugepage/enabled和defrag [ERROR] TokuDB: Huge pages are enabled, disable them before continuing [ERROR] ************************************************************ [ERROR] [ERROR] @@@@@@@@@@@ [ERROR] @@' '@@ [ERROR] @@ _ _ @@ [ERROR] | (.) (.) | [ERROR] | ` | [ERROR] | > ' | [ERROR] | .----. | [ERROR] .. |.----.| .. [ERROR] .. ' ' .. [ERROR] .._______,. [ERROR] [ERROR] TokuDB will not run with transparent huge pages enabled. [ERROR] Please disable them to continue. [ERROR] (echo never > /sys/kernel/mm/transparent_hugepage/enabled) [ERROR] [ERROR] ************************************************************ [ERROR] Plugin 'TokuDB' init function returned error. [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed. #### 配置malloc_lib放错位置，应该放在[mysqld_safe]下 [ERROR] /usr/local/Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101/bin/mysqld: unknown variable 'malloc_lib=/usr/local/Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101/lib/mysql/libjemalloc.so'
#### 配置malloc_lib缺项 [ERROR] TokuDB is not initialized because jemalloc is not loaded [ERROR] Plugin 'TokuDB' init function returned error. [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
#### 启动失败后删除TokuDB遗留文件 [ERROR] TokuDB: Persistent environment information is missing (but log exists) [ERROR] TokuDB unknown error 2 [ERROR] Plugin 'TokuDB' init function returned error. [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
然后进入MySQL确认TokuDB可用
mysql> show engines; +--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+ | Engine | Support | Comment | Transactions | XA | Savepoints | +--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+ | InnoDB | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES | YES | YES | | CSV | YES | CSV storage engine | NO | NO | NO | | MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO | | BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO | | MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO | | TokuDB | YES | Tokutek TokuDB Storage Engine with Fractal Tree(tm) Technology | YES | YES | YES | | PERFORMANCE_SCHEMA | YES | Performance Schema | NO | NO | NO | | ARCHIVE | YES | Archive storage engine | NO | NO | NO | | MyISAM | YES | MyISAM storage engine | NO | NO | NO | | FEDERATED | NO | Federated MySQL storage engine | NULL | NULL | NULL | +--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+ 10 rows in set (0.00 sec)
【性能测试】
由于使用的是Percona 5.6版本，默认performance_schema是开启的，所以关掉后再试。
建两张测试表，表字段类型长度一样引擎不一样
CREATE TABLE `test1_innodb` \ `test1_tokudb` ( `a` bigint(20) NOT NULL AUTO_INCREMENT, `b` varchar(255) DEFAULT NULL, `c` varchar(255) DEFAULT NULL, `d` text, `e` varchar(255) DEFAULT NULL, `f` varchar(255) DEFAULT NULL, `g` varchar(255) DEFAULT NULL, `h` varchar(255) DEFAULT NULL, `i` text, `j` varchar(255) DEFAULT NULL, `k` varchar(255) DEFAULT NULL, `l` varchar(255) DEFAULT NULL, `m` varchar(255) DEFAULT NULL, `n` varchar(255) DEFAULT NULL, PRIMARY KEY (`a`) ) ENGINE=InnoDB \ TokuDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8
##### 测试1
binlog_format = MIX sync_binlog = 0 innodb_flush_log_at_trx_commit = 1 innodb_buffer_pool_size = 1G tokudb_cache_size = 1G tokudb_row_format = tokudb_zlib tokudb_commit_sync = ON tokudb_fsync_log_period = 0 mysql> load data local infile 'test.csv' into table test_innodb ...; Query OK, 65536 rows affected (51.46 sec) Records: 65536 Deleted: 0 Skipped: 0 Warnings: 0 mysql> load data local infile 'test.csv' into table test_tokudb ...; Query OK, 65536 rows affected (31.86 sec) Records: 65536 Deleted: 0 Skipped: 0 Warnings: 0
##### 统计信息为information_schema.tables的table_length,index_length,total(table_length+index_length)，换算完单位为G mysql> test_innodb; +---------+---------+---------+ | DATA | idx | total | +---------+---------+---------+ | 2.66791 | 0.00000 | 2.66791 | +---------+---------+---------+ mysql> test_tokudb; +---------+---------+---------+ | DATA | idx | total | +---------+---------+---------+ | 1.44171 | 0.00000 | 1.44171 | +---------+---------+---------+ [root@test percona]# ll -h */*test_* -rw-rw---- 1 mysql mysql 17K Jun 30 11:11 test/test_innodb.frm -rw-rw---- 1 mysql mysql 2.8G Jun 30 11:24 test/test_innodb.ibd -rw-rw---- 1 mysql mysql 17K Jun 30 11:10 test/test_tokudb.frm -rw-rw---- 1 mysql mysql 16M Jun 30 11:26 toku_data/_test_test_tokudb_main_90_2_1b.tokudb -rw-rw---- 1 mysql mysql 64K Jun 30 11:27 toku_data/_test_test_tokudb_status_90_1_1b.tokudb
##### 插入数据后，发现系统中tokudb数据文件大小与实例中统计的表大小相差很多，暂不分析原因，后面测试实例中统计信息的依据性。
##### 测试2
tokudb_row_format = tokudb_small 其余配置不变 mysql> load data local infile 'test.csv' into table test_tokudb_small ...; Query OK, 65536 rows affected (33.91 sec) Records: 65536 Deleted: 0 Skipped: 0 Warnings: 0 mysql> test_tokudb_small; +---------+---------+---------+ | DATA | idx | total | +---------+---------+---------+ | 1.44171 | 0.00000 | 1.44171 | +---------+---------+---------+ 1 row in set (0.00 sec) [root@test percona]# ll -h */*test_toku* -rw-rw---- 1 mysql mysql 17K Jun 30 11:10 test/test_tokudb.frm -rw-rw---- 1 mysql mysql 17K Jun 30 11:36 test/test_tokudb_small.frm -rw-rw---- 1 mysql mysql 16M Jun 30 11:26 toku_data/_test_test_tokudb_main_90_2_1b.tokudb -rw-rw---- 1 mysql mysql 16M Jun 30 11:38 toku_data/_test_test_tokudb_small_main_a1_2_1b.tokudb -rw-rw---- 1 mysql mysql 64K Jun 30 11:38 toku_data/_test_test_tokudb_small_status_a1_1_1b.tokudb -rw-rw---- 1 mysql mysql 64K Jun 30 11:27 toku_data/_test_test_tokudb_status_90_1_1b.tokudb
##### 使用tokudb_small的表发现数据文件大小与tokudb_zlib相差并不多
##### 使用tokudb_small\tokudb_lzma CPU负载会比较高
##### tokudb_small, tokudb_lzma: Use the lzma library. the highest compression and highest CPU usage
##### 试了一下更换tokudb_row_format不影响表数据文件
##### 删除完数据使用optimize table测试磁盘回收，发现innodb表数据文件空间释放掉，而且速度很快，但是tokudb表数据文件空间没有释放，重启也未释放
【MariaDB 10编译TokuDB】
参考：http://blog.sina.com.cn/s/blog_4673e6030102v46l.html
参考：http://blog.sina.com.cn/s/blog_4673e6030102v46k.html
参考：http://www.ithov.com/linux/130504.shtml
按照上面的参考按照高版本的gcc和cmake。(C++11-compatible compiler (GCC >= 4.7 is recommended), as well as CMake >=2.8.8)
如果出现报错/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found参考上面链接的方法解决
如果报下面的按照jemalloc
CMake Warning at storage/tokudb/CMakeLists.txt:17 (MESSAGE):
TokuDB is enabled, but jemalloc is not. This configuration is not
supported
编译完的ha_tokudb.so放在官方同版本分发包中就可以使用了
MariaDB [(none)]> INSTALL SONAME 'ha_tokudb';
使用注意：
1.MariaDB分发包解压复制到/usr/local/下
2.MariaDB配置文件初始化时不要带TokuDB引擎参数
3.初始化实例系统表
4.启动实例执行“install soname 'ha_tokudb;'”
5.停止实例并将配置文件添加TokuDB引擎参数
如果使用默认参数可以不停实例
如果要修改tokudb_data_dir/tokudb_log_dir/tokudb_tmp_dir需要删除实例数据目录下面TokuDB加载文件，-BASH "rm -rf *tokudb*"
6.第一次加载成功TokuDB，之后启动自动加载，但是tokudb_*配置错误会导致引擎注册启动失败，TokuDB引擎表无法使用
【MySLQ 5.5.21实例升级Percona 5.6.24】
tokudb最新5.5下载地址：http://www.tokutek.com/download.php
官方安装都需要停止实例安装tokudb引擎
参考：http://docs.tokutek.com/tokudb/tokudb-index-installation.html
测试从MySQL-5.5.21升级到Percona-Server-5.6.24
1、停止实例
2、使用Percona-Server-5.6.24启动实例
3、使用Percona-Server-5.6.24的mysql_upgrade -u -p -S 升级系统表结构
4、启动实例
[root@test Percona]# /usr/local/Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101/bin/mysql_upgrade -uroot -p -S /tmp/mysql_3306.sock Enter password: Looking for 'mysql' as: /usr/local/Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101/bin/mysql Looking for 'mysqlcheck' as: /usr/local/Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101/bin/mysqlcheck Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock' Warning: Using a password on the command line interface can be insecure. Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock' Warning: Using a password on the command line interface can be insecure. mysql.columns_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.general_log OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.host OK mysql.ndb_binlog_index OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.servers OK mysql.slow_log OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.user OK Running 'mysql_fix_privilege_tables'... Warning: Using a password on the command line interface can be insecure. Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock' Warning: Using a password on the command line interface can be insecure. Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock' Warning: Using a password on the command line interface can be insecure. test.t1 OK tmp.tb OK tmp.tb_merge_goldeye_main_role OK OK
【MySQL 5.5.37实例升级MariaDB 10.0.20】
官网对于mysql_upgrade的说明：https://mariadb.com/kb/en/mariadb/mysql_upgrade/
1、MariaDB 10.0.20官网分发包，下载解压
2、MySQL 5.5.37挺实例
3、MariaDB 10.0.20启动实例，如果实例启动正常，errlog报错忽略
4、MariaDB 10.0.20分发包中的mysql_upgrade升级实例系统表，查看升级状态
5、实例停止再启动，errlog无报错
[root@test mariadb]# ./bin/mysql_upgrade -uroot -p -S /tmp/mysql_3306.sock Enter password: MySQL upgrade detected Phase 1/6: Checking and upgrading mysql database Processing databases mysql mysql.columns_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.host OK mysql.ndb_binlog_index OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.servers OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.user OK Phase 2/6: Fixing views from mysql Phase 3/6: Running 'mysql_fix_privilege_tables' Phase 4/6: Fixing table and database names Phase 5/6: Checking and upgrading tables Processing databases information_schema jiangxu jiangxu.jiangxu1 OK jiangxu.jiangxu2 OK jiangxu.jiangxu3 OK performance_schema test tmp Phase 6/6: Running 'FLUSH PRIVILEGES' OK
【功能特性说明】：支持辅助聚集索引、在线添加索引、在线修改列属性等
辅助聚集索引 --> 使得通过该索引的查询都为覆盖索引，速度更快
在线添加索引 --> tokudb对于alter table add index 和create index处理方式不同。
Using the ALTER TABLE command for creating an index will create the index offline (with the table unavailable for inserts or queries), regardless of the value of tokudb_create_index_online. The only way to hot create an index is to use the CREATE INDEX command.
Hot creating an index will be slower than creating the index offline, and progress depends how busy the mysqld server is with other tasks.
在线修改列属性 --> 有一些修改并不支持。
Hot column expansion operations are only supported to char, varchar, varbinary, and integer data types. Hot column expansion is not supported if the given column is part of the primary key or any secondary keys.
Hot column rename does not support the following data types: TIME, ENUM, BLOB, TINYBLOB, MEDIUMBLOB, LONGBLOB. Renaming columns of these types will revert to the standard MySQL blocking behavior.
上面只是官网的一小部分，详细看下面链接原文
参考：http://docs.tokutek.com/tokudb/tokudb-index-using-tokudb.html
【锁的说明】
实例主从复制建议使用binlog_format=ROW
参考：http://docs.tokutek.com/tokudb/tokudb-index-troubleshooting.html