mysql ddl 导致tmp空间溢出并报错

1.版本

1)操作系统

 cat /etc/issue
cat /etc/issue
CentOS release 6.6 (Final)
Kernel \r on an \m

 cat /proc/version
cat /proc/version
Linux version 2.6.32-504.el6.x86_64 ([email protected]) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Oct 15 04:27:16 UTC 2014

2)mysql数据库版本

MySQL --version
mysql  Ver 14.14 Distrib 5.6.26, for linux-glibc2.5 (x86_64) using  EditLine wrapper

2.问题描述

2.1 发现问题

  收到从库复制状态报警，show slave status\G;显示报错信息如下：

                   Last_Errno: 1034
                   Last_Error: Error 'Incorrect key file for table 'test_s'; try to repair it' on query. Default database: 'dbs1'. Query: 'alter table test add service_code varchar(64) COMMENT '业务编码''
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 547897990
              Relay_Log_Space: 631033469
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 1034
               Last_SQL_Error: Error 'Incorrect key file for table 'test_s'; try to repair it' on query. Default database: 'dbs1'. Query: 'alter table test_s add service_code varchar(64) COMMENT '业务编码''

3. 分析问题

   只通过上面的报错信息，我们很难确定具体是什么问题，所以我们继续查看mysql的error log。errorlog中相关报错信息如下：

2017-01-20 11:00:05 7fcc44742700 InnoDB: Error: Write to file (merge) failed at offset 10558111744.
InnoDB: 1048576 bytes should have been written, only 843776 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2017-01-20 11:00:06 27657 [ERROR] Slave SQL: Error 'Incorrect key file for table 'test_s'; try to repair it' on query. Default database: 'dbs1'. Query: 'alter table test_s add service_code varchar(64) COMMENT '业务编码'', Error_code: 1034
2017-01-20 11:00:06 27657 [Warning] Slave: Incorrect key file for table 'test_s'; try to repair it Error_code: 1034
2017-01-20 11:00:06 27657 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log '3306-bin.000167' position 547897990

  从errorlog的报错信息，我们很容易看出来是在给表add column的时候，磁盘空间满了，导致add column操作失败报错。关于mysql 各种ddl操作需要的空间问题，我会在另一篇博客中进行介绍。

4.问题解决方案

   知道了问题原因，处理就很简单了df -h找到使用率100%的挂在目录，然后添加磁盘空间(或者修改mysql tmpdir参数，如果是因为临时文件导致的话)

##mysql online ddl不但会在表所在目录下生成两个中间文件，而且会在tmpdri指定目录下生成临时排序文件，排序文件大概需要两倍表大小的空间(注意ls无法查看到这些临时排序文件，可以通过lsof|grep deleted观察)

ALTER TABLE operations that rebuild an InnoDB table using the ALGORITHM=INPLACE technique (online DDL) also create temporary sort files in the MySQL temporary directory ($TMPDIR on Unix, %TEMP% on Windows, or the directory specified by the --tmpdir configuration option).

参考：

https://dev.mysql.com/doc/refman/5.6/en/temporary-files.html