[MySQL] ERROR 2002(HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.

1 故障描述

测试环境Cacti页面登陆失败,sa找我排查问题,sa找我看是否是DB故障

 

2 去db服务器check。

[root@xxxx mysqldata]# ps-eaf|grep mysql

root     1422 12582  0 03:48 pts/13   00:00:00 grep mysql

root     1961     1  0 Sep02 ?        00:00:03 /bin/sh /usr/bin/mysqld_safe--datadir=/opt/mysqldata --socket=/var/lib/mysql/mysql.sock--pid-file=/opt/mysqldata/mysqld.pid --basedir=/usr --user=mysql

mysql   15117  1961  3 03:44 ?        00:00:08 /usr/libexec/mysqld--basedir=/usr --datadir=/opt/mysqldata --user=mysql--log-error=/opt/mysqldata/mysqld.log --pid-file=/opt/mysqldata/mysqld.pid--socket=/var/lib/mysql/mysql.sock

root    31480  6972  0 Sep17 pts/8    00:00:00 mysql -uroot -px xxxxxxx

[root@xxxx  mysqldata]#

OK，mysql进程一切正常，在后台运行着，没有被kill掉。

 

3 用root登陆check里面的mysql线程

[root@xxxx mysqldata]# mysql-uroot -p

Enter password:

ERROR 2002 (HY000): Can't connect to localMySQL server through socket '/var/lib/mysql/mysql.sock' (111)

[root@eanltrsutl001 mysqldata]#

 

4  去查看mysql的error日志

在2的步骤里面已经看到了--log-error=/opt/mysqldata/mysqld.log参数，所以去打开error日志文件/opt/mysqldata/mysqld.log

[root@xxxx mysqldata]# tail –f /opt/mysqldata/mysqld.log

130930 3:39:50  InnoDB: Started; logsequence number 3 3403486393

130930 3:39:50 [ERROR] /usr/libexec/mysqld: Error writing file'/opt/mysqldata/mysqld.pid' (Errcode: 28)

130930 3:39:50 [ERROR] Can't start server: can't create PID file: No space lefton device

130930 03:39:50 mysqld_safe Number ofprocesses running now: 0

130930 03:39:50 mysqld_safe mysqldrestarted

130930 3:39:50  InnoDB: Initializingbuffer pool, size = 15.0G

130930 3:39:51  InnoDB: Completedinitialization of buffer pool

InnoDB: The log sequence number in ibdatafiles does not match

InnoDB: the log sequence number in theib_logfiles!

130930 3:39:51  InnoDB: Database was notshut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information fromthe .ibd files...

InnoDB: Restoring possible half-writtendata pages from the doublewrite

InnoDB: buffer...

130930 3:39:51  InnoDB: Started; logsequence number 3 3403486393

130930  3:39:51 [ERROR] /usr/libexec/mysqld: Errorwriting file '/opt/mysqldata/mysqld.pid' (Errcode: 28)

130930  3:39:51 [ERROR] Can't start server: can'tcreate PID file: No space left on device

130930 03:39:51 mysqld_safe Number ofprocesses running now: 0

130930 03:39:51 mysqld_safe mysqldrestarted

130930 3:39:51  InnoDB: Initializingbuffer pool, size = 15.0G

看到了有没有，No space left ondevice，肯定是磁盘满了，导致无法wrote了。

 

5 去check磁盘空间

[root@xxxx mysqldata]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/Sys-root 1008M  513M 445M  54% /

tmpfs                  15G    0   15G   0% /dev/shm

/dev/mapper/Sys-applog

                       50G   14G  34G  29% /applog

/dev/vda1             194M   33M 152M  18% /boot

/dev/mapper/Sys-home  2.0G  68M  1.9G   4% /home

/dev/mapper/Sys-opt    20G  19G     0  100%/opt

/dev/mapper/Sys-tmp   7.9G 3.4G  4.2G  45% /tmp

/dev/mapper/Sys-usr   2.0G 1.9G   41M  98% /usr

/dev/mapper/Sys-var   7.9G 4.4G  3.2G  58% /var

/dev/mapper/Sys-crash

                      2.0G   68M 1.9G   4% /var/crash

/dev/mapper/Sys-log   7.9G 1.3G  6.3G  17% /var/log

/dev/mapper/Sys-vtmp 1008M   34M 924M   4% /var/tmp

//10.15.41.252/share  466G  23G  444G   5% /applog/winshare

果然是磁盘满了，/opt目录都100%了，赶紧通知sa清理磁盘空间，sa将磁盘扩充到了40G。

 

6 root登陆check mysql db状态

 [root@xxxx mysqldata]# mysql-uroot -p
 Enter password:
 ERROR 1045 (28000): Access denied for user'root'@'localhost' (using password: NO)
 [root@xxxx mysqldata]#

PS：OK，好了，磁盘满了之后，只要加大磁盘空间，mysql会自动恢复各种clients连接操作的。

 

 

7 善后

    由于这台是测试服务器的cacti监控应用db服务器，所以部署安装完db后，提醒sa添加磁盘监控，但是dev部门说没事量不大，sa忙碌别的事情就没有来得及加磁盘监控。

     看来还得及时提醒sa去添加磁盘监控，dev的经验也不是100%靠谱啊！