MySQL锁等待排查( Lock wait timeout exceeded)

问题现象

客户反馈，某个业务动作（添加商品）有时候成功，有时候会卡住。

排查

异常日志

2021-03-24 06:38:53.091 [] [http-nio-9999-exec-13] ERROR o.a.c.c.C.[.[localhost].[/].[dispatcherServlet][175] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.
dao.PessimisticLockingFailureException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.PessimisticLockException: could not execute statement] with root cause
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:123)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1040)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1347)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1025)
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeUpdate(FilterChainImpl.java:2797)
    at com.alibaba.druid.filter.FilterAdapter.preparedStatement_executeUpdate(FilterAdapter.java:1069)
    at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_executeUpdate(FilterEventAdapter.java:491)
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeUpdate(FilterChainImpl.java:2795)
    at com.alibaba.druid.filter.FilterAdapter.preparedStatement_executeUpdate(FilterAdapter.java:1069)
    at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_executeUpdate(FilterEventAdapter.java:491)
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeUpdate(FilterChainImpl.java:2795)
    at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.executeUpdate(PreparedStatementProxyImpl.java:163)
    at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:253)
    at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.MasterSlavePreparedStatement.executeUpdate(MasterSlavePreparedStatement.java:109)
    at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:175)
    at org.hibernate.dialect.identity.GetGeneratedKeysDelegate.executeAndExtract(GetGeneratedKeysDelegate.java:57)
    at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:42)
    at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3072)
    at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3663)
    at org.hibernate.action.internal.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:81)

MySQL锁信息查看

复现用户的行为，可以出现同样的情况。
查看MySQL的锁信息
当前事务：
SELECT * FROM information_schema.INNODB_TRX;

innodb上的锁
SELECT * FROM information_schema.INNODB_LOCKs;
可以看到176403706这个事务，从18:35开始，执行好几分钟了

锁的依赖关系
SELECT * FROM information_schema.INNODB_LOCK_waits;

可以看到176422531这个事务，被事务176403706阻塞。
干掉这个线程，kill 53 (176403706这个事务对应的trx_mysql_thread_id)

业务操作正常。过了一会，又出现类似现象。
查看运行的线程信息。SHOW FULL PROCESSLIST; （ps：图是后补的，仅供示意）

根据连接信息，去服务器上查看连接的进程，定位到对应的应用（是我们自己的服务）

临时开启MySQL实时SQL打印，根据threadId和时间，找到对应执行的SQL。

临时开启MySQL实时SQL打印

SET GLOBAL general_log = 'ON';
SET GLOBAL general_log_file = '/var/log/mysql/general_log.log';

查看sql，根据mysql thread id，找到对应的sql（下图中的111，就是mysql thread id，图是后补的，供参考）

根据sql，找到对应的业务

观察对应的业务方法执行，查看到错误

查看对应的业务代码，自己进行的事务管理，异常时没有回滚事务。图中的回滚是后面添加的。
相应的业务代码执行商品更新操作（insert into … on duplicate key update …），其主线是自增id。导致事务的锁定数据会出现“supermum pseudo-record”。

supremum pseudo-record :相当于比索引中所有值都大，但却不存在索引中，相当于最后一行之后的间隙锁
For the last interval, the next-key lock locks the gap above the largest value in the index and the “supremum”pseudo-record having a value higher than any value actually in the index. The supremum is not a real index record, so, in effect, this next-key lock locks only the gap following the largest index value.
来源： http://blog.itpub.net/26250550/viewspace-1070422/

参考资料

• MySQL死锁问题。http://xiaobaoqiu.github.io/blog/2016/07/22/%5B%3F%5D-ge-si-suo-wen-ti/
• MySQL锁介绍。https://blog.csdn.net/u010841296/article/details/84204701