【线上】SQLAlchemy数据库连接满导致的问题

这是刚做Superset项目时遇到的线上问题，现回顾并整理下。

原因和现象回顾

访问高峰qps增大，然后数据库连接占用超过了SQLAlchemy设置的默认最大数量，且达到超时时间，这导致线上很多用户看到如下报错

2018-11-04 11:27:50,370 /var/www/html/superset/views/base.py base.py[line:56] : ERROR  QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30
    (self.size(), self.overflow(), self._timeout))
    (self.size(), self.overflow(), self._timeout))
TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30
TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30
TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30
TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30
    (self.size(), self.overflow(), self._timeout))

此问题当时整理并复现

可将其称为如下问题，网上也有不少类似文章，不过不亲自遇到并实践，总是体会不到；我将其整理到github上，也借此复习下python基础知识

QueuePool limit of size overflow reached, connection timed out, timeout 问题

本次线上问题总结到的知识

主要是MySQL数据库方面的知识

1. 数据库的一些系统变量

show variables like '%connect%';

可能得到如下，对连接的一些超时，最大限制可以看到；连接数等也是DBA监控必然有的一项

+-----------------------------------------------+-----------------+
| Variable_name                                 | Value           |
+-----------------------------------------------+-----------------+
| character_set_connection                      | utf8            |
| collation_connection                          | utf8_general_ci |
| connect_timeout                               | 10              |
| disconnect_on_expired_password                | ON              |
| extra_max_connections                         | 1               |
| init_connect                                  |                 |
| max_connect_errors                            | 999999999       |
| max_connections                               | 10240           |
| max_user_connections                          | 0               |
| performance_schema_session_connect_attrs_size | -1              |
+-----------------------------------------------+-----------------+
10 rows in set

• 数据库本身也是tcp连接的, 你可以查看到连接信息的，一般DBA会配置这些监控告警

mysql> show full processlist;
+----+------+-----------------+------+---------+-------+-------+-----------------------+
| Id | User | Host            | db   | Command | Time  | State | Info                  |
+----+------+-----------------+------+---------+-------+-------+-----------------------+
|  2 | root | localhost       | test | Query   |     0 | init  | show full processlist |
|  8 | root | localhost:59270 | test | Sleep   | 38068 |       | NULL                  |
|  9 | root | localhost:59272 | test | Sleep   | 37914 |       | NULL                  |
| 10 | root | localhost:59288 | test | Sleep   | 37914 |       | NULL                  |
| 11 | root | localhost:59289 | test | Sleep   |   708 |       | NULL                  |
| 12 | root | localhost:59290 | test | Sleep   |   650 |       | NULL                  |
+----+------+-----------------+------+---------+-------+-------+-----------------------+
6 rows in set (0.00 sec)

mysql>

而所有这些含义都可以在mysql官方文档中找到，例如如下

其次应该是解决问题的思路：还是得弄懂本质，能复现，要整理归纳。