解决OutOfMemoryError: unable to create new native thread

问题

生产环境springboot工程卡死，检查日志发现了以下问题：

Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
    at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:84)
    at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:70)
    at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
    at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:84)
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
    at com.github.pagehelper.PageInterceptor.intercept(PageInterceptor.java:143)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
    at com.sun.proxy.$Proxy109.query(Unknown Source)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
    ... 34 common frames omitted
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20
    at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1265)
    at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1086)
    at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4544)
    at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:670)
    at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)
    at com.alibaba.druid.filter.FilterAdapter.dataSource_getConnection(FilterAdapter.java:2723)
    at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)
    at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1064)
    at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1056)
    at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:104)
    at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
    ... 46 common frames omitted
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:717)
    at java.util.Timer.(Timer.java:176)
    at org.postgresql.util.SharedTimer.getTimer(SharedTimer.java:45)
    at org.postgresql.jdbc.PgConnection.getTimer(PgConnection.java:1165)
    at org.postgresql.jdbc.PgConnection.addTimerTask(PgConnection.java:1178)
    at org.postgresql.jdbc.PgStatement.startTimer(PgStatement.java:889)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:429)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303)
    at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289)
    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266)
    at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:233)
    at com.alibaba.druid.pool.vendor.PGValidConnectionChecker.isValidConnection(PGValidConnectionChecker.java:65)
    at com.alibaba.druid.pool.DruidAbstractDataSource.validateConnection(DruidAbstractDataSource.java:1290)
    at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1535)
    at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2100)

分析原因

可能的原因

OutOfMemoryError: unable to create new native thread出现的情况有两种：

1. 服务器剩余内存不足（非JVM内存），不能创建新的线程
   能创建的线程数的具体计算公式如下：

(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads

MaxProcessMemory         指的是一个进程的最大内存
JVMMemory                JVM内存
ReservedOsMemory         保留的操作系统内存
ThreadStackSize          线程栈的大小

2. 超出服务器用户最大进程限制:
   通过以下命令可以查看（注意，不同用户，最大进程限制配置可能不一样）：

ulimit -u

image.png

参考资料：
https://blog.csdn.net/thwsir/article/details/86480956
https://www.cnblogs.com/svennee/p/4331549.html
https://blog.csdn.net/qq171563857/article/details/94590992

分析

通过检查jvm参数，发现没有设置-Xss，使用的默认大小，其次，使用free -m查看服务器可用内存，发现还有很多剩余，怀疑是超出服务器用户最大进程限制，由于需要先修复生产环境，没有过多时间分析定位错误，只保留了日志以及设置了用户的最大进程数，就重启了工程。

事后模拟

1. 在java工程中定义一个http接口，当调用时，不停的创建和启动线程。

@GetMapping("/testThread")
    public void testThread() {
        List list = new ArrayList<>();
        while(true) {
            Thread t = new Thread(() -> {
                try {
                    Thread.sleep(600000L);
                } catch (InterruptedException e) {
                    log.error("###",e);
                }
            });
            t.start();
            list.add(t);
            log.info(String.valueOf(list.size()));
        }
    }

2. 设置服务器java用户的最大进程数量为1000个。
   

   

   image.png

3. 在服务器启动java工程，并调用测试线程接口，成功模拟了生产异常。

image.png

4. 通过pstree查看java工程的线程数量

pstree -p java进程id | wc -l

image.png

修复

在文件/etc/security/limits.d/20-nproc.conf调整用户的最大进程数

*          soft    nproc     4096
java       soft    nproc     1000
root       soft    nproc     unlimited

说明：
root用户无限制
java用户1000个
其他用户4096个