记录一次由于 liquibase导致的内存溢出问题及解决方案

背景

我们的程序每当执行20天左右就会出现内存溢出的情况，或者很卡。终于有一天是在忍受不了20天就重启的麻烦是了，就开始了堆栈分析

1. 线上分析

# 找到出现问题的程序
ps -ef|grep java

# 查看特定程序的线程情况
top -p 2075605 -H
 
# 线程详情， nid 特定线程的16进制ID
jstack 2075605 | grep -A 100 nid=0x1fabd9

# 堆详情
jmap -heap 2075605

# 对象列表
jmap -histo 2075605
# 经过以上操作没有分析出来，那就只能dump heap 文件进行分析了
# 生成 heap dump 文件
jmap -dump:live,format=b,file=heap.hprof 2075605

2. 线下堆栈分析

使用 MAT 打开，很明显就能看到是 liquibase 里面的 ExecutorService 出了问题
可是我们的程序已经关闭了 liquibase的自动执行了呀，说明有别的调用者

就开始分析源码

public class ExecutorService {

    private static ExecutorService instance = new ExecutorService();
	// 这里面就只有这玩意会出现内存溢出的情况
    private Map<Database, Executor> executors = new ConcurrentHashMap<Database, Executor>();


    private ExecutorService() {
    }

    public static ExecutorService getInstance() {
        return instance;
    }

    public Executor getExecutor(Database database) {
        if (!executors.containsKey(database)) {
            try {
                Executor executor = (Executor) ServiceLocator.getInstance().newInstance(Executor.class);
                executor.setDatabase(database);
                executors.put(database, executor);
            } catch (Exception e) {
                throw new UnexpectedLiquibaseException(e);
            }
        }
        return executors.get(database);
    }

    public void setExecutor(Database database, Executor executor) {
        executors.put(database, executor);
    }

    public void clearExecutor(Database database) {
        executors.remove(database);
    }

    public void reset() {
        executors.clear();
    }
}

3. 添加线上日志

分析线下调用实在太多，只能采用 拦截器 加日志，看看到底是谁调用
结合线上的异常日志分析

	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2482)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2440)
	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1381)
	at net.sf.log4jdbc.sql.jdbcapi.StatementSpy.executeQuery(StatementSpy.java:731)
	at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:111)
	at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
	at liquibase.executor.jvm.JdbcExecutor$QueryStatementCallback.doInStatement(JdbcExecutor.java:345)
	at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:55)
	at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:126)
	at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:134)
	at liquibase.executor.jvm.JdbcExecutor.queryForList(JdbcExecutor.java:200)
	at liquibase.executor.jvm.JdbcExecutor.queryForList(JdbcExecutor.java:194)
	at liquibase.changelog.StandardChangeLogHistoryService.queryDatabaseChangeLogTable(StandardChangeLogHistoryService.java:317)
	at org.springframework.boot.actuate.endpoint.LiquibaseEndpoint.invoke(LiquibaseEndpoint.java:76)
	at org.springframework.boot.actuate.endpoint.LiquibaseEndpoint.invoke(LiquibaseEndpoint.java:43)
	at org.springframework.boot.actuate.endpoint.jmx.DataEndpointMBean.getData(DataEndpointMBean.java:46)
	at sun.reflect.GeneratedMethodAccessor276.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
	at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1252)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1246)
	at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1085)
	at org.springframework.jmx.export.SpringModelMBean.invoke(SpringModelMBean.java:90)
	at javax.management.modelmbean.RequiredModelMBean.getAttribute(RequiredModelMBean.java:1562)
	at org.springframework.jmx.export.SpringModelMBean.getAttribute(SpringModelMBean.java:109)
	at javax.management.modelmbean.RequiredModelMBean.getAttributes(RequiredModelMBean.java:1801)
	at org.springframework.jmx.export.SpringModelMBean.getAttributes(SpringModelMBean.java:126)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:709)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705)
	at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:151)

4. 问题原因及解决办法

发现
prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:151)
会定时的调用 spring actuator 的 liquibase endpoint .
好了，最终定位到原因了，修改 spring(1.5.12) 的配置文件，18的版本，不一定适合你们

endpoints:
    enabled: true
    sensitive: false
# 添加的配置
    liquibase:
        enabled: false