jdk11下ClassLoader的踩坑经历

背景

jdk11已发布多年（2018年发布），但在国内的使用情况却并不乐观，这次项目组顶着压力，把之前用到的oraclejdk8升级成openjdk11，期间遇到了不少问题，以下不探讨classloader机制在jdk8和jdk11的差异，根据实际遇到的场景，分析classloader的变化对spring项目的某些影响。

现象

2020-12-30 16:49:03.082 [ForkJoinPool.commonPool-worker-3] INFO  org.apache.hadoop.service.AbstractService - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to load class: [org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl]
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to load class: [org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl]
	at org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(RpcClientFactoryPBImpl.java:68)
	at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getProxy(HadoopYarnProtoRPC.java:48)
	at org.apache.hadoop.yarn.client.RMProxy$1.run(RMProxy.java:165)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1787)
	at org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:161)
	at org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal(ConfiguredRMFailoverProxyProvider.java:77)
	at org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxy(ConfiguredRMFailoverProxyProvider.java:90)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.(RetryInvocationHandler.java:195)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.(RetryInvocationHandler.java:304)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.(RetryInvocationHandler.java:298)
	at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59)
	at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:120)
	at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93)
	at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:210)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at ...
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
	at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
	at org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(RpcClientFactoryPBImpl.java:65)
	... 36 common frames omitted 

环境信息

1. linux： CentOS Linux release 7.4.1708
2. openjdk11：openjdk version “11.0.8” 2020-07-14 LTS
3. springcloud： Hostion

排查过程

1. 包冲突?
   使用 mvn dependency:tree 发现并没有冲突的包
   解压fatjar包，发现lib下也没有冲突的jar包
2. AccessController.doPrivilege 权限未开启？
   本轮发布有精简包的操作，排除掉了部分依赖包；检查了相应policy文件，未发现异常
3. remote debug环境信息
   发现堆栈位置，当前的classloader为 jdk.internal.loader.ClassLoaders.AppClassLoader，正常应该是org.springframework.boot.loader.LaunchedURLClassLoader
   问题定位清楚

根本原因

1. 并发流底层使用ForkJoinPool，在jdk11环境下，ForkJoinPool创建的线程对应的classloader为 ClassLoader.getSystemClassLoader()
2. ClassLoader.getSystemClassLoader()默认返回jdk.internal.loader.ClassLoaders.AppClassLoader
3. jdk.internal.loader.ClassLoaders.AppClassLoader加载的范围是classpath，所以没有找到 fatjar 下面的org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl

解决方案(任选其一)

1. 把并发流调用方式，改成串行流
2. 每次并发流调用前，设置ThreadContextClassLoader，（记得执行完恢复成之前的classloader，在springboot fatjar方式下，LaunchedURLClassLoader作为AppClassLoader，即使没有恢复成appClassLoader，业务上也没有影响，可能会有多种loader并存的怪异现象）
3. 并发流执行时，使用自定义的线程池

其他依赖ForkJoinPool的场景

1. java.util.concurrent.CompletableFuture
2. 主动调用ForkJoinPool的方法

参考

1. java -jar启动机制：https://blog.csdn.net/u012702547/article/details/99543936
2. springboot启动机制：https://blog.csdn.net/kangjnghang/article/details/107046925
3. https://github.com/spring-cloud/spring-cloud-netflix/issues/3101
4. https://www.it1352.com/1839800.html