记一次自定义Tomcat ClassLoader问题排查

问题描述

当前开发的项目需要隔离spark环境，因此自定义实现了SparkClassLoader。但是真正打包在服务器上运行的时候，应用需要初始化SparkSession，但是报出了如下错误：

20/04/09 16:57:02 ERROR SparkContext: Error initializing SparkContext.
java.lang.ClassCastException: org.apache.spark.serializer.JavaSerializer cannot be cast to org.apache.spark.serializer.Serializer
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:295)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:187)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.(SparkContext.scala:424)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2523)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
    at org.apache.spark.sql.SparderContext$$anonfun$initSpark$1$$anon$4.run(SparderContext.scala:128)
    at java.lang.Thread.run(Thread.java:748)

解决过程

1. 通过JVM -verbose:class参数查看两个类加载的jar包，看一下是不是包冲突的问题

[Loaded org.apache.spark.serializer.Serializer from file:/root/wangrupeng/spark/jars/spark-core_2.11-2.4.1-os-kylin-r3.jar]
[Loaded org.apache.spark.serializer.JavaSerializer from file:/root/wangrupeng/spark/jars/spark-core_2.11-2.4.1-os-kylin-r3.jar]

结果发现两个类都是从同一个jar包中加载的，排除依赖冲突的原因。

1. 借助阿里巴巴的强力工具Arthas

官方网址

curl -O https://alibaba.github.io/arthas/arthas-boot.jar
java -jar arthas-boot.jar
# 输入项目的进程ID
sc classloader #发现我们自定义的classloader有两个实例
sc -d org.apache.spark.serializer.JavaSerializer # 不出意外，改类被两个classloader实例分别加载了两次

原因找到了，是因为两个类被两个classloader实例加载了两次，然后class cast的时候是两个不同classloader加载的，所以导致了ClassCastException

1. 为什么classloader会初始化两次呢？

由于这个SparkClassLoader是我们自己定义的，所以我在其构造函数中打印了一下Stack信息，这样就能够看到这个类实例的初始化过程了

protected SparkClassLoader(ClassLoader parent) throws IOException {
        super(new URL[] {}, parent);
        init();
        Thread.dumpStack();
    }

最终日志中相关输出如下：

java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1336)
        at org.apache.kylin.spark.classloader.DebugTomcatClassLoader.(DebugTomcatClassLoader.java:75)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.catalina.loader.WebappLoader.createClassLoader(WebappLoader.java:753)
        at org.apache.catalina.loader.WebappLoader.startInternal(WebappLoader.java:598)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5581)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1016)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:992)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2020)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1336)
        at org.apache.kylin.spark.classloader.DebugTomcatClassLoader.(DebugTomcatClassLoader.java:75)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.catalina.loader.WebappLoader.createClassLoader(WebappLoader.java:753)
        at org.apache.catalina.loader.WebappLoader.startInternal(WebappLoader.java:598)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5581)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1016)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:992)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1296)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2038)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

可以看到是Tomcat在部署web实例的时候初始化的ClassLoader实例，然后重点关注，为什么deploy了两次，分别是DeployDirectory和DeployWar各一次，但是通过查看Tomcat官方文档可以知道部署一个web应用这两种方式只会选择一个，但为什么出现了两次？

1. 最终排查

最终发现是因为tomcat/webapp目录下有两个app目录，删掉一个没有用的就可以啦。