02 导入 spring-cloud-starter-zipkin 之后系统启动卡住
前言
呵呵 最近出现了一个这样的问题
系统添加了一个 spring-cloud-starter-zipkin 的依赖之后, 直接 启动不了, 卡在了
2020-07-18 09:13:06.390 INFO [,,,] 1088 [main] io.lettuce.core.EpollProvider : Starting without optional epoll library
2020-07-18 09:13:06.413 INFO [,,,] 1088 [main] io.lettuce.core.KqueueProvider : Starting with kqueue library当然 最后排查出来是 造成了死锁, 这个排查的过程 也是很费劲的
主要的流程大致是 main 线程在获取 redis 连接的时候 park 住了自己 并且持有 spring 容器里面的 singletonObjects 的锁, 并且 main线程 需要由另外一个线程 lettuce-kqueueEventLoop-4-1 来唤醒, 然后另外一个线程 lettuce-kqueueEventLoop-4-1, 在向 redis 服务器发送 auth 请求的时候, 由于 TraceRedisAutoConfiguration 配置了 BraveTracing, BraveTracing 需要 scopedTarget.defaultTraceSampler, scopedTarget.defaultTraceSampler 的创建依赖于 org.springframework.cloud.sleuth.sampler.SamplerAutoConfiguration$RefreshScopedSamplerConfiguration, 需要获取 spring 容器里面的 singletonObjects 的锁
总结一下 : main 线程持有 singletonObjects 的锁, 并且等待 lettuce-kqueueEventLoop-4-1 来 unpark, lettuce-kqueueEventLoop-4-1 线程需要 singletonObjects 的锁, 所以 就造成了死锁
测试用例
首先是一个 LockSupport 相关 api 的测试用例
执行发现 程序卡死了, 因为 LockSupport.park 的时候 没有释放持有的锁资源
/**
* CHMDeadLock
*
* @author Jerry.X.He <[email protected]>
* @version 1.0
* @date 2020-07-16 17:58
*/
public class Test13CHMDeadLock02 {
public static void main(String[] args) throws Exception {
Thread mainThread = Thread.currentThread();
Map map = new ConcurrentHashMap<>();
synchronized (map) {
new Thread(() -> {
System.out.println(" before ");
synchronized (map) {
map.get("12");
LockSupport.unpark(mainThread);
}
System.out.println(" after ");
}).start();
Thread.sleep(1000);
// LockSupport.park();
LockSupport.park(map);
}
}
}
再来看一下通过 wait, notify 的同样的情况
程序最终 正常完成了
/**
* CHMDeadLock
*
* @author Jerry.X.He <[email protected]>
* @version 1.0
* @date 2020-07-16 17:58
*/
public class Test13CHMDeadLock {
public static void main(String[] args) throws Exception {
Map map = new ConcurrentHashMap<>();
synchronized (map) {
new Thread(() -> {
System.out.println(" before ");
synchronized (map) {
map.get("12");
map.notify();
}
System.out.println(" after ");
}).start();
Thread.sleep(1000);
map.wait();
}
}
}
回到场景
回到我们这个场景的问题, 首先 jstack 查看一下 各个线程的具体的情况
然后 排除掉各个 runnable 的线程之后如下, 我们会发现 各个线程之间有关联的只有 lettuce-kqueueEventLoop-4-1 和 main
lettuce-kqueueEventLoop-4-1 等待的是oop 0x00000007407627d8, 是一个 ConcurrentHashMap, 结合具体的代码可以得知 这个 oop 是 spring 容器的 singletonObjects
main 线程 park 住了, blocker 是oop 0x00000007bfa548c8(一个 CompletableFuture$Signaller实例), 但是 它还持有 0x00000007407627d8 的锁
master:~ jerry$ jstack 1088
2020-07-18 09:16:08
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.211-b12 mixed mode):
"lettuce-kqueueEventLoop-4-1" #55 daemon prio=5 os_prio=31 tid=0x00007f92f8df1000 nid=0x9d03 waiting for monitor entry [0x000070000c022000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:205)
- waiting to lock <0x00000007407627d8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:409)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:359)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$977/615264607.getObject(Unknown Source)
at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:389)
- locked <0x0000000741552408> (a java.lang.String)
at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:356)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at brave.sampler.Sampler$$EnhancerBySpringCGLIB$$bec95542.isSampled()
at brave.Tracer.decorateContext(Tracer.java:279)
at brave.Tracer.newRootContext(Tracer.java:215)
at brave.Tracer.newTrace(Tracer.java:150)
at brave.Tracer.nextSpan(Tracer.java:502)
at io.lettuce.core.tracing.BraveTracing$BraveTracer.nextSpan(BraveTracing.java:259)
at io.lettuce.core.tracing.BraveTracing$BraveTracer.nextSpan(BraveTracing.java:266)
at io.lettuce.core.protocol.CommandHandler.writeSingleCommand(CommandHandler.java:392)
at io.lettuce.core.protocol.CommandHandler.write(CommandHandler.java:353)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:762)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:788)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:806)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:294)
at io.lettuce.core.protocol.DefaultEndpoint.channelWriteAndFlush(DefaultEndpoint.java:342)
at io.lettuce.core.protocol.DefaultEndpoint.writeToChannelAndFlush(DefaultEndpoint.java:282)
at io.lettuce.core.protocol.DefaultEndpoint.write(DefaultEndpoint.java:142)
at io.lettuce.core.protocol.CommandExpiryWriter.write(CommandExpiryWriter.java:112)
at io.lettuce.core.RedisChannelHandler.dispatch(RedisChannelHandler.java:184)
at io.lettuce.core.StatefulRedisConnectionImpl.dispatch(StatefulRedisConnectionImpl.java:152)
at io.lettuce.core.AbstractRedisAsyncCommands.dispatch(AbstractRedisAsyncCommands.java:467)
at io.lettuce.core.AbstractRedisAsyncCommands.dispatch(AbstractRedisAsyncCommands.java:454)
at io.lettuce.core.RedisClient.lambda$connectStatefulAsync$2(RedisClient.java:302)
at io.lettuce.core.RedisClient$$Lambda$962/878796532.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at io.lettuce.core.AbstractRedisClient.lambda$null$3(AbstractRedisClient.java:340)
at io.lettuce.core.AbstractRedisClient$$Lambda$967/316439120.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at io.lettuce.core.PlainChannelInitializer$1.userEventTriggered(PlainChannelInitializer.java:93)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:344)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:330)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:322)
at io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:344)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:330)
at io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913)
at io.lettuce.core.protocol.CommandHandler.lambda$channelActive$0(CommandHandler.java:281)
at io.lettuce.core.protocol.CommandHandler$$Lambda$971/1887070116.run(Unknown Source)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:293)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"container-0" #54 prio=5 os_prio=31 tid=0x00007f92fbe8c800 nid=0x6103 waiting on condition [0x000070000bf21000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:570)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run(TomcatWebServer.java:181)
"Catalina-utility-2" #53 prio=1 os_prio=31 tid=0x00007f92f8e11800 nid=0x9f2f waiting on condition [0x000070000be1e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007bf1ee8b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"RMI Scheduler(0)" #18 daemon prio=5 os_prio=31 tid=0x00007f92fcb06800 nid=0xa303 waiting on condition [0x000070000ba12000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074000d590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Attach Listener" #15 daemon prio=9 os_prio=31 tid=0x00007f92f9b17000 nid=0xa503 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #11 daemon prio=9 os_prio=31 tid=0x00007f92f89db000 nid=0x5503 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007f92f89da800 nid=0x3c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007f92f89d9800 nid=0x3f03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007f92f9afb000 nid=0x3a03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f92fc806000 nid=0x3203 in Object.wait() [0x000070000aae5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000074000d918> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000074000d918> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f92fc000000 nid=0x4d03 in Object.wait() [0x000070000a9e2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000740017620> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000740017620> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=31 tid=0x00007f92f8804000 nid=0x2703 waiting on condition [0x0000700009fc3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007bfa548c8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at io.lettuce.core.DefaultConnectionFuture.get(DefaultConnectionFuture.java:68)
at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:227)
at io.lettuce.core.RedisClient.connect(RedisClient.java:207)
at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.lambda$getConnection$1(StandaloneConnectionProvider.java:115)
at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider$$Lambda$943/449946392.get(Unknown Source)
at java.util.Optional.orElseGet(Optional.java:267)
at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.getConnection(StandaloneConnectionProvider.java:115)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1197)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1178)
- locked <0x00000007bf60d690> (a java.lang.Object)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:942)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:353)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:132)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:95)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:82)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:215)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
at org.springframework.data.redis.core.RedisTemplate.delete(RedisTemplate.java:713)
at com.spring.docker.component.InitializingBaseData.cacheDefaultTemplate(InitializingBaseData.java:82)
at com.spring.docker.component.InitializingBaseData.initializeDefaultTemplate(InitializingBaseData.java:44)
at com.spring.docker.component.InitializingBaseData.afterPropertiesSet(InitializingBaseData.java:36)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$167/2093319848.getObject(Unknown Source)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
- locked <0x00000007407627d8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
- locked <0x000000074095bfd0> (a java.lang.Object)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
at com.spring.dockerApplication.main(dockerApplication.java:20)
"VM Periodic Task Thread" os_prio=31 tid=0x00007f92f91b9800 nid=0xa603 waiting on condition
JNI global references: 40342
我们再看一下 这个 0x00000007407627d8
具体的原因
这个过程 就有点麻烦了
通过对比 是否 添加这个依赖来对比发现
不添加 spring-cloud-starter-zipkin 的时候 tracingEnabled 是 false, 添加了依赖之后 是 true, 走了里面的 "tracer.nextSpan(context)"
然后 里面去 spring 容器里面拿东西, 然后需要获取 singletonObjects 的锁
看一下这个 this.tracingEnabled = tracing.isEnabled();
tracingEnabled 来自于 tracing
然后 在对比一下 两种情况下的 tracing 分别是什么?
不加依赖的情况, tracing 是一个 NoOpTracing, 看名字就知道这是一个 工具实现
添加依赖的情况下 tracing 是一个 BraveTracing$BraveEndpoint 的实例
从代码中得知 BraveTracing$BraveEndpoint 来自于 BraveTracing
然后查看一下 BraveTracing 调用的地方, 发现有一个 TraceLettuceClientResourcesBeanPostProcessor, 他的实例是由 TraceRedisAutoConfiguration 创建的
TraceLettuceClientResourcesBeanPostProcessor, 的实例是由 TraceRedisAutoConfiguration 创建
然后你去掉 spring-cloud-starter-zipkin 依赖之后, 你会发现 没得这个 TraceRedisAutoConfiguration
加上这个依赖之后, TraceRedisAutoConfiguration 生效了, 然后 符合条件, 就创建了 TraceLettuceClientResourcesBeanPostProcessor
然后对 redis 的 ClientResource 做了拦截, 如果没有启动追踪, 则强制添加了一个 BraveTracing
所以看到这里, 你应该能够明白问题发生在哪里了吧?, 然后在结合你的场景, 来讨论解决方案
另外还可以回头再回顾一下上面的这个测试用例, 也是一个知识点
完