环境
com.alibaba.cloud
spring-cloud-starter-alibaba-nacos-config
2.2.0.RELEASE
现象
修改nacos配置文件后，保存后，一段时间后服务cpu告警。
jstack 日志
查看了一下 jstack 日志，有200个线程都是阻塞在同一个接口上，该接口访问量非常高。
可以看到都阻塞到了 LockedScopedProxyFactoryBean 的 invoke 方法
Thread 15906: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83, line=967 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10, line=1283 (Compiled frame)
- java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727 (Compiled frame)
- org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(org.aopalliance.intercept.MethodInvocation) @bci=132, line=494 (Compiled frame)
- org.springframework.aop.framework.ReflectiveMethodInvocation.proceed() @bci=120, line=186 (Compiled frame)
- org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed() @bci=1, line=747 (Compiled frame)
- org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(java.lang.Object, java.lang.reflect.Method, java.lang.Object[], org.springframework.cglib.proxy.MethodProxy) @bci=133, line=689 (Compiled frame)
原因
可以从如下的源码中看到阻塞到了 lock.lock() 方法，这是一个ReadWriteLock 获取readLock并尝试lock。
org.springframework.cloud.context.scope.GenericScope.LockedScopedProxyFactoryBean#invoke
public Object invoke(MethodInvocation invocation) throws Throwable {
Method method = invocation.getMethod();
if (AopUtils.isEqualsMethod(method) || AopUtils.isToStringMethod(method)
|| AopUtils.isHashCodeMethod(method)
|| isScopedObjectGetTargetObject(method)) {
return invocation.proceed();
}
Object proxy = getObject();
ReadWriteLock readWriteLock = this.scope.getLock(this.targetBeanName);
if (readWriteLock == null) {
if (logger.isDebugEnabled()) {
logger.debug("For bean with name [" + this.targetBeanName
+ "] there is no read write lock. Will create a new one to avoid NPE");
}
readWriteLock = new ReentrantReadWriteLock();
}
Lock lock = readWriteLock.readLock();
lock.lock();
try {
if (proxy instanceof Advised) {
Advised advised = (Advised) proxy;
ReflectionUtils.makeAccessible(method);
return ReflectionUtils.invokeMethod(method,
advised.getTargetSource().getTarget(),
invocation.getArguments());
}
return invocation.proceed();
}
// see gh-349. Throw the original exception rather than the
// UndeclaredThrowableException
catch (UndeclaredThrowableException e) {
throw e.getUndeclaredThrowable();
}
finally {
lock.unlock();
}
}
我们知道 ReadWriteLock 如果都是readLock，那不阻塞，可以直接lock。这个阻塞住了，那应该是被此时有个writeLock导致的
从jstack中找到了如下的writeLock 堆栈，可以看到，获取这个writeLock 是由于nacos更新了配置需要销毁这个bean，所以需要获取writeLock 。
org.springframework.cloud.context.scope.GenericScope.destroy()
Thread 9488: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)
- java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock() @bci=5, line=943 (Compiled frame)
- org.springframework.cloud.context.scope.GenericScope.destroy() @bci=69, line=140 (Interpreted frame)
- org.springframework.cloud.context.scope.refresh.RefreshScope.refreshAll() @bci=1, line=154 (Interpreted frame)
- org.springframework.cloud.context.refresh.ContextRefresher.refresh() @bci=9, line=86 (Interpreted frame)
- org.springframework.cloud.endpoint.event.RefreshEventListener.handle(org.springframework.cloud.endpoint.event.RefreshEvent) @bci=44, line=72 (Interpreted frame)
- org.springframework.cloud.endpoint.event.RefreshEventListener.onApplicationEvent(org.springframework.context.ApplicationEvent) @bci=30, line=61 (Interpreted frame)
- org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(org.springframework.context.ApplicationListener, org.springframework.context.ApplicationEvent) @bci=2, line=172 (Compiled frame)
- org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(org.springframework.context.ApplicationListener, org.springframework.context.ApplicationEvent) @bci=34, line=165 (Compiled frame)
- org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(org.springframework.context.ApplicationEvent, org.springframework.core.ResolvableType) @bci=83, line=139 (Compiled frame)
- org.springframework.context.support.AbstractApplicationContext.publishEvent(java.lang.Object, org.springframework.core.ResolvableType) @bci=70, line=403 (Compiled frame)
- org.springframework.context.support.AbstractApplicationContext.publishEvent(org.springframework.context.ApplicationEvent) @bci=3, line=360 (Compiled frame)
- com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1.innerReceive(java.lang.String, java.lang.String, java.lang.String) @bci=34, line=133 (Interpreted frame)
这个writeLock 为啥被阻塞住了，我们知道 writeLock 和 （readLock，writeLock ）都是互斥的，继续从jstack查看，没有发现其他的writeLock ，但找到了一个 readLock ，该readLock是由于Http请求外部服务导致的阻塞。
读写锁阻塞链路
解决方法
由于该bean中写了动态刷新的属性，并添加了@RefreshScope注解，所以该bean在nacos配置更新时需要销毁重新创建。
将该bean的属性单独抽出一个类，该配置类在nacos配置时刷新，并移除该bean的注解@RefreshScope。
总结
跟随nacos刷新的配置最好单独抽出一个类。