升级Spring Cloud Stream动态发送消息问题解析

升级Spring Cloud Stream动态发送消息问题解析

Spring Cloud Stream从2.0.1升级到3.1.3后，事件服务动态刷新发送消息、消费消息都出现了异常。由于动态发送、消费消息功能，是在源码的基础上做了一些封装和修改，新版本Spring Cloud Stream源码发送了变更导致之前的方式失效或出现异常

Unable to register MBean [bean ‘halt-target-test-20’]异常

复现步骤

此异常复现步骤如下

• 找到一个依赖事件服务客户端，并且能正常发送消息的服务，并启动服务，发送一条消息

• 修改事件参数，触发动态刷新发送通道操作

• 再次发送消息会报出Unable to register MBean [bean 'halt-target-test-20']异常

问题分析

根据执行栈分析可能出现问题点

将断点打在抛出异常处查看整个执行栈

我们可以看到红框框着的部分是我们自己的业务代码。点击resolveDestination部分我们可以看到如下内容

public synchronized MessageChannel resolveDestination(String channelName) {
    BindingServiceProperties bindingServiceProperties = this.bindingService.getBindingServiceProperties();
    String[] dynamicDestinations = bindingServiceProperties.getDynamicDestinations();
    boolean dynamicAllowed = ObjectUtils.isEmpty(dynamicDestinations) || ObjectUtils.containsElement(dynamicDestinations, channelName);

    MessageChannel channel;
    try {
        channel = super.resolveDestination(channelName);
    } catch (DestinationResolutionException var9) {
        if (!dynamicAllowed) {
            throw var9;
        }

        channel = (MessageChannel)this.bindingTargetFactory.createOutput(channelName);
        ProducerProperties producerProperties = bindingServiceProperties.getProducerProperties(channelName);
        if (this.newBindingCallback != null) {
            Object extendedProducerProperties = this.bindingService.getExtendedProducerProperties(channel, channelName);
            this.newBindingCallback.configure(channelName, channel, producerProperties, extendedProducerProperties);
        }

        bindingServiceProperties.updateProducerProperties(channelName, producerProperties);
        this.beanFactory.registerSingleton(channelName, channel);
        channel = (MessageChannel)this.beanFactory.initializeBean(channel, channelName);
        Binding binding = this.bindingService.bindProducer(channel, channelName);
        this.dynamicDestinationsBindable.addOutputBinding(channelName, binding);
    }

    return channel;
}

此方法是Spring Cloud Steam中动态获取发送通道的方法，这个方法我们在之前有做过分析，此方法会从上下文中获取已经存在的bean并返回。我们分析整个调用栈，后面在获取bean没有成功之后又去调用了创建bean的逻辑，从这处可以看出我们之前修改配置参数去销毁发送通道并没有彻底，因为正确的逻辑是，我们在页面上修改发送参数后，发送通道销毁，下次发送消息时在上述channel = super.resolveDestination(channelName);处应该是找不到bean直接进入到异常处理进行新的发送通道的创建。但我们看到的实际情况是在调用super.resolveDestination(channelName)方法里面，没有找到bean但却继续去创建bean去了，因此第一个需要研究的点便是，为什么会去执行bean的创建操作，而不是直接返回或抛出异常，更近一步我们需要知道我们销毁通道的时候少销毁了什么，导致没有按预期执行。之后我们继续分析后面的执行方法，在创建bean之后执行了IntegrationMBeanExporter的postProcessAfterInitialization方法

public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
    if (this.singletonsInstantiated) {
        try {
            if (bean instanceof MessageChannel) {
                MessageChannel monitor = (MessageChannel) extractTarget(bean);
                if (monitor instanceof IntegrationManagement) {
                    this.channels.put(beanName, (IntegrationManagement) monitor);
                    registerChannel((IntegrationManagement) monitor);
                    this.runtimeBeans.add(bean);
                }
            }
            else if (bean instanceof MessageProducer && bean instanceof Lifecycle) {
                registerProducer((MessageProducer) bean);
                this.runtimeBeans.add(bean);
            }
            else if (bean instanceof AbstractEndpoint) {
                postProcessAbstractEndpoint(bean);
            }
        }
        catch (Exception e) {
            logger.error("Could not register an MBean for: " + beanName, e);
        }
    }
    return bean;
}

此方法是bean初始化完成之后执行的方法，结合类名称IntegrationMBeanExporter，可以大概猜出这里是在处理监控Bean相关的操作，根据后面执行的方法名可以大概猜测出是在执行bean的监控注册，根据抛出异常的描述可以看出是注册的时候发生了异常，因此第二个要研究清楚的问题是为什么注册MBean会报错

研究问题点搞清楚问题具体原因

上面是根据执行栈猜测出的两个问题点，第一个点销毁通道的时候少销毁了什么，导致没有按预期执行；第二个点为什么注册MBean会报错。此处代码已经执行到异常处，我们要重新执行registerBeanNameOrInstance方法看具体原因。选择执行栈中的registerBeanNameOrInstance，点击"Drop Frame"重新进入registerBeanNameOrInstance

protected ObjectName registerBeanNameOrInstance(Object mapValue, String beanKey) throws MBeanExportException {
    try {
        if (mapValue instanceof String) {
            // Bean name pointing to a potentially lazy-init bean in the factory.
            if (this.beanFactory == null) {
                throw new MBeanExportException("Cannot resolve bean names if not running in a BeanFactory");
            }
            String beanName = (String) mapValue;
            if (isBeanDefinitionLazyInit(this.beanFactory, beanName)) {
                ObjectName objectName = registerLazyInit(beanName, beanKey);
                replaceNotificationListenerBeanNameKeysIfNecessary(beanName, objectName);
                return objectName;
            }
            else {
                Object bean = this.beanFactory.getBean(beanName);
                ObjectName objectName = registerBeanInstance(bean, beanKey);
                replaceNotificationListenerBeanNameKeysIfNecessary(beanName, objectName);
                return objectName;
            }
        }
        else {
            // Plain bean instance -> register it directly.
            if (this.beanFactory != null) {
                Map beansOfSameType =
                        this.beanFactory.getBeansOfType(mapValue.getClass(), false, this.allowEagerInit);
                for (Map.Entry entry : beansOfSameType.entrySet()) {
                    if (entry.getValue() == mapValue) {
                        String beanName = entry.getKey();
                        ObjectName objectName = registerBeanInstance(mapValue, beanKey);
                        replaceNotificationListenerBeanNameKeysIfNecessary(beanName, objectName);
                        return objectName;
                    }
                }
            }
            return registerBeanInstance(mapValue, beanKey);
        }
    }
    catch (Throwable ex) {
        throw new UnableToRegisterMBeanException(
                "Unable to register MBean [" + mapValue + "] with key '" + beanKey + "'", ex);
    }
}

根据调试可以发现代码具体报错地方是调用registerBeanInstance方法，调用此方法的代码块主要做的事情如下

• 判断beanFactory属性是否为空

• 不为空获取需要注入bean类型的bean集合

• 如果两个bean相同则进入registerBeanInstance方法

由上述执行流程可知我们在修改发送参数后，销毁发送通道bean，但并没有销毁监控对应的MBean。继续跟踪代码我们可以发现报错是在doRegister方法中抛出

protected void doRegister(Object mbean, ObjectName objectName) throws JMException {
    Assert.state(this.server != null, "No MBeanServer set");
    ObjectName actualObjectName;

    synchronized (this.registeredBeans) {
        ObjectInstance registeredBean = null;
        try {
            registeredBean = this.server.registerMBean(mbean, objectName);
        }
        catch (InstanceAlreadyExistsException ex) {
            if (this.registrationPolicy == RegistrationPolicy.IGNORE_EXISTING) {
                if (logger.isDebugEnabled()) {
                    logger.debug("Ignoring existing MBean at [" + objectName + "]");
                }
            }
            else if (this.registrationPolicy == RegistrationPolicy.REPLACE_EXISTING) {
                try {
                    if (logger.isDebugEnabled()) {
                        logger.debug("Replacing existing MBean at [" + objectName + "]");
                    }
                    this.server.unregisterMBean(objectName);
                    registeredBean = this.server.registerMBean(mbean, objectName);
                }
                catch (InstanceNotFoundException ex2) {
                    if (logger.isInfoEnabled()) {
                        logger.info("Unable to replace existing MBean at [" + objectName + "]", ex2);
                    }
                    throw ex;
                }
            }
            else {
                throw ex;
            }
        }

        // Track registration and notify listeners.
        actualObjectName = (registeredBean != null ? registeredBean.getObjectName() : null);
        if (actualObjectName == null) {
            actualObjectName = objectName;
        }
        this.registeredBeans.add(actualObjectName);
    }

    onRegister(actualObjectName, mbean);
}

此方法完成如下事情

• 调用sever属性的registerMBean方法完成MBean的注册

• 异常处理，检查注册策略，如果忽略已经注册则打印日志

• 异常处理，如果是替换现有的，则先移除注册的bean，再从新注册

• 异常处理，如果两种策略都不是则抛出异常

由上述方法以及调试结果可知，默认是直接抛出异常。我们可以配置策略使其不抛出异常，但策略改动可能会影响其他客户端，并且也会打印日志。也不是最优解决方案。但我们注意到server属性的unregisterMBean方法完成了MBean的移除注册，所以我们完全可以在我们修该发送参数，销毁发送通道时将对应MBean也删除。我们需要研究server属性的unregisterMBean如何可以在我们移除销毁通道时获取并调用

我们尝试找一下其他调用server属性unregisterMBean方法的地方，我们发现就在这个类中有一个doUnregister方法在调用

protected void doUnregister(ObjectName objectName) {
    Assert.state(this.server != null, "No MBeanServer set");
    boolean actuallyUnregistered = false;

    synchronized (this.registeredBeans) {
        if (this.registeredBeans.remove(objectName)) {
            try {
                // MBean might already have been unregistered by an external process
                if (this.server.isRegistered(objectName)) {
                    this.server.unregisterMBean(objectName);
                    actuallyUnregistered = true;
                }
                else {
                    if (logger.isInfoEnabled()) {
                        logger.info("Could not unregister MBean [" + objectName + "] as said MBean " +
                                "is not registered (perhaps already unregistered by an external process)");
                    }
                }
            }
            catch (JMException ex) {
                if (logger.isInfoEnabled()) {
                    logger.info("Could not unregister MBean [" + objectName + "]", ex);
                }
            }
        }
    }

    if (actuallyUnregistered) {
        onUnregister(objectName);
    }
}

我们发现其实这个方法主要就是在完成移除MBean对象的注册，此方法是protected，继续查看调用doUnregister方法的地方，我们发现postProcessBeforeDestruction方法在调用doUnregister，并且这个方法是公共的，传入参数bean及bean的名称。包含这个方法的bean我们可以直接注入使用，传入我们发送通道的bean及名称此方法将完成bean对应监听资源的释放

解决方案

由上面问题分析可知报错出现的原因是在销毁发送通道是没有销毁监听MBean导致的，我们需要注入IntegrationMBeanExporter类型的bean，并调用postProcessBeforeDestruction完成销毁发送通道的监控资源的释放

/**
 * 添加解绑定接口
 *
 * @param channelName
 */
public void unbindOutput(String channelName) {
    if (this.dynamicDestinationsBindable.getOutputs().contains(channelName)) {
        this.dynamicDestinationsBindable.unbindOutput(channelName);
        try {
            if (Objects.nonNull(integrationMBeanExporter)) {
                integrationMBeanExporter.postProcessBeforeDestruction(super.resolveDestination(channelName), channelName);
            }
        } catch (Exception e) {
            LOGGER.info("JMX Bean destroy fail");
        }
        ((DefaultListableBeanFactory) this.beanFactory).destroySingleton(channelName);
        ((DefaultListableBeanFactory) this.beanFactory).removeBeanDefinition(channelName);
    }
}

修改发送参数之后没有生效

问题分析

根据上面分析我们目前解决了一个问题点，还有另一个问题没有解决。销毁通道的时候少销毁了什么，导致没有按预期执行。根据上面的执行栈我们找到doGetBean方法

protected  T doGetBean(
        String name, @Nullable Class requiredType, @Nullable Object[] args, boolean typeCheckOnly)
        throws BeansException {

    String beanName = transformedBeanName(name);
    Object beanInstance;

    // Eagerly check singleton cache for manually registered singletons.
    Object sharedInstance = getSingleton(beanName);
    if (sharedInstance != null && args == null) {
        if (logger.isTraceEnabled()) {
            if (isSingletonCurrentlyInCreation(beanName)) {
                logger.trace("Returning eagerly cached instance of singleton bean '" + beanName +
                        "' that is not fully initialized yet - a consequence of a circular reference");
            }
            else {
                logger.trace("Returning cached instance of singleton bean '" + beanName + "'");
            }
        }
        beanInstance = getObjectForBeanInstance(sharedInstance, name, beanName, null);
    }

    else {
        // Fail if we're already creating this bean instance:
        // We're assumably within a circular reference.
        if (isPrototypeCurrentlyInCreation(beanName)) {
            throw new BeanCurrentlyInCreationException(beanName);
        }

        // Check if bean definition exists in this factory.
        BeanFactory parentBeanFactory = getParentBeanFactory();
        if (parentBeanFactory != null && !containsBeanDefinition(beanName)) {
            // Not found -> check parent.
            String nameToLookup = originalBeanName(name);
            if (parentBeanFactory instanceof AbstractBeanFactory) {
                return ((AbstractBeanFactory) parentBeanFactory).doGetBean(
                        nameToLookup, requiredType, args, typeCheckOnly);
            }
            else if (args != null) {
                // Delegation to parent with explicit args.
                return (T) parentBeanFactory.getBean(nameToLookup, args);
            }
            else if (requiredType != null) {
                // No args -> delegate to standard getBean method.
                return parentBeanFactory.getBean(nameToLookup, requiredType);
            }
            else {
                return (T) parentBeanFactory.getBean(nameToLookup);
            }
        }

        if (!typeCheckOnly) {
            markBeanAsCreated(beanName);
        }

        StartupStep beanCreation = this.applicationStartup.start("spring.beans.instantiate")
                .tag("beanName", name);
        try {
            if (requiredType != null) {
                beanCreation.tag("beanType", requiredType::toString);
            }
            RootBeanDefinition mbd = getMergedLocalBeanDefinition(beanName);
            checkMergedBeanDefinition(mbd, beanName, args);

            // Guarantee initialization of beans that the current bean depends on.
            String[] dependsOn = mbd.getDependsOn();
            if (dependsOn != null) {
                for (String dep : dependsOn) {
                    if (isDependent(beanName, dep)) {
                        throw new BeanCreationException(mbd.getResourceDescription(), beanName,
                                "Circular depends-on relationship between '" + beanName + "' and '" + dep + "'");
                    }
                    registerDependentBean(dep, beanName);
                    try {
                        getBean(dep);
                    }
                    catch (NoSuchBeanDefinitionException ex) {
                        throw new BeanCreationException(mbd.getResourceDescription(), beanName,
                                "'" + beanName + "' depends on missing bean '" + dep + "'", ex);
                    }
                }
            }

            // Create bean instance.
            if (mbd.isSingleton()) {
                sharedInstance = getSingleton(beanName, () -> {
                    try {
                        return createBean(beanName, mbd, args);
                    }
                    catch (BeansException ex) {
                        // Explicitly remove instance from singleton cache: It might have been put there
                        // eagerly by the creation process, to allow for circular reference resolution.
                        // Also remove any beans that received a temporary reference to the bean.
                        destroySingleton(beanName);
                        throw ex;
                    }
                });
                beanInstance = getObjectForBeanInstance(sharedInstance, name, beanName, mbd);
            }

            else if (mbd.isPrototype()) {
                // It's a prototype -> create a new instance.
                Object prototypeInstance = null;
                try {
                    beforePrototypeCreation(beanName);
                    prototypeInstance = createBean(beanName, mbd, args);
                }
                finally {
                    afterPrototypeCreation(beanName);
                }
                beanInstance = getObjectForBeanInstance(prototypeInstance, name, beanName, mbd);
            }

            else {
                String scopeName = mbd.getScope();
                if (!StringUtils.hasLength(scopeName)) {
                    throw new IllegalStateException("No scope name defined for bean ´" + beanName + "'");
                }
                Scope scope = this.scopes.get(scopeName);
                if (scope == null) {
                    throw new IllegalStateException("No Scope registered for scope name '" + scopeName + "'");
                }
                try {
                    Object scopedInstance = scope.get(beanName, () -> {
                        beforePrototypeCreation(beanName);
                        try {
                            return createBean(beanName, mbd, args);
                        }
                        finally {
                            afterPrototypeCreation(beanName);
                        }
                    });
                    beanInstance = getObjectForBeanInstance(scopedInstance, name, beanName, mbd);
                }
                catch (IllegalStateException ex) {
                    throw new ScopeNotActiveException(beanName, scopeName, ex);
                }
            }
        }
        catch (BeansException ex) {
            beanCreation.tag("exception", ex.getClass().toString());
            beanCreation.tag("message", String.valueOf(ex.getMessage()));
            cleanupAfterBeanCreationFailure(beanName);
            throw ex;
        }
        finally {
            beanCreation.end();
        }
    }

    return adaptBeanInstance(name, beanInstance, requiredType);
}

此方法是Spring IOC容器相关源码，如果对IOC容器有过研究应该对这个方法所做的事情会比较了解。这也是因为升级了Spring Cloud的版本导致此方法的实现相较于2.0.1的版本有了变化。我们来分析一下这个方法的代码

• 首先获取bean的转换后的名称，比如别名处理

• 调用getSingleton方法获取单例对象

• 判断获取的单例对象是否为空

• 不为空，通过getObjectForBeanInstance获取bean实例的对象

• 为空，判断父bean不为空，并且当前bean的定义中不包含该bean，则尝试从父bean中获取需要的bean

• 如果当前bean的定义中包含该bean，则获取bean的定义，并且根据bean的定义来创建bean

• 后续bean的处理

上面简单分析了一下这个方法都在干什么事情，我们根据执行栈，可以看出，我们发送通道的bean已经被销毁，但我们bean的定义并没有被移除，所以通过bean的定义又去创建了上一个发送通道的对象，导致我们修改参数之后的发送通道并没有去创建。因此没有生效。所以我们需要将bean的定义也一起移除才行。通过上面分析我们知道containsBeanDefinition方法就在判断是否包含bean的定义

@Override
public boolean containsBeanDefinition(String beanName) {
    Assert.notNull(beanName, "Bean name must not be null");
    return this.beanDefinitionMap.containsKey(beanName);
}

该方法直接判断beanDefinitionMap属性中是否有名称为给定bean名称的定义对象。所以我们需要在销毁发送通道的时候将beanDefinitionMap中的定义对象也一起移除掉。我们发现该类也就是DefaultListableBeanFactory中有一个removeBeanDefinition的方法

public void removeBeanDefinition(String beanName) throws NoSuchBeanDefinitionException {
    Assert.hasText(beanName, "'beanName' must not be empty");

    BeanDefinition bd = this.beanDefinitionMap.remove(beanName);
    if (bd == null) {
        if (logger.isTraceEnabled()) {
            logger.trace("No bean named '" + beanName + "' found in " + this);
        }
        throw new NoSuchBeanDefinitionException(beanName);
    }

    if (hasBeanCreationStarted()) {
        // Cannot modify startup-time collection elements anymore (for stable iteration)
        synchronized (this.beanDefinitionMap) {
            List updatedDefinitions = new ArrayList<>(this.beanDefinitionNames);
            updatedDefinitions.remove(beanName);
            this.beanDefinitionNames = updatedDefinitions;
        }
    }
    else {
        // Still in startup registration phase
        this.beanDefinitionNames.remove(beanName);
    }
    this.frozenBeanDefinitionNames = null;

    resetBeanDefinition(beanName);
}

此方法完成了对bean定义的移除及后续操作。那我们要如何调用此方法呢？根据之前的分析文档我们知道，BinderAwareChannelResolver的beanFactory属性就是DefaultListableBeanFactory的实例，而我们创建的发送通道的bean也是注册在beanFactory属性中的。之前我们调用destroySingleton方法完成了bean的销毁，但没有调用removeBeanDefinition完成bean定义的销毁

解决方案

根据上面分析我们可以知道解决方案就是在销毁发送bean的时候需要同时销毁其定义

/**
 * 添加解绑定接口
 *
 * @param channelName
 */
public void unbindOutput(String channelName) {
    if (this.dynamicDestinationsBindable.getOutputs().contains(channelName)) {
        this.dynamicDestinationsBindable.unbindOutput(channelName);
        try {
            if (Objects.nonNull(integrationMBeanExporter)) {
                integrationMBeanExporter.postProcessBeforeDestruction(super.resolveDestination(channelName), channelName);
            }
        } catch (Exception e) {
            LOGGER.info("JMX Bean destroy fail");
        }
        ((DefaultListableBeanFactory) this.beanFactory).destroySingleton(channelName);
        ((DefaultListableBeanFactory) this.beanFactory).removeBeanDefinition(channelName);
    }
}

结语

一切尽在源码中