RocketMQ同一Topic、消费组创建多个消费者失败问题
文章目录
-
- 业务场景
- 问题复现
- 解决方式
- 问题跟踪
业务场景
rocketmq建议一个服务对应一个topic，但是一个服务下会有多个不同的业务消息，同时rocketmq建议不同的业务消息对应不同的tag，当SpringBoot整合RocketMQ时，设置多个消费者发生报错
问题复现
RocketMQ创建多个消费者(同一个消费组)消费同一Topic的不同tag的消息发生报错
2021-05-27 11:10:12.862 ERROR 7636 --- [ main] o.a.r.s.a.ListenerContainerConfiguration : Started container failed. DefaultRocketMQListenerContainer{consumerGroup='MyConsumerGroup', nameServer='10.90.175.160:9876', topic='TestTopic', consumeMode=CONCURRENTLY, selectorType=TAG, selectorExpression='tag0', messageModel=CLUSTERING}
java.lang.IllegalStateException: Failed to start RocketMQ push consumer
at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer.start(DefaultRocketMQListenerContainer.java:281) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
at org.apache.rocketmq.spring.autoconfigure.ListenerContainerConfiguration.registerContainer(ListenerContainerConfiguration.java:120) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
at java.util.HashMap.forEach(HashMap.java:1289) ~[na:1.8.0_222]
at org.apache.rocketmq.spring.autoconfigure.ListenerContainerConfiguration.afterSingletonsInstantiated(ListenerContainerConfiguration.java:79) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:862) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:865) ~[spring-context-5.1.0.RELEASE.jar:5.1.0.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:548) ~[spring-context-5.1.0.RELEASE.jar:5.1.0.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:742) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:389) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:311) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1213) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1202) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at com.roy.rocketmq.RocketMQSBApplication.main(RocketMQSBApplication.java:16) ~[classes/:na]
Caused by: org.apache.rocketmq.client.exception.MQClientException: The consumer group[MyConsumerGroup] has been created before, specify another name please.
See http://rocketmq.apache.org/docs/faq/ for further details.
at org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl.start(DefaultMQPushConsumerImpl.java:634) ~[rocketmq-client-4.7.1.jar:4.7.1]
at org.apache.rocketmq.client.consumer.DefaultMQPushConsumer.start(DefaultMQPushConsumer.java:698) ~[rocketmq-client-4.7.1.jar:4.7.1]
at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer.start(DefaultRocketMQListenerContainer.java:279) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
... 13 common frames omitted
2021-05-27 11:10:12.866 INFO 7636 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2021-05-27 11:10:12.881 INFO 7636 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-05-27 11:10:12.892 ERROR 7636 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.RuntimeException: java.lang.IllegalStateException: Failed to start RocketMQ push consumer
at org.apache.rocketmq.spring.autoconfigure.ListenerContainerConfiguration.registerContainer(ListenerContainerConfiguration.java:123) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
at java.util.HashMap.forEach(HashMap.java:1289) ~[na:1.8.0_222]
at org.apache.rocketmq.spring.autoconfigure.ListenerContainerConfiguration.afterSingletonsInstantiated(ListenerContainerConfiguration.java:79) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:862) ~[spring-beans-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:865) ~[spring-context-5.1.0.RELEASE.jar:5.1.0.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:548) ~[spring-context-5.1.0.RELEASE.jar:5.1.0.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:742) [spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:389) [spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:311) [spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1213) [spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1202) [spring-boot-2.1.6.RELEASE.jar:2.1.6.RELEASE]
at com.roy.rocketmq.RocketMQSBApplication.main(RocketMQSBApplication.java:16) [classes/:na]
Caused by: java.lang.IllegalStateException: Failed to start RocketMQ push consumer
at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer.start(DefaultRocketMQListenerContainer.java:281) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
at org.apache.rocketmq.spring.autoconfigure.ListenerContainerConfiguration.registerContainer(ListenerContainerConfiguration.java:120) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
... 12 common frames omitted
Caused by: org.apache.rocketmq.client.exception.MQClientException: The consumer group[MyConsumerGroup] has been created before, specify another name please.
See http://rocketmq.apache.org/docs/faq/ for further details.
at org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl.start(DefaultMQPushConsumerImpl.java:634) ~[rocketmq-client-4.7.1.jar:4.7.1]
at org.apache.rocketmq.client.consumer.DefaultMQPushConsumer.start(DefaultMQPushConsumer.java:698) ~[rocketmq-client-4.7.1.jar:4.7.1]
at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer.start(DefaultRocketMQListenerContainer.java:279) ~[rocketmq-spring-boot-2.1.1.jar:2.1.1]
... 13 common frames omitted
解决方式
实现RocketMQPushConsumerLifecycleListener接口，重写prepareStart，consumer.setInstanceName(“testTopic-tag0”);设置唯一标识的instanceName即可
@Component
@RocketMQMessageListener(consumerGroup = "MyConsumerGroup",
topic = "TestTopic",
consumeMode= ConsumeMode.CONCURRENTLY,
selectorType = SelectorType.TAG,
selectorExpression = "tag0")
public class Tag0Consumer implements RocketMQListener<String>, RocketMQPushConsumerLifecycleListener {
@Override
public void onMessage(String message) {
System.out.println("Received tag0 message : "+ message);
}
@Override
public void prepareStart(DefaultMQPushConsumer consumer) {
//consumer.setInstanceName("testTopic-tag0");
}
}
@Component
@RocketMQMessageListener(consumerGroup = "MyConsumerGroup",
topic = "TestTopic",
consumeMode= ConsumeMode.CONCURRENTLY,
selectorType = SelectorType.TAG,
selectorExpression = "tag1")
public class Tag0Consumer implements RocketMQListener<String>, RocketMQPushConsumerLifecycleListener {
@Override
public void onMessage(String message) {
System.out.println("Received tag1 message : "+ message);
}
@Override
public void prepareStart(DefaultMQPushConsumer consumer) {
consumer.setInstanceName("testTopic-tag1");
}
}
问题跟踪
发现如果不给consumer指定instanceName，rocketmq就会给此consumer设置一个默认的instanceName，如果有多个消费者都在同一个消费组里，并且不指定instanceName，通过this.consumerTable.putIfAbsent(group, consumer)拿到的消费者对象就会是同一个，就会报错
log.warn(“the consumer group[” + group + “] exist already.”);
throw new MQClientException(“The consumer group[” + this.defaultMQPushConsumer.getConsumerGroup()+ “] has been created before, specify another name please.” + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL), null);
代码片段
DefaultMQPushConsumerImpl类
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown(defaultMQPushConsumer.getAwaitTerminationMillisWhenShutdown());
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
MQClientInstance类
public boolean registerConsumer(final String group, final MQConsumerInner consumer) {
if (null == group || null == consumer) {
return false;
}
MQConsumerInner prev = this.consumerTable.putIfAbsent(group, consumer);
if (prev != null) {
log.warn("the consumer group[" + group + "] exist already.");
return false;
}
return true;
}