com.alibaba.dubbo.remoting.RemotingException: Not found exported service

问题信息

问题描述

服务调用方同学反馈，XX应用调用我们对外提供的库存查询服务，找不到服务
···

错误信息

调用方输出的错误日志信息

[2019-01-24 16:46:43,985] INFO -:-:-:- [main] com.wwwarehouse.commons.dubbo.DubboConsumerAuthFilter.invoke(DubboConsumerAuthFilter.java:51) - dubboRpcContext:>>>
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method compositeQuery in the service com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService. Tried 3 times of the providers [192.168.72.158:20880] (1/1) from the registry 192.168.6.21:2181 on the consumer 192.168.72.158 using the dubbo version 2.8.4. Last error is: Failed to invoke remote method: compositeQuery, provider: dubbo://192.168.72.158:20880/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService?anyhost=true&application=commonindustry-biz-test&batchChangeInventory.timeout=60000&batchChangeInventoryQty.timeout=60000&changeInventory.timeout=600000&check=false&default.check=false&default.connections=10&default.group=mock&dubbo=2.8.4&eliminatePlanAdjustmentDatas.timeout=60000&generic=false&group=impl2&interface=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService&lockPlanAdjustmentDatas.timeout=5000&methods=selectSubItemsByGroupUkidAndItemUkid,queryConsumeRealityQTY,queryDeliveryFromStorageQTY,queryMaxQtyGroupForItem,unLockPlanAdjustmentDatas,selectAvailableListByGroupRelationUkid,inventoryQuery,selectItemsByGroupUkidAndIdentifyCode,compositeQuery,lockPlanAdjustmentDatas,getInventoryQueryFilterInfo,batchChangeInventoryQty,listByUkids,updateInventoryStatistics,sumInventoryQty,changeInventoryQty,changeInventory,deliveryFromStorage,selectQualitiesByGroupUkidAndItemUkid,batchChangeInventory,getGoodsStockInfo,warehouseEntry,selectConsumeListByIdentifyCode,addInventory,selectConsumeListByGroupUkid,selectConsumeListByGroupUkids,getInventoryQueryList,getStorageGoodsStock,selectConsumeListByGroupCode,eliminatePlanAdjustmentDatas&organization=wwwarehouse&owner=xdw&pid=17332&protocol=dubbo&revision=1.0-20190124.081517-966&side=consumer×tamp=1548319602539&updateInventoryStatistics.timeout=5000, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
 at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:222)
 at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:91)
 at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
 at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
 at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
 at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

 at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:108)
 at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
 at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
 at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
 at com.alibaba.dubbo.common.bytecode.proxy0.compositeQuery(proxy0.java)
 at com.wwwarehouse.xdw.commonindustry.ServiceTest.compositeQuery(ServiceTest.java:43)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
 at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
 at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
 at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
 at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
 at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
 at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
 at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

服务提供方的错误日志信息：

[2019-01-24 16:46:43,985] INFO -:-:-:- [main] com.wwwarehouse.commons.dubbo.DubboConsumerAuthFilter.invoke(DubboConsumerAuthFilter.java:51) - dubboRpcContext:>>>
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method compositeQuery in the service com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService. Tried 3 times of the providers [192.168.72.158:20880] (1/1) from the registry 192.168.6.21:2181 on the consumer 192.168.72.158 using the dubbo version 2.8.4. Last error is: Failed to invoke remote method: compositeQuery, provider: dubbo://192.168.72.158:20880/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService?anyhost=true&application=commonindustry-biz-test&batchChangeInventory.timeout=60000&batchChangeInventoryQty.timeout=60000&changeInventory.timeout=600000&check=false&default.check=false&default.connections=10&default.group=mock&dubbo=2.8.4&eliminatePlanAdjustmentDatas.timeout=60000&generic=false&group=impl2&interface=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService&lockPlanAdjustmentDatas.timeout=5000&methods=selectSubItemsByGroupUkidAndItemUkid,queryConsumeRealityQTY,queryDeliveryFromStorageQTY,queryMaxQtyGroupForItem,unLockPlanAdjustmentDatas,selectAvailableListByGroupRelationUkid,inventoryQuery,selectItemsByGroupUkidAndIdentifyCode,compositeQuery,lockPlanAdjustmentDatas,getInventoryQueryFilterInfo,batchChangeInventoryQty,listByUkids,updateInventoryStatistics,sumInventoryQty,changeInventoryQty,changeInventory,deliveryFromStorage,selectQualitiesByGroupUkidAndItemUkid,batchChangeInventory,getGoodsStockInfo,warehouseEntry,selectConsumeListByIdentifyCode,addInventory,selectConsumeListByGroupUkid,selectConsumeListByGroupUkids,getInventoryQueryList,getStorageGoodsStock,selectConsumeListByGroupCode,eliminatePlanAdjustmentDatas&organization=wwwarehouse&owner=xdw&pid=17332&protocol=dubbo&revision=1.0-20190124.081517-966&side=consumer×tamp=1548319602539&updateInventoryStatistics.timeout=5000, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880 in [impl2/com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService:20880], may be version or group mismatch , channel: consumer: /192.168.72.158:60792 --> provider: /192.168.72.158:20880, message:RpcInvocation [methodName=compositeQuery, parameterTypes=[class com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery], arguments=[null], attachments={path=com.wwwarehouse.xdw.resourcecenter.service.ImConsumeRealityService, input=829, name=OWNER_UKID, dubbo=2.8.4, version=0.0.0}]
 at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:222)
 at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:91)
 at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
 at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
 at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
 at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

 at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:108)
 at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
 at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
 at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
 at com.alibaba.dubbo.common.bytecode.proxy0.compositeQuery(proxy0.java)
 at com.wwwarehouse.xdw.commonindustry.ServiceTest.compositeQuery(ServiceTest.java:43)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
 at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
 at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
 at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
 at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
 at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
 at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
 at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

排查过程

dubbo admin上看了下，服务暴露是正常的。目标方法是新提供的，暂无业务调用。看了其他的方法，有正常消费。当时认为服务是ok的，让调用方自身排查下。
调用方排查环境配置ok，未找到原因。
自行测试了下，发现一个诡异的现象。查询对象不设置条件时，服务可以调用。如果设置，提示找不到服务。
测试代码如下

CompositeQuery compositeQuery = CompositeQuery.newQuery();
// 下面这行是设置条件，如果不设置，服务可以调用。如果设置，提示找不到服务
// compositeQuery.addExpression(new CompositeQuery.Expression(DBField.OWNER_UKID, Operator.EQ, "-100"));
try {
    System.out.println("results:" + imConsumeRealityService.compositeQuery(compositeQuery));
} catch (Exception e) {
    e.printStackTrace();
}

结合错误提示，看了下源码。暴露的服务有组名，请求调用的目标服务没有组名 (impl2是dubbo service group name)，调用方配置检查ok，有配置组名。如果配置有问题，应该都有问题。

重新查看服务方的日志，前后关联日志都看了下，dubbo输出了一个error级别的警告

[DUBBO] Decode argument failed: com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery.expressions: 
'com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery$Expression' could not be instantiated, 
dubbo version: biz, current host: 172.18.253.160 com.alibaba.com.caucho.hessian.io.HessianFieldException: 
com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery.expressions: 
'com.wwwarehouse.xdw.resourcecenter.model.consumerReality.CompositeQuery$Expression' could not be instantiated at 
com.alibaba.com.caucho.hessian.io.JavaDeserializer.logDeserializeError(JavaDeserializer.java:671) ~[resourcecenter-biz.jar:?] at 
com.alibaba.com.caucho.hessian.io.JavaDeserializer$ObjectFieldDeserializer.deserialize(JavaDeserializer.java:400) ~[resourcecenter-biz.jar:?] at 
com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:233) ~[resourcecenter-biz.jar:?] at 
com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:157) ~[resourcecenter-biz.jar:?] at 
com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2067) ~[resourcecenter-biz.jar:?] at 
com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1592) ~[resourcecenter-biz.jar:?] at 
com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1576) ~[resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94) ~[resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:135) [resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:74) [resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:138) [resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:134) [resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:95) [resourcecenter-biz.jar:?] at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:46) [resourcecenter-biz.jar:?] at
...

CompositeQuery$Expression, 这个对象，没有定义无参的构造器。新增公有无参构造器后问题解决。
为什么参数解析失败，会导致dubbo service拿不到group信息。
重新调试了下，服务方接收到请求后，对输入的字节流进行解析。先做的参数读取，然后读取附属信息(如group、version等)。参数解析时候出现了异常，导致获取到的附属信息不对，没有拿到group信息。

public class DecodeableRpcInvocation extends RpcInvocation implements Codec, Decodeable {

    public Object decode(Channel channel, InputStream input) throws IOException {
        
        // ...
                
        Object[] args = new Object[pts.length];
        for (int i = 0; i < args.length; i++) {
            try {
                // 参数解析，从输入流中读取参数对象
                args[i] = in.readObject(pts[i]);
            } catch (Exception e) {
                if (log.isWarnEnabled()) {
                    log.warn("Decode argument failed: " + e.getMessage(), e);
                }
            }
        }
           
        // ...
        // 读取附属信息
        Map map = (Map) in.readObject(Map.class);
        if (map != null && map.size() > 0) {
            Map attachment = getAttachments();
            if (attachment == null) {
                attachment = new HashMap();
            }
            attachment.putAll(map);
            // 设置附属参数信息
            setAttachments(attachment);
        }
        
        // ...
    }
}

读取到了正确的附属信息

读取到错误的附属信息

问题原因

参数对象里面有个内部类，没有定义公有无参构造器。序列化时候失败了

刚开始方向找偏了，这段时间dubbo服务暴露经常有问题，服务没有暴露出去。刚开始看到以为是服务暴露问题。确认dubbo admin服务ok，以为调用方出了问题，导致排查用了不少时间。
细心！！！ 善用经验！！！