k8s中dubbo注册IP和消费的问题
在系统迁移过程要考虑到平稳过度，先将部分应用迁移到k8s上，但是发现并没有那么简单
从dubbo-admin中可以看到注册服务的ip是容器组ip，如果所有应用都在k8s中自然是没有问题，但如果k8s与虚拟机中的服务相互调用，就存在问题了，因为找不到ip。
参考Docker 容器内运行 Dubbo 服务，启动容器,但是dubbo中还是容器的IP，并不是主机ip
docker run --name ab99755feae7 -d -p 9010:9010 -p 21716:21716 -e Dubbo_IP_TO_REGISTRY=10.101.7.66 ab99755feae7
# 再查看容器的环境变量，
[root@worker3 ~]# docker exec 1f559cde8604 env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=1f559cde8604
DUBBO_IP_TO_REGISTRY=10.101.7.66
LANG=C.UTF-8
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
JAVA_VERSION=8u111
JAVA_DEBIAN_VERSION=8u111-b14-2~bpo8+1
CA_CERTIFICATES_JAVA_VERSION=20140324
HOME=/root
接着翻阅Spring cloud dubbo docker部署IP注册问题，但我查看dubbo-2.8.4 的源码，没看到读取环境变量的地方，不知道他说的源码是哪里的。查看MavenRepository，最新的版本也才2.6.8，估计是当初使用当当的dubbox导致的，应为我的dubbo是2016年就可以使使用了。
通过docker cp f8f041b98c79:/eztserver.jar /appdata，下载查看后发现里面确实没有环境变量，而官方2.6.7版本确实有这个。于是想到了替换版本。
但并不是直接替换就可以了会出问题，Unable to locate Spring NamespaceHandler for XML schema namespace排错记录
spring.schemas的内容如下：
http\://dubbo.apache.org/schema/dubbo/dubbo.xsd=META-INF/dubbo.xsd
http\://code.alibabatech.com/schema/dubbo/dubbo.xsd=META-INF/compat/dubbo.xsd
启动服务，提示下面的错误，
\target\classes\dubbo\dubbo.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 9; columnNumber: 42; cvc-complex-type.2.4.c: 通配符的匹配很全面, 但无法找到元素 'dubbo:application' 的声明。
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
检查这些地方，都是正确的。为啥还提示这个错误呢？
如果放弃dubbo.xml的方式，采取springboot的方式，再调用docker run --name b448720570ed -d -p 9010:9010 -p 21716:21716 -e DUBBO_IP_TO_REGISTRY=10.101.7.65 b448720570ed,这个时候ip就是主机ip了
<properties>
<dubbo.version>2.7.5dubbo.version>
<zookeeper.version>3.4.14zookeeper.version>
properties>
<dependency>
<groupId>org.apache.dubbogroupId>
<artifactId>dubboartifactId>
<version>${dubbo.version}version>
<exclusions>
<exclusion>
<groupId>org.springframeworkgroupId>
<artifactId>springartifactId>
exclusion>
<exclusion>
<groupId>javax.servletgroupId>
<artifactId>servlet-apiartifactId>
exclusion>
<exclusion>
<groupId>log4jgroupId>
<artifactId>log4jartifactId>
exclusion>
exclusions>
dependency>
<dependency>
<groupId>org.apache.zookeepergroupId>
<artifactId>zookeeperartifactId>
<version>${zookeeper.version}version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12artifactId>
<groupId>org.slf4jgroupId>
exclusion>
exclusions>
dependency>
<dependency>
<groupId>com.github.sgroschupfgroupId>
<artifactId>zkclientartifactId>
<version>0.1version>
dependency>
<dependency>
<groupId>org.apache.curatorgroupId>
<artifactId>curator-recipesartifactId>
<version>4.2.0version>
dependency>
<dependency>
<groupId>org.apache.curatorgroupId>
<artifactId>curator-frameworkartifactId>
<version>4.2.0version>
dependency>
在springboot的启动类设置DubboComponentScan和EnableDubboConfig
import lombok.extern.slf4j.Slf4j;
import org.apache.dubbo.config.spring.context.annotation.DubboComponentScan;
import org.apache.dubbo.config.spring.context.annotation.EnableDubboConfig;
import org.mybatis.spring.annotation.MapperScan;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.scheduling.annotation.EnableScheduling;
@Slf4j
@SpringBootApplication
@ComponentScan(basePackages = { "com.test" })
@MapperScan("com.bwjf.**.dao")
@EnableScheduling
@EnableEurekaClient
@EnableFeignClients(basePackages = {"com.test"})
@EnableDubboConfig
@DubboComponentScan("com.test.bzt.dubbo.service.impl")
@EnableDiscoveryClient
public class Application {
private static Logger log = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) {
try {
System.setProperty("es.set.netty.runtime.available.processors", "false");
SpringApplication.run(Application.class, args);
} catch (Exception e) {
log.info(e.getMessage(), e);
} finally {
log.info("server start finish");
}
}
}
而在yaml中设置dubbo
#Dubbbo
domain: bztserver
dubbo:
owner: zhengbangshun
organization: bwjf
registry:
address: zookeeper://192.168.5.128:2181?backup=192.168.5.128:2182,192.168.5.128:2183
protocol:
name: dubbo
port: 21716
accesslog: dubbo-access.log
service:
validation: true
version: 1.0.0
timeout: 10000
retries: 2
loadbalance: random
actives: 150
reference:
check: false
version: 1.0.0
在k8s中并不能直接应用，因为你不知道他会将程序不出到那台机器上，故可参考Dubbo与Kubernetes集成,通过status.hostIP来制定主机ip
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: '3'
k8s.eip.work/ingress: 'false'
k8s.eip.work/service: none
k8s.eip.work/workload: eztserver
k8s.kuboard.cn/ingress: 'false'
k8s.kuboard.cn/service: none
k8s.kuboard.cn/workload: eztserver
creationTimestamp: '2020-06-28T05:01:08Z'
generation: 3
labels:
eztserver: eztserver
k8s.eip.work/layer: ''
k8s.eip.work/name: eztserver
name: eztserver
namespace: java
resourceVersion: '2785134'
selfLink: /apis/apps/v1/namespaces/java/deployments/eztserver
uid: 1470af0b-0b71-4faf-bdbb-4a3534c1795a
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
eztserver: eztserver
k8s.eip.work/layer: ''
k8s.eip.work/name: eztserver
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
eztserver: eztserver
k8s.eip.work/layer: ''
k8s.eip.work/name: eztserver
spec:
containers:
- env:
- name: DUBBO_IP_TO_REGISTRY
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
image: '192.168.5.101/java/eztserver:1.0.0'
imagePullPolicy: Always
lifecycle: {}
name: eztserver
ports:
- containerPort: 9010
hostPort: 9010
protocol: TCP
- containerPort: 21716
hostPort: 21716
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsConfig: {}
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harbor192.168.5.101
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
seLinuxOptions: {}
terminationGracePeriodSeconds: 30
容器的dubbo端口也无需要开放出来
注册问题算是解决了，新的问题来，k8s容器内部的应用反而无法调用了
但是在dubbo-admin中看到消费者是主机ip，而不是容器ip，所以消费失败
接着下载dubbo-demo,做简单的配置
<dependency>
<groupId>javax.validationgroupId>
<artifactId>validation-apiartifactId>
<version>1.1.0.Finalversion>
dependency>
<dependency>
<groupId>org.hibernategroupId>
<artifactId>hibernate-validatorartifactId>
<version>5.1.0.Finalversion>
dependency>
否则可能会有下面的错误,因为接口做了校验处理
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.AbstractMethodError: org.hibernate.validator.engine.ConfigurationImpl.getDefaultParameterNameProvider()Ljavax/validation/ParameterNameProvider;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$118/631673932.getObject(Unknown Source)
经过测试，在本地是可以调用到docker中发布的dubbo服务，但是docker内部的应用却不能访问，那么就可以断定是消费者注册ip的问题
跟踪UrlUtils中可以看到dubbo会将consumer和provider进行比对
继续跟踪，参照spring工程dubbo docker迁移做改造之后，在springboot工程中可以获取到接口返回，但是springmvc却找不到提供者
入下图，springboot工程和springmvc就可以成为对应的消费者，订阅了这个服务
springmvc工程就提示下面的异常
com.alibaba.dubbo.rpc.RpcException: No provider available in [invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632®ister.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000×tamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@3a917017, invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632®ister.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000×tamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@53fc870f]
at com.alibaba.dubbo.rpc.cluster.support.AvailableCluster$1.doInvoke(AvailableCluster.java:48)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:244)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.getMemberInfo(proxy0.java)
Dubbo：com.alibaba.dubbo.rpc.RpcException,实际netty-all并不会导致这个问题 ，不过没有会出现Dubbo 2.6.5启动报java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup,springmvc需要这个jar，但是springboot工程并需要
java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.startQosServer(QosProtocolWrapper.java:96)
at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.refer(QosProtocolWrapper.java:68)
at com.alibaba.dubbo.rpc.Protocol$Adaptive.refer(Protocol$Adaptive.java)
at com.alibaba.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:401)
at com.alibaba.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:335)
at com.alibaba.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:164)
将这个jar放在启动工程即可
<dependency>
<groupId>io.nettygroupId>
<artifactId>netty-allartifactId>
<version>4.1.32.Finalversion>
dependency>
如果dubbo的日志没有打印出来，那么有些日志也就看不出来，log4j-over-slf4j这个不要漏了，否则dubbo的日志就没有
!-- log -->
<dependency>
<groupId>org.slf4jgroupId>
<artifactId>slf4j-apiartifactId>
dependency>
<dependency>
<groupId>ch.qos.logbackgroupId>
<artifactId>logback-classicartifactId>
dependency>
<dependency>
<groupId>ch.qos.logbackgroupId>
<artifactId>logback-accessartifactId>
dependency>
<dependency>
<groupId>org.slf4jgroupId>
<artifactId>log4j-over-slf4jartifactId>
dependency>
打开日志后发现，在服务启动的时候抛出了异常,虽然在dubbo-admin中看到没有问题，但是已经出错了
18:03:06.026 ERROR com.alibaba.dubbo.qos.server.Server - [DUBBO] qos-server can not bind localhost:22222, dubbo version: 2.6.7, current host: 10.101.5.102
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:437)
at sun.nio.ch.Net.bind(Net.java:429)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:258)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)
18:03:06.027 WARN com.alibaba.dubbo.qos.protocol.QosProtocolWrapper - [DUBBO] Fail to start qos server: , dubbo version: 2.6.7, current host: 10.101.5.102
java.net.BindException: Address already in use: bind
知道问题了就好解决了，(3)Dubbo启动时qos-server can not bind localhost:22222错误解决,使用不同的qos.port就可以了，
<dubbo:application name="${domain}" owner="${dubbo.owner}"
organization="${dubbo.organization}" >
<dubbo:parameter key="qos.enable" value="true"/>
<dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
<dubbo:parameter key="qos.port" value="22200"/>
dubbo:application>
参考springboot+dubbo2.6.0关闭QOS服务，将qos.enable禁用掉也可以
<dubbo:application name="${domain}" owner="${dubbo.owner}"
organization="${dubbo.organization}" >
<dubbo:parameter key="qos.enable" value="false"/>
<dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
dubbo:application>