Hbase、elasticsearch整合中jar包冲突

问题背景

再数据平台中，项目搭建需要使用es和HBASE搭建数据查询接口，整合的过程中出现jar包冲突的bug :com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)
        ...
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil
        ...
        at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
        ... 45 more

---

解决办法

经排查，确认是因为 com.google.guava 这个包引起的冲突。es依赖于18及以上版本，而HBASE只支持16及以上版本。而guava这个包从17开始内部发生变化，方法发生改变，所以18并不会向下兼容16版本。在项目运行的过程中，如果同时引入16、18版本，es及hbase的调用过程会发生混乱。那么接下来就好办了，我们可以重新打包，将guava18打入es内部，再在pom文件中显示引用guava16版本的包。这样，es调用包内部打入的guava18，而hbase调用外部的guava16。

重新打包

新建maven工程，在pom文件中进行如下配置：

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0modelVersion>

    
    <groupId>douguo.shaded.elasticsearchgroupId>
    <artifactId>douguo_shaded_elasticsearchartifactId>
    <version>1.0-SNAPSHOTversion>

    <properties>
        <elasticsearch.version>2.4.1elasticsearch.version>
    properties>
    <dependencies>
        <dependency>
            <groupId>org.elasticsearchgroupId>
            <artifactId>elasticsearchartifactId>
            <version>${elasticsearch.version}version>
        dependency>
        <dependency>
            <groupId>org.elasticsearch.plugingroupId>
            <artifactId>shieldartifactId>
            <version>${elasticsearch.version}version>
        dependency>
    dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.pluginsgroupId>
                <artifactId>maven-shade-pluginartifactId>
                <version>2.4.1version>
                <configuration>
                    <createDependencyReducedPom>falsecreateDependencyReducedPom>
                configuration>
                <executions>
                    <execution>
                        <phase>packagephase>
                        <goals>
                            <goal>shadegoal>
                        goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>com.google.guavapattern>
                                    <shadedPattern>douguo.shaded.elasticsearch.guavashadedPattern>
                                relocation>
                                <relocation>
                                    <pattern>org.jodapattern>
                                    <shadedPattern>douguo.shaded.elasticsearch.jodashadedPattern>
                                relocation>
                                <relocation>
                                    <pattern>com.google.commonpattern>
                                    <shadedPattern>douguo.shaded.elasticsearch.commonshadedPattern>
                                relocation>
                                <relocation>
                                    <pattern>com.google.thirdpartypattern>
                                    <shadedPattern>douguo.shaded.elasticsearch.thirdpartyshadedPattern>
                                relocation>
                            relocations>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
                            transformers>
                        configuration>
                    execution>
                executions>
            plugin>
        plugins>
    build>

    <repositories>
        <repository>
            <id>elasticsearch-releasesid>
            <url>http://maven.elasticsearch.org/releasesurl>
            <releases>
                <enabled>trueenabled>
                <updatePolicy>dailyupdatePolicy>
            releases>
            <snapshots>
                <enabled>falseenabled>
            snapshots>
        repository>
    repositories>

project>

将org.joda等4个可能有冲突的jar包通过maven-shade-plugin插件迁移后重新打个jar包从而使得在引入这个jar包时能够使用该jar包自己的依赖而不是使用外部依赖。这里需要注意的是，需要将com.google.common等4个包全部重新迁移，否则会出现java.lang.IllegalAccessError: tried to access method com.google.common.base的错误

项目打包

mvn clean install

新的依赖包会在.m2 maven仓库中，如果公司搭建了仓库的话，需要上传jar包。如果直接运行jar包的话，记得重新编译项目，并替换lib目录

项目载入新包

只需要在pom文件中配置：

<dependency>
    <groupId>douguo.shaded.elasticsearchgroupId>
    <artifactId>douguo_shaded_elasticsearchartifactId>
    <version>1.0-SNAPSHOTversion>
    <exclusions>
        <exclusion>
            <groupId>org.elasticsearchgroupId>
            <artifactId>elasticsearchartifactId>
        exclusion>
    exclusions>
dependency>




<dependency>
    <groupId>com.google.guavagroupId>
    <artifactId>guavaartifactId>
    <version>16.0version>
dependency>

这样，guava18包就在douguo.shaded.elasticsearch下，es会优先调用。而外部配置的guava16就会被HBASE调用。2个版本的jar包互相独立存在！

到此，问题就解决了！

---

参考文章：

• http://blog.csdn.net/sunshine920103/article/details/51659936
• https://www.elastic.co/blog/to-shade-or-not-to-shade