本机环境
CentOS Linux release 7.6.1810 (Core) 64位
准备
下载hadoop3.1.0 ：
wget https://archive.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0-src.tar.gz
下载DynamoDBLocal ：
wget https://s3-us-west-2.amazonaws.com/dynamodb-local/release/com/amazonaws/DynamoDBLocal/1.11.86/DynamoDBLocal-1.11.86.jar
(国内常用的aliyun没有DynamoDBLocal这个jar包，需要从amazonaws下载。hadoop3.1.0 依赖DynamoDBLocal-1.11.86.jar。其他版本查看ws：https://s3-us-west-2.amazonaws.com/dynamodb-local，下载jar包加上key即可。如下载DynamoDBLocal-1.11.86.jar，对应的key为：release/com/amazonaws/DynamoDBLocal/1.11.86/DynamoDBLocal-1.11.86.jar)
安装到本地maven库：
mvn install:install-file -Dfile=DynamoDBLocal-1.11.86.jar -DgroupId=com.amazonaws -DartifactId=DynamoDBLocal -Dversion=1.11.86 -Dpackaging=jar
依赖
可以通过查看 hadoop-3.1.0-src/BUILDING.txt 查看
* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)
note：
1. cmake用yum安装是2.8的，需要通过编译安装3.1版本。本次实验采用cmake3（yum install cmake3; ln -s /usr/bin/cmake3 /usr/bin/cmake）
2. 编译目录磁盘空间 > 5G
编译
mvn -X clean package -Pdist,native -DskipTests -Dtar
(不能添加-Dmaven.test.skip=true)
bin包路径： hadoop-dist/target
问题
1. [ERROR] /appdata/hadoop/src/hadoop-3.1.0-src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/StagingTestBase.java:[647,23] cannot find symbol
[ERROR] symbol: method getArgumentAt(int,java.lang.Class
[ERROR] location: variable invocation of type org.mockito.invocation.InvocationOnMock
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-aws: Compilation failure
原因：InvocationOnMock.getArgumentAt(int,Class) 方法不存在
解决 ： 修改 hadoop-project的pom.xml → org.mockito:mockito-all:1.10.19
2. org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:get (default-cli) on project standalone-pom: Couldn't download artifact: Missing:
com.amazonaws:DynamoDBLocal:1.11.86:jar
原因：aliyun没有DynamoDBLocal包
解决：下载并添加到本地仓库，参考“准备”部分
总结
1. 3.1.0版本比之前编译过的2.7遇到的问题多一点。耗时10小时，2.7版本也就两三个小时。超过5个小时是在DynamoDBLocal依赖上。
其他
查看包冲突 ：
报错： "java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log"
mvn dependency:tree -Dverbose -Dincludes=log4j:log4j
java -verbose:class XXX | grep 'LocationAwareLogger'
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
-DrepoUrl=https://mvnrepository.com/artifact/
-Dartifact=redis.clients:jedis:2.6.2
mvn -DoutputDirectory=./lib
-DgroupId=com.it18zhang
-DartifactId=CallLogConsumerModule
-Dversion=1.0-SNAPSHOT
dependency:copy-dependencies