hadoop-3.0.1源码编译需要注意的事项
这次尝试了一下源码编译最新的hadoop3.0.1，发现了几个和原来不太一样的地方。记录下来：
1、需要的jdk不再是原来的1.7，直接jdk1.8就编译通过了；
2、以前安装需要安装编译依赖cmake,这次是需要cmake3，redhat7默认没有cmake3的package，需要先安装epel-release,然后安装cmake3,最后将cmake3做一个软链cmake；
yum install epel-release -y yum install cmake3 -y ln -s /usr/bin/cmake3 /usr/bin/cmake
3、编译hadoop amazon webservice时遇到缺少依赖包:DynamoDBLocal-1.11.86.jar。编译不通过，这时需要下载amazon提供的dynamodb_local_latest.tar.gz，解压之后，将其中的DynamoDBLocal.jar包拷贝到/path/maven/repo目录下的com/amazon/1.11.86/目录下，并命名为DynamoDBLocal-1.11.86.jar。接着编译就可以成功；
Failure to find com.amazonaws:DynamoDBLocal:jar
另外，网上有说更改hadoop-3.0.1-src目录下的pom.xm文件，增加dynamodb的仓库，我试了没有成功，不知道什么原因。
dynamodb-local-oregon DynamoDB Local Release Repository https://s3-ap-southeast-1.amazonaws.com/dynamodb-local-singapore/release
最终不得不下载dynamodb_local最新文件，解压出其中的DynamoDBLocal.jar。
4、其余的条件基本没什么变化。gcc-c++,openssl-devel,ncurses-devel和编译时需要的protobuf-2.5.0，再一个就是jdk,maven,编译时需要下载很多依赖包。
最后附上一个编译成功的打印信息：
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 0.606 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.187 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.588 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.350 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.093 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.144 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.612 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.159 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 3.331 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.520 s] [INFO] Apache Hadoop Common ............................... SUCCESS [ 36.657 s] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.024 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.565 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.030 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 10.304 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 33.907 s] [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 1.186 s] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 3.713 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.756 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.021 s] [INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.021 s] [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 6.282 s] [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 19.870 s] [INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.022 s] [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 5.239 s] [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.523 s] [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 5.820 s] [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.822 s] [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 2.954 s] [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.634 s] [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 9.594 s] [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.661 s] [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 2.662 s] [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.957 s] [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 2.036 s] [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 3.636 s] [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 1.781 s] [INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 2.504 s] [INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.020 s] [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.590 s] [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.189 s] [INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.019 s] [INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.019 s] [INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 5.579 s] [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.119 s] [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 9.071 s] [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 7.961 s] [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.258 s] [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 3.789 s] [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.843 s] [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 2.078 s] [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.134 s] [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 3.074 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.565 s] [INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.236 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 2.518 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.035 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.245 s] [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.160 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 2.598 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.777 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.260 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.078 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.092 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.321 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 4.232 s] [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 2.446 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.362 s] [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 2.752 s] [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 1.984 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.827 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.193 s] [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 4.739 s] [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 2.395 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 6.701 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.023 s] [INFO] Apache Hadoop Client API ........................... SUCCESS [01:20 min] [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:02 min] [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.592 s] [INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:31 min] [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.106 s] [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.072 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 16.725 s] [INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.018 s] [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.629 s] [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.014 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 08:51 min [INFO] Finished at: 2018-04-05T16:25:41+08:00 [INFO] Final Memory: 173M/931M [INFO] ------------------------------------------------------------------------
编译成功之后，编译结果在hadoop-3.0.1-src/hadoop-dist/target目录下，
我们可以查看一下编译之后，形成的lib/native/libhadoop.so.1.0.0。
[root@server target]# cd hadoop-3.0.1/lib/native/ [root@server native]# ls examples libhadooppipes.a libhadoop.so.1.0.0 libhdfs.a libhdfs.so.0.0.0 libnativetask.so libhadoop.a libhadoop.so libhadooputils.a libhdfs.so libnativetask.a libnativetask.so.1.0.0 [root@server native]# file libhadoop.so.1.0.0 libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=12cc31646dffc9651647fe12d8fe28b2dbc6280f, not stripped
本地依赖库有了之后，以后运行hadoop命令时就不会每次都提醒缺少本地库的警告了。