Windows 10编译Hadoop 2.6.0源码
转载自：https://www.zybuluo.com/ncepuwanghui/note/343755
原作者：Sam Wong向原文作者的辛勤劳动致敬！
版本控制
@Title Windows 10编译Hadoop 2.6.0源码
@Version v1.0
@Timestamp 2016-08-22 16:21
@Author Sam Wong
@Mail ncepuwanghui@foxmail.com系统环境
@OS Windows 10 Enterprise (64bit)
@JDK JDK 1.7.0_67
@Hadoop Hadoop 2.6.0下载相关软件包
Building on Windows
Requirements:
Windows System JDK 1.6+ Maven 3.0 or later Findbugs 1.3.9 (if running
findbugs) ProtocolBuffer 2.5.0 CMake 2.6 or newer Windows SDK or
Visual Studio 2010 Professional Unix command-line tools from GnuWin32
or Cygwin: sh, mkdir, rm, cp, tar, gzip zlib headers (if building
native code bindings for zlib) Internet connection for first build (to
fetch all Maven and Hadoop dependencies)If using Visual Studio, it must be Visual Studio 2010 Professional
(not 2012). Do not use Visual Studio Express. It does not support
compiling for 64-bit, which is problematic if running a 64-bit system.
JDK
jdk-7u67-windows-x64.exe
下载地址：http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html
Maven
apache-maven-3.3.9-bin.zip
下载地址：https://maven.apache.org/download.cgi
Findbugs
findbugs-1.3.9.zip
下载地址：https://sourceforge.net/projects/findbugs/files/findbugs/1.3.9/
ProtocolBuffer
protobuf-2.5.0.zip
protoc-2.5.0-win32.zip
下载地址：https://github.com/google/protobuf/releases/tag/v2.5.0
注：除了下载protobuf源码外，还需要下载相应版本的编译过的用于Windows平台的protoc命令（protoc-2.5.0-win32.zip）,该命令用于将.proto文件转化为Java或C++源文件。
CMake
cmake-3.6.1-win64-x64.zip
下载地址：https://cmake.org/download/
Visual Studio 2010 Professional
Cygwin
setup-x86_64.exe
下载地址：https://cygwin.com/install.html
Zlib headers
zlib128.zip
下载地址：http://www.zlib.net/
安装配置所需环境
JDK
① 安装JDK
② 新建系统变量JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_67
注：JAVA_HOME路径不能有空格，如果安装路径名中有空格，要使用旧式的8.3名字格式，即C:\Program Files\要写成C:\Progra~1\。
③ 编辑系统变量Path，新建%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin
Maven
① 解压Maven到指定目录
② 新建系统变量M2_HOME=C:\WorkSpace\apache-maven-3.3.9
② 编辑系统变量Path，新建%M2_HOME%\bin
Findbugs
① 解压Findbugs到指定目录
② 编辑系统变量Path，新建C:\WorkSpace\findbugs-1.3.9\bin
ProtocolBuffer
① 解压ProtocolBuffer到指定目录
② 解压protoc-2.5.0-win32.zip，将protoc.exe复制到C:\WorkSpace\protobuf-2.5.0\src目录下
③ 安装ProtocolBuffer，打开CMD命令行
cd C:\WorkSpace\protobuf-2.5.0\java
mvn test
mvn install
protoc --version若没有protoc.exe或复制目录不正确，会提示如下错误：
[INFO] Executing tasks
[mkdir] Created dir: C:\WorkSpace\protobuf-2.5.0\java\target\generated-sources
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:22 min
[INFO] Finished at: 2016-08-22T17:56:19+08:00
[INFO] Final Memory: 16M/439M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (generate-sources) on project protobuf-java: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "..\src\protoc": CreateProcess error=2, ????????? -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException④ 将protoc.exe所在路径C:\WorkSpace\protobuf-2.5.0\src，添加到系统变量Path
CMake
① 解压CMake到指定目录
② 编辑系统变量Path，新建C:\WorkSpace\cmake-3.6.1-win64-x64\bin
Cygwin
① 安装Cygwin64
② 编辑系统变量Path，新建C:\WorkSpace\cygwin64\bin
Zlib
① 解压Zlib到指定目录
② 新建系统变量ZLIB_HOME=C:\WorkSpace\zlib-1.2.8
编译Hadoop2.6.0
设置Platform
新建系统变量Platform=x64
编译Hadoop2.6.0
用右键以管理员身份运行Visual Studio x64 Win64 命令提示(2010)，开始菜单–>所有程序–>Microsoft Visual Studio 2010–>Visual Studio Tools–>Visual Studio x64 Win64 命令提示(2010)，执行如下命令：
cd C:\WorkSpace\hadoop-2.6.0-src
mvn package -Pdist,native-win -DskipTests -Dtar编译完成后，编译好的hadoo-2.6.0在目录C:\WorkSpace\hadoop-2.6.0-src\hadoop-dist\target\中。
注：本文采用Visual Studio 2015编译，开始–>所有应用–>Visual Studio 2015–>VS2015 x64
本机工具命令提示符
遇到的错误
① hadoop-common编译失败
[INFO] --- hadoop-maven-plugins:2.6.0:version-info (version-info) @ hadoop-common ---
[WARNING] [svn, info] failed: java.io.IOException: Cannot run program "svn": CreateProcess error=2, ?????????
[WARNING] [git, branch] failed with error code 128
[INFO] SCM: NONE
......
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Cpp.Platform.targets(55,5): error MSB8020: The build tools for Visual Studio 2010 (Platform Toolset = 'v100') cannot be found. To build using the v100 build tools, please install Visual Studio 2010 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".[C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]
已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj
”(默认目标)的操作 - 失败。
已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj”(默认目标)的操作 - 失败。
已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln”(默认
目标)的操作 - 失败。
生成失败。
......
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-comm
on: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
org.ap
......
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-common 解决方法，Visual Studio版本问题，官方默认使用Visual Studio 2010 Professional，但本文采用Visual Studio 2015，因此对于生成失败的项目，需要用Visual Studio 2015重新打开，升级项目至Visual Studio 2015版本即可：
C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln
C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln② hadoop-hdfs编译失败
main:
[mkdir] Created dir: C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native
[exec] -- The C compiler identification is unknown
[exec] -- The CXX compiler identification is unknown
[exec] CMake Error in CMakeLists.txt:
[exec] No CMAKE_C_COMPILER could be found.
[exec] CMake Error in CMakeLists.txt:
[exec] No CMAKE_CXX_COMPILER could be found.
[exec] -- Configuring incomplete, errors occurred!
[exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeOutput.log".
[exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log".
......
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-hdfs 问题原因是Visual Studio版本问题，解决方法：打开文件C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\pom.xml`，将
-G 'Visual Studio 10 Win64'修改为：
-G 'Visual Studio 14 Win64'若仍无法编译，可查看日志C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log。若日志提示找不到Microsoft.Cpp.Default.props，可在C:\Program Files (x86)中查找Microsoft.Cpp.Default.props，例如最后找到所在路径C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140
用文本编辑器打开项目文件
C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdC\CompilerIdC.vcxproj
C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdCXX\CompilerIdCXX.vcxproj查找Microsoft.Cpp.Default.props所在行，如下：
"$(VCTargetsPath)\Microsoft.Cpp.Default.props" /> 原因是没有系统变量VCTargetsPath，新建系统变量VCTargetsPath=C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140，然后重新编译：
mvn package -Pdist,native-win -DskipTests -Dtar -rf :hadoop-hdfs注：若还提示找不到，可能是环境变量未生效，关闭命令行窗口，重新打开执行编译命令，或者重启电脑试试。
参考链接
Build and Install Hadoop 2.x or newer on Windows
Windows 10下编译安装Hadoop2.6
在windows 7 64bit下编译Hadoop 2.6.0源码
编译本地64位版本的hadoop-2.6.0
在Windows下编译Hadoop2.5
Protobuf 的安装使用总结