WIN7下运行hadoop程序报：Failed to locate the winutils binary in the hadoop binary path

文章来源：http://blog.csdn.net/vickyrocker1/article/details/47341183

之前在mac上调试hadoop程序（mac之前配置过hadoop环境）一直都是正常的。因为工作需要，需要在windows上先调试该程序，然后再转到linux下。程序运行的过程中，报 Failed to locate the winutils binary in the hadoop binary path  java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

通过断点调试、查看源码发现程序需要根据HADOOP_HOME找到winutils.exe,由于win机器并没有配置该环境变量，所以程序报 null\bin\winutils.exe。

<span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">private</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">static</span></span><span style="color: rgb(0, 0, 0);"> String checkHadoopHome() {
  </span><span style="color: rgb(0, 128, 0);"><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;">//</span></span><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;"><span style="color: rgb(0, 128, 0);"> first check the Dflag hadoop.home.dir with JVM scope</span></span><span style="color: rgb(0, 128, 0);"></span>
  String home = System.getProperty(<span class="string" style="color: rgb(221, 17, 68);">"hadoop.home.dir"</span><span style="color: rgb(0, 0, 0);">);

  </span><span style="color: rgb(0, 128, 0);"><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;">//</span></span><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;"><span style="color: rgb(0, 128, 0);"> fall back to the system/user-global env variable</span></span><span style="color: rgb(0, 128, 0);"></span>
  <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span> (home == <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">null</span></span><span style="color: rgb(0, 0, 0);">) {
    home </span>= System.getenv(<span class="string" style="color: rgb(221, 17, 68);">"HADOOP_HOME"</span><span style="color: rgb(0, 0, 0);">);
  }
  </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">try</span></span><span style="color: rgb(0, 0, 0);"> {
     </span><span style="color: rgb(0, 128, 0);"><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;">//</span></span><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;"><span style="color: rgb(0, 128, 0);"> couldn't find either setting for hadoop's home directory</span></span><span style="color: rgb(0, 128, 0);"></span>
     <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span> (home == <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">null</span></span><span style="color: rgb(0, 0, 0);">) {
       </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">throw</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">new</span></span> IOException(<span class="string" style="color: rgb(221, 17, 68);">"HADOOP_HOME or hadoop.home.dir are not set."</span><span style="color: rgb(0, 0, 0);">);
     }
     </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span> (home.startsWith(<span class="string" style="color: rgb(221, 17, 68);">"\""</span>) && home.endsWith(<span class="string" style="color: rgb(221, 17, 68);">"\""</span><span style="color: rgb(0, 0, 0);">)) {
       home </span>= home.substring(<span class="number" style="color: rgb(0, 153, 153);">1</span>, home.length()-<span class="number" style="color: rgb(0, 153, 153);">1</span><span style="color: rgb(0, 0, 0);">);
     }
     </span><span style="color: rgb(0, 128, 0);"><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;">//</span></span><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;"><span style="color: rgb(0, 128, 0);"> check that the home setting is actually a directory that exists</span></span><span style="color: rgb(0, 128, 0);"></span>
     File homedir = <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">new</span></span><span style="color: rgb(0, 0, 0);"> File(home);
     </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span> (!homedir.isAbsolute() || !homedir.exists() || !<span style="color: rgb(0, 0, 0);">homedir.isDirectory()) {
       </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">throw</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">new</span></span> IOException(<span class="string" style="color: rgb(221, 17, 68);">"Hadoop home directory "</span> +<span style="color: rgb(0, 0, 0);"> homedir
         </span>+ <span class="string" style="color: rgb(221, 17, 68);">" does not exist, is not a directory, or is not an absolute path."</span><span style="color: rgb(0, 0, 0);">);
     }
     home </span>=<span style="color: rgb(0, 0, 0);"> homedir.getCanonicalPath();
  } </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">catch</span></span><span style="color: rgb(0, 0, 0);"> (IOException ioe) {
    </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span><span style="color: rgb(0, 0, 0);"> (LOG.isDebugEnabled()) {
      LOG.debug(</span><span class="string" style="color: rgb(221, 17, 68);">"Failed to detect a valid hadoop home directory"</span><span style="color: rgb(0, 0, 0);">, ioe);
    }
    home </span>= <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">null</span></span><span style="color: rgb(0, 0, 0);">;
  }    
  </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">return</span></span><span style="color: rgb(0, 0, 0);"> home;
}
</span>

private static String HADOOP_HOME_DIR = checkHadoopHome ();

<span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">public</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">static</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">final</span></span><span style="color: rgb(0, 0, 0);"> String getQualifiedBinPath(String executable) 
</span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">throws</span></span><span style="color: rgb(0, 0, 0);"> IOException {
  </span><span style="color: rgb(0, 128, 0);"><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;">//</span></span><span class="comment" style="color: rgb(153, 153, 136); font-style: italic;"><span style="color: rgb(0, 128, 0);"> construct hadoop bin path to the specified executable</span></span><span style="color: rgb(0, 128, 0);"></span>
  String fullExeName = HADOOP_HOME_DIR + File.separator + <span class="string" style="color: rgb(221, 17, 68);">"bin"</span> 
    + File.separator +<span style="color: rgb(0, 0, 0);"> executable;

  File exeFile </span>= <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">new</span></span><span style="color: rgb(0, 0, 0);"> File(fullExeName);
  </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span> (!<span style="color: rgb(0, 0, 0);">exeFile.exists()) {
    </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">throw</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">new</span></span> IOException(<span class="string" style="color: rgb(221, 17, 68);">"Could not locate executable "</span> +<span style="color: rgb(0, 0, 0);"> fullExeName
      </span>+ <span class="string" style="color: rgb(221, 17, 68);">" in the Hadoop binaries."</span><span style="color: rgb(0, 0, 0);">);
  }
  </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">return</span></span><span style="color: rgb(0, 0, 0);"> exeFile.getCanonicalPath();
}

</span><span style="color: rgb(0, 128, 0);"><span class="javadoc" style="color: rgb(153, 153, 136); font-style: italic;">/**</span></span><span class="javadoc" style="color: rgb(153, 153, 136); font-style: italic;"><span style="color: rgb(0, 128, 0);"> a Windows utility to emulate Unix commands </span><span style="color: rgb(0, 128, 0);">*/</span></span><span style="color: rgb(0, 128, 0);"></span>
<span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">public</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">static</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">final</span></span> String WINUTILS =<span style="color: rgb(0, 0, 0);"> getWinUtilsPath();

</span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">public</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">static</span></span> <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">final</span></span><span style="color: rgb(0, 0, 0);"> String getWinUtilsPath() {
  String winUtilsPath </span>= <span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">null</span></span><span style="color: rgb(0, 0, 0);">;

  </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">try</span></span><span style="color: rgb(0, 0, 0);"> {
    </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">if</span></span><span style="color: rgb(0, 0, 0);"> (WINDOWS) {
      winUtilsPath </span>= getQualifiedBinPath(<span class="string" style="color: rgb(221, 17, 68);">"winutils.exe"</span><span style="color: rgb(0, 0, 0);">);
    }
  } </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">catch</span></span><span style="color: rgb(0, 0, 0);"> (IOException ioe) {
     LOG.error(</span><span class="string" style="color: rgb(221, 17, 68);">"Failed to locate the winutils binary in the hadoop binary path"</span><span style="color: rgb(0, 0, 0);">,
       ioe);
  }

  </span><span style="color: rgb(0, 0, 255);"><span class="keyword" style="color: rgb(51, 51, 51); font-weight: bold;">return</span></span><span style="color: rgb(0, 0, 0);"> winUtilsPath;
}</span>

找到原因后就去网上问了度娘，找到了解决方案，很简单，如下：

1.下载winutils的windows版本

GitHub上，有人提供了winutils的windows的版本，项目地址是：https://github.com/srccodes/hadoop-common-2.2.0-bin ,直接下载此项目的zip包，下载后是文件名是hadoop-common-2.2.0-bin-master.zip,随便解压到一个目录

2.配置环境变量

增加用户变量HADOOP_HOME，值是下载的zip包解压的目录，然后在系统变量path里增加$HADOOP_HOME\bin 即可。

再次运行程序，正常执行。