hadoop 2.3.0-cdh5.0.0笔记

pom.xml

<propertiest>
    <hadoop.version>2.3.0-cdh5.0.0hadoop.version>
    <hadoop.mr.version>2.3.0-mr1-cdh5.0.0hadoop.mr.version>
properties>

<dependencies>
    <dependency>
        <groupId>org.apache.hadoopgroupId>
        <artifactId>hadoop-commonartifactId>
        <version>${hadoop.version}version>
    dependency>
    <dependency>
        <groupId>org.apache.hadoopgroupId>
        <artifactId>hadoop-coreartifactId>
        <version>${hadoop.version}version>
    dependency>
dependencies>

hadoop配置

将hadoop的三个配置core-site.xml、hdfs-site.xml、mapred-site.xml复制到resources/hadoop
程序中使用

JobConf jobconf = new JobConf(WordCount.class);
jobconf.setJobName("WordCount");
jobconf.addResource("classpath:/hadoop/core-site.xml");
jobconf.addResource("classpath:/hadoop/hdfs-site.xml");
jobconf.addResource("classpath:/hadoop/mapred-site.xml");

来加载

windows环境运行

抛出Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

1.下载winutils的windows版本
　　GitHub上，有人提供了winutils的windows的版本，项目地址是：https://github.com/srccodes/hadoop-common-2.2.0-bin，直接下载此项目的zip包，下载后是文件名是hadoop-common-2.2.0-bin-master.zip，随便解压到一个目录
2.配置环境变量
　　增加用户变量HADOOP_HOME，值是下载的zip包解压的目录，然后在系统变量path里增加$HADOOP_HOME\bin 即可