Hadoop学习-编译并运行maxTemperature

按照书中样例编写代码

其中的MaxTemperatureMapper类定义有错误，应该写为

public class MaxTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

使用mvn命令编译打包

mvn clean compile package

打包成功后生成jar包

maxTemperature-1.0-SNAPSHOT.jar

在Hadoop上创建输入路径及文件

hdfs dfs -mkdir input/ncdc
hdfs dfs -put input/ncdc/sample.txt /user/lianyz/input/ncdc/sample.txt

使用以下命令查看hadoop路径及文件

hdfs dfs -ls -R

删除多余的文件

hdfs dfs -rm -r ncdc

运行程序计算sample.txt

hadoop jar maxTemperature-1.0-SNAPSHOT.jar com.lianyz.hadoop.MaxTemperature input/ncdc/sample.txt output/maxTemperature

查看计算结果

hdfs dfs -cat output/maxTemperature/part-r-00000

计算结果如下

2020-05-03 08:07:07,462 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-03 08:07:08,044 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1949	111
1950	22

运行程序计算all

hdfs dfs -put input/ncdc/all/*.gz /user/lianyz/input/ncdc/
hdfs dfs -rm input/ncdc/sample.txt
hdfs dfs -rm output/maxTemperature
hadoop jar maxTemperature-1.0-SNAPSHOT.jar com.lianyz.hadoop.MaxTemperature input/ncdc output/maxTemperature

查看计算结果

hdfs dfs -cat output/maxTemperature/part-r-00000

计算结果如下

2020-05-03 08:13:31,537 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-03 08:13:32,133 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1901	333
1902	328