Hadoop权威指南环境搭建+实例运行
Hadoop权威指南---2.3.2MapReduce---环境搭建+实例运行
- 基于题目, 记录整个过程中所遇到的问题.
- 环境: windows10中创建的虚拟机Ubuntu16.04(VMware)
基于题目, 记录整个过程中所遇到的问题.
环境: windows10中创建的虚拟机Ubuntu16.04(VMware)
1.安装JDK, 安装目录: /usr/lib/jvm/java-8-oracle , 此JDK为以前所装,具体方式不详.
2.安装Hadoop(源代码编译等涉及到的目录放在后图):
(1) 官网下载Hadoop 2.8.5 二进制包
(2) 解压包: sudo tar zxvf hadoop-2.8.5.tar.gz -C /opt
(3) 创建软链接: sudo ln -snf /opt/hadoop-2.8.5 /opt/hadoop
(4) 更改环境变量: sudo vim /etc/profile
开头添加如下内容:
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export CLASSPATH=.:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.8.5.jar:$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.5.jar:$HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar:$CLASSPATH
注意: 最后一个export如果没有,在编译书中三个Java源码时javac *.java,便会找不到与Hadoop相关的类,详见文章最后错误1, 经网上多个教程踩坑, 得出上述结论.可参考:错误分析
(5) 更新环境变量: source /etc/profile
注意: 这样更新只会起临时效果, 再开一个Terminal将不起作用, 输入reboot重启即可解决.
(6) 将编译前后的文件整体打包jar -cvf MaxTemperature.jar . // 最后的 . 一定不要遗漏, 否则失败
(7) 准备数据: 因为书中数据是随便举例的,因此我在天气数据下载了008260-99999-2010.gz, 解压到 temdata重命名为008260-99999-2010.txt
(8) 新建目录, 然后建立HDFS输入路径: hadoop fs -mkdir -p example_1/data
(9) 将天气数据(008260-99999-2010.txt)复制到HDFS输入路径: hadoop fs -put temdata/008260-99999-2010.txt example_1/data
(10) 运行: hadoop jar MaxTemperature.jar MaxTemperature example_1/data/008260-99999-2010.txt example_1/dataoutput 也可以采用书中运行的方式, 更改为自己对应的文件即可.
(11) 查看两个生成文件(part-r-00000 和 _SUCCESS): hadoop fs -ls example_1/dataoutput
(12) 打印结果: hadoop fs -cat example_1/dataoutput/part-r-00000
最终显示2010 380 , 至此, 第一个例子完成!
如果对你有用, 欢迎点赞, 也欢迎评论交流.
源代码编译等涉及到的目录:
错误1:
MaxTemperature.java:3: error: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
^
MaxTemperature.java:4: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
^
MaxTemperature.java:5: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
MaxTemperature.java:6: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Job;
^
MaxTemperature.java:7: error: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
^
MaxTemperature.java:8: error: package org.apache.hadoop.mapreduce.lib.output does not exist
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
^
MaxTemperatureMapper.java:5: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
^
MaxTemperatureMapper.java:6: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.LongWritable;
^
MaxTemperatureMapper.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
MaxTemperatureMapper.java:8: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Mapper;
^
MaxTemperatureMapper.java:11: error: cannot find symbol
extends Mapper {
^
symbol: class Mapper
MaxTemperatureMapper.java:11: error: cannot find symbol
extends Mapper {
^
symbol: class LongWritable
MaxTemperatureMapper.java:11: error: cannot find symbol
extends Mapper {
^
symbol: class Text
MaxTemperatureMapper.java:11: error: cannot find symbol
extends Mapper {
^
symbol: class Text
MaxTemperatureMapper.java:11: error: cannot find symbol
extends Mapper {
^
symbol: class IntWritable
MaxTemperatureMapper.java:16: error: cannot find symbol
public void map(LongWritable key, Text value, Context context)
^
symbol: class LongWritable
location: class MaxTemperatureMapper
MaxTemperatureMapper.java:16: error: cannot find symbol
public void map(LongWritable key, Text value, Context context)
^
symbol: class Text
location: class MaxTemperatureMapper
MaxTemperatureMapper.java:16: error: cannot find symbol
public void map(LongWritable key, Text value, Context context)
^
symbol: class Context
location: class MaxTemperatureMapper
MaxTemperatureReducer.java:5: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
^
MaxTemperatureReducer.java:6: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
MaxTemperatureReducer.java:7: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Reducer;
^
MaxTemperatureReducer.java:10: error: cannot find symbol
extends Reducer {
^
symbol: class Reducer
MaxTemperatureReducer.java:10: error: cannot find symbol
extends Reducer {
^
symbol: class Text
MaxTemperatureReducer.java:10: error: cannot find symbol
extends Reducer {
^
symbol: class IntWritable
MaxTemperatureReducer.java:10: error: cannot find symbol
extends Reducer {
^
symbol: class Text
MaxTemperatureReducer.java:10: error: cannot find symbol
extends Reducer {
^
symbol: class IntWritable
MaxTemperatureReducer.java:13: error: cannot find symbol
public void reduce(Text key, Iterable values,
^
symbol: class Text
location: class MaxTemperatureReducer
MaxTemperatureReducer.java:13: error: cannot find symbol
public void reduce(Text key, Iterable values,
^
symbol: class IntWritable
location: class MaxTemperatureReducer
MaxTemperatureReducer.java:14: error: cannot find symbol
Context context)
^
symbol: class Context
location: class MaxTemperatureReducer
MaxTemperature.java:18: error: cannot find symbol
Job job = new Job();
^
symbol: class Job
location: class MaxTemperature
MaxTemperature.java:18: error: cannot find symbol
Job job = new Job();
^
symbol: class Job
location: class MaxTemperature
MaxTemperature.java:22: error: cannot find symbol
FileInputFormat.addInputPath(job, new Path(args[0]));
^
symbol: class Path
location: class MaxTemperature
MaxTemperature.java:22: error: cannot find symbol
FileInputFormat.addInputPath(job, new Path(args[0]));
^
symbol: variable FileInputFormat
location: class MaxTemperature
MaxTemperature.java:23: error: cannot find symbol
FileOutputFormat.setOutputPath(job, new Path(args[1]));
^
symbol: class Path
location: class MaxTemperature
MaxTemperature.java:23: error: cannot find symbol
FileOutputFormat.setOutputPath(job, new Path(args[1]));
^
symbol: variable FileOutputFormat
location: class MaxTemperature
MaxTemperature.java:28: error: cannot find symbol
job.setOutputKeyClass(Text.class);
^
symbol: class Text
location: class MaxTemperature
MaxTemperature.java:29: error: cannot find symbol
job.setOutputValueClass(IntWritable.class);
^
symbol: class IntWritable
location: class MaxTemperature
MaxTemperatureMapper.java:15: error: method does not override or implement a method from a supertype
@Override
^
MaxTemperatureMapper.java:29: error: cannot find symbol
context.write(new Text(year), new IntWritable(airTemperature));
^
symbol: class Text
location: class MaxTemperatureMapper
MaxTemperatureMapper.java:29: error: cannot find symbol
context.write(new Text(year), new IntWritable(airTemperature));
^
symbol: class IntWritable
location: class MaxTemperatureMapper
MaxTemperatureReducer.java:12: error: method does not override or implement a method from a supertype
@Override
^
MaxTemperatureReducer.java:18: error: cannot find symbol
for (IntWritable value : values) {
^
symbol: class IntWritable
location: class MaxTemperatureReducer
MaxTemperatureReducer.java:21: error: cannot find symbol
context.write(key, new IntWritable(maxValue));
^
symbol: class IntWritable
location: class MaxTemperatureReducer
43 errors
参考1
参考2
参考3
参考4
参考5