ubuntu下eclipse中hadoop中环境变量的设置
环境：ubuntu12.04（32位） eclipse-jee-luna-SR1-linux-gtk.tar.gz（32位） hadoop-2.2.0
1.安装hadoop
前一篇博客中有详细介绍 http://blog.csdn.net/nana_93/article/details/41912257
这里就不再详细说了
2.安装eclipse
下载软件：
解压后，将eclipse放在路径 /opt/中
tar -zxvf eclipse-jee-luna-SR1-linux-gtk.tar.gz
移动到/opt如下所示
给让当前用户分配opt/目录的权限
sudo chown hp:hp /opt/
进入eclipse的路径中，双击即可运行
3.在eclipse中配置hadoop环境
下载编译eclipse插件
或者直接下载已经编译好的插件：http://download.csdn.net/detail/zythy/6735167
配置hadoop插件
将下载的hadoop-eclipse-plugin-2.2.0.jar文件放到Eclipse的plugins目录下，重启Eclipse即可看到该插件已生效。
配置hadoop的安装路径windows->perspective
通过Open Perspective菜单打开Map Reduce视图，如下：
选中大象图标，右键点击New Hadoop Location编辑Hadoop配置信息：
填写正确的Map/Reduce和HDFS信息。（具体根据您的配置而定）
注意：MR Master和DFS Master配置必须和mapred-site.xml和core-site.xml等配置文件一致
打开Project Explorer,查看HDFS文件系统：
4.wordcount算法程序实现
1.新建Mapreduce项目
右键new一个WordCount的类
2.编写代码：
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {
public static class TokenizerMapper extends Mapper<Object,Text,Text,IntWritable>
{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key,Text value,Context context)
throws IOException,InterruptedException{
StringTokenizer itr = new StringTokenizer(value.toString());
while(itr.hasMoreTokens()){
word.set(itr.nextToken());
context.write(word, one);
}
}
public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable>
{
private IntWritable result = new IntWritable();
public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException,InterruptedException
{
int sum=0;
for(IntWritable val:values){
sum+=val.get();
}
result.set(sum);
context.write(key,result);
}
}
public static void main(String[] args) throws Exception
{
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: wordcount <in> <out>");
System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
}
3. 在外面新建一个文本文档word.txt，里面随便写点什么，如
java c++ python cjava c++ javascript helloworld hadoopmapreduce java hadoop hbase
然后在HDFS下新建文件夹 wordcount，将word.txt上传到wordcount下，如图所示。
hadoop fs -mkdir /tmp/wordcount
hadoop fs -copyFromLocal /home/hp/workspace/word.txt /tmp/wordcount/word.txt
4. 运行项目
选中WordCount，右击， Run AS--> Run Configuration. 在Java Application内new一个
是那个对话框内，Arguments那里写 hdfs://localhost:9000/tmp/wordcount/word.txt hdfs://localhost:9000/tmp/wordcount/out
点击run
可能会出现类似如下的这种警告
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
解决办法
在src下面新建file名为log4j.properties内容如下:文件名就是这个，一个字也不能错
log4j.rootLogger=info,stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
#dwr log config
log4j.logger.uk.ltd.getahead.dwr=
# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%c:%L) %d{yyyy-MM-dd HH:mm:ss,SSS} ---- %m%n
#log4j.appender.R=org.apache.log4j.RollingFileAppender
#log4j.appender.R.File=D:\\logs\\web_log.log
#log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
#log4j.appender.R.MaxBackupIndex=100
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %d{yyyy-MM-dd HH\:mm\:ss,SSS} %n%t %c ---- %m%n
#disable dwr log
#log4j.logger.org.directwebremoting=ERROR
再次运行则不会出现警告了
运行结果
