Hadoop学习

例子1： 统计一个文本文件里的单词数量

[hadoop@mylinux ~]$ cd hadoop-0.20.2/
[hadoop@mylinux hadoop-0.20.2]$ mkdir input   #在本地系统建立目录 input
[hadoop@mylinux hadoop-0.20.2]$ cd input       #进入input目录
[hadoop@mylinux input]$ vi file01                     #新建一个文件file01
hello hadoop 
this is first examples by huxin
hadoop
[hadoop@mylinux hadoop-0.20.2]$ cd
[hadoop@mylinux ~]$ cd hadoop-0.20.2/
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -mkdir input   在hadoop文件系统中建立目录input,注意和上面的本地系统目录input区分开
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -put ./input/file01 input   #将本地的file01放进hadoop文件系统的目录中
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -ls input
Found 1 items
-rw-r--r--   1 hadoop supergroup         54 2011-06-22 19:25 /user/hadoop/input/file01
[hadoop@mylinux hadoop-0.20.2]$ hadoop jar hadoop-0.20.2-examples.jar wordcount input  output #对hadoop文件系统进行count单词数，完成后自动生成一个output目录
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -ls output
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2011-06-22 19:26 /user/hadoop/output/_logs
-rw-r--r--   1 hadoop supergroup         61 2011-06-22 19:26 /user/hadoop/output/part-r-00000
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -cat output/part-r-00000   #查看part-r-00000
by      1
examples        1
first   1
hadoop  2
hello   1
huxin   1
is      1
this    1
-----------------------
Done!

 

Wordcount源码是在src/examples/org/apache/hadoop/examples/WordCount.java这里。
下面我们来手动编译和执行一遍这段原代码吧：
[hadoop@mylinux ~]$ cd hadoop-0.20.2/
[hadoop@mylinux hadoop-0.20.2]$ mkdir playground
[hadoop@mylinux hadoop-0.20.2] $mkdir playground/src
[hadoop@mylinux hadoop-0.20.2]$ mkdir playground/classes
[hadoop@mylinux hadoop-0.20.2]$ cp src/examples/org/apache/hadoop/examples/WordCount.java  playground/src/WordCount.java
[[hadoop@mylinux hadoop-0.20.2]$ javac -classpath hadoop-0.20.2-core.jar:lib/commons-cli-1.2.jar -d playground/classes/ playground/src/WordCount.java
[hadoop@mylinux hadoop-0.20.2]$ jar -cvf playground/wordcount.jar -C playground/classes/ .
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -rmr output  #记得先删除前面我们建立过的output目录
Deleted hdfs://master:9000/user/hadoop/output
[hadoop@mylinux hadoop-0.20.2]$ hadoop jar playground/wordcount.jar org.apache.hadoop.examples.WordCount input output
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -ls output
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2011-06-22 19:55 /user/hadoop/output/_logs
-rw-r--r--   1 hadoop supergroup         61 2011-06-22 19:55 /user/hadoop/output/part-r-00000
[hadoop@mylinux hadoop-0.20.2]$ hadoop fs -cat output/part-r-00000   #查看part-r-00000
by      1
examples        1
first   1
hadoop  2
hello   1
huxin   1
is      1
this    1
-----
done