Hadoop/spark安装实战（系列篇4） Hadoop MapReduce词频统计之小试牛刀

 Hadoop/spark安装实战（系列篇4） Hadoop MapReduce词频统计之小试牛刀

运行hadoop 自带的例子的MapReduce 计算

1 上传文件到hadoop的hdfs的根目录
[root@localhost hadoop-1.2.1]# hadoop fs -put README.txt /
检验
[root@localhost hadoop-1.2.1]# hadoop fs -ls /
Found 2 items
-rw-r--r--   1 root supergroup       1366 2015-09-12 06:45 /README.txt
drwxr-xr-x   - root supergroup          0 2015-09-12 06:34 /home
[root@localhost hadoop-1.2.1]#

2 运行MapReduce 分词统计

[root@localhost hadoop-1.2.1]# hadoop jar  hadoop-examples-1.2.1.jar  wordcount /README.txt /wordcountoutput
Warning: $HADOOP_HOME is deprecated.

15/09/12 06:48:00 INFO input.FileInputFormat: Total input paths to process : 1
15/09/12 06:48:00 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/09/12 06:48:00 WARN snappy.LoadSnappy: Snappy native library not loaded
15/09/12 06:48:03 INFO mapred.JobClient: Running job: job_201509120634_0001
15/09/12 06:48:04 INFO mapred.JobClient:  map 0% reduce 0%
15/09/12 06:48:27 INFO mapred.JobClient:  map 100% reduce 0%
15/09/12 06:48:34 INFO mapred.JobClient:  map 100% reduce 33%
15/09/12 06:48:36 INFO mapred.JobClient:  map 100% reduce 100%
15/09/12 06:48:36 INFO mapred.JobClient: Job complete: job_201509120634_0001
15/09/12 06:48:36 INFO mapred.JobClient: Counters: 29
15/09/12 06:48:36 INFO mapred.JobClient:   Job Counters
15/09/12 06:48:36 INFO mapred.JobClient:     Launched reduce tasks=1

3/运行结果
[root@localhost hadoop-1.2.1]# hadoop fs -ls /wordcountoutput
Warning: $HADOOP_HOME is deprecated.

Found 3 items
-rw-r--r--   1 root supergroup          0 2015-09-12 06:48 /wordcountoutput/_SUCCESS
drwxr-xr-x   - root supergroup          0 2015-09-12 06:48 /wordcountoutput/_logs
-rw-r--r--   1 root supergroup       1306 2015-09-12 06:48 /wordcountoutput/part-r-00000

[root@localhost hadoop-1.2.1]#  hadoop fs -cat  /wordcountoutput/part-r-00000
Warning: $HADOOP_HOME is deprecated.

(BIS),  1
(ECCN)  1
(TSU)   1
(see    1
5D002.C.1,      1
740.13) 1
     1
Administration  1
Apache  1
BEFORE  1
BIS     1
Bureau  1
Commerce,       1
Commodity       1
Control 1
Core    1
Department      1
ENC     1
Exception       1
Export  2
For     1

。。。。。

hadoop小试牛刀 OK