hadoop学习入门之伪分布式部署及测试

安装运行伪分布式Hadoop（以0.20.2版本为例）<wbr style="line-height:25px; font-family:Arial,Helvetica,simsun,u5b8bu4f53; font-size:14px"></wbr>

下载Hadoop：地址：http://www.apache.org/dyn/closer.cgi/hadoop/common/选择一个镜像地址。选择版本。

操作都在hadoop的home目录下。

准备工作

解压所下载的Hadoop发行版。编辑conf/hadoop-env.sh文件，至少需要将JAVA_HOME设置为Java安装根路径。

如下命令：
$ bin/hadoop
将会显示hadoop脚本的使用文档。

用以下三种支持的模式中的一种启动Hadoop集群：

• 单机模式
• 伪分布式模式
• 完全分布式模式

下面介绍伪分布式的配置。

伪分布式模式的操作方法

Hadoop可以在单节点上以所谓的伪分布式模式运行，此时每一个Hadoop守护进程都作为一个独立的Java进程运行。

配置

使用如下的conf/core-site.xml:

<configuration>
 <property>
 <name>fs.default.name</name>
 <value>hdfs://192.168.0.101:9000</value>
 </property>
</configuration>

conf/hdfs-site.xml:

<configuration>
 <property>
 <name>fs.replication</name>
 <value>1</value>
 </property>
</configuration>

conf/mapred-site.xml:

<configuration>
 <property>
 <name>mapred.job.tracker</name>
 <value>192.168.0.101:9001</value>  </property> </configuration>

首先，请求 namenode 对 DFS 文件系统进行格式化。在安装过程中完成了这个步骤，但是了解是否需要生成干净的文件系统是有用的。

bin/hadoop namenode -format

输出：

11/11/30 09:53:56 INFO namenode.NameNode: STARTUP_MSG:  /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = ubuntu1/192.168.0.101 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 0.20.2 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 ************************************************************/ 11/11/30 09:53:56 INFO namenode.FSNamesystem: fsOwner=root,root 11/11/30 09:53:56 INFO namenode.FSNamesystem: supergroup=supergroup 11/11/30 09:53:56 INFO namenode.FSNamesystem: isPermissionEnabled=true 11/11/30 09:53:56 INFO common.Storage: Image file of size 94 saved in 0 seconds. 11/11/30 09:53:57 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted. 11/11/30 09:53:57 INFO namenode.NameNode: SHUTDOWN_MSG:  /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ubuntu1/192.168.0.101 ************************************************************/

执行：bin/start-all.sh

输出：

starting namenode, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-ubuntu1.out localhost: starting datanode, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-datanode-ubuntu1.out localhost: starting secondarynamenode, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-secondarynamenode-ubuntu1.out starting jobtracker, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-ubuntu1.out localhost: starting tasktracker, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-tasktracker-ubuntu1.out

检查hdfs ：bin/hadoopfs -ls /

输出目录文件则正常。

hadoop文件系统操作：

bin/hadoop fs -mkdir test

bin/hadoop fs -ls test

bin/hadoop fs -rmr test

测试hadoop：

bin/hadoop fs -mkdir input

自己建立两个文本文件：file1和file2放在/opt/hadoop/sourcedata下

执行：bin/hadoopfs -put/opt/hadoop/sourcedata/file* input

执行：bin/hadoop jar hadoop-0.20.2-examples.jar wordcount input output

输出：

11/11/30 10:15:38 INFO input.FileInputFormat: Total input paths to process : 2 11/11/30 10:15:52 INFO mapred.JobClient: Running job: job_201111301005_0001 11/11/30 10:15:53 INFO mapred.JobClient: map 0% reduce 0% 11/11/30 10:19:07 INFO mapred.JobClient: map 50% reduce 0% 11/11/30 10:19:14 INFO mapred.JobClient: map 100% reduce 0% 11/11/30 10:19:46 INFO mapred.JobClient: map 100% reduce 100% 11/11/30 10:19:54 INFO mapred.JobClient: Job complete: job_201111301005_0001 11/11/30 10:19:59 INFO mapred.JobClient: Counters: 17 11/11/30 10:19:59 INFO mapred.JobClient: Job Counters  11/11/30 10:19:59 INFO mapred.JobClient: Launched reduce tasks=1 11/11/30 10:19:59 INFO mapred.JobClient: Launched map tasks=2 11/11/30 10:19:59 INFO mapred.JobClient: Data-local map tasks=2 11/11/30 10:19:59 INFO mapred.JobClient: FileSystemCounters 11/11/30 10:19:59 INFO mapred.JobClient: FILE_BYTES_READ=146 11/11/30 10:19:59 INFO mapred.JobClient: HDFS_BYTES_READ=64 11/11/30 10:19:59 INFO mapred.JobClient: FILE_BYTES_WRITTEN=362 11/11/30 10:19:59 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=60 11/11/30 10:19:59 INFO mapred.JobClient: Map-Reduce Framework 11/11/30 10:19:59 INFO mapred.JobClient: Reduce input groups=9 11/11/30 10:19:59 INFO mapred.JobClient: Combine output records=13 11/11/30 10:19:59 INFO mapred.JobClient: Map input records=2 11/11/30 10:19:59 INFO mapred.JobClient: Reduce shuffle bytes=102 11/11/30 10:19:59 INFO mapred.JobClient: Reduce output records=9 11/11/30 10:19:59 INFO mapred.JobClient: Spilled Records=26 11/11/30 10:19:59 INFO mapred.JobClient: Map output bytes=120 11/11/30 10:19:59 INFO mapred.JobClient: Combine input records=14 11/11/30 10:19:59 INFO mapred.JobClient: Map output records=14 11/11/30 10:19:59 INFO mapred.JobClient: Reduce input records=13

执行成功！

其他查看结果命令：

bin/hadoop fs -ls /user/root/output
bin/hadoop fs -cat output/part-r-00000
bin/hadoop fs -cat output/part-r-00000 | head -13
bin/hadoop fs -get output/part-r-00000 output.txt
cat output.txt | head -5
bin/hadoop fs -rmr output

也可以使用浏览器查看，地址：

http://192.168.0.101:50030(mapreduce的web页面)
http://192.168.0.101:50070(hdfs的web页面)

下面执行grep的mapreduce任务：

执行：bin/hadoop fs -rmr output

执行：bin/hadoop jar hadoop-0.20.2-examples.jar grep input output 'hadoop'

输出：

11/11/30 10:28:37 INFO mapred.FileInputFormat: Total input paths to process : 2 11/11/30 10:28:40 INFO mapred.JobClient: Running job: job_201111301005_0002 11/11/30 10:28:41 INFO mapred.JobClient: map 0% reduce 0% 11/11/30 10:34:16 INFO mapred.JobClient: map 66% reduce 0% 11/11/30 10:37:40 INFO mapred.JobClient: map 100% reduce 11% 11/11/30 10:37:50 INFO mapred.JobClient: map 100% reduce 22% 11/11/30 10:37:54 INFO mapred.JobClient: map 100% reduce 66% 11/11/30 10:38:15 INFO mapred.JobClient: map 100% reduce 100% 11/11/30 10:38:30 INFO mapred.JobClient: Job complete: job_201111301005_0002 11/11/30 10:38:32 INFO mapred.JobClient: Counters: 18 11/11/30 10:38:32 INFO mapred.JobClient: Job Counters  11/11/30 10:38:32 INFO mapred.JobClient: Launched reduce tasks=1 11/11/30 10:38:32 INFO mapred.JobClient: Launched map tasks=3 11/11/30 10:38:32 INFO mapred.JobClient: Data-local map tasks=3 11/11/30 10:38:32 INFO mapred.JobClient: FileSystemCounters 11/11/30 10:38:32 INFO mapred.JobClient: FILE_BYTES_READ=40 11/11/30 10:38:32 INFO mapred.JobClient: HDFS_BYTES_READ=77 11/11/30 10:38:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=188 11/11/30 10:38:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=109 11/11/30 10:38:32 INFO mapred.JobClient: Map-Reduce Framework 11/11/30 10:38:32 INFO mapred.JobClient: Reduce input groups=1 11/11/30 10:38:32 INFO mapred.JobClient: Combine output records=2 11/11/30 10:38:32 INFO mapred.JobClient: Map input records=2 11/11/30 10:38:32 INFO mapred.JobClient: Reduce shuffle bytes=46 11/11/30 10:38:32 INFO mapred.JobClient: Reduce output records=1 11/11/30 10:38:32 INFO mapred.JobClient: Spilled Records=4 11/11/30 10:38:32 INFO mapred.JobClient: Map output bytes=30 11/11/30 10:38:32 INFO mapred.JobClient: Map input bytes=64 11/11/30 10:38:32 INFO mapred.JobClient: Combine input records=2 11/11/30 10:38:32 INFO mapred.JobClient: Map output records=2 11/11/30 10:38:32 INFO mapred.JobClient: Reduce input records=2 11/11/30 10:38:36 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

执行：bin/hadoop fs -cat output/part-00000

输出：2hadoop

成功完成伪分布式的部署及测试。如有问题，请留言！