mapreduce 三种运行方式

mapreduce 三种运行方式

• Local (Standalone) Mode 本地模式
• Pseudo-Distributed Mode 尾分布式模式
• Fully-Distributed Mode 分布式模式

第一种 Local Mode

****Local (Standalone) Mode ****


cd 到hadoop安装目录：
  $ mkdir input
  $ cp etc/hadoop/*.xml input
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar grep input output 'dfs[a-z.]+'
  $ cat output/*
  
  经典案例：
 
xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0$ ls

bin  etc  include  input  lib  libexec  output  sbin  share

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0$ mkdir wcinput

mkdir: 无法创建目录"wcinput": 权限不够

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0$ sudo mkdir wcinput

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0$ ls

bin  etc  include  input  lib  libexec  output  sbin  share  wcinput

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0$ cd wcinput

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0/wcinput$ touch
wc.input

touch: 无法创建'wc.input': 权限不够

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0/wcinput$ sudo touch wc.input

xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.0/wcinput$ vim wc.input

第二种： Pseudo-Distributed Mode 伪分布式模式

etc/hadoop/core-site.xml:

配置一：
    
        fs.defaultFS
         hdfs://xiangkun-qin:8020
    

配置二：

    
        hadoop.tmp.dir
        /opt/modules/hadoop-2.5.0/data/tmp
    

etc/hadoop/hdfs-site.xml:

    
        dfs.replication
        1
    
### 

格式化：

  $ bin/hdfs namenode -format

启动：
xiangkun@xiangkun-X550LD:/opt/modules/hadoop-2.5.

[root@xiangkun hadoop-2.5.0]# sudo sbin/hadoop-daemon.sh start namenode

starting namenode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-root-namenode-xiangkun.out

[root@xiangkun hadoop-2.5.0]# sudo sbin/hadoop-daemon.sh start datanode

starting datanode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-root-datanode-xiangkun.out

[root@xiangkun hadoop-2.5.0]# jps

11875 DataNode
11799 NameNode
11946 Jps

hdfs文件系统运行之后，使用如下命令，创建一个多级目录（-P表示多级目录）

  [root@xiangkun-qin hadoop-2.5.0]# bin/hdfs dfs -mkdir -p /user/xiangkun_qin

##创建一个多级目录：
[root@xiangkun-qin hadoop-2.5.0]# bin/hdfs dfs -mkdir -p /user/xiangkun/mapreduce/wordcount/input

17/07/02 04:55:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

##将wc.input文件上传到 input目录下面（上传put 下载get ）
[root@xiangkun-qin hadoop-2.5.0]# bin/hdfs dfs -put wcinput/wc.input   /user/xiangkun/mapreduce/wordcount/input

##运行mapreduce程序：
 [root@xiangkun-qin hadoop-2.5.0]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar  wordcount /user/xiangkun/mapreduce/wordcount/input/ /user/xiangkun/mapreduce/wordcount/output