Hadoop 3.0.0-alpha2安装(三）之Python编写MapReduce

4.5.4 Python编写MapReduce

# mkdir -p /data/test  && cd /data/test

# vim input.txt

foo foo quux labs foo bar quux abx bar see you bytest welcome test abc labs foo me python hadoop ab ac bc bec python

# vim mapper.py    # 刘天斯 著《Python 自动化运维》中的代码

#!/usr/bin/env python

import sys

for line in sys.stdin:

    line =line.strip()

    words =line.split()

    for word inwords:

        print'%s\t%s' % (word, 1)

# vim reducer.py   # 刘天斯 著《Python 自动化运维》中的代码

#!/usr/bin/env python

from operator import itemgetter

import sys

 

current_word = None

current_count = 0

word = None

 

for line in sys.stdin:

    line =line.strip()

    word,count= line.split('\t', 1)

 

    try:

        count =int(count)

    except ValueError:

       continue

 

    ifcurrent_word == word:

       current_count += count

    else:

        ifcurrent_word:

           print '%s\t%s' % (current_word, current_count)

       current_count = count

       current_word = word

 

if current_word == word:

    print'%s\t%s' % (current_word, current_count)

# chmod +x *.py

# cat input.txt |./mapper.py    # 测试py文件

# cat input.txt |./mapper.py |sort -k1,1|./reducer.py

$ hdf dfs -mkdir /user/hduser/input/word

$ hdfs dfs -put /data/test/input.txt/user/hduser/input/word

$ hadoop jar../share/hadoop/tools/lib/hadoop-streaming-3.0.0-alpha2.jar -file/data/test/mapper.py -mapper /data/test/mapper.py -file /data/test/reducer.py-reducer /data/test/reducer.py -input /user/hduser/input/word -output/user/hadoop/word

$ hdfs dfs -ls /user/hadoop/word   # 查看生成的文件

Found 2 items

-rw-r--r--   2hadoop supergroup          0 2017-04-0509:53 /user/hadoop/word/_SUCCESS

-rw-r--r--   2hadoop supergroup        117 2017-04-0509:53 /user/hadoop/word/part-00000

$ hdfs dfs -cat/user/hadoop/word/part-00000

ab      1

abc     1

abx     1

ac      1

bar     2

bc      1

bec     1

by      1

foo     4

hadoop  1

labs    2

me      1

python  2

quux    2

see     1

test    2

welcomne       1

you     1

 ### good  成功了，继续努力……

### 以上python代码也可以这里看到：http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/