Hadoop Streaming Made Simple using Joins and Keys with Python

There are a lot of different ways to write MapReduce jobs!!!

Sample code for this post https://github.com/joestein/amaunet

I find streaming scripts a good way to interrogate data sets (especially when I have not worked with them yet or are creating new ones) and enjoy the lifecycle when the initial elaboration of the data sets lead to the construction of the finalized scripts for an entire job (or series of jobs as is often the case).

When doing streaming with Hadoop you do have a few library options.  If you are a Ruby programmer then wukong is awesome! For Python programmers you can use dumbo and more recently released mrjob.

I like working under the hood myself and getting down and dirty with the data and here is how you can too.

Lets start first with defining two simple sample data sets.

Data set 1:  countries.dat

name|key

United States|US
Canada|CA
United Kingdom|UK
Italy|IT

Data set 2: customers.dat

name|type|country

Alice Bob|not bad|US
Sam Sneed|valued|CA
Jon Sneed|valued|CA
Arnold Wesise|not so good|UK
Henry Bob|not bad|US
Yo Yo Ma|not so good|CA
Jon York|valued|CA
Alex Ball|valued|UK
Jim Davis|not so bad|JA

The requirements: you need to find out grouped by type of customer how many of each type are in each country with the name of the country listed in the countries.dat in the final result (and not the 2 digit country name).

To-do this you need to:

1) Join the data sets
2) Key on country
3) Count type of customer per country
4) Output the results

So first lets code up a quick mapper called smplMapper.py (you can decide if smpl is short for simple or sample).

Now in coding the mapper and reducer in Python the basics are explained nicely herehttp://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/ but I am going to dive a bit deeper to tackle our example with some more tactics.

#!/usr/bin/env python

import sys

# input comes from STDIN (standard input)
for line in sys.stdin:
    try: #sometimes bad data can cause errors use this how you like to deal with lint and bad data

        personName = "-1" #default sorted as first
        personType = "-1" #default sorted as first
        countryName = "-1" #default sorted as first
        country2digit = "-1" #default sorted as first

        # remove leading and trailing whitespace
        line = line.strip()

        splits = line.split("|")

        if len(splits) == 2: #country data
            countryName = splits[0]
            country2digit = splits[1]
        else: #people data
            personName = splits[0]
            personType = splits[1]
            country2digit = splits[2]

        print '%s^%s^%s^%s' % (country2digit,personType,personName,countryName)
    except: #errors are going to make your job fail which you may or may not want
        pass

Don’t forget:

chmod a+x smplMapper.py

Great! We just took care of #1 but time to test and see what is going to the reducer.

From the command line run:

cat customers.dat countries.dat|./smplMapper.py|sort

Which will result in:

CA^-1^-1^Canada
CA^not so good^Yo Yo Ma^-1
CA^valued^Jon Sneed^-1
CA^valued^Jon York^-1
CA^valued^Sam Sneed^-1
IT^-1^-1^Italy
JA^not so bad^Jim Davis^-1
UK^-1^-1^United Kingdom
UK^not so good^Arnold Wesise^-1
UK^valued^Alex Ball^-1
US^-1^-1^United States
US^not bad^Alice Bob^-1
US^not bad^Henry Bob^-1

Notice how this is sorted so the country is first and the people in that country after it (so we can grab the correct country name as we loop) and with the type of customer also sorted (but within country) so we can properly count the types within the country. =8^)

Let us hold off on #2 for a moment (just hang in there it will all come together soon I promise) and get smplReducer.py working first.

#!/usr/bin/env python

import sys

# maps words to their counts
foundKey = ""
foundValue = ""
isFirst = 1
currentCount = 0
currentCountry2digit = "-1"
currentCountryName = "-1"
isCountryMappingLine = False

# input comes from STDIN
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()

    try:
        # parse the input we got from mapper.py
        country2digit,personType,personName,countryName = line.split('^')

        #the first line should be a mapping line, otherwise we need to set the currentCountryName to not known
        if personName == "-1": #this is a new country which may or may not have people in it
            currentCountryName = countryName
            currentCountry2digit = country2digit
            isCountryMappingLine = True
        else:
            isCountryMappingLine = False # this is a person we want to count

        if not isCountryMappingLine: #we only want to count people but use the country line to get the right name

            #first check to see if the 2digit country info matches up, might be unkown country
            if currentCountry2digit != country2digit:
                currentCountry2digit = country2digit
                currentCountryName = '%s - Unkown Country' % currentCountry2digit

            currentKey = '%s\t%s' % (currentCountryName,personType)

            if foundKey != currentKey: #new combo of keys to count
                if isFirst == 0:
                    print '%s\t%s' % (foundKey,currentCount)
                    currentCount = 0 #reset the count
                else:
                    isFirst = 0

                foundKey = currentKey #make the found key what we see so when we loop again can see if we increment or print out

            currentCount += 1 # we increment anything not in the map list