Week3 Homework: Analyse Data

Appendix

https://docs.mongodb.com/manual/core/aggregation-pipeline/
Manual for Pipeline
https://docs.mongodb.com/manual/reference/operator/aggregation/
Usage for bulletin operators
https://github.com/qianjiahao/MongoDB/wiki/MongoDB之索引
Chinese Index Book for mongoDB

Target

target1

target2

---

Here we Go

Analysis of a record entry, and get design ideas from it.

{'pub_date': '2016.01.13',
'look': '-', 'time': 0,
'price': 260, 'url': 'http://bj.58.com/jiadian/24652878967613x.shtml',
'_id': ObjectId('5698f525a98063dbe6e91ca8'),
'area': ['西城', '西单'], 'title': '【图】很新的海信冰箱 - 西城西单二手家电 - 北京58同城', 'cates': ['北京58同城', '北京二手市场', '北京二手家电', '北京二手冰箱']}

Basic Moves

For both targets, we have to do these basic moves first.

import pymongo
from datetime import date
from datetime import timedelta
import charts
client = pymongo.MongoClient('localhost',27017)
myDB = client['ganjiDB']
myCollection = myDB['bjGanji']

Target1

Split into the following sub-goals：

1. Get post counts of each category in one day.
2. Sum them up in a time period.
3. Sort and find out Top3.
4. Draw a histogram.

Coding:

DateDict = {}
for eachDay in date_generate(sDay,3):
    p1 = [
    {'$match':{'pub_date':eachDay}},
    {'$group':{'_id':{'$slice':['$cates',2,1]},'countsPday':{'$sum':1}}},
    {'$sort':{'countsPday':-1}}
    ]
    for i in myCollection.aggregate(p1):
        if DateDict.get(i['_id'][0]) == None:
            DateDict[i['_id'][0]] = i['countsPday']
        else:
            DateDict[i['_id'][0]] += i['countsPday']
print(DateDict)

newDict = sorted(DateDict.items(), key=lambda d:d[1], reverse = True)
print(newDict)

options = {
    'title':{'text':'Top3 category'}
}
series = []
for index in range(0,3):
    each = newDict[index]
    dat = {
        'name':each[0],
        'data':[each[1]],
        'type':'column'
    }   
    print(dat)
    series.append(dat)

charts.plot(series=series, show='inline', options=options)

Target1 result

---

Target2

pipe1 = [
    {'$match':{'$and':[{'pub_date':{'$gte':'2015.12.25','$lte':'2015.12.29'}},
                       {'cates':{'$all':['北京二手手机']}},
                       {'look':{'$nin':['-']}}
                      ]}},
    {'$group':{'_id':"$look",'avgPrice':{'$avg':"$price"}}},
    {'$sort':{'avgPrice':-1}}
]

priceList = [i['avgPrice'] for i in myCollection.aggregate(pipe1) ]
print(priceList)
series = [
    {
        'name':'北京二手手机',
        'data':priceList,
        'type':'line'
    }
]
options = {
    'chart':{'zoomType':'xy'},
    'title':{'text':'Line Chart'},
    'subtitle':{'text':'made by Jet'},
    'xAxis':{'categories':[i['_id'] for i in myCollection.aggregate(pipe1)]},
    'yAxis':{'title': {'text': 'AverangePrice'}}
}
charts.plot(series,show='inline',options=options)

Target2 result

Export collection to CSV file

mongoexport -d database -c collection -o output/path.csv

This command is used in your terminal.

mongoexport -d ganjiDB -c bjGanji -o  User/aaa.csv
or
mongoexport -d ganjiDB -c bjGanji -o  User/bbb.json