MovieLens 1M之python数据分析练习
数据集来源https://grouplens.org/datasets/movielens/1m/
代码区：
import pandas as pd
uname=['user_id','gender','age','occupation','zip']
users=pd.read_table(r'D:\demo1\ml-1m\users.dat',sep='::',header=None,names=uname,engine = 'python')
'''
sep : str, default ‘,’
指定分隔符。如果不指定参数，则会尝试使用逗号分隔。分隔符长于一个字符并且不是‘\s+’,
将使用python的语法分析器。并且忽略数据中的逗号。正则表达式例子：'\r\t'
header : int or list of ints, default ‘infer’指定行数用来作为列名，数据开始行数。
names : array-like, default None
用于结果的列名列表，如果数据文件中没有列标题行，就需要执行header=None。
engine解析器引擎使用。C引擎速度更快，而python引擎目前更加完善。除去警告
'''
rnames=['user_id','movie_id','rating','timestamp']
ratings=pd.read_table(r'D:\demo1\ml-1m\ratings.dat',sep='::',header=None,names=rnames,engine = 'python')
mname=['movie_id','title','genres']
movies=pd.read_table(r'D:\demo1\ml-1m\movies.dat',sep='::',header=None,names=mname,engine = 'python')
data=pd.merge(pd.merge(movies,ratings),users)
print data.loc[0]#ix[0]已经deprecated弃用
结果：
movie_id 1
title Toy Story (1995)
genres Animation|Children's|Comedy
user_id 1
rating 5
timestamp 978824268
gender F
age 1
occupation 10
zip 48067'''
#枢轴表pandas.pivot_table(data, values=None,
index=None, columns=None, aggfunc='mean', fill_value=None, margins=False, dropna=True, margins_name='All')
'''
mean_ratings=data.pivot_table('rating',index='title',columns='gender',aggfunc='mean')
print mean_ratings[:5]result:
gender F M
title
$1,000,000 Duck (1971) 3.375000 2.761905
'Night Mother (1986) 3.388889 3.352941
'Til There Was You (1997) 2.675676 2.733333
'burbs, The (1989) 2.793478 2.962085
...And Justice for All (1979) 3.828571 3.689024#过滤数据不足200条的电影
ratings_groupby_title=data.groupby('title').size()
print ratings_groupby_title[:5]
reslut:
title
$1,000,000 Duck (1971) 37
'Night Mother (1986) 70
'Til There Was You (1997) 52
'burbs, The (1989) 303
...And Justice for All (1979) 199
dtype: int64
active_titles=data.groupby('title').size().index[data.groupby('title').size()>=200]
print active_titlesresult:
Index([u''burbs, The (1989)', u'10 Things I Hate About You (1999)',
u'101 Dalmatians (1961)', u'101 Dalmatians (1996)',
u'12 Angry Men (1957)', u'13th Warrior, The (1999)',
u'2 Days in the Valley (1996)', u'20,000 Leagues Under the Sea (1954)',
u'2001: A Space Odyssey (1968)', u'2010 (1984)',
...
u'Year of Living Dangerously (1982)', u'Yellow Submarine (1968)',
u'Yojimbo (1961)', u'You've Got Mail (1998)',
u'Young Frankenstein (1974)', u'Young Guns (1988)',
u'Young Guns II (1990)', u'Young Sherlock Holmes (1985)',
u'Zero Effect (1998)', u'eXistenZ (1999)'],
dtype='object', name=u'title', length=1426)mean_ratings=mean_ratings.loc[active_titles]
#对F列进行降序
top_female_rating=mean_ratings.sort_values(by='F',ascending='False')
print top_female_rating[:10]result：
gender F M
title
Battlefield Earth (2000) 1.574468 1.616949
Barb Wire (1996) 1.585366 2.100386
Showgirls (1995) 1.709091 2.166667
Jaws 3-D (1983) 1.863636 1.851064
Rocky V (1990) 1.878788 2.132780
Speed 2: Cruise Control (1997) 1.906667 1.863014
Avengers, The (1998) 1.915254 2.017467
Anaconda (1997) 2.000000 2.248447
Nightmare on Elm Street 5: The Dream Child, A (... 2.052632 1.981481
Howard the Duck (1986) 2.074627 2.103542计算评分分歧
mean_ratings['diff']=mean_ratings['M']-mean_ratings['F']
sorted_by_diff=mean_ratings.sort_values(by='diff')
print sorted_by_diff[:5]result：
gender F M
title
Dirty Dancing (1987) 3.790378 2.959596
To Wong Foo, Thanks for Everything! Julie Newma... 3.486842 2.795276
Jumpin' Jack Flash (1986) 3.254717 2.578358
Grease (1978) 3.975265 3.367041
Relic, The (1997) 3.309524 2.723077
gender diff
title
Dirty Dancing (1987) -0.830782
To Wong Foo, Thanks for Everything! Julie Newma... -0.691567
Jumpin' Jack Flash (1986) -0.676359
Grease (1978) -0.608224
Relic, The (1997) -0.586447 记一个笔记：脚本实现txt替换
#把文件内容替换
#把file3.txt 的 :: 替换为 ,,并保存到file4.txt
import re
fp3=open("file3.txt","r")
fp4=open("file4.txt","w")
for s in fp3.readlines():#先读出来
fp4.write(s.replace("::",",")) #替换 并写入
fp3.close()
fp4.close()