pandas教程：MovieLens 1M Dataset MovieLens 1M数据集
文章目录
- 14.2 MovieLens 1M Dataset（MovieLens 1M数据集）
- 1 Measuring Rating Disagreement（计算评分分歧）
14.2 MovieLens 1M Dataset（MovieLens 1M数据集）
这个数据集是电影评分数据：包括电影评分，电影元数据（风格类型，年代）以及关于用户的人口统计学数据（年龄，邮编，性别，职业等）。
MovieLens 1M数据集含有来自6000名用户对4000部电影的100万条评分数据。分为三个表：评分，用户信息，电影信息。这些数据都是dat文件格式，可以通过pandas.read_table将各个表分别读到一个pandas DataFrame对象中：
import pandas as pd
# Make display smaller
pd.options.display.max_rows = 10
unames = ['user_id', 'gender', 'age', 'occupation', 'zip']
users = pd.read_table('../datasets/movielens/users.dat', sep='::',
header=None, names=unames)
/Users/xu/anaconda/envs/py35/lib/python3.5/site-packages/ipykernel/__main__.py:3: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
app.launch_new_instance()
因为sep='::'有点像是正则表达式，于是有了上面的错误。在这个帖子找到了解决方法，设置engine为python即可。
Looks like on Python 2.7 Pandas just doesn’t handle separators that
look regexish. The initial “error” can be worked around by adding
engine=‘python’ as a named parameter in the call, as suggested in the
warning.
users = pd.read_table('../datasets/movielens/users.dat', sep='::',
header=None, names=unames, engine='python')
rnames = ['user_id', 'movie_id', 'rating', 'timestamp']
ratings = pd.read_table('../datasets/movielens/ratings.dat', sep='::', header=None, names=rnames, engine='python')
mnames = ['movie_id', 'title', 'genres']
movies = pd.read_table('../datasets/movielens/movies.dat', sep='::', header=None, names=mnames, engine='python')
加载前几行验证一下数据加载工作是否顺利
users[:5]
| user_id | gender | age | occupation | zip | |
|---|---|---|---|---|---|
| 0 | 1 | F | 1 | 10 | 48067 |
| 1 | 2 | M | 56 | 16 | 70072 |
| 2 | 3 | M | 25 | 15 | 55117 |
| 3 | 4 | M | 45 | 7 | 02460 |
| 4 | 5 | M | 25 | 20 | 55455 |
ratings[:5]
| user_id | movie_id | rating | timestamp | |
|---|---|---|---|---|
| 0 | 1 | 1193 | 5 | 978300760 |
| 1 | 1 | 661 | 3 | 978302109 |
| 2 | 1 | 914 | 3 | 978301968 |
| 3 | 1 | 3408 | 4 | 978300275 |
| 4 | 1 | 2355 | 5 | 978824291 |
movies[:5]
| movie_id | title | genres | |
|---|---|---|---|
| 0 | 1 | Toy Story (1995) | Animation|Children's|Comedy |
| 1 | 2 | Jumanji (1995) | Adventure|Children's|Fantasy |
| 2 | 3 | Grumpier Old Men (1995) | Comedy|Romance |
| 3 | 4 | Waiting to Exhale (1995) | Comedy|Drama |
| 4 | 5 | Father of the Bride Part II (1995) | Comedy |
注意，年龄和职业是以编码形式给出的，它们的具体含义请参考改数据集的README文件。分析散布在三个表中的数据不是一件轻松的事情。假设我们想要根据性别和年龄来计算某部电影的平均得分，如果将所有的数据都合并到一个表中的话，问题就简单多了。我们先用pandas的merge函数将ratings和users合并到一起，然后再将movies也合并进去。pandas会根据列名的重叠情况推断出哪些列是合并（或连接）键：
data = pd.merge(pd.merge(ratings, users), movies)
data.head()
| user_id | movie_id | rating | timestamp | gender | age | occupation | zip | title | genres | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 1193 | 5 | 978300760 | F | 1 | 10 | 48067 | One Flew Over the Cuckoo's Nest (1975) | Drama |
| 1 | 2 | 1193 | 5 | 978298413 | M | 56 | 16 | 70072 | One Flew Over the Cuckoo's Nest (1975) | Drama |
| 2 | 12 | 1193 | 4 | 978220179 | M | 25 | 12 | 32793 | One Flew Over the Cuckoo's Nest (1975) | Drama |
| 3 | 15 | 1193 | 4 | 978199279 | M | 25 | 7 | 22903 | One Flew Over the Cuckoo's Nest (1975) | Drama |
| 4 | 17 | 1193 | 5 | 978158471 | M | 50 | 1 | 95350 | One Flew Over the Cuckoo's Nest (1975) | Drama |
data.iloc[0]
user_id 1
movie_id 1193
rating 5
timestamp 978300760
gender F
age 1
occupation 10
zip 48067
title One Flew Over the Cuckoo's Nest (1975)
genres Drama
Name: 0, dtype: object
现在，只要稍微熟悉一下pandas，就能轻松地根据任意个用户或电影属性对评分数据进行聚合操作了。为了按性别计算每部电影的平均得分，我们可以使用pivot_table方法：
mean_ratings = data.pivot_table('rating', index='title',
columns='gender', aggfunc='mean')
mean_ratings[:5]
| gender | F | M |
|---|---|---|
| title | ||
| $1,000,000 Duck (1971) | 3.375000 | 2.761905 |
| 'Night Mother (1986) | 3.388889 | 3.352941 |
| 'Til There Was You (1997) | 2.675676 | 2.733333 |
| 'burbs, The (1989) | 2.793478 | 2.962085 |
| ...And Justice for All (1979) | 3.828571 | 3.689024 |
该操作产生了另一个DataFrame，其内容为电影平均得分，行标为电影名称，列表为性别。现在，我们打算过滤掉评分数据不够250条的电影（这个数字可以自己设定）。为了达到这个目的，我们先对title进行分组，然后利用size()得到一个含有各电影分组大小的Series对象：
ratings_by_title = data.groupby('title').size()
ratings_by_title[:10]
title
$1,000,000 Duck (1971) 37
'Night Mother (1986) 70
'Til There Was You (1997) 52
'burbs, The (1989) 303
...And Justice for All (1979) 199
1-900 (1994) 2
10 Things I Hate About You (1999) 700
101 Dalmatians (1961) 565
101 Dalmatians (1996) 364
12 Angry Men (1957) 616
dtype: int64
active_titles = ratings_by_title.index[ratings_by_title >= 250]
print(active_titles)
Index([''burbs, The (1989)', '10 Things I Hate About You (1999)',
'101 Dalmatians (1961)', '101 Dalmatians (1996)', '12 Angry Men (1957)',
'13th Warrior, The (1999)', '2 Days in the Valley (1996)',
'20,000 Leagues Under the Sea (1954)', '2001: A Space Odyssey (1968)',
'2010 (1984)',
...
'X-Men (2000)', 'Year of Living Dangerously (1982)',
'Yellow Submarine (1968)', 'You've Got Mail (1998)',
'Young Frankenstein (1974)', 'Young Guns (1988)',
'Young Guns II (1990)', 'Young Sherlock Holmes (1985)',
'Zero Effect (1998)', 'eXistenZ (1999)'],
dtype='object', name='title', length=1216)
上面的active_titles中的电影，都是评论是大于250条以上的。我们可以用这些标题作为索引，从mean_ratings中选出这些评论大于250条的电影：
mean_ratings = mean_ratings.loc[active_titles]
mean_ratings
| gender | F | M |
|---|---|---|
| title | ||
| 'burbs, The (1989) | 2.793478 | 2.962085 |
| 10 Things I Hate About You (1999) | 3.646552 | 3.311966 |
| 101 Dalmatians (1961) | 3.791444 | 3.500000 |
| 101 Dalmatians (1996) | 3.240000 | 2.911215 |
| 12 Angry Men (1957) | 4.184397 | 4.328421 |
| ... | ... | ... |
| Young Guns (1988) | 3.371795 | 3.425620 |
| Young Guns II (1990) | 2.934783 | 2.904025 |
| Young Sherlock Holmes (1985) | 3.514706 | 3.363344 |
| Zero Effect (1998) | 3.864407 | 3.723140 |
| eXistenZ (1999) | 3.098592 | 3.289086 |
1216 rows × 2 columns
想要查看女性观众喜欢的电影，可以按F列进行降序操作：
top_female_ratings = mean_ratings.sort_values(by='F', ascending=False)
top_female_ratings[:10]
| gender | F | M |
|---|---|---|
| title | ||
| Close Shave, A (1995) | 4.644444 | 4.473795 |
| Wrong Trousers, The (1993) | 4.588235 | 4.478261 |
| Sunset Blvd. (a.k.a. Sunset Boulevard) (1950) | 4.572650 | 4.464589 |
| Wallace & Gromit: The Best of Aardman Animation (1996) | 4.563107 | 4.385075 |
| Schindler's List (1993) | 4.562602 | 4.491415 |
| Shawshank Redemption, The (1994) | 4.539075 | 4.560625 |
| Grand Day Out, A (1992) | 4.537879 | 4.293255 |
| To Kill a Mockingbird (1962) | 4.536667 | 4.372611 |
| Creature Comforts (1990) | 4.513889 | 4.272277 |
| Usual Suspects, The (1995) | 4.513317 | 4.518248 |
1 Measuring Rating Disagreement（计算评分分歧）
假设我们想要找出男性和女性观众分歧最大的电影。一个办法是给mean_ratings加上一个用于存放平均得分之差的列，并对其进行排序：
mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
按‘diff’排序即可得到分歧最大且女性观众更喜欢的电影：
sorted_by_diff = mean_ratings.sort_values(by='diff')
sorted_by_diff[:15]
| gender | F | M | diff |
|---|---|---|---|
| title | |||
| Dirty Dancing (1987) | 3.790378 | 2.959596 | -0.830782 |
| Jumpin' Jack Flash (1986) | 3.254717 | 2.578358 | -0.676359 |
| Grease (1978) | 3.975265 | 3.367041 | -0.608224 |
| Little Women (1994) | 3.870588 | 3.321739 | -0.548849 |
| Steel Magnolias (1989) | 3.901734 | 3.365957 | -0.535777 |
| ... | ... | ... | ... |
| French Kiss (1995) | 3.535714 | 3.056962 | -0.478752 |
| Little Shop of Horrors, The (1960) | 3.650000 | 3.179688 | -0.470312 |
| Guys and Dolls (1955) | 4.051724 | 3.583333 | -0.468391 |
| Mary Poppins (1964) | 4.197740 | 3.730594 | -0.467147 |
| Patch Adams (1998) | 3.473282 | 3.008746 | -0.464536 |
15 rows × 3 columns
对行进行反序操作，并取出前15行，得到的则是男性更喜欢，而女性评价较低的电影：
# Reverse order of rows, take first 10 rows
sorted_by_diff[::-1][:10]
| gender | F | M | diff |
|---|---|---|---|
| title | |||
| Good, The Bad and The Ugly, The (1966) | 3.494949 | 4.221300 | 0.726351 |
| Kentucky Fried Movie, The (1977) | 2.878788 | 3.555147 | 0.676359 |
| Dumb & Dumber (1994) | 2.697987 | 3.336595 | 0.638608 |
| Longest Day, The (1962) | 3.411765 | 4.031447 | 0.619682 |
| Cable Guy, The (1996) | 2.250000 | 2.863787 | 0.613787 |
| Evil Dead II (Dead By Dawn) (1987) | 3.297297 | 3.909283 | 0.611985 |
| Hidden, The (1987) | 3.137931 | 3.745098 | 0.607167 |
| Rocky III (1982) | 2.361702 | 2.943503 | 0.581801 |
| Caddyshack (1980) | 3.396135 | 3.969737 | 0.573602 |
| For a Few Dollars More (1965) | 3.409091 | 3.953795 | 0.544704 |
如果只是想要找出分歧最大的电影（不考虑性别因素），则可以计算得分数据的方差或标准差：
# 根据电影名称分组的得分数据的标准差
rating_std_by_title = data.groupby('title')['rating'].std()
# 根据active_titles进行过滤
rating_std_by_title = rating_std_by_title.loc[active_titles]
# Order Series by value in descending order
rating_std_by_title.sort_values(ascending=False)[:10]
title
Dumb & Dumber (1994) 1.321333
Blair Witch Project, The (1999) 1.316368
Natural Born Killers (1994) 1.307198
Tank Girl (1995) 1.277695
Rocky Horror Picture Show, The (1975) 1.260177
Eyes Wide Shut (1999) 1.259624
Evita (1996) 1.253631
Billy Madison (1995) 1.249970
Fear and Loathing in Las Vegas (1998) 1.246408
Bicentennial Man (1999) 1.245533
Name: rating, dtype: float64
这里我们注意到，电影分类是以竖线|分割的字符串形式给出的。如果想对不同的电影分类进行分析的话，就需要先将其转换成更有用的形式才行。