光于前裕于后

“联创黔线”杯大数据应用创新大赛

文章目录

赛题介绍
代码

1 特征工程

1.1 正样本
1.2 负样本

2 建模
3 预测

3.1 测试集

4 提交结果
0 查看数据

0.1 训练数据

0.1.1 正样本
0.1.2 负样本
0.1.3 天气数据

0.2 测试数据

0.2.1 测试集
0.2.2 天气数据

赛题地址：https://www.kesci.com/home/competition/5be92233954d6e001063649a

又打了个酱油，最终成绩是39/205。说出来挺丢人的，因为本次比赛采用AUC来评判模型的效果，不用建模一半预测为去，另一半预测为不去就能得0.5分。

赛题介绍

赛题描述
参赛选手需要根据2017年贵阳市常住居民的部分用户的历史数据（训练集），以及2018年6月、7月的数据（测试集），对2018年8月贵阳市常住居民前往黔东南州进行省内旅游的可能性进行预测。

本比赛任务为：

训练：使用所提供的训练集，即用户使用2017年6、7月的历史数据与8月是否前往黔东南州进行省内旅游的数据，建立预测模型
输出结果：使用所提供的测试集，即用户使用2018年6月、7月的历史数据，通过所建立的模型，预测用户在2018年8月是否会前往黔东南州进行省内旅游的概率。在科赛网，提交测评，得到AUC分数
数据说明
训练集(training_set)约2.3G，其中包含 201708n，201708q 和 weather_data_2017三个文件夹，分别记录了对应的2017年6、7月用户历史数据和天气历史数据。

在201708n和201708q两个文件夹中，各包含7个txt文件，201708n文件夹中的用户在2017年8月都没有去过黔东南目标区域，201708q文件夹中的用户在2017年8月都去过黔东南目标景区
训练集中，除以下列示字段外，最后还有一个字段“label”：“0”表示其为负样本，即该用户在2017年8月没有去过黔东南目标区域；“1”表示其为正样本，即该用户在2017年8月去过黔东南目标区域
用户身份属性表（201708n1.txt, 201708q1.txt)
用户手机终端信息表（201708n2.txt, 201708q2.txt)
用户漫游行为表（201708n3.txt, 201708q3.txt)
用户漫出省份表（201708n4.txt, 201708q4.txt)
用户地理位置表（201708n6.txt, 201708q6.txt)
用户APP使用情况表（201708n7.txt, 201708q7.txt)
在weather_data_2017文件夹中包含两个txt文件，“weather_reported_2017”记录了2017年6月、7月的实际天气，“weather_forecast_2017”，记录了2017年6月、7月的预报天气，以及一个“天气现象编码表.xlsx”文件。
2017实况天气表（weather_reported_2017.txt）
2017预测天气表（weather_forecast_2017.txt）
测试集(testing_set)共约1G，其中包含201808和weather_data_2018两个文件夹

在201808文件夹中包含7个txt文件，命名依次为2018_1.txt,2018_2.txt, … ,2018_7.txt，字段信息与训练集相对应
在weather_data_2018文件夹中包含两个txt文件，命“weather_reported_2018”记录了2018年6月、7月的实际天气，“weather_forecast_2018”记录了2018年6月、7月的预报天气，字段信息与训练集相对应。
备注：

每个文件夹中的7个表可以通过虚拟ID互相关联；但不是每个虚拟ID都可以被关联，选手自行判断如何处理和使用
不同表中的虚拟ID存在格式不同的情况，需选手自行处理，并保证提交虚拟ID格式为string
由于表的数量较多，信息维度不同，应用方法多种，数据可能存在异常和缺失，选手需自行处理可能遇到的异常状况
欢迎选手用不同的方法进行尝试，如迁移学习等前沿方法
本次竞赛数据经过了脱敏处理，数据和实际信息有一定差距，但是不会影响问题的解决
评审说明
1、初赛评分规则

本次比赛采用AUC来评判模型的效果。AUC即以False Positive Rate为横轴，True Positive Rate为纵轴的ROC （Receiver Operating Characteristic）曲线下方的面积大小。

2、评审说明

测评排行榜采用Private/Public机制，其中，Private榜对应所提交结果文件中一定比例数据的成绩，Public榜对应剩余数据的成绩。

提供给每个队伍每天5次提交与测评排名的机会，实时更新Public排行榜，从高到低排序，若队伍一天内多次提交结果，新结果版本将覆盖原版本。
由于受到使用模型的泛化性能的影响，在 Public 榜获得最高分的提交在 Private 的分数不一定最高，因此需要选手从自己的有效提交里，选择两个觉得兼顾了泛化性能与模型评分的结果文件进入 Private 榜测评
Private 排行榜在比赛结束后会揭晓，比赛的最终有效成绩与有效排名将以 Private 榜为准。

代码

# 显示cell运行时长
%load_ext klab-autotime

import pandas as pd
import numpy as np

time: 311 ms

# 减少内存使用

def reduce_mem_usage(df, verbose=True):

    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']

    start_mem = df.memory_usage().sum() / 1024 ** 2

    for col in df.columns:

        col_type = df[col].dtypes

        if col_type in numerics:

            c_min = df[col].min()

            c_max = df[col].max()

            if str(col_type)[:3] == 'int':

                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:

                    df[col] = df[col].astype(np.int8)

                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:

                    df[col] = df[col].astype(np.int16)

                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:

                    df[col] = df[col].astype(np.int32)

                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:

                    df[col] = df[col].astype(np.int64)

            else:

                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:

                    df[col] = df[col].astype(np.float16)

                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:

                    df[col] = df[col].astype(np.float32)

                else:

                    df[col] = df[col].astype(np.float64)

    end_mem = df.memory_usage().sum() / 1024 ** 2

    if verbose:

        print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))

    return df

time: 3.85 ms

1 特征工程

正样本

q1
将两月金额相加

q1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q1.txt', sep='\t', header=None))
q1.columns = ['year_month', 'id', 'consume', 'label']

Mem. usage decreased to  0.16 Mb (53.1% reduction)
time: 39.2 ms

q1.describe()

	year_month	id	consume	label
count	11200.000000	1.120000e+04	1.086500e+04	11200.0
mean	201706.500000	5.416583e+15	inf	1.0
std	0.500022	2.642827e+15	inf	0.0
min	201706.000000	1.448104e+12	4.998779e-02	1.0
25%	201706.000000	3.117220e+15	4.068750e+01	1.0
50%	201706.500000	5.456254e+15	9.837500e+01	1.0
75%	201707.000000	7.702940e+15	1.785000e+02	1.0
max	201707.000000	9.997949e+15	1.324000e+03	1.0

time: 37.3 ms

q1.info()


RangeIndex: 11200 entries, 0 to 11199
Data columns (total 4 columns):
year_month    11200 non-null int32
id            11200 non-null int64
consume       10865 non-null float16
label         11200 non-null int8
dtypes: float16(1), int32(1), int64(1), int8(1)
memory usage: 164.1 KB
time: 6.91 ms

q1.consume.min()

0.05



time: 2.64 ms

q1 = q1.fillna(98.0)

time: 2.75 ms

q1.info()


RangeIndex: 11200 entries, 0 to 11199
Data columns (total 4 columns):
year_month    11200 non-null int32
id            11200 non-null int64
consume       11200 non-null float16
label         11200 non-null int8
dtypes: float16(1), int32(1), int64(1), int8(1)
memory usage: 164.1 KB
time: 6.71 ms

q1 = q1[['id', 'consume']]
q1_groupbyid = q1.groupby(['id']).agg({'consume': pd.Series.sum})

time: 709 ms

q2
特征1 使用过的top9+其它手机品牌共10个
特征2 使用的不同品牌数量

q2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q2.txt', sep='\t', header=None))
q2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time', 'label']

Mem. usage decreased to 11.31 Mb (14.6% reduction)
time: 2.46 s

q2.info()


RangeIndex: 289203 entries, 0 to 289202
Data columns (total 6 columns):
id                 289203 non-null int64
brand              197376 non-null object
type               197380 non-null object
first_use_time     289203 non-null int64
recent_use_time    289203 non-null int64
label              289203 non-null int8
dtypes: int64(3), int8(1), object(2)
memory usage: 11.3+ MB
time: 62.6 ms

q2.type = q2.type.fillna('其它')

time: 18.4 ms

brand_series = pd.Series({'苹果' : 'iphone', '华为' : "huawei", '欧珀' : 'oppo', '维沃' : 'vivo', '三星' : 'san', '小米' : 'mi', '金立' : 'jinli', '魅族' : 'mei', '乐视' : 'le', '四季恒美' : 'siji'})

q2.brand = q2.brand.map(brand_series)

time: 42.4 ms

q2.brand = q2.brand.fillna('其它')

time: 17.4 ms

q2.head()

	id	brand	type	first_use_time	recent_use_time	label
0	1752398069509000	其它	其它	20161209134530	20161209190636	1
1	1752398069509000	huawei	PLK-AL10	20170609223138	20170609224345	1
2	1752398069509000	le	LETV X501	20160924102711	20160924112425	1
3	1752398069509000	jinli	金立 GN800	20150331210255	20150630131232	1
4	1752398069509000	jinli	GIONEE M5	20170508191216	20170605192347	1

time: 18.7 ms

q2['brand_type'] = q2['brand'] + q2['type']

time: 109 ms

q2.head()

	id	brand	type	first_use_time	recent_use_time	label	brand_type
0	1752398069509000	其它	其它	20161209134530	20161209190636	1	其它其它
1	1752398069509000	huawei	PLK-AL10	20170609223138	20170609224345	1	huaweiPLK-AL10
2	1752398069509000	le	LETV X501	20160924102711	20160924112425	1	leLETV X501
3	1752398069509000	jinli	金立 GN800	20150331210255	20150630131232	1	jinli金立 GN800
4	1752398069509000	jinli	GIONEE M5	20170508191216	20170605192347	1	jinliGIONEE M5

time: 9.75 ms

groupbybrand_type = q2['brand_type'].value_counts()

time: 51.8 ms

groupbybrand_type.head(10)

其它其它                     91823
iphoneA1586              14898
iphoneA1524              10330
iphoneA1700               9246
iphoneA1699               8277
iphoneIPHONE6S(A1633)     6271
oppoOPPO R9M              4725
iphoneA1530               4640
oppoOPPO R9TM             2978
vivoVIVO X7               2516
Name: brand_type, dtype: int64



time: 3.44 ms

q2_brand_type = q2[['id', 'brand_type']]
q2_brand_type = q2_brand_type.drop_duplicates()
q2_groupbyid = q2_brand_type['id'].value_counts()
q2_groupbyid = q2_groupbyid.reset_index()
q2_groupbyid.columns = ['id', 'phone_nums']
q2_groupbyid.head()

	id	phone_nums
0	8707678197418467	422
1	9196501153454276	409
2	3900535090108175	389
3	4104535378288025	352
4	1106540188374027	350

time: 90 ms

q2_groupbyid.info()


RangeIndex: 5600 entries, 0 to 5599
Data columns (total 2 columns):
id            5600 non-null int64
phone_nums    5600 non-null int64
dtypes: int64(2)
memory usage: 87.6 KB
time: 5.91 ms

q2_brand = q2[['id', 'brand']]
q2_brand = q2_brand.drop_duplicates()
q2_brand_one_hot = pd.get_dummies(q2_brand)
q2_brand_one_hot.head()

	id	brand_huawei	brand_jinli	brand_le	brand_san	brand_其它
0	1752398069509000	0	0	0	0	1
1	1752398069509000	1	0	0	0	0
2	1752398069509000	0	0	1	0	0
3	1752398069509000	0	1	0	0	0
8	1752398069509000	0	0	0	1	0

time: 48.9 ms

q2_one_hot = q2_brand_one_hot.groupby(['id']).agg({'brand_huawei': pd.Series.max, 
                                                   'brand_iphone': pd.Series.max,
                                                   'brand_jinli': pd.Series.max, 
                                                   'brand_le': pd.Series.max,
                                                   'brand_mei': pd.Series.max, 
                                                   'brand_mi': pd.Series.max,
                                                   'brand_oppo': pd.Series.max, 
                                                   'brand_san': pd.Series.max,
                                                   'brand_siji': pd.Series.max, 
                                                   'brand_vivo': pd.Series.max,
                                                   'brand_其它': pd.Series.max
})
q2_one_hot.head()

	brand_huawei	brand_iphone	brand_jinli	brand_le	brand_mei	brand_mi	brand_oppo	brand_san	brand_siji	brand_vivo	brand_其它
id
1448103998000	1	1	0	1	1	0	1	1	0	0	1
17398718813730	1	1	1	1	1	1	1	1	0	1	1
61132623486000	1	0	0	0	0	0	0	0	0	0	1
68156596675520	0	1	1	1	0	0	0	0	0	0	1
76819334576430	1	1	1	0	1	1	1	1	0	1	1

time: 6.57 s

pos_set = q1_groupbyid.merge(q2_groupbyid, on=['id'])
pos_set.info()


Int64Index: 5600 entries, 0 to 5599
Data columns (total 3 columns):
id            5600 non-null int64
consume       5600 non-null float16
phone_nums    5600 non-null int64
dtypes: float16(1), int64(2)
memory usage: 142.2 KB
time: 11.6 ms

pos_set = pos_set.merge(q2_one_hot, on=['id'])
pos_set.info()


Int64Index: 5600 entries, 0 to 5599
Data columns (total 14 columns):
id              5600 non-null int64
consume         5600 non-null float16
phone_nums      5600 non-null int64
brand_huawei    5600 non-null uint8
brand_iphone    5600 non-null uint8
brand_jinli     5600 non-null uint8
brand_le        5600 non-null uint8
brand_mei       5600 non-null uint8
brand_mi        5600 non-null uint8
brand_oppo      5600 non-null uint8
brand_san       5600 non-null uint8
brand_siji      5600 non-null uint8
brand_vivo      5600 non-null uint8
brand_其它        5600 non-null uint8
dtypes: float16(1), int64(2), uint8(11)
memory usage: 202.3 KB
time: 98.6 ms

q3
1.将两月联络圈规模求和
2.将两月出省求和是:1 否:0
3.将两月出国求和是:1 否:0

q3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q3.txt', sep='\t', header=None))
q3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational', 'label']

Mem. usage decreased to  0.18 Mb (64.6% reduction)
time: 85.8 ms

q3.info()


RangeIndex: 11200 entries, 0 to 11199
Data columns (total 6 columns):
year_month             11200 non-null int32
id                     11200 non-null int64
call_nums              11200 non-null int16
is_trans_provincial    11200 non-null int8
is_transnational       11200 non-null int8
label                  11200 non-null int8
dtypes: int16(1), int32(1), int64(1), int8(3)
memory usage: 186.0 KB
time: 7.49 ms

q3_groupbyid_call = q3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})
q3_groupbyid_provincial = q3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})
q3_groupbyid_trans = q3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})

pos_set = pos_set.merge(q3_groupbyid_call, on=['id'])
pos_set = pos_set.merge(q3_groupbyid_provincial, on=['id'])
pos_set = pos_set.merge(q3_groupbyid_trans, on=['id'])
pos_set.info()


Int64Index: 5600 entries, 0 to 5599
Data columns (total 17 columns):
id                     5600 non-null int64
consume                5600 non-null float16
phone_nums             5600 non-null int64
brand_huawei           5600 non-null uint8
brand_iphone           5600 non-null uint8
brand_jinli            5600 non-null uint8
brand_le               5600 non-null uint8
brand_mei              5600 non-null uint8
brand_mi               5600 non-null uint8
brand_oppo             5600 non-null uint8
brand_san              5600 non-null uint8
brand_siji             5600 non-null uint8
brand_vivo             5600 non-null uint8
brand_其它               5600 non-null uint8
call_nums              5600 non-null int16
is_trans_provincial    5600 non-null int8
is_transnational       5600 non-null int8
dtypes: float16(1), int16(1), int64(2), int8(2), uint8(11)
memory usage: 224.2 KB
time: 1.95 s

q4
1.两月内漫出省次数
2.所有省份one-hot或top10省份+其它省份
3.两月内漫出不同省个数

q4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q4.txt', sep='\t', header=None))
q4.columns = ['year_month', 'id', 'province', 'label']
q4.info()

Mem. usage decreased to  0.15 Mb (34.4% reduction)

RangeIndex: 7289 entries, 0 to 7288
Data columns (total 4 columns):
year_month    7289 non-null int32
id            7289 non-null int64
province      7218 non-null object
label         7289 non-null int8
dtypes: int32(1), int64(1), int8(1), object(1)
memory usage: 149.6+ KB
time: 18.4 ms

q4.head()

	year_month	id	province	label
0	201707	6062475264825100	广东	1
1	201707	5627768389537500	北京	1
2	201707	2000900444179600	山西	1
3	201707	5304502776817600	四川	1
4	201707	5304502776817600	四川	1

time: 7.16 ms

q4_groupbyid = q4.groupby(['province']).size()

time: 61.3 ms

q4_groupbyid.sort_values()

province
宁夏      15
吉林      20
内蒙古     22
黑龙江     27
青海      35
天津      39
辽宁      44
西藏      69
山西      70
甘肃      73
新疆      74
安徽      86
海南     100
陕西     114
山东     121
福建     150
河北     168
江苏     182
湖北     208
上海     215
河南     237
北京     247
江西     364
重庆     428
浙江     483
云南     530
广西     536
四川     793
广东     835
湖南     933
dtype: int64



time: 4.04 ms

q4.province = q4.province.fillna('湖南')
q4.info()


RangeIndex: 7289 entries, 0 to 7288
Data columns (total 4 columns):
year_month    7289 non-null int32
id            7289 non-null int64
province      7289 non-null object
label         7289 non-null int8
dtypes: int32(1), int64(1), int8(1), object(1)
memory usage: 149.6+ KB
time: 8.09 ms

q4_groupbyid = q4[['id', 'province']].groupby(['id']).size()
q4_groupbyid = q4_groupbyid.reset_index()
q4_groupbyid.columns = ['id', 'province_out_cnt']

pos_set = pos_set.merge(q4_groupbyid, how='left', on=['id'])
pos_set.info()


Int64Index: 5600 entries, 0 to 5599
Data columns (total 18 columns):
id                     5600 non-null int64
consume                5600 non-null float16
phone_nums             5600 non-null int64
brand_huawei           5600 non-null uint8
brand_iphone           5600 non-null uint8
brand_jinli            5600 non-null uint8
brand_le               5600 non-null uint8
brand_mei              5600 non-null uint8
brand_mi               5600 non-null uint8
brand_oppo             5600 non-null uint8
brand_san              5600 non-null uint8
brand_siji             5600 non-null uint8
brand_vivo             5600 non-null uint8
brand_其它               5600 non-null uint8
call_nums              5600 non-null int16
is_trans_provincial    5600 non-null int8
is_transnational       5600 non-null int8
province_out_cnt       1942 non-null float64
dtypes: float16(1), float64(1), int16(1), int64(2), int8(2), uint8(11)
memory usage: 268.0 KB
time: 19.6 ms

pos_set = pos_set.fillna(0)
pos_set['label'] = 1
pos_set.info()


Int64Index: 5600 entries, 0 to 5599
Data columns (total 19 columns):
id                     5600 non-null int64
consume                5600 non-null float16
phone_nums             5600 non-null int64
brand_huawei           5600 non-null uint8
brand_iphone           5600 non-null uint8
brand_jinli            5600 non-null uint8
brand_le               5600 non-null uint8
brand_mei              5600 non-null uint8
brand_mi               5600 non-null uint8
brand_oppo             5600 non-null uint8
brand_san              5600 non-null uint8
brand_siji             5600 non-null uint8
brand_vivo             5600 non-null uint8
brand_其它               5600 non-null uint8
call_nums              5600 non-null int16
is_trans_provincial    5600 non-null int8
is_transnational       5600 non-null int8
province_out_cnt       5600 non-null float64
label                  5600 non-null int64
dtypes: float16(1), float64(1), int16(1), int64(3), int8(2), uint8(11)
memory usage: 311.7 KB
time: 12.7 ms

q6 暂时忽略
q7
1.使用总流量
2.使用不同APP数量
3.某些特定（旅游相关）APP是否使用

1.1 正样本

q1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q1.txt', sep='\t', header=None))

q1.columns = ['year_month', 'id', 'consume', 'label']

q1 = q1.fillna(98.0)

q1 = q1[['id', 'consume']]

q1_groupbyid = q1.groupby(['id']).agg({'consume': pd.Series.sum})



q2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q2.txt', sep='\t', header=None))

q2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time', 'label']

q2.type = q2.type.fillna('其它')

brand_series = pd.Series({'苹果' : 'iphone', '华为' : "huawei", '欧珀' : 'oppo', '维沃' : 'vivo', '三星' : 'san', '小米' : 'mi', '金立' : 'jinli', '魅族' : 'mei', '乐视' : 'le', '四季恒美' : 'siji'})

q2.brand = q2.brand.map(brand_series)

q2.brand = q2.brand.fillna('其它')

q2['brand_type'] = q2['brand'] + q2['type']

q2_brand_type = q2[['id', 'brand_type']]

q2_brand_type = q2_brand_type.drop_duplicates()

q2_groupbyid = q2_brand_type['id'].value_counts()

q2_groupbyid = q2_groupbyid.reset_index()

q2_groupbyid.columns = ['id', 'phone_nums']

q2_brand = q2[['id', 'brand']]

q2_brand = q2_brand.drop_duplicates()

q2_brand_one_hot = pd.get_dummies(q2_brand)

q2_one_hot = q2_brand_one_hot.groupby(['id']).agg({'brand_huawei': pd.Series.max, 

                                                   'brand_iphone': pd.Series.max,

                                                   'brand_jinli': pd.Series.max, 

                                                   'brand_le': pd.Series.max,

                                                   'brand_mei': pd.Series.max, 

                                                   'brand_mi': pd.Series.max,

                                                   'brand_oppo': pd.Series.max, 

                                                   'brand_san': pd.Series.max,

                                                   'brand_siji': pd.Series.max, 

                                                   'brand_vivo': pd.Series.max,

                                                   'brand_其它': pd.Series.max

})

q2_one_hot.head()

pos_set = q1_groupbyid.merge(q2_groupbyid, on=['id'])

pos_set = pos_set.merge(q2_one_hot, on=['id'])



q3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q3.txt', sep='\t', header=None))

q3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational', 'label']

q3_groupbyid_call = q3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})

q3_groupbyid_provincial = q3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})

q3_groupbyid_trans = q3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})



pos_set = pos_set.merge(q3_groupbyid_call, on=['id'])

pos_set = pos_set.merge(q3_groupbyid_provincial, on=['id'])

pos_set = pos_set.merge(q3_groupbyid_trans, on=['id'])



q4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q4.txt', sep='\t', header=None))

q4.columns = ['year_month', 'id', 'province', 'label']

q4.province = q4.province.fillna('湖南')

q4_groupbyid = q4[['id', 'province']].groupby(['id']).size()

q4_groupbyid = q4_groupbyid.reset_index()

q4_groupbyid.columns = ['id', 'province_out_cnt']



pos_set = pos_set.merge(q4_groupbyid, how='left', on=['id'])

pos_set = pos_set.fillna(0)

pos_set['label'] = 1

pos_set.info()

Mem. usage decreased to  0.16 Mb (53.1% reduction)
Mem. usage decreased to 11.31 Mb (14.6% reduction)
Mem. usage decreased to  0.18 Mb (64.6% reduction)
Mem. usage decreased to  0.15 Mb (34.4% reduction)

Int64Index: 5600 entries, 0 to 5599
Data columns (total 19 columns):
id                     5600 non-null int64
consume                5600 non-null float16
phone_nums             5600 non-null int64
brand_huawei           5600 non-null uint8
brand_iphone           5600 non-null uint8
brand_jinli            5600 non-null uint8
brand_le               5600 non-null uint8
brand_mei              5600 non-null uint8
brand_mi               5600 non-null uint8
brand_oppo             5600 non-null uint8
brand_san              5600 non-null uint8
brand_siji             5600 non-null uint8
brand_vivo             5600 non-null uint8
brand_其它               5600 non-null uint8
call_nums              5600 non-null int16
is_trans_provincial    5600 non-null int8
is_transnational       5600 non-null int8
province_out_cnt       5600 non-null float64
label                  5600 non-null int64
dtypes: float16(1), float64(1), int16(1), int64(3), int8(2), uint8(11)
memory usage: 311.7 KB
time: 10.1 s

1.2 负样本

n1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n1.txt', sep='\t', header=None))

n1.columns = ['year_month', 'id', 'consume', 'label']

n1 = n1.fillna(98.0)

n1_groupbyid = n1[['id', 'consume']].groupby(['id']).agg({'consume': pd.Series.sum})



n2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n2.txt', sep='\t', header=None))

n2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time', 'label']

n2.type = n2.type.fillna('其它')

brand_series = pd.Series({'苹果' : 'iphone', '华为' : "huawei", '欧珀' : 'oppo', '维沃' : 'vivo', '三星' : 'san', '小米' : 'mi', '金立' : 'jinli', '魅族' : 'mei', '乐视' : 'le', '四季恒美' : 'siji'})

n2.brand = n2.brand.map(brand_series)

n2.brand = n2.brand.fillna('其它')

n2['brand_type'] = n2['brand'] + n2['type']

n2_brand_type = n2[['id', 'brand_type']]

n2_brand_type = n2_brand_type.drop_duplicates()

n2_groupbyid = n2_brand_type['id'].value_counts()

n2_groupbyid = n2_groupbyid.reset_index()

n2_groupbyid.columns = ['id', 'phone_nums']

n2_brand = n2[['id', 'brand']]

n2_brand = n2_brand.drop_duplicates()

n2_brand_one_hot = pd.get_dummies(n2_brand)

n2_one_hot = n2_brand_one_hot.groupby(['id']).agg({'brand_huawei': pd.Series.max, 

                                                   'brand_iphone': pd.Series.max,

                                                   'brand_jinli': pd.Series.max, 

                                                   'brand_le': pd.Series.max,

                                                   'brand_mei': pd.Series.max, 

                                                   'brand_mi': pd.Series.max,

                                                   'brand_oppo': pd.Series.max, 

                                                   'brand_san': pd.Series.max,

                                                   'brand_siji': pd.Series.max, 

                                                   'brand_vivo': pd.Series.max,

                                                   'brand_其它': pd.Series.max

})



neg_set = n1_groupbyid.merge(n2_groupbyid, on=['id'])

neg_set = neg_set.merge(n2_one_hot, on=['id'])

n3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n3.txt', sep='\t', header=None))

n3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational', 'label']



n3_groupbyid_call = n3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})

n3_groupbyid_provincial = n3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})

n3_groupbyid_trans = n3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})

neg_set = neg_set.merge(n3_groupbyid_call, on=['id'])

neg_set = neg_set.merge(n3_groupbyid_provincial, on=['id'])

neg_set = neg_set.merge(n3_groupbyid_trans, on=['id'])



n4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n4.txt', sep='\t', header=None))

n4.columns = ['year_month', 'id', 'province', 'label']

n4.province = n4.province.fillna('湖南')

n4_groupbyid = n4[['id', 'province']].groupby(['id']).size()

n4_groupbyid = n4_groupbyid.reset_index()

n4_groupbyid.columns = ['id', 'province_out_cnt']

neg_set = neg_set.merge(n4_groupbyid, how='left', on=['id'])

neg_set = neg_set.fillna(0)



neg_set['label'] = 0

neg_set.info()

Mem. usage decreased to  2.67 Mb (53.1% reduction)
Mem. usage decreased to 51.13 Mb (14.6% reduction)
Mem. usage decreased to  3.03 Mb (64.6% reduction)
Mem. usage decreased to  0.73 Mb (34.4% reduction)

Int64Index: 93375 entries, 0 to 93374
Data columns (total 19 columns):
id                     93375 non-null int64
consume                93375 non-null float16
phone_nums             93375 non-null int64
brand_huawei           93375 non-null uint8
brand_iphone           93375 non-null uint8
brand_jinli            93375 non-null uint8
brand_le               93375 non-null uint8
brand_mei              93375 non-null uint8
brand_mi               93375 non-null uint8
brand_oppo             93375 non-null uint8
brand_san              93375 non-null uint8
brand_siji             93375 non-null uint8
brand_vivo             93375 non-null uint8
brand_其它               93375 non-null uint8
call_nums              93375 non-null int16
is_trans_provincial    93375 non-null int8
is_transnational       93375 non-null int8
province_out_cnt       93375 non-null float64
label                  93375 non-null int64
dtypes: float16(1), float64(1), int16(1), int64(3), int8(2), uint8(11)
memory usage: 5.1 MB
time: 2min 48s

train_set = pos_set.append(neg_set)
train_set.info()


Int64Index: 98975 entries, 0 to 93374
Data columns (total 19 columns):
id                     98975 non-null int64
consume                98975 non-null float16
phone_nums             98975 non-null int64
brand_huawei           98975 non-null uint8
brand_iphone           98975 non-null uint8
brand_jinli            98975 non-null uint8
brand_le               98975 non-null uint8
brand_mei              98975 non-null uint8
brand_mi               98975 non-null uint8
brand_oppo             98975 non-null uint8
brand_san              98975 non-null uint8
brand_siji             98975 non-null uint8
brand_vivo             98975 non-null uint8
brand_其它               98975 non-null uint8
call_nums              98975 non-null int16
is_trans_provincial    98975 non-null int8
is_transnational       98975 non-null int8
province_out_cnt       98975 non-null float64
label                  98975 non-null int64
dtypes: float16(1), float64(1), int16(1), int64(3), int8(2), uint8(11)
memory usage: 5.4 MB
time: 62.5 ms

2 建模

import numpy as np

import pandas as pd

import lightgbm as lgb

from sklearn import metrics

from sklearn.model_selection import train_test_split



X = train_set[['consume', 'phone_nums', 'call_nums', 'is_trans_provincial', 'is_transnational', 'province_out_cnt']].values

y = train_set['label'].values



x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2)



lgb_train = lgb.Dataset(x_train, y_train)

lgb_eval = lgb.Dataset(x_test, y_test, reference = lgb_train)

params = {

        'boosting_type':'gbdt',  #提升器的类型

        'objective':'binary',   

        'metric':{'auc'},

        'num_leaves':100,

        'reg_alpha':0,

        'reg_lambda':0.01,

        'max_depth':6,

        'n_estimators':100,

        'subsample':0.9,

        'colsample_bytree':0.85,

        'subsample_freq':1,

        'min_child_samples':25,

        'learning_rate':0.1,

        'random_state':2019

        #'feature_fraction':0.9,  #每棵树训练之前选择90%的特征

        #'bagging_fraction':0.8,  #类似于feature_fraction，加速训练，处理过拟合

        #'bagging_freq':5,

        #'verbose':0

}

gbm = lgb.train(params,

                lgb_train,

                num_boost_round = 2000, # 4000 number of boosting iterations,

                valid_sets = lgb_eval,

                verbose_eval=250,

                early_stopping_rounds=50)

                

y_pred = gbm.predict(X, num_iteration=gbm.best_iteration)

print('AUC: %.4f' % metrics.roc_auc_score(y, y_pred))



y_pred = gbm.predict(x_test, num_iteration=gbm.best_iteration)

print('Test AUC: %.4f' % metrics.roc_auc_score(y_test, y_pred))

Training until validation scores don't improve for 50 rounds.
Early stopping, best iteration is:
[18]	valid_0's auc: 0.786865
AUC: 0.7981
Test AUC: 0.7869
time: 772 ms

from sklearn.model_selection import train_test_split

from xgboost import XGBClassifier

from collections import Counter



X = train_set[['consume', 'phone_nums', 'call_nums', 'is_trans_provincial', 'is_transnational', 'province_out_cnt']].values

y = train_set['label'].values



X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)



c = Counter(y_train)

'''

params={'booster':'gbtree',

    'objective': 'binary:logistic',

    'eval_metric': 'auc',

    'max_depth':4,

    'lambda':10,

    'subsample':0.75,

    'colsample_bytree':0.75,

    'min_child_weight':2,

    'eta': 0.025,

    'seed':0,

    'nthread':8,

     'silent':1}

'''

clf = XGBClassifier(max_depth=5, eval_metric='auc', min_child_weight=6, scale_pos_weight=c[0] / 16 / c[1], 

                    nthread=12, num_boost_round=1000, seed=2019

                    )

                    

print('fit start...')

clf.fit(X_train, y_train)

print('fit finish')



'''

train_score = clf.score(X_train, y_train)

test_score = clf.score(X_test, y_test)

print('train score:{}\ntest score:{}'.format(train_score, test_score))

'''



y_pred=clf.predict(X)

from sklearn import metrics

print('AUC: %.4f' % metrics.roc_auc_score(y, y_pred))



y_pred=clf.predict(X_test)

print('Test AUC: %.4f' % metrics.roc_auc_score(y_test, y_pred))

fit start...
fit finish
AUC: 0.5134
Test AUC: 0.5082
time: 3.11 s

import xgboost as xgb

import pandas as pd

from sklearn.model_selection import GridSearchCV

from collections import Counter





X_train = train_set[['consume', 'phone_nums', 'call_nums', 'is_trans_provincial', 'is_transnational', 'province_out_cnt']].values

y_train = train_set['label'].values

c = Counter(y_train)



# n = c[0] / c[1]  # 13.98

# nn = c[0] / 16 / c[1] # 0.8738

# print(n, nn)



parameters = {

    'max_depth': [5, 10, 15],

    'learning_rate': [0.01, 0.02, 0.05],

    'n_estimators': [500, 1000, 2000],

    'min_child_weight': [0, 2, 5],

    'max_delta_step': [0, 0.2, 0.6],

    'subsample': [0.6, 0.7, 0.8],

    'colsample_bytree': [0.5, 0.6, 0.7],

    'reg_alpha': [0, 0.25, 0.5],

    'reg_lambda': [0.2, 0.4, 0.6],

    'scale_pos_weight': [0.8, 8, 14]



}



xlf = xgb.XGBClassifier(max_depth=10,

                        learning_rate=0.01,

                        n_estimators=2000,

                        silent=True,

                        objective='binary:logistic',

                        nthread=12,

                        gamma=0,

                        min_child_weight=1,

                        max_delta_step=0,

                        subsample=0.85,

                        colsample_bytree=0.7,

                        colsample_bylevel=1,

                        reg_alpha=0,

                        reg_lambda=1,

                        scale_pos_weight=1,

                        seed=2019,

                        missing=None)



# 有了gridsearch我们便不需要fit函数

gsearch = GridSearchCV(xlf, param_grid=parameters, scoring='accuracy', cv=3)

gsearch.fit(X_train, y_train)



print("Best score: %0.3f" % gsearch.best_score_)

print("Best parameters set:")

best_parameters = gsearch.best_estimator_.get_params()

for param_name in sorted(parameters.keys()):

    print("\t%s: %r" % (param_name, best_parameters[param_name]))

3 预测

3.1 测试集

t1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_1.txt', sep='\t', header=None))

t1.columns = ['year_month', 'id', 'consume']

t1 = t1.fillna(81.0)

# t1 = t1.dropna(axis=0)

t1_groupbyid = t1[['id', 'consume']].groupby(['id']).agg({'consume': pd.Series.sum})



t2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_2.txt', sep='\t', header=None))

t2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time']

t2 = t2.fillna('其它')

# t2 = t2.dropna(axis=0)

brand_series = pd.Series({'苹果' : 'iphone', '华为' : "huawei", '欧珀' : 'oppo', '维沃' : 'vivo', '三星' : 'san', '小米' : 'mi', '金立' : 'jinli', '魅族' : 'mei', '乐视' : 'le', '四季恒美' : 'siji'})

t2.brand = t2.brand.map(brand_series)

t2.brand = t2.brand.fillna('其它')

t2['brand_type'] = t2['brand'] + t2['type']

t2_brand_type = t2[['id', 'brand_type']]

t2_brand_type = t2_brand_type.drop_duplicates()

t2_groupbyid = t2_brand_type['id'].value_counts()

t2_groupbyid = t2_groupbyid.reset_index()

t2_groupbyid.columns = ['id', 'phone_nums']

t2_brand = t2[['id', 'brand']]

t2_brand = t2_brand.drop_duplicates()

t2_brand_one_hot = pd.get_dummies(t2_brand)

t2_one_hot = t2_brand_one_hot.groupby(['id']).agg({'brand_huawei': pd.Series.max, 

                                                   'brand_iphone': pd.Series.max,

                                                   'brand_jinli': pd.Series.max, 

                                                   'brand_le': pd.Series.max,

                                                   'brand_mei': pd.Series.max, 

                                                   'brand_mi': pd.Series.max,

                                                   'brand_oppo': pd.Series.max, 

                                                   'brand_san': pd.Series.max,

                                                   'brand_siji': pd.Series.max, 

                                                   'brand_vivo': pd.Series.max,

                                                   'brand_其它': pd.Series.max

})



test_set = t1_groupbyid.merge(t2_groupbyid, on=['id'])

test_set = test_set.merge(t2_one_hot, on=['id'])



t3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_3.txt', sep='\t', header=None))

t3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational']



t3_groupbyid_call = t3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})

t3_groupbyid_provincial = t3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})

t3_groupbyid_trans = t3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})

test_set = test_set.merge(t3_groupbyid_call, on=['id'])

test_set = test_set.merge(t3_groupbyid_provincial, on=['id'])

test_set = test_set.merge(t3_groupbyid_trans, on=['id'])



t4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_4.txt', sep='\t', header=None))

t4.columns = ['year_month', 'id', 'province']

t4 = t4.fillna('湖南')

# t4 = t4.dropna(axis=0)

t4_groupbyid = t4[['id', 'province']].groupby(['id']).size()

t4_groupbyid = t4_groupbyid.reset_index()

t4_groupbyid.columns = ['id', 'province_out_cnt']

test_set = test_set.merge(t4_groupbyid, how='left', on=['id'])



test_set = test_set.fillna(0)

test_set.info()

Mem. usage decreased to  1.34 Mb (41.7% reduction)
Mem. usage decreased to 60.50 Mb (0.0% reduction)
Mem. usage decreased to  1.53 Mb (60.0% reduction)
Mem. usage decreased to  0.85 Mb (16.7% reduction)

Int64Index: 48668 entries, 0 to 48667
Data columns (total 18 columns):
id                     48668 non-null int64
consume                48668 non-null float16
phone_nums             48668 non-null int64
brand_huawei           48668 non-null uint8
brand_iphone           48668 non-null uint8
brand_jinli            48668 non-null uint8
brand_le               48668 non-null uint8
brand_mei              48668 non-null uint8
brand_mi               48668 non-null uint8
brand_oppo             48668 non-null uint8
brand_san              48668 non-null uint8
brand_siji             48668 non-null uint8
brand_vivo             48668 non-null uint8
brand_其它               48668 non-null uint8
call_nums              48668 non-null int16
is_trans_provincial    48668 non-null int8
is_transnational       48668 non-null int8
province_out_cnt       48668 non-null float64
dtypes: float16(1), float64(1), int16(1), int64(2), int8(2), uint8(11)
memory usage: 2.3 MB
time: 1min 39s

# lightgbm
X_test = test_set[['consume', 'phone_nums', 'call_nums', 'is_trans_provincial', 'is_transnational', 'province_out_cnt']].values
y_predict = gbm.predict(X_test, num_iteration=gbm.best_iteration)
submit = test_set[['id']]
submit['pred'] = y_predict

time: 108 ms


/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  """

type(y_predict)

numpy.ndarray



time: 2.3 ms

y_predict[:5]

array([0.10280227, 0.08214867, 0.06905468, 0.07655945, 0.11238844])



time: 2.9 ms

# xgboost
X_test = test_set[['consume', 'phone_nums', 'call_nums', 'is_trans_provincial', 'is_transnational', 'province_out_cnt']].values
y_predict = clf.predict_proba(X_test)[:, 1]
submit_xgb = test_set[['id']]
submit_xgb['pred'] = y_predict

time: 208 ms


/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  """

4 提交结果

tt1 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_1.txt', sep='\t', header=None)
tt1.columns = ['year_month', 'id', 'consume']

time: 41.6 ms

xgb_t1_id = tt1[['id']].drop_duplicates()

time: 13 ms

xgb_t1_id.info()


Int64Index: 50200 entries, 0 to 99852
Data columns (total 1 columns):
id    50200 non-null int64
dtypes: int64(1)
memory usage: 784.4 KB
time: 5.46 ms

t1_id = tt1[['id']].drop_duplicates()

time: 12.5 ms

t1_id.info()


Int64Index: 50200 entries, 0 to 99852
Data columns (total 1 columns):
id    50200 non-null int64
dtypes: int64(1)
memory usage: 784.4 KB
time: 5.67 ms

submit_xgb.info()


Int64Index: 48668 entries, 0 to 48667
Data columns (total 2 columns):
id      48668 non-null int64
pred    48668 non-null float32
dtypes: float32(1), int64(1)
memory usage: 950.5 KB
time: 7.8 ms

submit.info()


Int64Index: 48668 entries, 0 to 48667
Data columns (total 2 columns):
id      48668 non-null int64
pred    48668 non-null float64
dtypes: float64(1), int64(1)
memory usage: 1.1 MB
time: 8.33 ms

tt_xgb = t1_id.merge(submit_xgb, on=['id'], how='left')

time: 17.6 ms

tt_xgb.info()


Int64Index: 50200 entries, 0 to 50199
Data columns (total 2 columns):
id      50200 non-null int64
pred    48668 non-null float32
dtypes: float32(1), int64(1)
memory usage: 980.5 KB
time: 8.14 ms

tt = t1_id.merge(submit, on=['id'], how='left')

time: 19.3 ms

tt.info()


Int64Index: 50200 entries, 0 to 50199
Data columns (total 2 columns):
id      50200 non-null int64
pred    48668 non-null float64
dtypes: float64(1), int64(1)
memory usage: 1.1 MB
time: 8.06 ms

xgboost

# fill 0 0.469  dropna-0.50005  addfeat-0.46  addfeat dropna-0.4549
# fill 1 0.436  addfeat dropna-0.419
# fill mean0.088458 0.43048757
submit_xgb = tt_xgb.fillna(0.0)

time: 1.92 ms

lightgbm

# fill 0 addfeat-0.4491 0.4539  addfeat dropna-0.4512
submit_gbm = tt.fillna(0.0)

time: 1.96 ms

1.模型融合求和得分0.4558
2.全为1.0/0.0 得分0.5
3.大于0.5改为1.0，小于0.5改为0.0 应有2800人左右去 xgb0.26 得分0.50153 gbm0.17 得分0.50554

submit_xgb.describe()

	id	pred
count	5.020000e+04	50200.000000
mean	5.449990e+15	0.092590
std	2.628886e+15	0.088487
min	5.959412e+11	0.000000
25%	3.177008e+15	0.034837
50%	5.441108e+15	0.063993
75%	7.726328e+15	0.125547
max	9.999920e+15	0.754152

time: 22.4 ms

submit_xgb[submit_xgb['pred']>=0.26].describe()

	id	pred
count	2.818000e+03	2818.000000
mean	5.523494e+15	0.350387
std	2.632627e+15	0.083545
min	7.736480e+13	0.260060
25%	3.193231e+15	0.287803
50%	5.528103e+15	0.324941
75%	7.801996e+15	0.386373
max	9.999505e+15	0.754152

time: 16.7 ms

xgb_yes = submit_xgb[submit_xgb['pred']>=0.26] 
xgb_yes['pred'] = 1.0
xgb_yes.describe()

/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

	id	pred
count	2.818000e+03	2818.0
mean	5.523494e+15	1.0
std	2.632627e+15	0.0
min	7.736480e+13	1.0
25%	3.193231e+15	1.0
50%	5.528103e+15	1.0
75%	7.801996e+15	1.0
max	9.999505e+15	1.0

time: 347 ms

xgb_no = submit_xgb[submit_xgb['pred']<0.26] 
xgb_no['pred'] = 0.0
xgb_no.describe()

/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

	id	pred
count	4.738200e+04	47382.0
mean	5.445619e+15	0.0
std	2.628626e+15	0.0
min	5.959412e+11	0.0
25%	3.175890e+15	0.0
50%	5.435288e+15	0.0
75%	7.722863e+15	0.0
max	9.999920e+15	0.0

time: 380 ms

submit = xgb_yes.append(xgb_no)

time: 2.29 ms

submit.describe()

	id	pred
count	5.020000e+04	50200.000000
mean	5.449990e+15	0.056135
std	2.628886e+15	0.230185
min	5.959412e+11	0.000000
25%	3.177008e+15	0.000000
50%	5.441108e+15	0.000000
75%	7.726328e+15	0.000000
max	9.999920e+15	1.000000

time: 19.6 ms

submit_xgb[submit_xgb['pred']>=0.2].describe()

	id	pred
count	5.547000e+03	5547.000000
mean	5.508672e+15	0.289829
std	2.641133e+15	0.086438
min	5.399382e+12	0.200014
25%	3.195841e+15	0.225862
50%	5.489831e+15	0.261552
75%	7.813588e+15	0.326278
max	9.999505e+15	0.754152

time: 18.5 ms

5600/98975*50200

2840.3132104066685



time: 2.17 ms

submit_gbm[submit_gbm['pred']>=0.23].describe()

	id	pred
count	2.539000e+03	2539.000000
mean	5.482621e+15	0.298836
std	2.625965e+15	0.062903
min	7.736480e+13	0.230013
25%	3.200866e+15	0.253366
50%	5.471503e+15	0.279145
75%	7.742764e+15	0.326900
max	9.999505e+15	0.632138

time: 19 ms

submit_gbm[submit_gbm['pred']>=0.22].describe()

	id	pred
count	2.859000e+03	2859.000000
mean	5.493943e+15	0.290563
std	2.630246e+15	0.063701
min	7.736480e+13	0.220121
25%	3.195841e+15	0.244933
50%	5.501943e+15	0.270700
75%	7.743865e+15	0.321506
max	9.999505e+15	0.632138

time: 19.6 ms

gbm_yes = submit_gbm[submit_gbm['pred']>=0.23] 
gbm_yes['pred'] = 1.0
gbm_yes.describe()

/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

	id	pred
count	2.539000e+03	2539.0
mean	5.482621e+15	1.0
std	2.625965e+15	0.0
min	7.736480e+13	1.0
25%	3.200866e+15	1.0
50%	5.471503e+15	1.0
75%	7.742764e+15	1.0
max	9.999505e+15	1.0

time: 82.2 ms

gbm_no = submit_gbm[submit_gbm['pred']<0.23] 
gbm_no['pred'] = 0.0
gbm_no.describe()

/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

	id	pred
count	4.766100e+04	47661.0
mean	5.448252e+15	0.0
std	2.629058e+15	0.0
min	5.959412e+11	0.0
25%	3.175232e+15	0.0
50%	5.439911e+15	0.0
75%	7.725629e+15	0.0
max	9.999920e+15	0.0

time: 58.7 ms

submit = gbm_yes.append(gbm_no)

time: 4.19 ms

submit.describe()

	id	pred
count	5.020000e+04	50200.000000
mean	5.449990e+15	0.018745
std	2.628886e+15	0.135625
min	5.959412e+11	0.000000
25%	3.177008e+15	0.000000
50%	5.441108e+15	0.000000
75%	7.726328e+15	0.000000
max	9.999920e+15	1.000000

time: 20.4 ms

submit_gbm.describe()

	id	pred
count	5.020000e+04	50200.000000
mean	5.449990e+15	0.085097
std	2.628886e+15	0.071304
min	5.959412e+11	0.000000
25%	3.177008e+15	0.036845
50%	5.441108e+15	0.062206
75%	7.726328e+15	0.113462
max	9.999920e+15	0.632138

time: 20.8 ms

submit.info()


Int64Index: 50200 entries, 91 to 50199
Data columns (total 2 columns):
id      50200 non-null int64
pred    50200 non-null float64
dtypes: float64(1), int64(1)
memory usage: 1.1 MB
time: 9.36 ms

submit = submit_xgb.append(submit_gbm)
submit = submit.groupby(by='id').sum().reset_index()
submit.describe()

	id	pred
count	5.020000e+04	50200.000000
mean	5.449990e+15	0.169012
std	2.628886e+15	0.139313
min	5.959412e+11	0.000000
25%	3.177008e+15	0.076237
50%	5.441108e+15	0.125893
75%	7.726328e+15	0.222622
max	9.999920e+15	1.124561

time: 41.7 ms

submit.head()

	id	pred
4	9297165066591558	1.0
14	8168181097053542	1.0
18	6473515505643555	1.0
25	4641233171005560	1.0
29	6759757036024682	1.0

time: 6.16 ms

submit_xgb[submit_xgb['id']==595941207920]

	id	pred
8048	595941207920	0.185561

time: 7.07 ms

submit_gbm[submit_gbm['id']==595941207920]

	id	pred
8048	595941207920	0.114782

time: 6.33 ms

submit.info()


Int64Index: 50200 entries, 14 to 50199
Data columns (total 2 columns):
id      50200 non-null int64
pred    50200 non-null float64
dtypes: float64(1), int64(1)
memory usage: 1.1 MB
time: 8 ms

全为1

t1_id['pred'] = 1.0

submit = t1_id.copy()
submit.info()


Int64Index: 50200 entries, 0 to 99852
Data columns (total 2 columns):
id      50200 non-null int64
pred    50200 non-null float64
dtypes: float64(1), int64(1)
memory usage: 1.1 MB
time: 8.79 ms

submit.head()

	id	pred
0	6401824160010748	1.0
1	6506134548135499	1.0
2	5996920884619954	1.0
3	1187209424543713	1.0
4	9297165066591558	1.0

time: 13.1 ms

submit.columns = ['ID', 'Pred']
submit['ID'] = submit['ID'].astype(str)

time: 36.7 ms

submit.info()


Int64Index: 50200 entries, 14 to 50199
Data columns (total 2 columns):
ID      50200 non-null object
Pred    50200 non-null float64
dtypes: float64(1), object(1)
memory usage: 1.1+ MB
time: 10.1 ms

submit.to_csv('../submit.csv')

time: 126 ms

!wget -O kesci_submit https://www.heywhale.com/kesci_submit&&chmod +x kesci_submit

wget: /opt/conda/lib/libcrypto.so.1.0.0: no version information available (required by wget)
wget: /opt/conda/lib/libssl.so.1.0.0: no version information available (required by wget)
wget: /opt/conda/lib/libssl.so.1.0.0: no version information available (required by wget)
--2019-07-31 08:15:56--  https://www.heywhale.com/kesci_submit
Resolving www.heywhale.com (www.heywhale.com)... 106.15.25.147
Connecting to www.heywhale.com (www.heywhale.com)|106.15.25.147|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6528405 (6.2M) [application/octet-stream]
Saving to: ‘kesci_submit’

kesci_submit        100%[===================>]   6.23M  12.1MB/s    in 0.5s    

2019-07-31 08:15:57 (12.1 MB/s) - ‘kesci_submit’ saved [6528405/6528405]

time: 1.83 s

!https_proxy="http://klab-external-proxy" ./kesci_submit -file ../submit.csv -token 578549794d544bff

Kesci Submit Tool 3.0

> 已验证Token
> 提交文件 ../submit.csv (1312.26 KiB)
> 文件已上传        
> 提交完成
time: 1.7 s

!./kesci_submit -token 578549794d544bff -file ../submit.csv

Kesci Submit Tool
Result File: ../submit.csv (1.28 MiB)
Uploading: 7%====================
Submit Failed.
Serevr Response:
 400 - {"message":"当前提交工具版本过旧，请参考比赛提交页面信息下载新的提交工具"}

time: 1 s

!ls ../

input  pred.csv  work
time: 665 ms

!wget -nv -O kesci_submit https://www.heywhale.com/kesci_submit&&chmod +x kesci_submit

wget: /opt/conda/lib/libcrypto.so.1.0.0: no version information available (required by wget)
wget: /opt/conda/lib/libssl.so.1.0.0: no version information available (required by wget)
wget: /opt/conda/lib/libssl.so.1.0.0: no version information available (required by wget)
2019-07-02 08:08:23 URL:https://www.heywhale.com/kesci_submit [7842088/7842088] -> "kesci_submit" [1]
time: 1.47 s

0 查看数据

0.1 训练数据

0.1.1 正样本

q1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q1.txt', sep='\t', header=None))

Mem. usage decreased to  0.16 Mb (53.1% reduction)
time: 23 ms

q1.columns = ['year_month', 'id', 'consume', 'label']

time: 1.21 ms

q1 = q1.dropna(axis=0)

time: 6.72 ms

q1.head()

	year_month	id	consume	label
2	201706	8160829951314300	82.75000	1
3	201707	8160829951314300	37.68750	1
4	201706	1508075698521400	68.00000	1
5	201707	1508075698521400	49.59375	1
6	201706	1686251204809800	200.75000	1

time: 6.82 ms

q1.describe()

	year_month	id	consume	label
count	10865.000000	1.086500e+04	1.086500e+04	10865.0
mean	201706.499678	5.417732e+15	inf	1.0
std	0.500023	2.635784e+15	inf	0.0
min	201706.000000	1.448104e+12	4.998779e-02	1.0
25%	201706.000000	3.118365e+15	4.068750e+01	1.0
50%	201706.000000	5.456594e+15	9.837500e+01	1.0
75%	201707.000000	7.687339e+15	1.785000e+02	1.0
max	201707.000000	9.997949e+15	1.324000e+03	1.0

time: 37.1 ms

q1.info()


Int64Index: 10865 entries, 2 to 11199
Data columns (total 4 columns):
year_month    10865 non-null int32
id            10865 non-null int64
consume       10865 non-null float16
label         10865 non-null int8
dtypes: float16(1), int32(1), int64(1), int8(1)
memory usage: 244.0 KB
time: 6.9 ms

%matplotlib inline

# 按index日期排序

q1.consume.plot()

Matplotlib is building the font cache using fc-list. This may take a moment.

time: 11.3 s

q1[q1.consume == 1323.74]

	year_month	id	consume	label
4867	201707	5510977603357000	1324.0	1

time: 11.1 ms

q2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q2.txt', sep='\t', header=None))

Mem. usage decreased to 11.31 Mb (14.6% reduction)
time: 291 ms

q2 = q2.dropna(axis=0)

time: 77.7 ms

q2.head()

	0	1	2	3	4	5
1	1752398069509000	华为	PLK-AL10	20170609223138	20170609224345	1
2	1752398069509000	乐视	LETV X501	20160924102711	20160924112425	1
3	1752398069509000	金立	金立 GN800	20150331210255	20150630131232	1
4	1752398069509000	金立	GIONEE M5	20170508191216	20170605192347	1
5	1752398069509000	华为	PLK-AL10	20160618182839	20170731235959	1

time: 8.16 ms

q2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time', 'label']

time: 1.15 ms

q2.head()

	id	brand	type	first_use_time	recent_use_time	label
1	1752398069509000	华为	PLK-AL10	20170609223138	20170609224345	1
2	1752398069509000	乐视	LETV X501	20160924102711	20160924112425	1
3	1752398069509000	金立	金立 GN800	20150331210255	20150630131232	1
4	1752398069509000	金立	GIONEE M5	20170508191216	20170605192347	1
5	1752398069509000	华为	PLK-AL10	20160618182839	20170731235959	1

time: 8.58 ms

q2.describe()

	id	first_use_time	recent_use_time	label
count	1.973760e+05	1.973760e+05	1.973760e+05	197376.0
mean	5.436228e+15	2.015597e+13	2.015684e+13	1.0
std	2.642924e+15	2.685010e+11	2.685124e+11	0.0
min	1.448104e+12	-1.000000e+00	-1.000000e+00	1.0
25%	3.227267e+15	2.015122e+13	2.016013e+13	1.0
50%	5.353833e+15	2.016052e+13	2.016060e+13	1.0
75%	7.764521e+15	2.016102e+13	2.016112e+13	1.0
max	9.997949e+15	2.017073e+13	2.017073e+13	1.0

time: 64.7 ms

q2.info()


Int64Index: 197376 entries, 1 to 289201
Data columns (total 6 columns):
id                 197376 non-null int64
brand              197376 non-null object
type               197376 non-null object
first_use_time     197376 non-null int64
recent_use_time    197376 non-null int64
label              197376 non-null int8
dtypes: int64(3), int8(1), object(2)
memory usage: 9.2+ MB
time: 41.7 ms

q3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q3.txt', sep='\t', header=None))

Mem. usage decreased to  0.18 Mb (64.6% reduction)
time: 18.4 ms

q3 = q3.dropna(axis=0)

time: 6.41 ms

q3.head()

	0	1	2	3	5
0	201707	6062475264825100	88	1	1
1	201707	8160829951314300	27	0	1
2	201707	1508075698521400	19	0	1
3	201707	1686251204809800	207	0	1
4	201707	5627768389537500	133	1	1

time: 7.62 ms

q3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational', 'label']

time: 1.16 ms

q3.head()

	year_month	id	call_nums	is_trans_provincial	label
0	201707	6062475264825100	88	1	1
1	201707	8160829951314300	27	0	1
2	201707	1508075698521400	19	0	1
3	201707	1686251204809800	207	0	1
4	201707	5627768389537500	133	1	1

time: 7.37 ms

q3.describe()

	year_month	id	call_nums	is_trans_provincial	is_transnational	label
count	11200.000000	1.120000e+04	11200.000000	11200.000000	11200.000000	11200.0
mean	201706.500000	5.416583e+15	70.562232	0.235446	0.014464	1.0
std	0.500022	2.642827e+15	61.820144	0.424296	0.119400	0.0
min	201706.000000	1.448104e+12	-1.000000	0.000000	0.000000	1.0
25%	201706.000000	3.117220e+15	25.000000	0.000000	0.000000	1.0
50%	201706.500000	5.456254e+15	54.000000	0.000000	0.000000	1.0
75%	201707.000000	7.702940e+15	99.250000	0.000000	0.000000	1.0
max	201707.000000	9.997949e+15	727.000000	1.000000	1.000000	1.0

time: 79.6 ms

q3.info()


Int64Index: 11200 entries, 0 to 11199
Data columns (total 6 columns):
year_month             11200 non-null int32
id                     11200 non-null int64
call_nums              11200 non-null int16
is_trans_provincial    11200 non-null int8
is_transnational       11200 non-null int8
label                  11200 non-null int8
dtypes: int16(1), int32(1), int64(1), int8(3)
memory usage: 273.4 KB
time: 7.47 ms

q4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q4.txt', sep='\t', header=None))
q4 = q4.dropna(axis=0)
q4.columns = ['year_month', 'id', 'province', 'label']

time: 935 µs

q4.head()

	year_month	id	province	label
0	201707	6062475264825100	广东	1
1	201707	5627768389537500	北京	1
2	201707	2000900444179600	山西	1
3	201707	5304502776817600	四川	1
4	201707	5304502776817600	四川	1

time: 6.84 ms

q4.describe()

	year_month	id	label
count	7218.000000	7.218000e+03	7218.0
mean	201706.538515	5.341915e+15	1.0
std	0.498549	2.631231e+15	0.0
min	201706.000000	1.739872e+13	1.0
25%	201706.000000	3.037311e+15	1.0
50%	201707.000000	5.367106e+15	1.0
75%	201707.000000	7.545199e+15	1.0
max	201707.000000	9.987407e+15	1.0

time: 22.2 ms

q4.info()


Int64Index: 7218 entries, 0 to 7288
Data columns (total 4 columns):
year_month    7218 non-null int32
id            7218 non-null int64
province      7218 non-null object
label         7218 non-null int8
dtypes: int32(1), int64(1), int8(1), object(1)
memory usage: 204.4+ KB
time: 6.74 ms

!ls /home/kesci/input/gzlt/train_set/201708q/

201708q1.txt  201708q3.txt  201708q6.txt
201708q2.txt  201708q4.txt  201708q7.txt
time: 667 ms

q6 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q6.txt', sep='\t', header=None))

Mem. usage decreased to 62.58 Mb (52.1% reduction)
time: 3.9 s

q6.columns = ['date', 'hour', 'id', 'user_longitude', 'user_latitude', 'label']

time: 868 µs

q6.head()

	date	hour	id	user_longitude	user_latitude	label
0	2017-07-18	8.0	9239265006758100	106.467545	26.58625	1
1	2017-07-10	0.0	3859201812337600	106.708213	26.57854	1
2	2017-07-16	18.0	3859201812337600	106.545690	26.56724	1
3	2017-07-17	8.0	3859201812337600	106.545690	26.56724	1
4	2017-07-27	16.0	3859201812337600	106.545690	26.56724	1

time: 16.7 ms

q6.describe()

	hour	id	user_longitude	user_latitude	label
count	2.852871e+06	2.852871e+06	2.851527e+06	2.851527e+06	2852871.0
mean	1.141897e+01	5.415213e+15	1.068143e+02	2.659968e+01	1.0
std	6.632995e+00	2.634349e+15	5.580043e-01	2.852525e-01	0.0
min	0.000000e+00	1.448104e+12	1.036700e+02	2.470664e+01	1.0
25%	6.000000e+00	3.135488e+15	1.066656e+02	2.654610e+01	1.0
50%	1.200000e+01	5.442594e+15	1.067027e+02	2.658143e+01	1.0
75%	1.800000e+01	7.687963e+15	1.067373e+02	2.662629e+01	1.0
max	2.200000e+01	9.997949e+15	1.095277e+02	2.909348e+01	1.0

time: 775 ms

q6.info()


RangeIndex: 2852871 entries, 0 to 2852870
Data columns (total 6 columns):
date              object
hour              float64
id                int64
user_longitude    float64
user_latitude     float64
label             int64
dtypes: float64(3), int64(2), object(1)
memory usage: 130.6+ MB
time: 3.24 ms

q7 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708q/201708q7.txt', sep='\t', header=None))

Mem. usage decreased to  3.80 Mb (42.5% reduction)
time: 137 ms

q7 = q7.dropna(axis=0)

time: 35.4 ms

q7.columns = ['year_month', 'id', 'app', 'flow', 'label']

time: 1.54 ms

q7.head()

	year_month	id	app	flow	label
0	201707	6610350034824100	腾讯手机管家	0.010002	1
1	201707	6997210664840100	喜马拉雅FM	27.390625	1
2	201707	3198621664927300	网易新闻	0.029999	1
3	201707	9987406611703100	喜马拉雅FM	0.000000	1
4	201707	1785540174324200	天气通	0.020004	1

time: 8.14 ms

q7.describe()

	year_month	id	flow	label
count	173117.000000	1.731170e+05	173117.000000	173117.0
mean	201706.539699	5.403100e+15	NaN	1.0
std	0.498423	2.667026e+15	NaN	0.0
min	201706.000000	1.448104e+12	0.000000	1.0
25%	201706.000000	3.056260e+15	0.010002	1.0
50%	201707.000000	5.429056e+15	0.080017	1.0
75%	201707.000000	7.730223e+15	1.599609	1.0
max	201707.000000	9.997949e+15	7828.000000	1.0

time: 70.4 ms

q7.info()


Int64Index: 173117 entries, 0 to 173116
Data columns (total 5 columns):
year_month    173117 non-null int32
id            173117 non-null int64
app           173117 non-null object
flow          173117 non-null float16
label         173117 non-null int8
dtypes: float16(1), int32(1), int64(1), int8(1), object(1)
memory usage: 5.1+ MB
time: 29.8 ms

q1
将两月金额相加

q1.head()

	year_month	id	consume	label
2	201706	8160829951314300	82.75000	1
3	201707	8160829951314300	37.68750	1
4	201706	1508075698521400	68.00000	1
5	201707	1508075698521400	49.59375	1
6	201706	1686251204809800	200.75000	1

time: 7.05 ms

q1 = q1[['id', 'consume']]

time: 2.91 ms

q1_groupbyid = q1.groupby(['id']).agg({'consume': pd.Series.sum})

time: 747 ms

len(q1)

10865



time: 8.1 ms

q1[q1['id']==1448103998000]

	id	consume
3532	1448103998000	18.09375
3533	1448103998000	44.28125

time: 8.84 ms

q1_groupbyid[:10]

	consume
id
1448103998000	62.37500
17398718813730	460.75000
61132623486000	12.28125
68156596675520	903.50000
76819334576430	282.25000
78745100940550	531.00000
110229638660000	253.00000
122134826301000	138.75000
132923269304000	26.81250
138204830829320	387.50000

time: 5.8 ms

q2
特征1 使用过的top9+其它手机品牌共10个
特征2 使用的不同品牌数量

q2 = q2[['id', 'brand']]

time: 4.86 ms

q2.head(10)

	id	brand
1	1752398069509000	华为
2	1752398069509000	乐视
3	1752398069509000	金立
4	1752398069509000	金立
5	1752398069509000	华为
6	1752398069509000	华为
7	1752398069509000	金立
8	1752398069509000	三星
9	4799656026499908	三星
10	4799656026499908	华为

time: 6.36 ms

groupbybrand = q2['brand'].value_counts()

time: 18.7 ms

len(groupbybrand)

750



time: 2.09 ms

%matplotlib inline

groupbybrand.plot()

time: 454 ms

groupbybrand[:10]

苹果      62347
华为      22266
欧珀      20516
维沃      17158
三星      13435
小米      10632
金立       9922
魅族       9708
乐视       5609
四季恒美     2163
Name: brand, dtype: int64



time: 3.52 ms

q2 = q2.drop_duplicates()
groupbyid = q2['id'].value_counts()

time: 19.6 ms

len(groupbyid)

5597



time: 2.23 ms

%matplotlib inline

groupbyid.plot()

time: 294 ms

groupbyid[:10]

4104535378288025    115
8707678197418467    108
3900535090108175    104
3986280749497468     93
9196501153454276     88
5510977603357000     84
8569492566715454     78
1106540188374027     71
4091371962011072     71
4874962666674313     71
Name: id, dtype: int64



time: 3.27 ms

q1[q1['id']==4104535378288025]

	year_month	id	consume	label
10576	201706	4104535378288025	208.000	1
10577	201707	4104535378288025	205.125	1

time: 7.63 ms

# q2[q2['id']==4104535378288025]

time: 364 µs

type(groupbyid)

pandas.core.series.Series



time: 2.14 ms

type(groupbyid.to_frame())

pandas.core.frame.DataFrame



time: 3.13 ms

q2_groupbyid = groupbyid.reset_index()

time: 2.34 ms

q2_groupbyid.columns = ['id', 'phone_nums']

time: 1.19 ms

q2_groupbyid.head()

	id	phone_nums
0	4104535378288025	115
1	8707678197418467	108
2	3900535090108175	104
3	3986280749497468	93
4	9196501153454276	88

time: 6.12 ms

type(q1_groupbyid)

pandas.core.frame.DataFrame



time: 2.15 ms

pos_set = q1_groupbyid.merge(q2_groupbyid, on=['id'])

time: 6.42 ms

pos_set.head()

	id	consume	phone_nums
0	1448103998000	62.37500	6
1	17398718813730	460.75000	23
2	61132623486000	12.28125	1
3	68156596675520	903.50000	4
4	76819334576430	282.25000	21

time: 7.11 ms

pos_set.info()


Int64Index: 5473 entries, 0 to 5472
Data columns (total 3 columns):
id            5473 non-null int64
consume       5473 non-null float16
phone_nums    5473 non-null int64
dtypes: float16(1), int64(2)
memory usage: 139.0 KB
time: 6.27 ms

q3
1.将两月联络圈规模求和
2.将两月出省求和是:1 否:0
3.将两月出国求和是:1 否:0

q3.head()

	year_month	id	call_nums	is_trans_provincial	label
0	201707	6062475264825100	88	1	1
1	201707	8160829951314300	27	0	1
2	201707	1508075698521400	19	0	1
3	201707	1686251204809800	207	0	1
4	201707	5627768389537500	133	1	1

time: 7.69 ms

q3_groupbyid_call = q3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})
q3_groupbyid_provincial = q3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})
q3_groupbyid_trans = q3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})

time: 1.95 s

pos_set = pos_set.merge(q3_groupbyid_call, on=['id'])

time: 5.14 ms

pos_set.head()

	id	consume	phone_nums	call_nums
0	1448103998000	62.37500	6	21
1	17398718813730	460.75000	23	217
2	61132623486000	12.28125	1	61
3	68156596675520	903.50000	4	353
4	76819334576430	282.25000	21	431

time: 7.94 ms

pos_set = pos_set.merge(q3_groupbyid_provincial, on=['id'])
pos_set = pos_set.merge(q3_groupbyid_trans, on=['id'])

time: 9.61 ms

pos_set.info()


Int64Index: 5473 entries, 0 to 5472
Data columns (total 6 columns):
id                     5473 non-null int64
consume                5473 non-null float16
phone_nums             5473 non-null int64
call_nums              5473 non-null int16
is_trans_provincial    5473 non-null int8
is_transnational       5473 non-null int8
dtypes: float16(1), int16(1), int64(2), int8(2)
memory usage: 160.3 KB
time: 7.3 ms

q4
1.两月内漫出省次数
2.所有省份one-hot或top10省份+其它省份
3.两月内漫出不同省个数

q4.head(10)

	year_month	id	province	label
0	201707	6062475264825100	广东	1
1	201707	5627768389537500	北京	1
2	201707	2000900444179600	山西	1
3	201707	5304502776817600	四川	1
4	201707	5304502776817600	四川	1
5	201707	5304502776817600	四川	1
6	201707	5304502776817600	重庆	1
7	201707	8594396491246200	广西	1
8	201707	8594396491246200	广西	1
9	201707	8594396491246200	广西	1

time: 8.78 ms

q4_groupbyid = q4[['id', 'province']].groupby(['id']).agg({'province': pd.Series.unique})
q4_groupbyid.head()

	province
id
17398718813730	重庆
61132623486000	[福建, 河南, 江苏, 安徽]
68156596675520	[辽宁, 广东]
132923269304000	江西
138204830829320	浙江

time: 322 ms

q4_groupbyid = q4[['id', 'province']].groupby(['id']).size()
q4_groupbyid.head()

id
17398718813730     1
61132623486000     8
68156596675520     3
132923269304000    1
138204830829320    2
dtype: int64



time: 6.52 ms

q4[q4['id']==61132623486000]

	year_month	id	province	label
461	201707	61132623486000	福建	1
462	201707	61132623486000	福建	1
463	201707	61132623486000	福建	1
4363	201706	61132623486000	河南	1
4364	201706	61132623486000	江苏	1
4365	201706	61132623486000	安徽	1
4366	201706	61132623486000	安徽	1
4367	201706	61132623486000	江苏	1

time: 8.26 ms

type(q4_groupbyid.reset_index())

pandas.core.frame.DataFrame



time: 4.03 ms

q4_groupbyid = q4_groupbyid.reset_index()
q4_groupbyid.columns = ['id', 'province_out_cnt']

time: 2.73 ms

q4_groupbyid.head()

	id	province_out_cnt
0	17398718813730	1
1	61132623486000	8
2	68156596675520	3
3	132923269304000	1
4	138204830829320	2

time: 5.73 ms

pos_set = pos_set.merge(q4_groupbyid, how='left', on=['id'])
pos_set.head()

	id	consume	phone_nums	call_nums	is_trans_provincial	province_out_cnt
0	1448103998000	62.37500	6	21	0	NaN
1	17398718813730	460.75000	23	217	1	1.0
2	61132623486000	12.28125	1	61	2	8.0
3	68156596675520	903.50000	4	353	2	3.0
4	76819334576430	282.25000	21	431	0	NaN

time: 14.6 ms

pos_set.info()


Int64Index: 5473 entries, 0 to 5472
Data columns (total 7 columns):
id                     5473 non-null int64
consume                5473 non-null float16
phone_nums             5473 non-null int64
call_nums              5473 non-null int16
is_trans_provincial    5473 non-null int8
is_transnational       5473 non-null int8
province_out_cnt       1913 non-null float64
dtypes: float16(1), float64(1), int16(1), int64(2), int8(2)
memory usage: 203.1 KB
time: 7.53 ms

pos_set = pos_set.fillna(0)

time: 2.46 ms

pos_set.info()


Int64Index: 5473 entries, 0 to 5472
Data columns (total 7 columns):
id                     5473 non-null int64
consume                5473 non-null float16
phone_nums             5473 non-null int64
call_nums              5473 non-null int16
is_trans_provincial    5473 non-null int8
is_transnational       5473 non-null int8
province_out_cnt       5473 non-null float64
dtypes: float16(1), float64(1), int16(1), int64(2), int8(2)
memory usage: 203.1 KB
time: 8.02 ms

# inner
# pos_set.info()


Int64Index: 1913 entries, 0 to 1912
Data columns (total 7 columns):
id                     1913 non-null int64
consume                1913 non-null float16
phone_nums             1913 non-null int64
call_nums              1913 non-null int16
is_trans_provincial    1913 non-null int8
is_transnational       1913 non-null int8
province_out_cnt       1913 non-null int64
dtypes: float16(1), int16(1), int64(3), int8(2)
memory usage: 71.0 KB
time: 6.67 ms

q6 暂时忽略
q7
1.使用总流量
2.使用不同APP数量
3.某些特定（旅游相关）APP是否使用

q7.head()

	year_month	id	app	flow	label
0	201707	6610350034824100	腾讯手机管家	0.010002	1
1	201707	6997210664840100	喜马拉雅FM	27.390625	1
2	201707	3198621664927300	网易新闻	0.029999	1
3	201707	9987406611703100	喜马拉雅FM	0.000000	1
4	201707	1785540174324200	天气通	0.020004	1

time: 7.94 ms

q7_groupbyapp = q7.groupby(['app']).agg({'flow': pd.Series.sum})

time: 135 ms

len(q7_groupbyapp)

762



time: 2.04 ms

q7_groupbyapp.sort_values(by='flow', ascending=False)

	flow
app
网易云音乐	inf
爱奇艺视频	inf
微信	inf
新浪微博	inf
QQ音乐	inf
今日头条	inf
QQ	57856.0
手机百度	53408.0
陌陌	43488.0
iTunes	35392.0
腾讯新闻	25952.0
快手	24256.0
手机淘宝	18400.0
UC浏览器	16608.0
酷狗音乐	15360.0
高德地图	14984.0
酷我音乐	13488.0
新浪新闻	13432.0
唯品会	11504.0
腾讯视频	10760.0
优酷视频	10736.0
汽车之家	9984.0
百度地图	9816.0
美团	9400.0
网易新闻	8648.0
AppStore	7776.0
中国联通手机营业厅	6736.0
百度贴吧	6104.0
凤凰新闻	5504.0
虾米音乐	5020.0
...	...
百才招聘网	0.0
碰碰	0.0
禾文阿思看图购	0.0
科学作息时间表	0.0
章鱼输入法	0.0
米折	0.0
约会吧	0.0
网易微博	0.0
表情大全	0.0
欢乐互娱	0.0
博客大巴	0.0
查快递	0.0
邮储银行	0.0
号簿助手	0.0
司机邦	0.0
壁纸多多	0.0
天天聊	0.0
天翼阅读	0.0
安全管家	0.0
安卓游戏盒子	0.0
安软市场	0.0
车网互联	0.0
宜搜搜索	0.0
工程师爸爸	0.0
彩票控	0.0
贝瓦儿歌	0.0
搜狗壁纸	0.0
智远一户通	0.0
诚品快拍	0.0
07073手游中心	0.0

762 rows × 1 columns

time: 12.4 ms

pos_set.describe()

	id	consume	phone_nums	call_nums	is_trans_provincial	is_transnational	province_out_cnt
count	5.473000e+03	5473.000000	5473.000000	5473.000000	5473.000000	5473.000000	5473.000000
mean	5.417038e+15	inf	8.228942	141.201900	0.474511	0.029600	1.300018
std	2.637784e+15	inf	8.551830	121.262826	0.706162	0.187904	3.110401
min	1.448104e+12	0.099976	1.000000	-2.000000	0.000000	0.000000	0.000000
25%	3.113785e+15	82.000000	3.000000	52.000000	0.000000	0.000000	0.000000
50%	5.457364e+15	198.250000	6.000000	108.000000	0.000000	0.000000	0.000000
75%	7.688781e+15	355.250000	10.000000	198.000000	1.000000	0.000000	1.000000
max	9.997949e+15	2392.000000	115.000000	1035.000000	2.000000	2.000000	42.000000

time: 126 ms

pos_set['label'] = 1

	id	consume	phone_nums	call_nums	is_trans_provincial	province_out_cnt	label
0	1448103998000	62.37500	6	21	0	NaN	1
1	17398718813730	460.75000	23	217	1	1.0	1
2	61132623486000	12.28125	1	61	2	8.0	1
3	68156596675520	903.50000	4	353	2	3.0	1
4	76819334576430	282.25000	21	431	0	NaN	1

time: 10.5 ms

pos_set.fillna(0)
pos_set.head()

	id	consume	phone_nums	call_nums	is_trans_provincial	province_out_cnt	label
0	1448103998000	62.37500	6	21	0	NaN	1
1	17398718813730	460.75000	23	217	1	1.0	1
2	61132623486000	12.28125	1	61	2	8.0	1
3	68156596675520	903.50000	4	353	2	3.0	1
4	76819334576430	282.25000	21	431	0	NaN	1

time: 23.5 ms

0.1.2 负样本

n1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n1.txt', sep='\t', header=None))
n1.columns = ['year_month', 'id', 'consume', 'label']
n1 = n1.dropna(axis=0)
n1_groupbyid = n1[['id', 'consume']].groupby(['id']).agg({'consume': pd.Series.sum})

n2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n2.txt', sep='\t', header=None))
n2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time', 'label']
n2 = n2.dropna(axis=0)
n2 = n2[['id', 'brand']]
n2 = n2.drop_duplicates()
n2_groupbyid = n2['id'].value_counts()
n2_groupbyid = n2_groupbyid.reset_index()
n2_groupbyid.columns = ['id', 'phone_nums']

neg_set = n1_groupbyid.merge(n2_groupbyid, on=['id'])
neg_set.head()

Mem. usage decreased to  2.67 Mb (53.1% reduction)
Mem. usage decreased to 51.13 Mb (14.6% reduction)

	id	consume	phone_nums
0	1009387204000	225.000000	4
1	1167316303000	1.199219	4
2	1883071709000	213.500000	8
3	3393143830010	517.500000	6
4	4568973162000	18.078125	3

time: 10.8 s

neg_set.info()


Int64Index: 76515 entries, 0 to 76514
Data columns (total 3 columns):
id            76515 non-null int64
consume       76515 non-null float16
phone_nums    76515 non-null int64
dtypes: float16(1), int64(2)
memory usage: 1.9 MB
time: 11.1 ms

n3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n3.txt', sep='\t', header=None))
n3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational', 'label']

n3_groupbyid_call = n3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})
n3_groupbyid_provincial = n3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})
n3_groupbyid_trans = n3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})
neg_set = neg_set.merge(n3_groupbyid_call, on=['id'])
neg_set = neg_set.merge(n3_groupbyid_provincial, on=['id'])
neg_set = neg_set.merge(n3_groupbyid_trans, on=['id'])

n4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n4.txt', sep='\t', header=None))
n4.columns = ['year_month', 'id', 'province', 'label']

n4_groupbyid = n4[['id', 'province']].groupby(['id']).size()
n4_groupbyid = n4_groupbyid.reset_index()
n4_groupbyid.columns = ['id', 'province_out_cnt']
neg_set = neg_set.merge(n4_groupbyid, how='left', on=['id'])
neg_set = neg_set.fillna(0)
neg_set.head()

Mem. usage decreased to  3.03 Mb (64.6% reduction)
Mem. usage decreased to  0.73 Mb (34.4% reduction)

	id	consume	phone_nums	call_nums	is_trans_provincial	province_out_cnt
0	1009387204000	225.000000	4	19	0	0.0
1	1167316303000	1.199219	4	6	0	0.0
2	1883071709000	213.500000	8	40	0	0.0
3	3393143830010	517.500000	6	205	1	2.0
4	4568973162000	18.078125	3	17	0	0.0

time: 32.5 s

neg_set['label'] = 0

time: 1.83 ms

neg_set.info()


Int64Index: 76515 entries, 0 to 76514
Data columns (total 8 columns):
id                     76515 non-null int64
consume                76515 non-null float16
phone_nums             76515 non-null int64
call_nums              76515 non-null int16
is_trans_provincial    76515 non-null int8
is_transnational       76515 non-null int8
province_out_cnt       76515 non-null float64
label                  76515 non-null int64
dtypes: float16(1), float64(1), int16(1), int64(3), int8(2)
memory usage: 3.4 MB
time: 18.9 ms

n1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n1.txt', sep='\t', header=None))

Mem. usage decreased to  2.67 Mb (53.1% reduction)
time: 484 ms

n1.columns = ['year_month', 'id', 'consume', 'label']

time: 1.28 ms

n1.head()

	year_month	id	consume
0	201707	8570518832906100	9.00
1	201707	2182640938718700	10.00
2	201707	783614344429000	8.38
3	201707	2007036960106400	100.00
4	201707	9482847959399300	226.05

time: 7.22 ms

n1.describe()

	year_month	id	consume	label
count	186800.000000	1.868000e+05	150750.000000	186800.0
mean	201706.500000	5.464219e+15	63.580028	0.0
std	0.500001	2.633848e+15	84.063600	0.0
min	201706.000000	1.009387e+12	-70.660000	0.0
25%	201706.000000	3.192389e+15	12.930000	0.0
50%	201706.500000	5.486486e+15	34.000000	0.0
75%	201707.000000	7.744140e+15	82.500000	0.0
max	201707.000000	9.999717e+15	3979.940000	0.0

time: 52.5 ms

n1.info()


RangeIndex: 186800 entries, 0 to 186799
Data columns (total 4 columns):
year_month    186800 non-null int64
id            186800 non-null int64
consume       150750 non-null float64
label         186800 non-null int64
dtypes: float64(1), int64(3)
memory usage: 5.7 MB
time: 21.7 ms

n2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n2.txt', sep='\t', header=None))

Mem. usage decreased to 51.13 Mb (14.6% reduction)
time: 7.76 s

n2.head()

	0	1	2	3	4
0	5227696575283900	苹果	A1699	20150331210636	20150701063017
1	6279759720262000	NaN	NaN	20160725112240	20170731235959
2	6279759720262000	NaN	NaN	20161205220417	20161205220417
3	6279759720262000	三星	SM-A9000	20161128231001	20161128231001
4	6279759720262000	NaN	NaN	20161220102623	20170306173713

time: 8.15 ms

n2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time', 'label']

time: 1.2 ms

n2.head()

	id	brand	type	first_use_time	recent_use_time
0	5227696575283900	苹果	A1699	20150331210636	20150701063017
1	6279759720262000	NaN	NaN	20160725112240	20170731235959
2	6279759720262000	NaN	NaN	20161205220417	20161205220417
3	6279759720262000	三星	SM-A9000	20161128231001	20161128231001
4	6279759720262000	NaN	NaN	20161220102623	20170306173713

time: 8.3 ms

n2.describe()

	id	first_use_time	recent_use_time	label
count	1.307608e+06	1.307608e+06	1.307608e+06	1307608.0
mean	5.460966e+15	1.999810e+13	1.999992e+13	0.0
std	2.619222e+15	1.801007e+12	1.801171e+12	0.0
min	1.009387e+12	-1.000000e+00	-1.000000e+00	0.0
25%	3.196695e+15	2.015112e+13	2.016022e+13	0.0
50%	5.477102e+15	2.016071e+13	2.016101e+13	0.0
75%	7.728047e+15	2.016123e+13	2.017023e+13	0.0
max	9.999717e+15	2.017073e+13	2.017073e+13	0.0

time: 252 ms

n2.info()


RangeIndex: 1307608 entries, 0 to 1307607
Data columns (total 6 columns):
id                 1307608 non-null int64
brand              894190 non-null object
type               894205 non-null object
first_use_time     1307608 non-null int64
recent_use_time    1307608 non-null int64
label              1307608 non-null int64
dtypes: int64(4), object(2)
memory usage: 59.9+ MB
time: 251 ms

n3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n3.txt', sep='\t', header=None))

Mem. usage decreased to  3.03 Mb (64.6% reduction)
time: 584 ms

n3.head()

	0	1	2	3
0	201707	4295277677437000	36	1
1	201707	9121335969062000	37	0
2	201707	9438277095447300	-1	0
3	201707	6749854876532500	20	0
4	201707	1545361809381400	26	0

time: 7.82 ms

n3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational', 'label']

time: 1.13 ms

n3.head()

	year_month	id	call_nums	is_trans_provincial
0	201707	4295277677437000	36	1
1	201707	9121335969062000	37	0
2	201707	9438277095447300	-1	0
3	201707	6749854876532500	20	0
4	201707	1545361809381400	26	0

time: 7.49 ms

n3.describe()

	year_month	id	call_nums	is_trans_provincial	is_transnational	label
count	186800.000000	1.868000e+05	186800.000000	186800.000000	186800.000000	186800.0
mean	201706.500000	5.464219e+15	32.674797	0.093292	0.005054	0.0
std	0.500001	2.633848e+15	46.054929	0.290842	0.070909	0.0
min	201706.000000	1.009387e+12	-1.000000	0.000000	0.000000	0.0
25%	201706.000000	3.192389e+15	4.000000	0.000000	0.000000	0.0
50%	201706.500000	5.486486e+15	19.000000	0.000000	0.000000	0.0
75%	201707.000000	7.744140e+15	43.000000	0.000000	0.000000	0.0
max	201707.000000	9.999717e+15	1807.000000	1.000000	1.000000	0.0

time: 75.7 ms

n3.info()


RangeIndex: 186800 entries, 0 to 186799
Data columns (total 6 columns):
year_month             186800 non-null int64
id                     186800 non-null int64
call_nums              186800 non-null int64
is_trans_provincial    186800 non-null int64
is_transnational       186800 non-null int64
label                  186800 non-null int64
dtypes: int64(6)
memory usage: 8.6 MB
time: 26.6 ms

n4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n4.txt', sep='\t', header=None))

Mem. usage decreased to  0.73 Mb (34.4% reduction)
time: 88.8 ms

n4.columns = ['year_month', 'id', 'province', 'label']

time: 1.15 ms

n4.head()

	year_month	id	province
0	201707	4295277677437000	重庆
1	201707	5560109665240300	广西
2	201707	5560109665240300	广东
3	201707	5560109665240300	广东
4	201707	5705601521649600	重庆

time: 7.14 ms

n4.describe()

	year_month	id	label
count	36499.000000	3.649900e+04	36499.0
mean	201706.539193	5.471019e+15	0.0
std	0.498468	2.639006e+15	0.0
min	201706.000000	3.393144e+12	0.0
25%	201706.000000	3.203830e+15	0.0
50%	201707.000000	5.468480e+15	0.0
75%	201707.000000	7.753756e+15	0.0
max	201707.000000	9.999305e+15	0.0

time: 24.4 ms

n4.info()


RangeIndex: 36499 entries, 0 to 36498
Data columns (total 4 columns):
year_month    36499 non-null int64
id            36499 non-null int64
province      36099 non-null object
label         36499 non-null int64
dtypes: int64(3), object(1)
memory usage: 1.1+ MB
time: 9.97 ms

!ls /home/kesci/input/gzlt/train_set/201708n/

201708n1.txt  201708n3.txt  201708n6.txt
201708n2.txt  201708n4.txt  201708n7.txt
time: 669 ms

n6 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n6.txt', sep='\t', header=None))

Mem. usage decreased to 798.26 Mb (52.1% reduction)
time: 2min 59s

n6.columns = ['date', 'hour', 'id', 'user_longitude', 'user_latitude', 'label']

time: 1.51 ms

n6.head()

	date	hour	id	user_longitude	user_latitude
0	2017-07-02	10.0	7748777616409800	106.680816	26.563650
1	2017-07-10	0.0	7748777616409800	106.719520	26.576370
2	2017-07-31	14.0	7748777616409800	106.683060	26.654663
3	2017-07-01	0.0	6633710902197900	106.697440	26.613930
4	2017-07-08	14.0	6633710902197900	106.715700	26.609710

time: 9.14 ms

q6.describe()

	hour	id	user_longitude	user_latitude	label
count	2.852871e+06	2.852871e+06	2.851527e+06	2.851527e+06	2852871.0
mean	1.141897e+01	5.415213e+15	1.068143e+02	2.659968e+01	1.0
std	6.632995e+00	2.634349e+15	5.580043e-01	2.852525e-01	0.0
min	0.000000e+00	1.448104e+12	1.036700e+02	2.470664e+01	1.0
25%	6.000000e+00	3.135488e+15	1.066656e+02	2.654610e+01	1.0
50%	1.200000e+01	5.442594e+15	1.067027e+02	2.658143e+01	1.0
75%	1.800000e+01	7.687963e+15	1.067373e+02	2.662629e+01	1.0
max	2.200000e+01	9.997949e+15	1.095277e+02	2.909348e+01	1.0

time: 979 ms

n6.info()


RangeIndex: 36393070 entries, 0 to 36393069
Data columns (total 6 columns):
date              object
hour              float64
id                int64
user_longitude    float64
user_latitude     float64
label             int64
dtypes: float64(3), int64(2), object(1)
memory usage: 1.6+ GB
time: 3.76 ms

n7 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/train_set/201708n/201708n7.txt', sep='\t', header=None))

Mem. usage decreased to 17.98 Mb (31.2% reduction)
time: 3.14 s

n7.columns = ['year_month', 'id', 'app', 'flow']

time: 1.44 ms

n7.head()

	year_month	id	app	flow
0	201707	4011022166491000	米聊	0.01
1	201707	8544172893207700	百度地图	2.07
2	201707	9856572220983403	搜狗输入法	0.00
3	201707	6441300393946200	爱奇艺视频	0.00
4	201707	8751918977379700	开心消消乐	0.03

time: 7.51 ms

# n7['label'] = 0

time: 2.94 ms

# n7.head()

	year_month	id	app	flow
0	201707	4011022166491000	米聊	0.01
1	201707	8544172893207700	百度地图	2.07
2	201707	9856572220983403	搜狗输入法	0.00
3	201707	6441300393946200	爱奇艺视频	0.00
4	201707	8751918977379700	开心消消乐	0.03

time: 8.46 ms

n7.describe()

	year_month	id	flow	label
count	856961.000000	8.569610e+05	856961.000000	856961.0
mean	201706.535881	5.432556e+15	9.942533	0.0
std	0.498711	2.643712e+15	68.096944	0.0
min	201706.000000	1.009387e+12	0.000000	0.0
25%	201706.000000	3.134290e+15	0.000000	0.0
50%	201707.000000	5.440495e+15	0.060000	0.0
75%	201707.000000	7.727765e+15	1.130000	0.0
max	201707.000000	9.999717e+15	10986.150000	0.0

time: 170 ms

n7.info()


RangeIndex: 856961 entries, 0 to 856960
Data columns (total 5 columns):
year_month    856961 non-null int64
id            856961 non-null int64
app           856961 non-null object
flow          856961 non-null float64
label         856961 non-null int64
dtypes: float64(1), int64(3), object(1)
memory usage: 32.7+ MB
time: 116 ms

0.1.3 天气数据

!ls /home/kesci/input/gzlt/train_set/weather_data_2017/

weather_forecast_2017.txt  weather_reported_2017.txt  天气现象编码.xlsx
time: 669 ms

weather_reported = pd.read_csv('/home/kesci/input/gzlt/train_set/weather_data_2017/weather_reported_2017.txt', sep='\t')

time: 6.15 ms

weather_reported.head()

	Station_Name	VACODE	Year	Month	Day	TEM_Avg	TEM_Max	TEM_Min	PRE_Time_2020	WEP_Record
0	麻江	522635	2017	6	1	23.00	24.5	20.9	0.6	( 01 60 ) 60 .
1	三穗	522624	2017	6	1	21.13	25.6	19.4	9.0	( 01 10 80 ) 80 60 .
2	镇远	522625	2017	6	1	22.68	26.5	21.3	8.9	( 60 ) 60 .
3	雷山	522634	2017	6	1	23.80	26.1	20.4	5.1	( 10 ) 60 .
4	剑河	522629	2017	6	1	23.53	27.1	22.0	6.8	( 01 10 80 ) 80 10 .

time: 12.2 ms

# weather_reported.columns = ['Station_Name', 'VACODE', 'Year', 'Month', 'Day', 'TEM_Avg', 'TEM_Max', 'TEM_Min', 'PRE_Time_2020', 'WEP_Record']

time: 1.25 ms

weather_reported.describe()

	Station_Name	VACODE	Year	Month	Day	TEM_Avg	TEM_Max	TEM_Min	PRE_Time_2020	WEP_Record
count	1404	1404	1404	1404	1404	1404	1404	1404	1404	1404
unique	24	25	2	3	32	448	214	109	330	305
top	贵阳	520000	2017	7	4	22.83	30.5	20.5	0.0	( 01 ) 01 .
freq	61	360	1403	713	46	10	18	35	625	197

time: 49.9 ms

weather_reported.info()


RangeIndex: 1404 entries, 0 to 1403
Data columns (total 10 columns):
Station_Name     1404 non-null object
VACODE           1404 non-null object
Year             1404 non-null object
Month            1404 non-null object
Day              1404 non-null object
TEM_Avg          1404 non-null object
TEM_Max          1404 non-null object
TEM_Min          1404 non-null object
PRE_Time_2020    1404 non-null object
WEP_Record       1404 non-null object
dtypes: object(10)
memory usage: 109.8+ KB
time: 6.32 ms

weather_forecast = pd.read_csv('/home/kesci/input/gzlt/train_set/weather_data_2017/weather_forecast_2017.txt', sep='\t')

time: 10.8 ms

weather_forecast.head()

	Station_Name	VACODE	Year	Mon	Day	TEM_Max_24h	TEM_Min_24h	WEP_24h	TEM_Max_48h	TEM_Min_48h	...	TEM_Max_120h	TEM_Min_120h	WEP_120h	TEM_Max_144h	TEM_Min_144h	WEP_144h	TEM_Max_168h	TEM_Min_168h，WEP_168h	Unnamed: 24	Unnamed: 25
0	白云	520113	2017	6	1	25.0	17.0	(2)1	24.0	19.0	...	(4)2	25.0	15.0	(2)1	27.0	15.0	(1)0	26.0	16.0	(1)0
1	岑巩	522626	2017	6	1	31.3	19.4	(1)1	31.0	22.0	...	(4)1	32.0	19.4	(1)1	32.0	22.8	(1)1	32.0	21.0	(1)1
2	从江	522633	2017	6	1	33.4	22.0	(1)1	30.0	23.0	...	(4)3	34.0	22.0	(1)1	34.0	23.8	(1)1	34.0	22.0	(1)1
3	丹寨	522636	2017	6	1	27.5	18.0	(1)1	24.5	20.0	...	(4)1	28.5	18.0	(1)1	28.5	21.0	(1)1	28.5	20.0	(1)1
4	贵阳	520103	2017	6	1	26.0	18.0	(2)1	25.0	20.0	...	(4)2	26.0	16.0	(2)1	28.0	16.0	(1)0	27.0	17.0	(1)0

5 rows × 26 columns

time: 86.4 ms

weather_forecast.describe()

	VACODE	Year	Mon	Day	TEM_Max_24h	TEM_Min_24h	TEM_Max_48h	TEM_Min_48h	TEM_Max_72h	TEM_Min_72h，WEP_72h	TEM_Min_96h	WEP_96h	TEM_Min_120h	WEP_120h	TEM_Min_144h	WEP_144h	TEM_Min_168h，WEP_168h	Unnamed: 24
count	1464.000000	1464.0	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000	1464.000000
mean	521792.583333	2017.0	6.508197	15.754098	28.374658	20.721585	28.375820	20.872814	28.283811	21.112432	28.539481	21.408128	28.702254	21.454713	29.142623	21.485656	29.131626	21.589003
std	1180.891163	0.0	0.500104	8.809966	4.300391	2.290850	4.379771	2.232788	4.329132	2.204980	4.154188	5.203525	4.167441	5.238257	4.124026	2.180222	4.033227	2.391945
min	520103.000000	2017.0	6.000000	1.000000	17.300000	13.800000	17.300000	13.600000	17.000000	10.000000	19.000000	14.300000	19.000000	15.000000	18.000000	15.000000	18.000000	2.000000
25%	520122.750000	2017.0	6.000000	8.000000	25.000000	19.000000	25.000000	19.400000	25.000000	19.600000	25.500000	19.700000	26.000000	19.700000	26.000000	20.000000	26.500000	20.000000
50%	522624.500000	2017.0	7.000000	16.000000	28.500000	21.000000	28.500000	21.000000	28.000000	21.000000	28.500000	21.500000	28.500000	21.500000	29.000000	22.000000	29.000000	22.000000
75%	522630.250000	2017.0	7.000000	23.000000	31.800000	22.500000	31.600000	22.500000	31.500000	23.000000	31.500000	23.000000	32.000000	23.000000	32.000000	23.000000	32.000000	23.500000
max	522636.000000	2017.0	7.000000	31.000000	39.000000	25.700000	39.500000	25.500000	38.000000	25.800000	39.000000	200.000000	39.000000	202.000000	38.800000	25.800000	37.500000	26.000000

time: 121 ms

weather_forecast.info()


RangeIndex: 1464 entries, 0 to 1463
Data columns (total 26 columns):
Station_Name             1464 non-null object
VACODE                   1464 non-null int64
Year                     1464 non-null int64
Mon                      1464 non-null int64
Day                      1464 non-null int64
TEM_Max_24h              1464 non-null float64
TEM_Min_24h              1464 non-null float64
WEP_24h                  1464 non-null object
TEM_Max_48h              1464 non-null float64
TEM_Min_48h              1464 non-null float64
WEP_48h                  1464 non-null object
TEM_Max_72h              1464 non-null float64
TEM_Min_72h，WEP_72h      1464 non-null float64
TEM_Max_96h              1464 non-null object
TEM_Min_96h              1464 non-null float64
WEP_96h                  1464 non-null float64
TEM_Max_120h             1464 non-null object
TEM_Min_120h             1464 non-null float64
WEP_120h                 1464 non-null float64
TEM_Max_144h             1464 non-null object
TEM_Min_144h             1464 non-null float64
WEP_144h                 1464 non-null float64
TEM_Max_168h             1464 non-null object
TEM_Min_168h，WEP_168h    1464 non-null float64
Unnamed: 24              1464 non-null float64
Unnamed: 25              1464 non-null object
dtypes: float64(14), int64(4), object(8)
memory usage: 297.5+ KB
time: 9.2 ms

0.2 测试数据

0.2.1 测试集

t1 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_1.txt', sep='\t', header=None))
t1.columns = ['year_month', 'id', 'consume']
t1 = t1.dropna(axis=0)
t1_groupbyid = t1[['id', 'consume']].groupby(['id']).agg({'consume': pd.Series.sum})

t2 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_2.txt', sep='\t', header=None))
t2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time']
t2 = t2.dropna(axis=0)
t2 = t2[['id', 'brand']]
t2 = t2.drop_duplicates()
t2_groupbyid = t2['id'].value_counts()
t2_groupbyid = t2_groupbyid.reset_index()
t2_groupbyid.columns = ['id', 'phone_nums']

test_set = t1_groupbyid.merge(t2_groupbyid, on=['id'])
test_set.head()

Mem. usage decreased to  1.34 Mb (41.7% reduction)
Mem. usage decreased to 60.50 Mb (0.0% reduction)

	id	consume	phone_nums
0	595941207920	220.000	10
1	901845022650	662.000	6
2	1868765858840	143.375	4
3	5058794512580	200.000	7
4	5399381591230	192.000	29

time: 7.86 s

test_set.info()


Int64Index: 43977 entries, 0 to 43976
Data columns (total 3 columns):
id            43977 non-null int64
consume       43977 non-null float16
phone_nums    43977 non-null int64
dtypes: float16(1), int64(2)
memory usage: 1.1 MB
time: 9.02 ms

t3 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_3.txt', sep='\t', header=None))
t3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational']

t3_groupbyid_call = t3[['id', 'call_nums']].groupby(['id']).agg({'call_nums': pd.Series.sum})
t3_groupbyid_provincial = t3[['id', 'is_trans_provincial']].groupby(['id']).agg({'is_trans_provincial': pd.Series.sum})
t3_groupbyid_trans = t3[['id', 'is_transnational']].groupby(['id']).agg({'is_transnational': pd.Series.sum})
test_set = test_set.merge(t3_groupbyid_call, on=['id'])
test_set = test_set.merge(t3_groupbyid_provincial, on=['id'])
test_set = test_set.merge(t3_groupbyid_trans, on=['id'])

t4 = reduce_mem_usage(pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_4.txt', sep='\t', header=None))
t4.columns = ['year_month', 'id', 'province']

t4_groupbyid = t4[['id', 'province']].groupby(['id']).size()
t4_groupbyid = t4_groupbyid.reset_index()
t4_groupbyid.columns = ['id', 'province_out_cnt']
test_set = test_set.merge(t4_groupbyid, how='left', on=['id'])
test_set = test_set.fillna(0)
test_set.head()

Mem. usage decreased to  1.53 Mb (60.0% reduction)
Mem. usage decreased to  0.85 Mb (16.7% reduction)

	id	consume	phone_nums	call_nums	is_trans_provincial	province_out_cnt
0	595941207920	220.000	10	68	1	1.0
1	901845022650	662.000	6	278	0	0.0
2	1868765858840	143.375	4	107	2	3.0
3	5058794512580	200.000	7	128	0	0.0
4	5399381591230	192.000	29	61	0	0.0

time: 17.4 s

!ls /home/kesci/input/gzlt/test_set/

201808	weather_data_2018
time: 704 ms

!ls /home/kesci/input/gzlt/test_set/201808

2018_1.txt  2018_2.txt	2018_3.txt  2018_4.txt	2018_6.txt  2018_7.txt
time: 702 ms

t1 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_1.txt', sep='\t', header=None)

time: 527 ms

t1.columns = ['year_month', 'id', 'consume']

time: 1.27 ms

t1.head()

	year_month	id	consume
0	201807	6401824160010748	618.40
1	201807	6506134548135499	NaN
2	201807	5996920884619954	22.05
3	201806	1187209424543713	7.20
4	201807	9297165066591558	124.00

time: 99.9 ms

t1.describe()

	year_month	id	consume
count	100402.000000	1.004020e+05	86787.000000
mean	201806.500000	5.449905e+15	103.357399
std	0.500002	2.628916e+15	311.428596
min	201806.000000	5.959412e+11	0.010000
25%	201806.000000	3.176902e+15	36.500000
50%	201806.500000	5.440931e+15	81.000000
75%	201807.000000	7.726318e+15	132.125000
max	201807.000000	9.999920e+15	61465.900000

time: 50.6 ms

t1.info()


RangeIndex: 100402 entries, 0 to 100401
Data columns (total 3 columns):
year_month    100402 non-null int64
id            100402 non-null int64
consume       86787 non-null float64
dtypes: float64(1), int64(2)
memory usage: 2.3 MB
time: 12.6 ms

%matplotlib inline

# 按index日期排序

t1.consume.plot()

Matplotlib is building the font cache using fc-list. This may take a moment.

time: 17 s

t1[t1.consume == 61465.9]

	year_month	id	consume
11962	201807	4827806860301307	61465.9

time: 7.15 ms

t2 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_2.txt', sep='\t', header=None)

time: 11.8 s

t2.columns = ['id', 'brand', 'type', 'first_use_time', 'recent_use_time']

time: 1.18 ms

t2.head()

	id	brand	type	first_use_time	recent_use_time
0	3179771753483280	魅族	M575	20180601151052	20180601151054
1	4185007692177509	NaN	NaN	20171021182915	20171021183000
2	4972845789896505	NaN	NaN	20180624003647	20180624003656
3	4207293827582218	NaN	NaN	20171224165902	20180306175444
4	2628020151876580	NaN	NaN	20170820111053	20171207020159

time: 7.95 ms

t2.describe()

	id	first_use_time	recent_use_time
count	1.586024e+06	1.586024e+06	1.586024e+06
mean	5.410516e+15	2.017033e+13	2.017156e+13
std	2.618994e+15	6.902153e+09	6.865591e+09
min	5.959412e+11	2.016032e+13	2.016033e+13
25%	3.140763e+15	2.016122e+13	2.017021e+13
50%	5.389338e+15	2.017063e+13	2.017080e+13
75%	7.660413e+15	2.017122e+13	2.018013e+13
max	9.999920e+15	2.018073e+13	2.018073e+13

time: 353 ms

t2.info()


RangeIndex: 1586024 entries, 0 to 1586023
Data columns (total 5 columns):
id                 1586024 non-null int64
brand              1098244 non-null object
type               1098250 non-null object
first_use_time     1586024 non-null int64
recent_use_time    1586024 non-null int64
dtypes: int64(3), object(2)
memory usage: 60.5+ MB
time: 291 ms

t3 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_3.txt', sep='\t', header=None)

time: 451 ms

t3.columns = ['year_month', 'id', 'call_nums', 'is_trans_provincial', 'is_transnational']

time: 1.14 ms

t3.head()

	year_month	id	call_nums
0	201806	3690814703003361	49
1	201807	4315823592069831	-1
2	201806	5199170013029443	-1
3	201806	1387658205895203	35
4	201807	3280240784164442	-1

time: 7.12 ms

t3.describe()

	year_month	id	call_nums	is_trans_provincial	is_transnational
count	100400.000000	1.004000e+05	100400.000000	100400.000000	100400.000000
mean	201806.500000	5.449990e+15	51.642102	0.206116	0.012809
std	0.500002	2.628873e+15	90.705957	0.404516	0.112449
min	201806.000000	5.959412e+11	-1.000000	0.000000	0.000000
25%	201806.000000	3.177008e+15	6.000000	0.000000	0.000000
50%	201806.500000	5.441108e+15	31.000000	0.000000	0.000000
75%	201807.000000	7.726328e+15	71.000000	0.000000	0.000000
max	201807.000000	9.999920e+15	6537.000000	1.000000	1.000000

time: 46.4 ms

t3.info()


RangeIndex: 100400 entries, 0 to 100399
Data columns (total 5 columns):
year_month             100400 non-null int64
id                     100400 non-null int64
call_nums              100400 non-null int64
is_trans_provincial    100400 non-null int64
is_transnational       100400 non-null int64
dtypes: int64(5)
memory usage: 3.8 MB
time: 15.1 ms

t4 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_4.txt', sep='\t', header=None)

time: 240 ms

t4.columns = ['year_month', 'id', 'province']

time: 1.2 ms

t4.head()

	year_month	id	province
0	201807	8445647072009305	广东
1	201806	9414872397547413	浙江
2	201806	2272887111818372	广东
3	201807	224368910874770	湖北
4	201807	6081677258986878	NaN

time: 6.81 ms

t4.describe()

	year_month	id
count	44543.000000	4.454300e+04
mean	201806.530319	5.448788e+15
std	0.499086	2.640390e+15
min	201806.000000	5.959412e+11
25%	201806.000000	3.118911e+15
50%	201807.000000	5.430117e+15
75%	201807.000000	7.751481e+15
max	201807.000000	9.999505e+15

time: 20.3 ms

t4.info()


RangeIndex: 44543 entries, 0 to 44542
Data columns (total 3 columns):
year_month    44543 non-null int64
id            44543 non-null int64
province      44119 non-null object
dtypes: int64(2), object(1)
memory usage: 1.0+ MB
time: 9.73 ms

t6 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_6.txt', sep='\t', header=None)

time: 2min 7s

t6.columns = ['date', 'hour', 'id', 'user_longitude', 'user_latitude']

time: 1.22 ms

t6.head()

	date	hour	id	user_longitude	user_latitude
0	2018-06-10	20	1929821481825935	106.289902	26.837687
1	2018-07-14	18	5450093661688579	106.641975	26.627846
2	2018-07-16	2	4617571498633816	106.230420	27.466980
3	2018-06-15	22	2826359445811398	106.693610	26.591110
4	2018-06-22	10	3526202744290054	107.032570	27.715830

time: 8.4 ms

t6.describe()

	hour	id	user_longitude	user_latitude
count	1.655899e+07	1.655899e+07	1.655081e+07	1.655081e+07
mean	1.144987e+01	5.461505e+15	1.066642e+02	2.662386e+01
std	6.742805e+00	2.629564e+15	4.626476e-01	3.195807e-01
min	0.000000e+00	5.959412e+11	1.036700e+02	2.469706e+01
25%	6.000000e+00	3.191837e+15	1.066328e+02	2.655164e+01
50%	1.200000e+01	5.475087e+15	1.066902e+02	2.658444e+01
75%	1.800000e+01	7.732384e+15	1.067199e+02	2.663778e+01
max	2.200000e+01	9.999920e+15	1.095534e+02	2.916468e+01

time: 6.3 s

t6.info()


RangeIndex: 16558993 entries, 0 to 16558992
Data columns (total 5 columns):
date              object
hour              int64
id                int64
user_longitude    float64
user_latitude     float64
dtypes: float64(2), int64(2), object(1)
memory usage: 631.7+ MB
time: 3.04 ms

t7 = pd.read_csv('/home/kesci/input/gzlt/test_set/201808/2018_7.txt', sep='\t', header=None)

time: 8.75 s

t7.columns = ['year_month', 'id', 'app', 'flow']

time: 1.18 ms

t7.head()

	year_month	id	app	flow
0	201806	9813651010156104	OPPO软件商店	14545.00
1	201806	2338567014163500	腾讯新闻	0.19
2	201807	1133512913801798	讯飞输入法	0.01
3	201807	7739596338372898	手机百度	1615.00
4	201807	5724269192271018	百度贴吧	1301953.00

time: 15.6 ms

t7.describe()

	year_month	id	flow
count	1.493733e+06	1.493733e+06	1.492434e+06
mean	2.018065e+05	5.468351e+15	8.991198e+07
std	4.999895e-01	2.628382e+15	8.503798e+08
min	2.018060e+05	5.959412e+11	0.000000e+00
25%	2.018060e+05	3.196619e+15	6.519000e+03
50%	2.018070e+05	5.477012e+15	2.883350e+05
75%	2.018070e+05	7.737568e+15	7.842132e+06
max	2.018070e+05	9.999920e+15	3.341152e+11

time: 226 ms

t7.info()


RangeIndex: 1493733 entries, 0 to 1493732
Data columns (total 4 columns):
year_month    1493733 non-null int64
id            1493733 non-null int64
app           1457137 non-null object
flow          1492434 non-null float64
dtypes: float64(1), int64(2), object(1)
memory usage: 45.6+ MB
time: 178 ms

0.2.2 天气数据

!ls /home/kesci/input/gzlt/test_set/weather_data_2018/

weather_forecast_2018.txt  weather_reported_2018.txt
time: 830 ms

weather_reported_2018 = pd.read_csv('/home/kesci/input/gzlt/test_set/weather_data_2018/weather_reported_2018.txt', sep='\t')

time: 8.57 ms

weather_reported_2018.head()

	Station_Name	VACODE	Year	Month	Day	TEM_Avg	TEM_Max	TEM_Min	PRE_Time_2020	WEP_Record
0	镇远	522625	2018	6	1	19.0	21.0	17.8	0.1	( 60 01 ) 01 60 10 .
1	丹寨	522636	2018	6	1	17.0	19.9	15.3	4.3	( 60 80 ) 80 .
2	三穗	522624	2018	6	1	17.8	19.2	17.0	0.6	( 80 10 ) 60 10 .
3	台江	522630	2018	6	1	18.8	21.1	17.5	1.4	( 60 01 ) 01 60 10 .
4	剑河	522629	2018	6	1	19.2	21.6	17.9	2.1	( 60 ) 60 10 .

time: 12.6 ms

weather_reported_2018.describe()

	VACODE	Year	Month	Day	TEM_Avg	TEM_Max	TEM_Min	PRE_Time_2020
count	1403.000000	1403.0	1403.000000	1403.000000	1403.000000	1403.000000	1403.000000	1403.000000
mean	521862.934426	2018.0	6.508197	15.754098	737.393799	742.297577	734.011119	4.922594
std	1155.972144	0.0	0.500111	8.810097	26696.850268	26696.719415	26696.940604	15.090986
min	520103.000000	2018.0	6.000000	1.000000	15.100000	16.200000	11.800000	0.000000
25%	520122.000000	2018.0	6.000000	8.000000	22.900000	27.300000	20.000000	0.000000
50%	522625.000000	2018.0	7.000000	16.000000	25.100000	30.100000	21.600000	0.000000
75%	522631.000000	2018.0	7.000000	23.000000	26.900000	32.550000	23.050000	2.100000
max	522636.000000	2018.0	7.000000	31.000000	999999.000000	999999.000000	999999.000000	281.700000

time: 118 ms

weather_reported_2018.info()


RangeIndex: 1403 entries, 0 to 1402
Data columns (total 10 columns):
Station_Name     1403 non-null object
 VACODE          1403 non-null int64
 Year            1403 non-null int64
 Month           1403 non-null int64
 Day             1403 non-null int64
 TEM_Avg         1403 non-null float64
 TEM_Max         1403 non-null float64
 TEM_Min         1403 non-null float64
PRE_Time_2020    1403 non-null float64
WEP_Record       1403 non-null object
dtypes: float64(4), int64(4), object(2)
memory usage: 109.7+ KB
time: 6.7 ms

weather_forecast_2018 = pd.read_csv('/home/kesci/input/gzlt/test_set/weather_data_2018/weather_forecast_2018.txt', sep='\t')

time: 12 ms

weather_forecast_2018.head()

	Station_Name	VACODE	Year	Mon	Day	TEM_Max_24h	TEM_Min_24h	WEP_24h	TEM_Max_48h	TEM_Min_48h	...	TEM_Max_120h	TEM_Min_120h	WEP_120h	TEM_Max_144h	TEM_Min_144h	WEP_144h	TEM_Max_168h	TEM_Min_168h，WEP_168h	Unnamed: 24	Unnamed: 25
0	白云	520113	2018	6	1	20.2	14.8	(3)2	23.2	15.8	...	(2)1	27.5	13.5	(1)1	26.0	14.0	(2)1	24.0	16.0	(1)1
1	岑巩	522626	2018	6	1	25.5	17.5	(2)2	28.5	20.2	...	(2)0	31.0	17.0	(0)0	31.0	18.5	(0)1	31.0	21.5	(1)1
2	从江	522633	2018	6	1	27.3	19.0	(7)2	29.5	22.0	...	(21)0	33.5	19.6	(0)0	33.5	20.2	(0)1	31.5	23.0	(1)1
3	丹寨	522636	2018	6	1	23.0	15.5	(2)2	26.0	19.2	...	(2)0	28.0	16.2	(0)0	28.0	17.2	(0)1	27.0	19.5	(1)1
4	贵阳	520103	2018	6	1	20.9	14.9	(3)2	24.0	16.4	...	(2)1	28.0	14.0	(1)1	26.0	14.0	(2)1	24.0	16.0	(1)1

5 rows × 26 columns

time: 54.2 ms

weather_forecast_2018.describe()

	VACODE	Year	Mon	Day	TEM_Max_24h	TEM_Min_24h	TEM_Max_48h	TEM_Min_48h	TEM_Max_72h	TEM_Min_72h，WEP_72h	TEM_Min_96h	WEP_96h	TEM_Min_120h	WEP_120h	TEM_Min_144h	WEP_144h	TEM_Min_168h，WEP_168h	Unnamed: 24
count	1463.000000	1463.0	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000
mean	521793.738209	2018.0	6.508544	15.759398	29.724607	21.244703	29.724470	21.385236	29.694463	21.655434	29.924949	21.886945	29.891183	22.010936	30.027341	22.055229	30.192960	21.985373
std	1180.467638	0.0	0.500098	8.810643	3.470128	2.536103	3.232737	2.385237	3.167789	2.270505	3.130886	2.131020	3.191721	2.066640	3.199460	2.092155	3.167676	2.227871
min	520103.000000	2018.0	6.000000	1.000000	17.800000	10.800000	18.000000	12.000000	16.500000	12.500000	16.500000	14.000000	14.500000	13.000000	17.000000	13.200000	16.000000	15.000000
25%	520123.000000	2018.0	6.000000	8.000000	27.500000	20.000000	27.500000	20.000000	27.500000	20.200000	28.000000	20.500000	27.500000	21.000000	28.000000	21.000000	28.000000	20.850000
50%	522625.000000	2018.0	7.000000	16.000000	30.000000	22.000000	29.900000	22.000000	29.500000	22.000000	30.000000	22.000000	30.000000	22.200000	30.000000	22.100000	30.000000	22.200000
75%	522630.500000	2018.0	7.000000	23.000000	32.350000	23.000000	32.000000	23.000000	32.300000	23.300000	32.500000	23.500000	32.500000	23.500000	32.500000	23.700000	32.600000	24.000000
max	522636.000000	2018.0	7.000000	31.000000	37.500000	27.000000	37.000000	25.900000	36.500000	26.000000	36.500000	26.000000	36.500000	26.200000	37.000000	26.000000	37.000000	30.000000

time: 74 ms

weather_forecast_2018.info()


RangeIndex: 1463 entries, 0 to 1462
Data columns (total 26 columns):
Station_Name             1463 non-null object
VACODE                   1463 non-null int64
Year                     1463 non-null int64
Mon                      1463 non-null int64
Day                      1463 non-null int64
TEM_Max_24h              1463 non-null float64
TEM_Min_24h              1463 non-null float64
WEP_24h                  1463 non-null object
TEM_Max_48h              1463 non-null float64
TEM_Min_48h              1463 non-null float64
WEP_48h                  1463 non-null object
TEM_Max_72h              1463 non-null float64
TEM_Min_72h，WEP_72h      1463 non-null float64
TEM_Max_96h              1463 non-null object
TEM_Min_96h              1463 non-null float64
WEP_96h                  1463 non-null float64
TEM_Max_120h             1463 non-null object
TEM_Min_120h             1463 non-null float64
WEP_120h                 1463 non-null float64
TEM_Max_144h             1463 non-null object
TEM_Min_144h             1463 non-null float64
WEP_144h                 1463 non-null float64
TEM_Max_168h             1463 non-null object
TEM_Min_168h，WEP_168h    1463 non-null float64
Unnamed: 24              1463 non-null float64
Unnamed: 25              1463 non-null object
dtypes: float64(14), int64(4), object(8)
memory usage: 297.2+ KB
time: 11 ms

!jupyter nbconvert --to markdown "“联创黔线”杯大数据应用创新大赛.ipynb"

0.000000

25%
520122.000000
2018.0
6.000000
8.000000
22.900000
27.300000
20.000000
0.000000

50%
522625.000000
2018.0
7.000000
16.000000
25.100000
30.100000
21.600000
0.000000

75%
522631.000000
2018.0
7.000000
23.000000
26.900000
32.550000
23.050000
2.100000

max
522636.000000
2018.0
7.000000
31.000000
999999.000000
999999.000000
999999.000000
281.700000

time: 118 ms

weather_reported_2018.info()


RangeIndex: 1403 entries, 0 to 1402
Data columns (total 10 columns):
Station_Name     1403 non-null object
 VACODE          1403 non-null int64
 Year            1403 non-null int64
 Month           1403 non-null int64
 Day             1403 non-null int64
 TEM_Avg         1403 non-null float64
 TEM_Max         1403 non-null float64
 TEM_Min         1403 non-null float64
PRE_Time_2020    1403 non-null float64
WEP_Record       1403 non-null object
dtypes: float64(4), int64(4), object(2)
memory usage: 109.7+ KB
time: 6.7 ms

weather_forecast_2018 = pd.read_csv('/home/kesci/input/gzlt/test_set/weather_data_2018/weather_forecast_2018.txt', sep='\t')

time: 12 ms

weather_forecast_2018.head()

	Station_Name	VACODE	Year	Mon	Day	TEM_Max_24h	TEM_Min_24h	WEP_24h	TEM_Max_48h	TEM_Min_48h	...	TEM_Max_120h	TEM_Min_120h	WEP_120h	TEM_Max_144h	TEM_Min_144h	WEP_144h	TEM_Max_168h	TEM_Min_168h，WEP_168h	Unnamed: 24	Unnamed: 25
0	白云	520113	2018	6	1	20.2	14.8	(3)2	23.2	15.8	...	(2)1	27.5	13.5	(1)1	26.0	14.0	(2)1	24.0	16.0	(1)1
1	岑巩	522626	2018	6	1	25.5	17.5	(2)2	28.5	20.2	...	(2)0	31.0	17.0	(0)0	31.0	18.5	(0)1	31.0	21.5	(1)1
2	从江	522633	2018	6	1	27.3	19.0	(7)2	29.5	22.0	...	(21)0	33.5	19.6	(0)0	33.5	20.2	(0)1	31.5	23.0	(1)1
3	丹寨	522636	2018	6	1	23.0	15.5	(2)2	26.0	19.2	...	(2)0	28.0	16.2	(0)0	28.0	17.2	(0)1	27.0	19.5	(1)1
4	贵阳	520103	2018	6	1	20.9	14.9	(3)2	24.0	16.4	...	(2)1	28.0	14.0	(1)1	26.0	14.0	(2)1	24.0	16.0	(1)1

5 rows × 26 columns

time: 54.2 ms

weather_forecast_2018.describe()

	VACODE	Year	Mon	Day	TEM_Max_24h	TEM_Min_24h	TEM_Max_48h	TEM_Min_48h	TEM_Max_72h	TEM_Min_72h，WEP_72h	TEM_Min_96h	WEP_96h	TEM_Min_120h	WEP_120h	TEM_Min_144h	WEP_144h	TEM_Min_168h，WEP_168h	Unnamed: 24
count	1463.000000	1463.0	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000	1463.000000
mean	521793.738209	2018.0	6.508544	15.759398	29.724607	21.244703	29.724470	21.385236	29.694463	21.655434	29.924949	21.886945	29.891183	22.010936	30.027341	22.055229	30.192960	21.985373
std	1180.467638	0.0	0.500098	8.810643	3.470128	2.536103	3.232737	2.385237	3.167789	2.270505	3.130886	2.131020	3.191721	2.066640	3.199460	2.092155	3.167676	2.227871
min	520103.000000	2018.0	6.000000	1.000000	17.800000	10.800000	18.000000	12.000000	16.500000	12.500000	16.500000	14.000000	14.500000	13.000000	17.000000	13.200000	16.000000	15.000000
25%	520123.000000	2018.0	6.000000	8.000000	27.500000	20.000000	27.500000	20.000000	27.500000	20.200000	28.000000	20.500000	27.500000	21.000000	28.000000	21.000000	28.000000	20.850000
50%	522625.000000	2018.0	7.000000	16.000000	30.000000	22.000000	29.900000	22.000000	29.500000	22.000000	30.000000	22.000000	30.000000	22.200000	30.000000	22.100000	30.000000	22.200000
75%	522630.500000	2018.0	7.000000	23.000000	32.350000	23.000000	32.000000	23.000000	32.300000	23.300000	32.500000	23.500000	32.500000	23.500000	32.500000	23.700000	32.600000	24.000000
max	522636.000000	2018.0	7.000000	31.000000	37.500000	27.000000	37.000000	25.900000	36.500000	26.000000	36.500000	26.000000	36.500000	26.200000	37.000000	26.000000	37.000000	30.000000

time: 74 ms

weather_forecast_2018.info()


RangeIndex: 1463 entries, 0 to 1462
Data columns (total 26 columns):
Station_Name             1463 non-null object
VACODE                   1463 non-null int64
Year                     1463 non-null int64
Mon                      1463 non-null int64
Day                      1463 non-null int64
TEM_Max_24h              1463 non-null float64
TEM_Min_24h              1463 non-null float64
WEP_24h                  1463 non-null object
TEM_Max_48h              1463 non-null float64
TEM_Min_48h              1463 non-null float64
WEP_48h                  1463 non-null object
TEM_Max_72h              1463 non-null float64
TEM_Min_72h，WEP_72h      1463 non-null float64
TEM_Max_96h              1463 non-null object
TEM_Min_96h              1463 non-null float64
WEP_96h                  1463 non-null float64
TEM_Max_120h             1463 non-null object
TEM_Min_120h             1463 non-null float64
WEP_120h                 1463 non-null float64
TEM_Max_144h             1463 non-null object
TEM_Min_144h             1463 non-null float64
WEP_144h                 1463 non-null float64
TEM_Max_168h             1463 non-null object
TEM_Min_168h，WEP_168h    1463 non-null float64
Unnamed: 24              1463 non-null float64
Unnamed: 25              1463 non-null object
dtypes: float64(14), int64(4), object(8)
memory usage: 297.2+ KB
time: 11 ms

!jupyter nbconvert --to markdown "“联创黔线”杯大数据应用创新大赛.ipynb"

“联创黔线”杯大数据应用创新大赛

文章目录

赛题介绍

代码

1 特征工程

1.1 正样本

1.2 负样本

2 建模

3 预测

3.1 测试集

4 提交结果

0 查看数据

0.1 训练数据

0.1.1 正样本

0.1.2 负样本

0.1.3 天气数据

0.2 测试数据

0.2.1 测试集

0.2.2 天气数据

你可能感兴趣的:(大数据竞赛,python,Python,数据挖掘)