练手项目——Click-Through Rate Prediction 逻辑回归

系统环境:

  • 操作系统:Windows 8.1 64-bit
  • CPU:Intel i7-4200HQ 3.60GHz
  • RAM:8GB
  • GPU: GeForce GTX 970M (CUDA 10.1)

目录

  • 读取数据
    • File descriptions:
    • Data fields:
  • 数据预处理
    • 特征离散/因子化
  • 训练
    • 小部分特征训练
    • 全特征训练
    • 选择部分特征
    • 标准化
    • 模型调参

库环境:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib
import sklearn


print('pandas version:',pd.__version__)
print('matplotlib version:',matplotlib.__version__)
print('seaborn version:',sns.__version__)
print('sklearn version:',sklearn.__version__)
pandas version: 0.23.4
matplotlib version: 2.2.3
seaborn version: 0.9.0
sklearn version: 0.22.2.post1

读取数据

  • 数据下载地址:kaggle_Click-Through Rate Prediction
  • 由于数据量大,笔记本内存有限,无法读取和运算如此大量的数据。因此使用dump_data.py进行采样,选取10000条数据进行训练,10000条数据进行测试

File descriptions:

  • train - Training set. 10 days of click-through data, ordered chronologically. Non-clicks and clicks are subsampled according to different strategies.
  • test - Testing set. 1 day of ads to for testing your model predictions.

Data fields:

  • id: ad identifier
  • click: 0/1 for non-click/click
  • hour: format is YYMMDDHH, so 14091123 means 23:00 on Sept. 11, 2014 UTC.
  • C1 – anonymized categorical variable
  • banner_pos
  • site_id
  • site_domain
  • site_category
  • app_id
  • app_domain
  • app_category
  • device_id
  • device_ip
  • device_model
  • device_type
  • device_conn_type
  • C14-C21 – anonymized categorical variables
data_train = pd.read_csv(r'./dataset/ctr/train_sample_ctr_8450.csv')
data_test = pd.read_csv(r'.\dataset\ctr\test_sample_ctr_155.csv')
data_train.info()
data_test.info()

RangeIndex: 8450 entries, 0 to 8449
Data columns (total 24 columns):
id                  8450 non-null uint64
click               8450 non-null int64
hour                8450 non-null int64
C1                  8450 non-null int64
banner_pos          8450 non-null int64
site_id             8450 non-null object
site_domain         8450 non-null object
site_category       8450 non-null object
app_id              8450 non-null object
app_domain          8450 non-null object
app_category        8450 non-null object
device_id           8450 non-null object
device_ip           8450 non-null object
device_model        8450 non-null object
device_type         8450 non-null int64
device_conn_type    8450 non-null int64
C14                 8450 non-null int64
C15                 8450 non-null int64
C16                 8450 non-null int64
C17                 8450 non-null int64
C18                 8450 non-null int64
C19                 8450 non-null int64
C20                 8450 non-null int64
C21                 8450 non-null int64
dtypes: int64(14), object(9), uint64(1)
memory usage: 1.5+ MB

RangeIndex: 155 entries, 0 to 154
Data columns (total 23 columns):
id                  155 non-null uint64
hour                155 non-null int64
C1                  155 non-null int64
banner_pos          155 non-null int64
site_id             155 non-null object
site_domain         155 non-null object
site_category       155 non-null object
app_id              155 non-null object
app_domain          155 non-null object
app_category        155 non-null object
device_id           155 non-null object
device_ip           155 non-null object
device_model        155 non-null object
device_type         155 non-null int64
device_conn_type    155 non-null int64
C14                 155 non-null int64
C15                 155 non-null int64
C16                 155 non-null int64
C17                 155 non-null int64
C18                 155 non-null int64
C19                 155 non-null int64
C20                 155 non-null int64
C21                 155 non-null int64
dtypes: int64(13), object(9), uint64(1)
memory usage: 27.9+ KB
data_train.describe()
id click hour C1 banner_pos device_type device_conn_type C14 C15 C16 C17 C18 C19 C20 C21
count 8.450000e+03 8450.000000 8.450000e+03 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000 8450.000000
mean 9.179895e+18 0.177751 1.410255e+07 1004.959763 0.287219 1.010178 0.346982 18833.676805 319.027219 61.371124 2110.164734 1.446154 222.525680 52899.882604 83.929231
std 5.358954e+18 0.382326 2.981426e+02 1.085978 0.510966 0.514568 0.869565 5012.525716 23.672810 52.690929 615.540273 1.327442 343.077724 49979.530350 70.765039
min 1.275883e+15 0.000000 1.410210e+07 1001.000000 0.000000 0.000000 0.000000 375.000000 216.000000 36.000000 112.000000 0.000000 33.000000 -1.000000 13.000000
25% 4.536772e+18 0.000000 1.410230e+07 1005.000000 0.000000 1.000000 0.000000 16920.000000 320.000000 50.000000 1863.000000 0.000000 35.000000 -1.000000 23.000000
50% 9.077211e+18 0.000000 1.410252e+07 1005.000000 0.000000 1.000000 0.000000 20352.000000 320.000000 50.000000 2325.000000 2.000000 39.000000 100035.500000 61.000000
75% 1.377866e+19 0.000000 1.410281e+07 1005.000000 1.000000 1.000000 0.000000 21894.000000 320.000000 50.000000 2526.000000 3.000000 171.000000 100103.000000 110.000000
max 1.844664e+19 1.000000 1.410302e+07 1012.000000 7.000000 5.000000 5.000000 24041.000000 768.000000 1024.000000 2756.000000 3.000000 1839.000000 100248.000000 253.000000
data_test.describe()
id hour C1 banner_pos device_type device_conn_type C14 C15 C16 C17 C18 C19 C20 C21
count 1.550000e+02 1.550000e+02 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000 155.000000
mean 8.928224e+18 1.410311e+07 1005.051613 0.187097 1.051613 0.380645 21478.096774 318.941935 53.780645 2437.064516 1.309677 153.864516 56847.980645 92.683871
std 5.552391e+18 5.682679e+00 0.938315 0.391253 0.507010 0.913613 4764.198700 8.769219 27.678645 607.895654 1.331768 215.958632 49765.040950 87.115519
min 7.968314e+16 1.410310e+07 1002.000000 0.000000 0.000000 0.000000 4687.000000 216.000000 36.000000 423.000000 0.000000 33.000000 -1.000000 13.000000
25% 4.190929e+18 1.410311e+07 1005.000000 0.000000 1.000000 0.000000 22104.000000 320.000000 50.000000 2545.000000 0.000000 35.000000 -1.000000 23.000000
50% 9.154648e+18 1.410311e+07 1005.000000 0.000000 1.000000 0.000000 23141.000000 320.000000 50.000000 2664.000000 1.000000 39.000000 100077.000000 51.000000
75% 1.394021e+19 1.410312e+07 1005.000000 0.000000 1.000000 0.000000 24095.500000 320.000000 50.000000 2761.000000 3.000000 175.000000 100148.000000 221.000000
max 1.835151e+19 1.410312e+07 1010.000000 1.000000 4.000000 3.000000 24320.000000 320.000000 250.000000 2790.000000 3.000000 1327.000000 100233.000000 251.000000

观察上述数据,可以发现数据整体不存在缺失值,数据类型为int或object, 需要对object类型数据进行向量化/离散化。对于ID这个特征,可以看到其方差极大,说明是一个连续的数据,结合经验,可以直接删除该特征

数据预处理

  • 删除id特征
data_train = data_train.drop('id',axis=1)
data_test = data_test.drop('id',axis=1)
  • 观察object类型特征并进行向量化/离散化处理
df_object = data_train.filter(regex='site_id|site_domain|site_category|app_id|app_domain|app_category|device_id|device_ip|device_model')
for itm, column in enumerate(df_object.columns):
    category = df_object.iloc[itm].unique()
    num = len(category)
    print('{}: {} \n num: {}'.format(column, category, num))
site_id: ['6399eda6' '968765cd' 'f028772b' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' 'ffd71479' '76dc4769'] 
 num: 9
site_domain: ['8fda644b' '25d4cfcd' 'f028772b' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' '34674ab0' '6569eeb3'] 
 num: 9
site_category: ['5b4d2eda' '16a36ef3' 'f028772b' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' 'cdc5cc94' '00b1f3a7'] 
 num: 9
app_id: ['d8bb8687' '98e6755b' '3e814130' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' '799d2fa5' 'be6db1d7'] 
 num: 9
app_domain: ['1fbe01fe' 'f3845767' '28905ebd' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' '0a945a46' '8a4875bd'] 
 num: 9
app_category: ['83a0ad1a' '5c9ae867' 'f028772b' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' '01bd946f' 'c48dce72'] 
 num: 9
device_id: ['85f751fd' 'c4e18dd6' '50e219e0' 'e2a1ca37' '2347f47a' '8ded1f7a'
 '324d6925' '51b00dde' 'dc70b0f9'] 
 num: 9
device_ip: ['1fbe01fe' 'f3845767' '28905ebd' 'ecad2386' '7801e8d9' '07d7df22'
 'a99f214a' '2d905ba5' 'c6263d8a'] 
 num: 9
device_model: ['85f751fd' 'c4e18dd6' '50e219e0' '685d1c4c' '2347f47a' '8ded1f7a'
 'a99f214a' 'a0ec533d' '88fbaa46'] 
 num: 9

特征离散/因子化

site_id = pd.get_dummies(data_train['site_id'], prefix= 'site_id')
site_domain = pd.get_dummies(data_train['site_domain'], prefix= 'site_domain')
site_category = pd.get_dummies(data_train['site_category'], prefix= 'site_category')
app_id = pd.get_dummies(data_train['app_id'], prefix= 'app_id')
app_domain = pd.get_dummies(data_train['app_domain'], prefix= 'app_domain')
app_category = pd.get_dummies(data_train['app_category'], prefix= 'app_category')
device_id = pd.get_dummies(data_train['device_id'], prefix= 'device_id')
device_ip = pd.get_dummies(data_train['device_ip'], prefix= 'device_ip')
device_model = pd.get_dummies(data_train['device_model'], prefix= 'device_model')

df = pd.concat([site_id, site_domain, site_category, 
                app_id, app_domain, app_category, 
                device_id, device_ip, device_model], axis=1)
df.columns
Index(['site_id_021cd138', 'site_id_02296256', 'site_id_023f3644',
       'site_id_0273c5ad', 'site_id_02d5151c', 'site_id_030440fe',
       'site_id_05c65e53', 'site_id_06a0ac14', 'site_id_070ca277',
       'site_id_079325ff',
       ...
       'device_model_feacaaee', 'device_model_feb70d53',
       'device_model_ff065cf0', 'device_model_ff0f1aca',
       'device_model_ff16d623', 'device_model_ff2a3543',
       'device_model_ff607a1a', 'device_model_ff91ea03',
       'device_model_ffc70ef9', 'device_model_ffe69079'],
      dtype='object', length=11919)

训练

小部分特征训练

  • 只选择int类型数据进行训练
def train(train_np):
    from sklearn import linear_model
    from sklearn.metrics import accuracy_score
    from sklearn.model_selection import train_test_split
    import time
    # y即Survival结果
    y = train_np[:, 0]
    # X即特征属性值
    X = train_np[:, 1:]


    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

    clf = linear_model.LogisticRegression(C=1,penalty="l2")
    t1 = time.time()
    clf.fit(X_train,y_train)
    t2 = time.time()
    y_pred = clf.predict(X_test)
#     print('accuracy:{:.3f}%, tims:{:.3f}s'.format(accuracy_score(y_test, y_pred)*100,(t2-t1)))
    return accuracy_score(y_test, y_pred), (t2-t1)
def train_cv(train_np):
    from sklearn import linear_model
    from sklearn.metrics import accuracy_score
    from sklearn.model_selection import cross_val_score
    import time
    # y即Survival结果
    y = train_np[:, 0]
    # X即特征属性值
    X = train_np[:, 1:]

    clf = linear_model.LogisticRegression(C=1,penalty="l2")
    t1 = time.time()
    scores = cross_val_score(clf, X, y, cv=5)
    t2 = time.time()
    return np.mean(scores), (t2-t1)/5
import time
import matplotlib.pyplot as plt 
import seaborn as sns
plt.figure(figsize=(16,9))
sns.heatmap(data_train[["click","hour","banner_pos","C1","C14","C15","C16",
                        "C17","C18","C19","C20","C21","device_type","device_conn_type"]].corr(),
            annot=True, fmt = ".2f", cmap = "coolwarm") 
plt.show()

练手项目——Click-Through Rate Prediction 逻辑回归_第1张图片

通过上图可以看出单一数值型的特征与标签的线性关系并不是很大,下面看一下训练的效果

few_feature_df = data_train.filter(regex='click|hour|C.*|banner_pos|device_type|device_conn_type')
print('num of feature:',len(few_feature_df.columns))
few_feature_np = few_feature_df.values
acc, t = train_cv(few_feature_np)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
num of feature: 14
accuracy:82.225%, tims:0.035s

全特征训练

  • 使用数值类特征以及特征离散化过后的特征(共计:11931个)
all_feature_df = pd.concat([few_feature_df, df], axis=1)
print('num of feature:',len(all_feature_df.columns))
all_feature_np = all_feature_df.values
acc, t = train_cv(all_feature_np)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
num of feature: 11933
accuracy:82.225%, tims:7.074s

选取部分特征和全特征进行训练最终的结果一直,但时间消耗相差数倍。因此我们应当进行特征选择

选择部分特征

  • site_domain, site_category
df_1 = pd.concat([data_train, site_domain, site_category], axis=1)
df_1.drop(['site_domain', 'site_category'], axis=1, inplace=True)
train_df = df_1.filter(regex='click|site_domain_.*|hour|site_category_.*|C.*|banner_pos')
train_np = train_df.values
acc, t = train_cv(train_np)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
accuracy:82.225%, tims:1.448s

看到结果还是没有什么太大的变化,通过之前观察数据,部分特征的量纲之间差距很大,接下来进行特征数据标准化,来看一下效果。

标准化

  • StandardScaler() → 均值为0,方差为1
  • MinMaxScaler() → [0, 1]
  • MaxAbsScaler() → [-1, 1]
from sklearn.preprocessing import StandardScaler,MinMaxScaler,MaxAbsScaler
import warnings
warnings.filterwarnings('ignore')

scalers = [MinMaxScaler(),
         StandardScaler(),
         MaxAbsScaler()]
names = ['MinMaxScaler','StandardScaler()','MaxAbsScaler']
for name, scaler in zip(names,scalers):
    print(name,':')
    temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos|device_type|device_conn_type'))
    temp = pd.DataFrame(temp, 
                        columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                                 'C19_s','C20_s','C21_s','banner_pos_s','device_type_s','device_conn_type_s'])
    df_1_s = pd.concat([data_train['click'], temp, site_domain, site_category], axis=1)
    acc, t = train_cv(df_1_s.values)
    print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
MinMaxScaler :
accuracy:82.178%, tims:2.935s
StandardScaler() :
accuracy:82.142%, tims:2.540s
MaxAbsScaler :
accuracy:82.166%, tims:2.866s

通过效果看出,对于该问题使用MinMaxScaler()效果比较好,也是因为其与特征都进行离散化([0, 1]),这样量纲统一了。

接下来尝试再增加特征:
由于ip信息大部分均为不同的数据,做为特征时对结果会产生负影响。因此排除ip类的特征,选取其与特征

scaler = MinMaxScaler()
temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos|device_type|device_conn_type'))
trian_data_scalered = pd.DataFrame(temp, 
                    columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                            'C19_s','C20_s','C21_s','banner_pos_s','device_type_s','device_conn_type_s'])
selected_feature_df = pd.concat([data_train['click'], trian_data_scalered,site_domain,site_category,
                                 app_domain,app_category,device_model], axis=1)
acc, t = train_cv(selected_feature_df.values)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
accuracy:82.154%, tims:7.976s
  • 在尝试减少特征:删去domain类特征
scaler = MinMaxScaler()
temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos|device_type|device_conn_type'))
trian_data_scalered = pd.DataFrame(temp, 
                    columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                            'C19_s','C20_s','C21_s','banner_pos_s','device_type_s','device_conn_type_s'])
selected_feature_df = pd.concat([data_train['click'], trian_data_scalered,site_category,
                                 app_category,device_model], axis=1)
acc, t = train_cv(selected_feature_df.values)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
accuracy:82.059%, tims:1.255s
  • 选取site类特征以及其他特征,不选取app和device类特征
scaler = MinMaxScaler()
temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos'))
trian_data_scalered = pd.DataFrame(temp, 
                    columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                            'C19_s','C20_s','C21_s','banner_pos_s'])
selected_feature_df = pd.concat([data_train['click'], trian_data_scalered,site_id, site_domain,site_category], axis=1)
acc, t = train_cv(selected_feature_df.values)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
accuracy:82.130%, tims:0.973s
  • 选取app类特征以及其他特征,不选取site和device类特征
scaler = MinMaxScaler()
temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos'))
trian_data_scalered = pd.DataFrame(temp, 
                    columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                            'C19_s','C20_s','C21_s','banner_pos_s'])
selected_feature_df = pd.concat([data_train['click'], trian_data_scalered,app_id, app_domain, app_category], axis=1)
acc, t = train_cv(selected_feature_df.values)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
accuracy:82.178%, tims:0.546s
  • 选取device类特征以及其他特征,不选取app和site类特征
scaler = MinMaxScaler()
temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos|device_type|device_conn_type'))
trian_data_scalered = pd.DataFrame(temp, 
                    columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                            'C19_s','C20_s','C21_s','banner_pos_s','device_type_s','device_conn_type_s'])
selected_feature_df = pd.concat([data_train['click'], trian_data_scalered,device_id, device_ip, device_model], axis=1)
acc, t = train_cv(selected_feature_df.values)
print('accuracy:{:.3f}%, tims:{:.3f}s'.format(acc*100,t))
accuracy:82.012%, tims:8.948s

通过特征选择,准确率提升的提升并不是很大,因此尝试通过对逻辑回归模型调参

模型调参

from sklearn.model_selection import GridSearchCV
from sklearn import linear_model

# set data
scaler = MinMaxScaler()
temp = scaler.fit_transform(df_1.filter(regex='hour|C.*|banner_pos|device_type|device_conn_type'))
trian_data_scalered = pd.DataFrame(temp, 
                    columns=['hour_s','C1_s','C14_s','C15_s','C16_s','C17_s','C18_s',
                            'C19_s','C20_s','C21_s','banner_pos_s','device_type_s','device_conn_type_s'])
selected_feature_df = pd.concat([data_train['click'], trian_data_scalered,site_domain,site_category,
                                 app_domain,app_category,device_model], axis=1)
X = selected_feature_df.values[:,1:]
y = selected_feature_df.values[:,0]
# set train model
clf = linear_model.LogisticRegression()
param_LR = {'C':[0.01, 0.1, 0.5, 1, 5, 20, 50],
           'penalty':['l1'],
           'solver':['liblinear','saga'],
           'class_weight':['balanced','None']}
clf_gscv_l1 = GridSearchCV(clf, param_grid = param_LR, cv=5, scoring="accuracy", n_jobs= 8, verbose = 1)
clf_gscv_l1.fit(X, y)
Fitting 5 folds for each of 28 candidates, totalling 140 fits
[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers.
[Parallel(n_jobs=8)]: Done  34 tasks      | elapsed:   43.2s
[Parallel(n_jobs=8)]: Done 140 out of 140 | elapsed:  6.2min finished
print(clf_gscv_l1.best_estimator_)
print(clf_gscv_l1.best_score_)
LogisticRegression(C=0.1, class_weight='None', dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l1',
                   random_state=None, solver='saga', tol=0.0001, verbose=0,
                   warm_start=False)
0.8223668639053254
clf = linear_model.LogisticRegression()
param_LR = {'C':[0.01, 0.1, 0.5, 1, 5, 20, 50],
           'penalty':['l2'],
           'solver':['newton-cg','lbfgs','liblinear'],
           'class_weight':['balanced','None']}
clf_gscv_l2 = GridSearchCV(clf, param_grid = param_LR, cv=5, scoring="accuracy", n_jobs= 8, verbose = 1)
clf_gscv_l2.fit(X, y) 
Fitting 5 folds for each of 42 candidates, totalling 210 fits
[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers.
[Parallel(n_jobs=8)]: Done  34 tasks      | elapsed:   27.4s
[Parallel(n_jobs=8)]: Done 184 tasks      | elapsed:  4.8min
[Parallel(n_jobs=8)]: Done 210 out of 210 | elapsed:  5.9min finished
print(clf_gscv_l2.best_estimator_)
print(clf_gscv_l2.best_score_)
LogisticRegression(C=0.01, class_weight='None', dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='newton-cg', tol=0.0001, verbose=0,
                   warm_start=False)
0.8222485207100592

由于逻辑回归中L1和L2近适用于不同的优化算法(solver),所以分开进行调参。因为数据的标签是不均衡的,所以在调参中增添了标签均衡,但由于开启标签均衡后会导致训练集准确率下降,所以这里最佳参数中class_weight=None。但为了预防过拟合的情况,还需将将参数class_weight=‘balanced’

你可能感兴趣的:(练手项目,机器学习,数据分析,逻辑回归,算法,人工智能)