python stacking_模型融合之Stacking（原理+Python代码）

数据来源于天池赛题：零基础入门数据挖掘 - 二手车交易价格预测

地址：https://tianchi.aliyun.com/competition/entrance/231784/introduction?spm=5176.12281957.1004.1.38b02448ausjSX

目录

一、原理介绍

二、代码实现

三、结果解读

一、原理介绍

在数据挖掘过程中，单个模型的泛化能力往往比较单薄，而模型融合的方法可以结合多个模型的优点，提升模型的预测精度。典型的模型融合的方法有加权融合、Stacking/Blending、提升树。下面将以Stacking为例，做一个详细介绍。

Stacking是一种多层模型，将已训练好的多个模型作为基分类器。然后将这几个学习器的预测结果作为新的训练集，来学习一个新的学习器。

即可以看成是一种结合策略，使用另外一个机器学习算法来将个体机器学习器的结果结合在一起。

我们称第一层学习器为初级学习器，称第二层学习器为次级学习器。

通常情况下，为了防止过拟合，次级学习器宜选用简单模型。如在回归问题中，可以使用线性回归；在分类问题中，可以使用logistic。

二、代码实现

#加载需要的模块

import warnings

warnings.filterwarnings('ignore')

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

from sklearn.model_selection import cross_val_score

from sklearn.metrics import mean_absolute_error, make_scorer

from xgboost.sklearn import XGBRegressor

from lightgbm.sklearn import LGBMRegressor

from sklearn.model_selection import train_test_split

from sklearn.ensemble import GradientBoostingRegressor

import lightgbm as lgb

import xgboost as xgb

from sklearn.model_selection import GridSearchCV,cross_val_score

from sklearn import linear_model

#数据读取

Train_data = pd.read_csv('F:/data/used_car_train_20200313.csv', sep=' ')

TestA_data = pd.read_csv('F:/data/used_car_testA_20200313.csv', sep=' ')

#选择前面特征工程过程中筛选出的特征

Train_data=Train_data[['v_12','v_10','v_9','v_11','price']]

TestA_data=TestA_data[['v_12','v_10','v_9','v_11']]

上述特征的筛选过程可以参考: 特征工程之嵌入式 随机森林度量各指标的重要性

#划分自变量和目标变量

X_data = Train_data.drop('price', axis=1) #删除列

Y_data = Train_data['price']

X_test = TestA_data

#定义模型

def build_model_gbdt(x_train,y_train):

estimator =GradientBoostingRegressor(loss='ls',subsample= 0.85,max_depth= 5,n_estimators = 100)

param_grid = {

'learning_rate': [0.05,0.08,0.1,0.2],

}

gbdt = GridSearchCV(estimator, param_grid,cv=3)

gbdt.fit(x_train,y_train)

print(gbdt.best_params_)

# print(gbdt.best_estimator_ )

return gbdt

def build_model_xgb(x_train,y_train):

model = xgb.XGBRegressor(n_estimators=120, learning_rate=0.08, gamma=0, subsample=0.8,\

colsample_bytree=0.9, max_depth=5) #, objective ='reg:squarederror'

model.fit(x_train, y_train)

return model

def build_model_lgb(x_train,y_train):

estimator = lgb.LGBMRegressor(num_leaves=63,n_estimators = 100)

param_grid = {

'learning_rate': [0.01, 0.05, 0.1],

}

gbm = GridSearchCV(estimator, param_grid)

gbm.fit(x_train, y_train)

return gbm

def build_model_lr(x_train,y_train):

reg_model = linear_model.LinearRegression()

reg_model.fit(x_train,y_train)

return reg_model

#交叉验证

#划分训练集和测试集

x_train,x_val,y_train,y_val = train_test_split(X_data,Y_data,test_size=0.3)

#训练模型

print('Predict GBDT...')

model_gbdt = build_model_gbdt(x_train,y_train)

val_gbdt = model_gbdt.predict(x_val)

subA_gbdt = model_gbdt.predict(X_test)

print('predict XGB...')

model_xgb = build_model_xgb(x_train,y_train)

val_xgb = model_xgb.predict(x_val)

subA_xgb = model_xgb.predict(X_test)

print('predict lgb...')

model_lgb = build_model_lgb(x_train,y_train)

val_lgb = model_lgb.predict(x_val)

subA_lgb = model_lgb.predict(X_test)

Predict GBDT…

{‘learning_rate’: 0.1}

predict XGB…

predict lgb…

在初级学习器中，一共建立了三个模型，分别是LightGBM、GBDT、XGBoost。

关于XGBoost模型的原理和代码实现，可以参考: 集成学习之XGBoost算法

#Starking

#第一层

train_lgb_pred = model_lgb.predict(x_train)

train_xgb_pred = model_xgb.predict(x_train)

train_gbdt_pred = model_gbdt.predict(x_train)

Strak_X_train = pd.DataFrame()

Strak_X_train['Method_1'] = train_lgb_pred

Strak_X_train['Method_2'] = train_xgb_pred

Strak_X_train['Method_3'] = train_gbdt_pred

Strak_X_val = pd.DataFrame()

Strak_X_val['Method_1'] = val_lgb

Strak_X_val['Method_2'] = val_xgb

Strak_X_val['Method_3'] = val_gbdt

Strak_X_test = pd.DataFrame()

Strak_X_test['Method_1'] = subA_lgb

Strak_X_test['Method_2'] = subA_xgb

Strak_X_test['Method_3'] = subA_gbdt

这里将线性回归作为次级学习器

#第二层

model_lr_Stacking = build_model_lr(Strak_X_train,y_train)

#训练集

train_pre_Stacking = model_lr_Stacking.predict(Strak_X_train)

print('MAE of Stacking-LR:',mean_absolute_error(y_train,train_pre_Stacking))

#验证集

val_pre_Stacking = model_lr_Stacking.predict(Strak_X_val)

print('MAE of Stacking-LR:',mean_absolute_error(y_val,val_pre_Stacking))

#预测集

print('Predict Stacking-LR...')

subA_Stacking = model_lr_Stacking.predict(Strak_X_test)

MAE of Stacking-LR: 914.5652539316941

MAE of Stacking-LR: 961.6758318716319

Predict Stacking-LR…

三、结果解读

从模型结果可以看出，Stacking融合之后模型的MAE达到了914.5652539316941。这相较于前文使用的单个XGBoost模型，平均绝对误差有所减小。说明Stacking在一定程度上提升了模型精度。

在验证集中，MAE=961.6758318716319，略大于训练集上的MAE。说明模型存在轻微的过拟合，这也是后面模型改进的一个方向。