Day12-集成学习-机器学习-Blending(DataWhale)

一、Blending算法

1.1 原理

(1) 将数据划分为训练集和测试集(test_set)，其中训练集再划分为训练集(train_set)和验证集(val_set)

(2) 创建第一层的多个模型，模型可以是同质的也可以是异质的

(3) 使用train_set训练步骤2中的多个模型，然后用训练好的模型预测val_set和test_set得到val_predict和test_predict1

(4) 创建第二层的模型，使用val_predict作为训练集训练第二层的模型

(5) 使用第二层训练好的模型对第二层测试集test_predict1进行预测，该结果为整个测试集的结果

1.2 过程

• 在第(1)步中，将数据集分成训练集(80%)和测试集(20%)，再将训练集(80%)拆分为训练集(70%)和验证集(30%)，最后得到训练集(80% * 70%)，测试集(20%)，验证集(80% * 30%)。训练集来训练模型，测试集来调整模型(调参)，验证集来检验模型的优度。
  
• 在第(2)-(3)步中，使用训练集创建K个模型，如SVM、random forests、XGBoost，这是第一层的模型。训练好模型后将验证集输入模型进行预测，得到K组不同的输出，记作$A_1,A_2,\dots \dots ,A_K$，然后将测试集输入K个模型得到K组输出，记作$B_1,\dots \dots ,B_K$，其中$A_i$的样本数与验证集一致，$B_i$的样本数与从测试集一致
• 在第(4)步中，使用验证集结果$A_1,\dots \dots ,A_K$来训练第二层分类器
• 在第(5)步中，将测试集结果$B_1,\dots \dots ,B_K$放入第二层分类器，得到预测结果

1.3 评价

• 优点：简单粗暴
• 缺点：Blending只使用了一部分数据集作为留出集进行验证

1.4 实现

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use("ggplot")
%matplotlib inline
import seaborn as sns

# 创建数据
from sklearn import datasets
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split

data, target = make_blobs(n_samples=10000, centers=2, random_state=1, cluster_std=1.0)
'''
make_blobs 为聚类产生数据集
n_festures：数据的维度，默认为2
centers：数据的中心点，默认为3
cluster_std：数据集的标准差，默认1
center_box：中心确认之后的数据边界，默认(-10,10)
shuffle：洗乱
'''

# 创建训练集和测试集
X_train1, X_test, y_train1, y_test = train_test_split(data, target, test_size=0.3, random_state=1)
# 创建训练集和验证集
X_train, X_val, y_train, y_val = train_test_split(X_train1, y_train1, test_size=0.3, random_state=1)

print("The shape of training X:", X_train.shape)
print("The shape of training y:", y_train.shape)
print("The shape of test X:", X_test.shape)
print("The shape of test y:", y_test.shape)
print("The shape of validation X:", X_val.shape)
print("The shape of validation y:", y_val.shape)

The shape of training X: (4900, 2)
The shape of training y: (4900,)
The shape of test X: (3000, 2)
The shape of test y: (3000,)
The shape of validation X: (2100, 2)
The shape of validation y: (2100,)

# 设置第一层分类器
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

clfs = [SVC(probability = True), RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'), KNeighborsClassifier()]

# 设置第二层分类器
from sklearn.linear_model import LogisticRegression
lg = LogisticRegression()

# 输出第一层的验证集结果与测试集结果
val_features = np.zeros((X_val.shape[0], len(clfs))) #初始化验证结果
test_features = np.zeros((X_test.shape[0], len(clfs))) #初始化测试结果

for i,clf in enumerate(clfs):
    clf.fit(X_train,y_train)
    val_feature = clf.predict_proba(X_val)[:, 1]
    test_feature = clf.predict_proba(X_test)[:, 1]
    val_features[: ,i] = val_feature
    test_features[:,i] = test_feature

# 将第一层的验证集结果输入第二层，训练第二层分类器
lg.fit(val_features, y_val)

# 输出预测的结果
from sklearn.model_selection import cross_val_score
cross_val_score(lg, test_features, y_test, cv=5)

array([1., 1., 1., 1., 1.])

使用iris数据集

from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target
features = iris.feature_names
iris_data = pd.DataFrame(X, columns=features)
iris_data['target'] = y

iris_data.head(1)

	sepal length (cm)	sepal width (cm)	petal length (cm)	petal width (cm)	target
0	5.1	3.5	1.4	0.2	0

## 创建训练集和测试集
X_train1,X_test,y_train1,y_test = train_test_split(iris_data.iloc[:,:2], iris_data.iloc[:,-1], test_size=0.2, random_state=1)
## 创建训练集和验证集
X_train,X_val,y_train,y_val = train_test_split(X_train1, y_train1, test_size=0.3, random_state=1)
print("The shape of training X:",X_train.shape)
print("The shape of training y:",y_train.shape)
print("The shape of test X:",X_test.shape)
print("The shape of test y:",y_test.shape)
print("The shape of validation X:",X_val.shape)
print("The shape of validation y:",y_val.shape)

The shape of training X: (84, 2)
The shape of training y: (84,)
The shape of test X: (30, 2)
The shape of test y: (30,)
The shape of validation X: (36, 2)
The shape of validation y: (36,)

#  设置第一层分类器
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

clfs = [SVC(probability = True),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),KNeighborsClassifier()]

# 设置第二层分类器
from sklearn.linear_model import LogisticRegression
lg = LogisticRegression()

# 输出第一层的验证集结果与测试集结果
val_features = np.zeros((X_val.shape[0],len(clfs)))  # 初始化验证集结果
test_features = np.zeros((X_test.shape[0],len(clfs)))  # 初始化测试集结果

for i,clf in enumerate(clfs):
    clf.fit(X_train,y_train)
    val_feature = clf.predict_proba(X_val)[:, 1]
    test_feature = clf.predict_proba(X_test)[:,1]
    val_features[:,i] = val_feature
    test_features[:,i] = test_feature

# 将第一层的验证集的结果输入第二层训练第二层分类器
lg.fit(val_features,y_val)
# 输出预测的结果
from sklearn.model_selection import cross_val_score
cross_val_score(lg,test_features,y_test,cv=5)

array([0.83333333, 0.5       , 0.83333333, 0.83333333, 0.66666667])

x_min,x_max = X_test.iloc[:,0].min() - 1,X_test.iloc[:,0].max() + 1
y_min,y_max = X_test.iloc[:,1].min() - 1,X_test.iloc[:,1].max() + 1
cmap_light = ListedColormap(['#AAAAFF','#AAFFAA','#FFAAAA'])
h = .02
xx,yy = np.meshgrid(np.arange(x_min,x_max,h),np.arange(y_min,y_max,h))
Z = lg.predict(np.c_[xx.ravel(),yy.ravel()])
Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx,yy,Z,cmap=cmap_light)
plt.scatter(x[:,0],x[:,1],c=y)
plt.xlim(xx.min(),xx.max())
plt.ylim(yy.min(),yy.max())
plt.show()