kaggle 房价预测比赛项目（一）：数据处理，特征选择

房价预测是 kaggle 上的入门比赛，总的来说就是给你 79 个关于房价的特征，然后根据特征预测房价。房价预测的评价指标是均方根误差(RMSE)，即：

1. 数据探索性分析

首先使用 pandas 模块读取数据

import pandas as pd

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

分别显示训练集和测试集的前几行数据

train.head()

test.head()

上面的特征数据中存在离散值和连续值，还有大量的缺失值，由于特征数量较多，这里只基于建造年份来可视化房价数据。

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(10,8))
sns.boxplot(train.YearBuilt, train.SalePrice)

单从建造年份这个特征来看，大致趋势是：老房子便宜，新房子贵。

2. 数据清洗

异常点处理

首先根据房子的面积可视化房价数据，观察是否存在线性关系

plt.figure(figsize=(12,6))
plt.scatter(x=train.GrLivArea, y=train.SalePrice)##可以用来观察存在线型的关系
plt.xlabel("GrLivArea", fontsize=13)
plt.ylabel("SalePrice", fontsize=13)
plt.ylim(0,800000)

从上面的散点图可以看出，房子面积越大，房价也高，但图中右下角存在两个异常点违反了这一线性关系，需要去除

# 去除 "GrLivArea">4000 且 "SalePrice"<300000 的数据
train.drop(train[(train["GrLivArea"]>4000)&(train["SalePrice"]<300000)].index,inplace=True)

数据处理

为了方便，将训练集和测试集数据放在一起处理

full = pd.concat([train,test],ignore_index=True)

首先去除 Id 这一列，对预测没有用

full.drop("Id",axis=1,inplace=True)
print(full.shape)  # (2917, 80)

包含训练集测试集在内共有 2917 个样本，每个样本实际包含 79 个特征，由于训练集数据包含最终的房价数据，所以这里显示 80

空值填充和删除

前面说过样本中存在许多缺失值，这里需要进行缺失值处理，首先查看那些特征存在缺失值，并将缺失的个数从低到高排序

miss = full.isnull().sum() #统计出空值的个数
miss[miss>0].sort_values(ascending=True)#由低到高排好序

GarageArea         1
SaleType           1
KitchenQual        1
BsmtFinSF1         1
BsmtFinSF2         1
GarageCars         1
TotalBsmtSF        1
Exterior2nd        1
Exterior1st        1
BsmtUnfSF          1
Electrical         1
Functional         2
Utilities          2
BsmtHalfBath       2
BsmtFullBath       2
MSZoning           4
MasVnrArea        23
MasVnrType        24
BsmtFinType1      79
BsmtFinType2      80
BsmtQual          81
BsmtCond          82
BsmtExposure      82
GarageType       157
GarageYrBlt      159
GarageFinish     159
GarageCond       159
GarageQual       159
LotFrontage      486
FireplaceQu     1420
SalePrice       1459
Fence           2346
Alley           2719
MiscFeature     2812
PoolQC          2908
dtype: int64

由于特征值包含字符型和数值型，因此需要分开处理
对以下几个字符类型的特征使用 None进行填充：

cols1 = ["PoolQC" , "MiscFeature", "Alley", "Fence", "FireplaceQu", "GarageQual", "GarageCond", "GarageFinish", "GarageYrBlt", "GarageType", "BsmtExposure", "BsmtCond", "BsmtQual", "BsmtFinType2", "BsmtFinType1", "MasVnrType"]
for col in cols1:
    full[col].fillna("None",inplace=True)

对以下几个数值类型特征使用 0 进行填充：

cols=["MasVnrArea", "BsmtUnfSF", "TotalBsmtSF", "GarageCars", "BsmtFinSF2", "BsmtFinSF1", "GarageArea"]
for col in cols:
    full[col].fillna(0, inplace=True)

对 lotfrontage 的空值进行填充（用这一列的均值）

full["LotFrontage"].fillna(np.mean(full["LotFrontage"]),inplace=True)

对以下这些列进行众数填充

cols2 = ["MSZoning", "BsmtFullBath", "BsmtHalfBath", "Utilities", "Functional", "Electrical", "KitchenQual", "SaleType","Exterior1st", "Exterior2nd"]
for col in cols2:
    full[col].fillna(full[col].mode()[0], inplace=True)

接下来查看哪些是还没填充好的，发现只有 test 数据没有标签(房价)列

full.isnull().sum()[full.isnull().sum()>0] # 至此我们已经把空值填充好了

SalePrice    1459
dtype: int64

3. 数据预处理- -将字符型转换为数值型

将一些数字特征转换为类别特征。最好使用 LabelEncoder 和 get_dummies 来实现这些功能。

from sklearn.preprocessing import LabelEncoder #标签编码

lab = LabelEncoder()

对以下这些特征进行标签编码

full["Alley"] = lab.fit_transform(full.Alley)
full["PoolQC"] = lab.fit_transform(full.PoolQC)
full["MiscFeature"] = lab.fit_transform(full.MiscFeature)
full["Fence"] = lab.fit_transform(full.Fence)
full["FireplaceQu"] = lab.fit_transform(full.FireplaceQu)
full["GarageQual"] = lab.fit_transform(full.GarageQual)
full["GarageCond"] = lab.fit_transform(full.GarageCond)
full["GarageFinish"] = lab.fit_transform(full.GarageFinish)
full["GarageYrBlt"] = lab.fit_transform(full.GarageYrBlt)
full["GarageType"] = lab.fit_transform(full.GarageType)
full["BsmtExposure"] = lab.fit_transform(full.BsmtExposure)
full["BsmtCond"] = lab.fit_transform(full.BsmtCond)
full["BsmtQual"] = lab.fit_transform(full.BsmtQual)
full["BsmtFinType2"] = lab.fit_transform(full.BsmtFinType2)
full["BsmtFinType1"] = lab.fit_transform(full.BsmtFinType1)
full["MasVnrType"] = lab.fit_transform(full.MasVnrType)
full["BsmtFinType1"] = lab.fit_transform(full.BsmtFinType1)

full["MSZoning"] = lab.fit_transform(full.MSZoning)
full["BsmtFullBath"] = lab.fit_transform(full.BsmtFullBath)
full["BsmtHalfBath"] = lab.fit_transform(full.BsmtHalfBath)
full["Utilities"] = lab.fit_transform(full.Utilities)
full["Functional"] = lab.fit_transform(full.Functional)
full["Electrical"] = lab.fit_transform(full.Electrical)
full["KitchenQual"] = lab.fit_transform(full.KitchenQual)
full["SaleType"] = lab.fit_transform(full.SaleType)
full["Exterior1st"] = lab.fit_transform(full.Exterior1st)
full["Exterior2nd"] = lab.fit_transform(full.Exterior2nd)

这样处理后大部分特征都转换为了数值类型

full.head()

最后将房价对应的列删除

full.drop("SalePrice",axis=1,inplace=True) #删除
full.shape  # (2917, 79)

4. 管道建设- -pipeline

方便组合各种特征以及对特征的处理，方便后续的机器学习的特征的重做
首先构建转换函数：

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline, make_pipeline #构建管道
from scipy.stats import skew #偏度


class labelenc(BaseEstimator, TransformerMixin):
    def __init__(self):
        pass
    
    def fit(self,X,y=None):
        return self
    ##对三个年份来进行一个标签编码,这里可以随便自己添加
    def transform(self,X):
        lab=LabelEncoder()
        X["YearBuilt"] = lab.fit_transform(X["YearBuilt"])
        X["YearRemodAdd"] = lab.fit_transform(X["YearRemodAdd"])
        X["GarageYrBlt"] = lab.fit_transform(X["GarageYrBlt"])
        X["BldgType"] = lab.fit_transform(X["BldgType"])
        
        return X

class skew_dummies(BaseEstimator, TransformerMixin):
    def __init__(self,skew=0.5):#偏度
        self.skew = skew
    
    def fit(self,X,y=None):
        return self
    
    def transform(self,X):
        #去除了包含了对象数据类型，取出来绝大部分是数值型
        X_numeric=X.select_dtypes(exclude=["object"])
        #匿名函数，做成字典的形式
        skewness = X_numeric.apply(lambda x: skew(x))
        #通过条件来涮选出skew>=0.5的索引的条件，取到了全部数据，防止数据的丢失
        skewness_features = skewness[abs(skewness) >= self.skew].index
        #求对数，进一步让他更符合正态分布
        X[skewness_features] = np.log1p(X[skewness_features])
        ##一键独热，独热编码
        X = pd.get_dummies(X)
        return X

构建管道

pipe = Pipeline([##构建管道的意思
    ('labenc', labelenc()),
    ('skew_dummies', skew_dummies(skew=2)),
    ])

将数据进行管道处理

full2 = full.copy()
pipeline_data = pipe.fit_transform(full2)
pipeline_data.shape  # (2917, 178)

将处理好的数据重新划分为训练集和测试集

n_train=train.shape[0] # 训练集的行数
X = pipeline_data[:n_train] # 取出处理之后的训练集
test_X = pipeline_data[n_train:] # 取出n_train后的数据作为测试集
y= train.SalePrice
X_scaled = StandardScaler().fit(X).transform(X) # 做转换
y_log = np.log(train.SalePrice) # 这里使用log，更符合正态分布
#得到测试集
test_X_scaled = StandardScaler().fit_transform(test_X)

5. 特征选择- -基于特征重要性来选择

运用算法来进行训练集的特征选择

from sklearn.linear_model import Lasso

lasso=Lasso(alpha=0.001)
lasso.fit(X_scaled,y_log)

FI_lasso = pd.DataFrame({"Feature Importance":lasso.coef_}, index=pipeline_data.columns)

FI_lasso.sort_values("Feature Importance",ascending=False)#由高到低进行排序
FI_lasso[FI_lasso["Feature Importance"]!=0].sort_values("Feature Importance").plot(kind="barh",figsize=(15,25))
plt.xticks(rotation=90)
plt.show()##画图显示

得到特征重要性图之后就可以进行特征选择与重做

class add_feature(BaseEstimator, TransformerMixin):#自己定义转换函数--fit_transform由自己定义
    def __init__(self,additional=1):
        self.additional = additional
    
    def fit(self,X,y=None):
        return self
    
    def transform(self,X):
        if self.additional==1:
            X["TotalHouse"] = X["TotalBsmtSF"] + X["1stFlrSF"] + X["2ndFlrSF"]   
            X["TotalArea"] = X["TotalBsmtSF"] + X["1stFlrSF"] + X["2ndFlrSF"] + X["GarageArea"]
            
        else:
            X["TotalHouse"] = X["TotalBsmtSF"] + X["1stFlrSF"] + X["2ndFlrSF"]   
            X["TotalArea"] = X["TotalBsmtSF"] + X["1stFlrSF"] + X["2ndFlrSF"] + X["GarageArea"]
            
            X["+_TotalHouse_OverallQual"] = X["TotalHouse"] * X["OverallQual"]
            X["+_GrLivArea_OverallQual"] = X["GrLivArea"] * X["OverallQual"]
            X["+_oMSZoning_TotalHouse"] = X["MSZoning"] * X["TotalHouse"]
            X["+_oMSZoning_OverallQual"] = X["MSZoning"] + X["OverallQual"]
            X["+_oMSZoning_YearBuilt"] = X["MSZoning"] + X["YearBuilt"]
            X["+_oNeighborhood_TotalHouse"] = X["Neighborhood"] * X["TotalHouse"]
            X["+_oNeighborhood_OverallQual"] = X["Neighborhood"] + X["OverallQual"]
            X["+_oNeighborhood_YearBuilt"] = X["Neighborhood"] + X["YearBuilt"]
            X["+_BsmtFinSF1_OverallQual"] = X["BsmtFinSF1"] * X["OverallQual"]
            
            X["-_oFunctional_TotalHouse"] = X["Functional"] * X["TotalHouse"]
            X["-_oFunctional_OverallQual"] = X["Functional"] + X["OverallQual"]
            X["-_LotArea_OverallQual"] = X["LotArea"] * X["OverallQual"]
            X["-_TotalHouse_LotArea"] = X["TotalHouse"] + X["LotArea"]
            X["-_oCondition1_TotalHouse"] = X["Condition1"] * X["TotalHouse"]
            X["-_oCondition1_OverallQual"] = X["Condition1"] + X["OverallQual"]
            
           
            X["Bsmt"] = X["BsmtFinSF1"] + X["BsmtFinSF2"] + X["BsmtUnfSF"]
            X["Rooms"] = X["FullBath"]+X["TotRmsAbvGrd"]
            X["PorchArea"] = X["OpenPorchSF"]+X["EnclosedPorch"]+X["3SsnPorch"]+X["ScreenPorch"]
            X["TotalPlace"] = X["TotalBsmtSF"] + X["1stFlrSF"] + X["2ndFlrSF"] + X["GarageArea"] + X["OpenPorchSF"]+X["EnclosedPorch"]+X["3SsnPorch"]+X["ScreenPorch"]

    
            return X

将所有的特征处理都加到管道里来

pipe = Pipeline([
    ('labenc', labelenc()),
    ('add_feature', add_feature(additional=2)),
    ('skew_dummies', skew_dummies(skew=4)),
    ])

full3 = full.copy()
pipeline_data = pipe.fit_transform(full3)
pipeline_data.shape  # (2917, 159)

n_train=train.shape[0]#训练集的行数
X = pipeline_data[:n_train]#取出处理之后的训练集
test_X = pipeline_data[n_train:]#取出n_train后的数据作为测试集
y= train.SalePrice
X_scaled = StandardScaler().fit(X).transform(X)#做转换
y_log = np.log(train.SalePrice)##这里要注意的是，更符合正态分布
#得到测试集
test_X_scaled = StandardScaler().fit_transform(test_X)

6. 构建决策树模型

from sklearn.tree import DecisionTreeRegressor#导入模型

model = DecisionTreeRegressor()
model1 =model.fit(X_scaled,y_log)

predict = np.exp(model1.predict(test_X_scaled)) #np.exp是对上面的对数变换之后的反变换
result=pd.DataFrame({'Id':test.Id, 'SalePrice':predict})
result.to_csv("submission.csv",index=False)

将得到的 submission.csv 上传，大概得分为 0.19760