sklearn 缺失值处理

1. 查看dataframe所有列缺失情况

   df = pd.read_excel('real_estate_sample.xlsx')
   df.isna().sum()
   

   

2. 查看列特殊字符的缺失
   df.列名.value_counts()
   

解决方案:

• 删除

• 填充(缺失值<10%)

• 均值填充:
  适用于正态分布

  # 选择均值填充
  price_mean = df['average_price'].mean()
  # 使用均值进行缺失值填充
  df.loc[df['average_price'].isna(),'average_price'] = price_mean
  print(df.isnull().sum())
  

• 中位数填充:
  适用于偏态分布

  # 选择中位数填充
  area_median = df['area'].median()
  # 使用均值进行缺失值填充
  df.loc[df['area'].isna(),'area'] = area_median
  print(df.isnull().sum())
  

• 算法填充:

  1. 查看相关系数

     # 查看相关性系数
     print(df[['daypop','nightpop','night20-39']].corr())
     

     

  2. 构建模型

     # 训练线性回归模型对夜间人口进行填补
     from sklearn.linear_model import LinearRegression
     from sklearn.metrics import mean_absolute_error, r2_score
     train = df.copy().loc[~df['nightpop'].isna(),['daypop','nightpop']].reset_index(drop=True)#波浪号取反
     x = train[['daypop']]
     y = train[['nightpop']]
     model = LinearRegression()
     model.fit(x,y)
     print(f'R squared is: {r2_score(y, x)}')#查看模型效果
     

     

  3. 模型效果理想则可进行预测填充

     df.loc[df['nightpop'].isna(),['nightpop']] = model.predict(df.loc[df['nightpop'].isna(),['daypop']])
     print(df.isnull().sum())