pandas处理缺失值
pandas 用 nan 表示缺失值，可以用 isnull() 检测
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: string_data = pd.Series(['aardvark', 'artichoke', np.nan, 'avocado'])
In [4]: string_data
Out[4]:
0 aardvark
1 artichoke
2 NaN
3 avocado
dtype: object
In [5]: string_data.isnull()
Out[5]:
0 False
1 False
2 True
3 False
dtype: bool内建的Python None值也被当做NA：
In [6]: string_data[0] = None
In [7]: string_data.isnull()
Out[7]:
0 True
1 False
2 True
3 False
dtype: bool一、Filtering Out Missing Data（过滤缺失值）
1 . pd.dropna()
In [9]: from numpy import nan as NA
In [10]: data = pd.Series([1,5,6,NA,5,NA])
In [11]: data.dropna()
Out[11]:
0 1.0
1 5.0
2 6.0
4 5.0
dtype: float64上面等同于
In [12]: data[data.notnull()]
Out[12]:
0 1.0
1 5.0
2 6.0
4 5.0
dtype: float64对于DataFrame，会复杂一些，dropna默认会删除包含有缺失值的row：
In [13]: data = pd.DataFrame([[1., 6.5, 3.], [1., NA, NA],
...: [NA, NA, NA], [NA, 6.5, 3.]])
...:
In [14]: data
Out[14]:
0 1 2
0 1.0 6.5 3.0
1 1.0 NaN NaN
2 NaN NaN NaN
3 NaN 6.5 3.0
In [15]: data.dropna()
Out[15]:
0 1 2
0 1.0 6.5 3.0设定how=all只会删除那些全是NA的行：
In [16]: data.dropna(how = 'all')
Out[16]:
0 1 2
0 1.0 6.5 3.0
1 1.0 NaN NaN
3 NaN 6.5 3.0删除列也一样，设置axis=1:
In [17]: data[4] = NA
In [18]: data
Out[18]:
0 1 2 4
0 1.0 6.5 3.0 NaN
1 1.0 NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN 6.5 3.0 NaN
In [19]: data.dropna(axis=1,how = 'all')
Out[19]:
0 1 2
0 1.0 6.5 3.0
1 1.0 NaN NaN
2 NaN NaN NaN
3 NaN 6.5 3.0thresh参数,它的作用是，比如 thresh=2，则一行中至少有两个非nan的值，否则会被删除。
In [23]: data
Out[23]:
0 1 2 4
0 1.0 6.5 3.0 NaN
1 1.0 NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN 6.5 3.0 NaN
In [24]: data.dropna(thresh = 2)
Out[24]:
0 1 2 4
0 1.0 6.5 3.0 NaN
3 NaN 6.5 3.0 NaN2 Filling In Missing Data（填补缺失值）
不是删除缺失值，而是用一些数字填补。对于大部分目的，fillna是可以用的。调用fillna的时候设置好一个常用用来替换缺失值：
In [25]: data
Out[25]:
0 1 2 4
0 1.0 6.5 3.0 NaN
1 1.0 NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN 6.5 3.0 NaN
In [26]: data.fillna(0)
Out[26]:
0 1 2 4
0 1.0 6.5 3.0 0.0
1 1.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0
3 0.0 6.5 3.0 0.0给fillna传入一个dict，可以给不同列替换不同的值：
In [27]: data.fillna({1:1,2:2})
Out[27]:
0 1 2 4
0 1.0 6.5 3.0 NaN
1 1.0 1.0 2.0 NaN
2 NaN 1.0 2.0 NaN
3 NaN 6.5 3.0 NaNfillna 还可以使用 method 参数
method 可以使用下面的方法
1 . pad/ffill：用前一个非缺失值去填充该缺失值
2 . backfill/bfill：用下一个非缺失值填充该缺失值
In [50]: data
Out[50]:
0 1 2 4
0 1.0 6.5 3.0 2.0
1 1.0 NaN NaN 8.0
2 NaN NaN NaN NaN
3 NaN 6.5 3.0 NaN
In [51]: data.fillna(method='ffill')
Out[51]:
0 1 2 4
0 1.0 6.5 3.0 2.0
1 1.0 6.5 3.0 8.0
2 1.0 6.5 3.0 8.0
3 1.0 6.5 3.0 8.0可以使用limit控制填充的数量
In [52]: data.fillna(method='ffill',limit=1)
Out[52]:
0 1 2 4
0 1.0 6.5 3.0 2.0
1 1.0 6.5 3.0 8.0
2 1.0 NaN NaN 8.0
3 NaN 6.5 3.0 NaN使用fillna可以我们做一些颇有创造力的事情。比如，可以传入一个series的平均值或中位数：
In [53]: data = pd.Series([1., NA, 3.5, NA, 7])
In [54]: data.fillna(data.mean())
Out[54]:
0 1.000000
1 3.833333
2 3.500000
3 3.833333
4 7.000000
dtype: float64在改变DataFrame 和 Series 的操作时，会返回一个新的对象，原对象不变，如果要改变原对象，可以添加参数 inplace = True。
END。