pandas 缺失值处理和统计操作

缺失值处理

在Pandas中使用浮点值NaN表示数组中的缺失数据

1. 使用reindex()方法可以对指定轴上的索引进行改变/增加/删除操作，这将返回原始数据的一个拷贝。

In [88]: df1 = df.reindex(index=dates[0:4],columns=list(df.columns)+['E'])

In [89]: df1
Out[89]:
                   A         B         C         D   E
2013-01-01 -2.130791  0.903688  0.645726 -0.776207 NaN
2013-01-02 -0.622650  0.499566 -0.022492  1.326563 NaN
2013-01-03  2.140337  0.605600 -1.312784  1.059143 NaN
2013-01-04 -1.125467 -0.200313 -0.082067 -0.523501 NaN

In [90]: df1.loc[dates[0]:dates[1],'E']=1

In [91]: df1
Out[91]:
                   A         B         C         D   E
2013-01-01 -2.130791  0.903688  0.645726 -0.776207   1
2013-01-02 -0.622650  0.499566 -0.022492  1.326563   1
2013-01-03  2.140337  0.605600 -1.312784  1.059143 NaN
2013-01-04 -1.125467 -0.200313 -0.082067 -0.523501 NaN

2. 去掉包含缺失值的行，不改变原来的值 dropna

In [92]: df1.dropna(how='any')
Out[92]:
                   A         B         C         D  E
2013-01-01 -2.130791  0.903688  0.645726 -0.776207  1
2013-01-02 -0.622650  0.499566 -0.022492  1.326563  1

3. 对缺失值进行填充 fillna

In [94]: df1.fillna(value=5)
Out[94]:
                   A         B         C         D  E
2013-01-01 -2.130791  0.903688  0.645726 -0.776207  1
2013-01-02 -0.622650  0.499566 -0.022492  1.326563  1
2013-01-03  2.140337  0.605600 -1.312784  1.059143  5
2013-01-04 -1.125467 -0.200313 -0.082067 -0.523501  5

4. 对数据进行布尔填充 isnull notnull

In [95]: pd.isnull(df1)
Out[95]:
                A      B      C      D      E
2013-01-01  False  False  False  False  False
2013-01-02  False  False  False  False  False
2013-01-03  False  False  False  False   True
2013-01-04  False  False  False  False   True

In [96]: df1.isnull()
Out[96]:
                A      B      C      D      E
2013-01-01  False  False  False  False  False
2013-01-02  False  False  False  False  False
2013-01-03  False  False  False  False   True
2013-01-04  False  False  False  False   True

统计操作

1. 统计操作

执行描述性统计默认x轴

In [98]: df.mean()
Out[98]:
A   -0.234022
B    0.433988
C   -0.224383
D    0.164193
dtype: float64

在其他轴上进行相同的操作

In [99]: df.mean(1)
Out[99]:
2013-01-01   -0.339396
2013-01-02    0.295247
2013-01-03    0.623074
2013-01-04   -0.482837
2013-01-05    0.580012
2013-01-06   -0.466437
Freq: D, dtype: float64

对不拥有不同维度，需要对齐的对象进行操作pandas会自动的沿着指定的维度进行广播 shift

In [101]: s
Out[101]:
2013-01-01     1
2013-01-02     3
2013-01-03     5
2013-01-04   NaN
2013-01-05     5
2013-01-06     8
Freq: D, dtype: float64

In [102]: s=pd.Series([1,3,5,np.nan,5,8],index=dates).shift(1)

In [103]: s
Out[103]:
2013-01-01   NaN
2013-01-02     1
2013-01-03     3
2013-01-04     5
2013-01-05   NaN
2013-01-06     5
Freq: D, dtype: float64

2. Apply对数据也应用函数

In [104]: df.apply(np.cumsum)
Out[104]:
                   A         B         C         D
2013-01-01 -2.130791  0.903688  0.645726 -0.776207
2013-01-02 -2.753441  1.403254  0.623234  0.550356
2013-01-03 -0.613104  2.008854 -0.689550  1.609499
2013-01-04 -1.738571  1.808542 -0.771617  1.085997
2013-01-05 -1.206559  3.211595 -0.650754  1.350118
2013-01-06 -1.404133  2.603928 -1.346298  0.985156
In [107]: df.apply(lambda x:x.max()-x.min())
Out[107]:
A    4.271128
B    2.010720
C    1.958509
D    2.102770
dtype: float64

3. 直方图 value_counts()

In [108]: s=pd.Series(np.random.randint(0,7,size=10))

In [109]: s
Out[109]:
0    4
1    6
2    1
3    1
4    2
5    6
6    6
7    4
8    5
9    1
dtype: int32
In [111]: s.value_counts()
Out[111]:
6    3
1    3
4    2
5    1
2    1
dtype: int64

4. 字符串方法

In [112]: s=pd.Series(['A','B','C','Aaba','Baca',np.nan,'CABA','dog','cat'])

In [113]: s
Out[113]:
0       A
1       B
2       C
3    Aaba
4    Baca
5     NaN
6    CABA
7     dog
8     cat
dtype: object

In [114]: s.str.lower()
Out[114]:
0       a
1       b
2       c
3    aaba
4    baca
5     NaN
6    caba
7     dog
8     cat
dtype: object

In [115]: s.str.upper()
Out[115]:
0       A
1       B
2       C
3    AABA
4    BACA
5     NaN
6    CABA
7     DOG
8     CAT
dtype: object