python自学篇十六[pandas——数据分析 （二）：读取文件+索引+NaNs处理方法]

概括：Numpy+Scipy+pandas+matplotlib

pandas基本功能

一.数据文件读取/文本数据读取

1.pandas：数据文件读取

通过pandas提供的read_xxx相关的函数可以读取文件中的数据，并形成DataFrame,常用的数据读取方法为：read_csv，主要可以读取文本类型的数据

help(pd.read_csv)

2.读取csv文件

先创建一个data1.csv文件，里面内容写：

name,age,source
Peter,18,98.5
Tom,21,78.2
Bob,24,98.5
Wangdachui,20,89.2

Jupyter notebook代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
##读取csv文件
df=pd.read_csv("data1.csv")
df

3.读取txt文件

先创建一个data01.txt文件，里面内容写：

王大锤;18;100;99;98
王大锤;18;100;99;98
王大锤;18;100;99;98

Jupyter notebook代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
#读取文本数据，指定";"为分隔符，不读取头部数据
df=pd.read_csv("data01.txt",sep=';',header=None)
df

二.索引、选取和数据过滤

pandas：数据过滤获取
通过DataFrame的相关方式可以获取对应的列或者数据形成一个新的DataFrame, 方便后续进行统计计算。

对于DataFrame/Series中的NaN一般采取的方式为删除对应的列/行或者填充一个默认值

1.指定行头部

创建一个data01.txt文件，里面内容写：

王大锤;18;100;99;98
王大锤;18;100;99;98
王大锤;18;100;99;98

Jupyter notebook代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
columns=['name','age',u'语文',u'数学',u'英语']
df.columns=columns
df

2.切片获取内容

创建一个data01.txt文件，里面内容写：

王大锤;18;100;99;98
王大锤;18;100;99;98
王大锤;18;100;99;98

Jupyter notebook代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
columns=['name','age',u'语文',u'数学',u'英语']
df=df[columns[2:]]
df

三.pandas：缺省值NaN处理方法

对于DataFrame/Series中的NaN一般采取的方式为删除对应的列/行或者填充一个默认值
代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
df2=DataFrame([
    ['Tom',np.nan,456.67,'M'],
    ['Merry',34,345.56,np.nan],
    ['Gerry',np.nan,np.nan,np.nan],
    ['Jom',np.nan,456.67,'M'],
    ['Jone',18,35.12,'F']],
    columns=['name','age','salary','gender']
)
df2

结果：

name	age	salary	gender
0	Tom	NaN	456.67	M
1	Merry	34.0	345.56	NaN
2	Gerry	NaN	NaN	NaN
3	Jom	NaN	456.67	M
4	Jone	18.0	35.12	F

1.dropna

根据标签的值中是否存在缺失数据对轴标签进行过滤(删除), 可以通过阈值的调节对缺失值的容忍度

1.dropna()

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
df2=DataFrame([
    ['Tom',np.nan,456.67,'M'],
    ['Merry',34,345.56,np.nan],
    ['Gerry',np.nan,np.nan,np.nan],
    ['Jom',np.nan,456.67,'M'],
    ['Jone',18,35.12,'F']],
    columns=['name','age','salary','gender']
)
df2.dropna()#默认丢弃只要包含缺失值的行（去掉有NaN的数据）

结果：

name	age	salary	gender
4	Jone	18.0	35.12	F

2.dropna(how=‘all’)

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
df2=DataFrame([
    ['Tom',np.nan,456.67,'M'],
    ['Merry',34,345.56,np.nan],
    ['Gerry',np.nan,np.nan,np.nan],
    ['Jom',np.nan,456.67,'M'],
    ['Jone',18,35.12,'F']],
    columns=['name','age','salary','gender']
)
df2.dropna(how='all')#给定只丢弃值全部为缺失值的行

结果：

	name	age	salary	gender
0	Tom	NaN	456.67	M
1	Merry	34.0	345.56	NaN
2	Gerry	NaN	NaN	NaN
3	Jom	NaN	456.67	M
4	Jone	18.0	35.12	F

3.dropna(axis=1)

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
df2=DataFrame([
    ['Tom',np.nan,456.67,'M'],
    ['Merry',34,345.56,np.nan],
    ['Gerry',np.nan,np.nan,np.nan],
    ['Jom',np.nan,456.67,'M'],
    ['Jone',18,35.12,'F']],
    columns=['name','age','salary','gender']
)
df2.dropna(axis=1)#丢弃列

结果：

name
0	Tom
1	Merry
2	Gerry
3	Jom
4	Jone

2.fillna

用指定值或者插值的方式填充缺失数据，比如: ffill或者bfill

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd

df=DataFrame(np.random.randn(7,3))
df.loc[:4,1]=np.nan
df.loc[:2,2]=np.nan
df

结果：

0	1	2
0	-0.031422	NaN	NaN
1	-0.916141	NaN	NaN
2	0.427765	NaN	NaN
3	0.242490	NaN	0.200289
4	-0.214651	NaN	0.533594
5	0.302438	-0.228859	-0.883538
6	-0.356205	0.154669	-0.448864

1.fillna(0)

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd

df=DataFrame(np.random.randn(7,3))
df.loc[:4,1]=np.nan
df.loc[:2,2]=np.nan
print(df.fillna(0))
print()
print(df.fillna('fill'))

结果：

         0         1         2
0 -1.034991  0.000000  0.000000
1 -1.923518  0.000000  0.000000
2 -1.314832  0.000000  0.000000
3  0.167929  0.000000 -0.806852
4  0.900060  0.000000  1.443878
5  0.312364  0.222698 -1.000081
6 -0.291597  1.095243  0.678713

          0         1         2
0  1.093552      fill      fill
1  2.208307      fill      fill
2  0.319327      fill      fill
3 -0.525311      fill  -0.11486
4  1.539547      fill  -1.30771
5 -3.682927  0.588243   1.15384
6  0.255435  -2.06252 -0.808872

2.fillna({1:0.5,2:-1,3:1})

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd

df=DataFrame(np.random.randn(7,3))
df.loc[:4,1]=np.nan
df.loc[:2,2]=np.nan
df.fillna({1:0.5,2:-1,3:1})
结果：
```python
0	1	2
0	1.317635	-1.574467	1.399366
1	-0.690984	0.823191	-0.138721
2	2.840376	-0.522517	0.347104
3	-0.036683	0.767751	-0.646185
4	0.676305	-1.961409	-1.337382
5	1.752402	-1.192964	-0.057789
6	0.171615	0.554056	-1.322705

3.isnul

返回一个含有布尔值的对象，这些布尔值表示那些值是缺失值NA
代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
df2=DataFrame([
    ['Tom',np.nan,456.67,'M'],
    ['Merry',34,345.56,np.nan],
    ['Gerry',np.nan,np.nan,np.nan],
    ['Jom',np.nan,456.67,'M'],
    ['Jone',18,35.12,'F']],
    columns=['name','age','salary','gender']
)
df2.isnull()

结果：

	name	age	salary	gender
0	False	True	False	False
1	False	False	False	True
2	False	True	True	True
3	False	True	False	False
4	False	False	False	False

4. notnull

isnull的否定式

代码：

import numpy as np
from pandas import Series, DataFrame
import pandas as pd
df2=DataFrame([
    ['Tom',np.nan,456.67,'M'],
    ['Merry',34,345.56,np.nan],
    ['Gerry',np.nan,np.nan,np.nan],
    ['Jom',np.nan,456.67,'M'],
    ['Jone',18,35.12,'F']],
    columns=['name','age','salary','gender']
)
df2.notnull()

结果：

name	age	salary	gender
0	True	False	True	True
1	True	True	True	False
2	True	False	False	False
3	True	False	True	True
4	True	True	True	True