In [9]: ints=np.ones(10,dtype=np.uint16)
In [10]: floats=np.ones(10,dtype=np.float32)
In [11]: np.issubdtype(ints.dtype,np.integer)#判断数据类型，同python的isinstance
Out[11]: True
In [12]: np.issubdtype(floats.dtype,np.floating)
Out[12]: True
#调用dtype的mro方法即可查看其所有的父类
In [13]: np.float64.mro()
Out[13]:
[numpy.float64,
numpy.floating,
numpy.inexact,
numpy.number,
numpy.generic,
float,
object]
下图简单说明了dtype体系以及父子类关系
1.1 创建数组
import numpy as np
In [15]: np.array([1,2,3])
Out[15]: array([1, 2, 3])#传入列表，产生一维数组
In [16]: np.array([[1,2,3],[4,5,6]])
Out[16]:
array([[1, 2, 3],
[4, 5, 6]])#传入嵌套列表，产生二维数组
#传入一个数组
In [22]: rnd=np.random.randn(12)
In [23]: rnd_re=rnd.reshape(3,4)#reshape,重新构造数组结构
In [24]: rnd
Out[24]:
array([ 0.65986137, 0.59780991, -0.29968381, -0.13900897, -0.91233434,
-1.11650251, 0.10361076, -0.99113903, -0.18876077, -1.07867959,
-1.35017413, -0.02826018])
In [25]: rnd_re
Out[25]:
array([[ 0.65986137, 0.59780991, -0.29968381, -0.13900897],
[-0.91233434, -1.11650251, 0.10361076, -0.99113903],
[-0.18876077, -1.07867959, -1.35017413, -0.02826018]])
In [26]: np.array(rnd_re)
Out[26]:
array([[ 0.65986137, 0.59780991, -0.29968381, -0.13900897],
[-0.91233434, -1.11650251, 0.10361076, -0.99113903],
[-0.18876077, -1.07867959, -1.35017413, -0.02826018]])
1.2 shape和dtype
shape表示各维度大小的元组，tdype用于说明数组数据类型的对象
In [1]: import numpy as np
In [2]: data=np.array([1,2,3])
In [3]: data.shape
Out[3]: (3,)
In [4]: data=np.array([[1,2,3],[4,5,6]])
In [5]: data.shape
Out[5]: (2, 3)
In [6]: data.dtype
Out[6]: dtype('int64')
#数值型dtype的命名方式相同:一个类型名(如int，float)，后面跟一个表示各元素位长的数字，如float64
In [36]: np.array([1,2,3]).dtype
Out[36]: dtype('int64')
In [37]: data=np.array([1,2,3],dtype=np.float64)
In [38]: data.dtype
Out[38]: dtype('float64')
1.6 astype
通过astype可以显式的转换数组的数据类型
In [41]: data=np.array(['1','2','3'])
In [42]: data.dtype
Out[42]: dtype('
In [84]: data_arr=np.array([[x for x in range(10)],[x for x in range(10,20)]])
In [85]: data_arr
Out[85]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])
In [86]: data_arr[1][1]
Out[86]: 11
In [87]: data_arr[1,1]
Out[87]: 11
三维数组,切片取值可以理解为三层嵌套列表，同样可以使用两种方法表达
In [90]: data_arr=np.array([[[x for x in range(10)],[x for x in range(10,20)]],[
...: [x for x in range(20,30)],[x for x in range(40,50)]]])
In [91]: data_arr
Out[91]:
array([[[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]]])
In [95]: data_arr[1,1,1]
Out[95]: 41
In [92]: data_arr[1]
Out[92]:
array([[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]])
In [93]: data_arr[1][1]
Out[93]: array([40, 41, 42, 43, 44, 45, 46, 47, 48, 49])
In [94]: data_arr[1][1][1]
Out[94]: 41
In [105]: data=np.array(['a','b','c'])
In [106]: data=='d'
Out[106]: array([False, False, False])
In [107]: data!='d'
Out[107]: array([ True, True, True])
布尔型数组和整数数组混合使用
In [108]: data1=np.array(['a','b','c'])
In [109]: data2=np.array([1,2,3])
In [110]: data_boolean=data1=='a'
In [111]: data_boolean
Out[111]: array([ True, False, False])
In [112]: data2[data_boolean]
Out[112]: array([1])
布尔算术运算符：& |
python中的and和or在numpy中无效
In [108]: data1=np.array(['a','b','c'])
In [109]: data2=np.array([1,2,3])
In [121]: data_boolean=(data1=='a')&(data1=='b')
In [122]: data_boolean
Out[122]: array([False, False, False])
In [123]: data_boolean=(data1=='a')|(data1=='b')
In [124]: data_boolean
Out[124]: array([ True, True, False])
In [125]: data2[data_boolean]
Out[125]: array([1, 2])
花式索引
花式索引(Fancy indexing)是一个Numpy术语，它指利用整数数组进行索引
#子集赋值
In [126]: arr=np.empty((8,4))
In [127]: arr
Out[127]:
array([[1.78873028e-316, 6.92799192e-310, 6.92799191e-310,
6.92799192e-310],
[6.92799191e-310, 6.92799191e-310, 6.92799191e-310,
6.92799191e-310],
[6.92799191e-310, 6.92799191e-310, 6.92799192e-310,
6.92799191e-310],
[6.92799192e-310, 6.92799191e-310, 6.92799191e-310,
6.92799192e-310],
[6.92799191e-310, 6.92799191e-310, 6.92799192e-310,
6.92799192e-310],
[6.92799191e-310, 6.92799191e-310, 5.41734680e-317,
6.92799191e-310],
[6.92799191e-310, 6.92799191e-310, 6.92799191e-310,
6.92799191e-310],
[6.92799192e-310, 6.92799192e-310, 6.92799192e-310,
6.92799191e-310]])
In [128]: for i in range(8):
...: arr[i]=i
...:
In [129]: arr
Out[129]:
array([[0., 0., 0., 0.],
[1., 1., 1., 1.],
[2., 2., 2., 2.],
[3., 3., 3., 3.],
[4., 4., 4., 4.],
[5., 5., 5., 5.],
[6., 6., 6., 6.],
[7., 7., 7., 7.]])
#为了以特定的顺序选取子集，只需要传入一个用于指定顺序的整数列表或ndarray即可
In [130]: arr[[4,3,0,6]]
Out[130]:
array([[4., 4., 4., 4.],
[3., 3., 3., 3.],
[0., 0., 0., 0.],
[6., 6., 6., 6.]])
#一次传入多个索引数组会有一点特别，他返回一个一维数组，其中的元素对应各个索引元组
In [133]: arr[[1,2,3,4],[5,6,7,0]]#第二个列表表示列的排序，不能超过3
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
in ()
----> 1 arr[[1,2,3,4],[5,6,7,0]]
IndexError: index 5 is out of bounds for axis 1 with size 4
In [134]: arr[[1,4,5,7],[0,3,1,2]]#返回的是数组的第一列数据，即默认的索引
Out[134]: array([1., 4., 5., 7.])
In [143]: arr[[1,4,5,7]][:,[0,3,1,2]]#前面的列表表示子集，后面列表表示选取的子集的数据排列
Out[143]:
array([[ 4, 7, 5, 6],
[16, 19, 17, 18],
[20, 23, 21, 22],
[28, 31, 29, 30]])
#也可以使用np.ix_函数，它可以将两个一维整数数组转换为一个用于选取方形区域的索引
In [144]: arr[np.ix_([1,4,5,7],[0,3,1,2])]
Out[144]:
array([[ 4, 7, 5, 6],
[16, 19, 17, 18],
[20, 23, 21, 22],
[28, 31, 29, 30]])
In [22]: arr1=np.array([1,2,3])
In [23]: arr2=np.array([4,5,6])
In [24]: np.add(arr1,arr2)#数组1和数组2相加
Out[24]: array([5, 7, 9])
In [29]: arr1=np.array([1,3,5])
In [30]: arr2=np.array([1,2,6])
In [31]: np.maximum(arr1,arr2)#取数组1和数组2对应元素的最大值，组成新数组
Out[31]: array([1, 3, 6])
In [72]: arr=np.arange(10)
In [73]: np.save('save_arr',arr)
In [75]: np.load('save_arr.npy')
Out[75]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
np.savez
将多个数组保存到一个压缩文件中，将数组以关键字参数的形式传入即可，后缀名为.npz的文件，
加载.npz文件时，你会得到一个类似字典的对象，该对象会对各个数组进行延迟加载：
In [77]: np.savez('savez_arr',aaa=np.array([12,34]),bbb=np.array([3,4,2,2]))
In [79]: np.load('savez_arr.npz')
Out[79]:
In [80]: ar=np.load('savez_arr.npz')
In [81]: ar['aaa']
Out[81]: array([12, 34])
In [136]: import random
In [137]: position=0
In [138]: walk=[position]
In [139]: steps=1000
#纯python方法
In [178]: for i in range(steps):
...: step=1 if random.randint(0,2) else -1
...: position+=step
...: walk.append(position)
#numpy方法
In [180]: nsteps=1000
In [181]: draws=np.random.randint(0,2,size=nsteps)
In [182]: steps=np.where(draws,1,-1)
In [183]: walk=np.cumsum(steps)
In [184]: walk
In [61]: arr=np.random.randn(5,2)
In [62]: arr
Out[62]:
array([[-0.82230236, 0.62723821],
[ 2.18883671, -0.27052682],
[-0.45726504, -1.66183692],
[-0.42266318, 0.84994138],
[-0.30175674, 1.25396072]])
In [63]: first,second,third=np.split(arr,[1,3])
In [64]: first
Out[64]: array([[-0.82230236, 0.62723821]])#半开放区间，边界轴在下一个数组分区
In [65]: second
Out[65]:
array([[ 2.18883671, -0.27052682],
[-0.45726504, -1.66183692]])
In [66]: third
Out[66]:
array([[-0.42266318, 0.84994138],
[-0.30175674, 1.25396072]])
In [199]: dtype=[('x',np.float64),('y',np.int32)]
In [200]: sarr=np.array([(1.5,6),(np.pi,-2)],dtype=dtype)#np.pi 圆周率
In [201]: sarr
Out[201]:
array([(1.5 , 6), (3.14159265, -2)],
dtype=[('x', ' in ()
----> 1 sarr[2]
IndexError: index 2 is out of bounds for axis 0 with size 2
In [205]: sarr[0]['x']
Out[205]: 1.5
In [206]: sarr[0]['y']
Out[206]: 6
In [207]: sarr['y']
Out[207]: array([ 6, -2], dtype=int32)
#字段名保存在dtype.name属性中，在访问结构化数组在的某个字段时，返回的是该数据的视图，所以不会发生数据复制
嵌套dtype和多维字段
在定义结构化dtype时，可以再设置一个形状(可以是个整数，也可以是一个元组)
In [208]: dtype=[('x',np.float64,3),('',np.int32)]
In [209]: arr=np.zeros(4,dtype=dtype)
In [210]: arr
Out[210]:
array([([0., 0., 0.], 0), ([0., 0., 0.], 0), ([0., 0., 0.], 0),
([0., 0., 0.], 0)], dtype=[('x', '
from numpy cimport ndarray,float64_t
def sum_element(ndarray[float64_t] arr):
cdef py_ssize_t i,n=len(arr)
cdef float64_t result=0
for i in range(n):
result+=arr[i]
return result
#Cpython处理这段代码时，先将其翻译为C代码，然后编译这些C代码并创建一个python扩展，一般的工作流程是：得到能在python中运行的算法，然后再将其翻译为Cpython(只需添加类型定义并完成一些其他必要的工作即可--只需你妹！！)
二 pandas模块
１．pandas的数据结构：
Series
Series 是一维标签数组(Data must be 1-dimension)，能够保存任何数据类型（整型，浮点型，字符串或其他Python对象类型）。轴标签被称为索引
>>> import numpy as np
>>> import pandas as pd
>>> pd.Series(np.random.randn(12))#Series函数用于创建Series
0 0.445457
1 1.410470
2 -0.669725
3 -0.907310
4 0.081393
5 0.034115
6 1.219609
7 0.135690
8 -0.353035
9 0.640904
10 0.218566
11 0.836471
dtype: float64
>>> pd.Series(np.random.randn(12).reshape(3,4))#只能传如一维数组，否则会报错
Traceback (most recent call last):
File "", line 1, in
File "/home/zelin/.local/lib/python3.6/site-packages/pandas/core/series.py", line 275, in __init__
raise_cast_failure=True)
File "/home/zelin/.local/lib/python3.6/site-packages/pandas/core/series.py", line 4165, in _sanitize_array
raise Exception('Data must be 1-dimensional')
Exception: Data must be 1-dimensional
In [199]: pd.Series({'a':111,'b':222})
Out[199]:
a 111
b 222
dtype: int64
In [200]:
In [200]: pd.Series({'a':111,'b':222},index=['ccc','ddd'])
Out[200]:
ccc NaN
ddd NaN
dtype: float64
In [201]: ser1=pd.Series({'a':1,'b':2,'c':3})
In [202]: ser2=pd.Series({'c':3,'d':4,'e':5})
In [203]: ser1+ser2
Out[203]:
a NaN
b NaN
c 6.0
d NaN
e NaN
dtype: float64
isnull和notnull
判断Series数据是否为NaN值
In [212]: ser=pd.Series({'a':1,'b':2,'c':3,'d':np.nan})
In [213]: pd.isnull(ser)
Out[213]:
a False
b False
c False
d True
dtype: bool
In [214]: pd.notnull(ser)
Out[214]:
a True
b True
c True
d False
dtype: bool
data_dict={'states':['oooo','ssss','dddd'],'age':[11,12,12]}
In [3]: df=pd.DataFrame(data_dict,index
...: =['aaa','bbb','ccc'])
In [4]: df
Out[4]:
states age
aaa oooo 11
bbb ssss 12
ccc dddd 12
#如果制定类列的序列，则会根据序列排序
In [8]: df=pd.DataFrame(data_dict,index=['aaa','bbb','ccc'],columns=['age','stat
...: es'])
In [9]: df
Out[9]:
age states
aaa 11 oooo
bbb 12 ssss
ccc 12 dddd
#如果传入的列序列在数据中找不到，则会产生NaN值
In [10]: df=pd.DataFrame(data_dict,index=['aaa','bbb','ccc'],columns=['age','sta
...: tes','NAN'])
In [11]: df
Out[11]:
age states NAN
aaa 11 oooo NaN
bbb 12 ssss NaN
ccc 12 dddd NaN
In [10]: df=pd.DataFrame(data_dict,index=['aaa','bbb','ccc'],columns=['age','sta
...: tes','NAN'])
In [11]: df
Out[11]:
age states NAN
aaa 11 oooo NaN
bbb 12 ssss NaN
ccc 12 dddd NaN
In [12]: df['age']
Out[12]:
aaa 11
bbb 12
ccc 12
Name: age, dtype: int64
行也可以通过位置或者名称的方式进行获取，比如用索引字段ix
In [15]: df.ix['aaa']
Out[15]:
age 11
states oooo
NAN NaN
Name: aaa, dtype: object
列可以通过赋值的方式进行修改(整列发生改变)，若赋值的列名不存在，则会新增一列
In [16]: df['age']=100
In [17]: df
Out[17]:
age states NAN
aaa 100 oooo NaN
bbb 100 ssss NaN
ccc 100 dddd NaN
In [18]: df['num']=200
In [19]: df
Out[19]:
age states NAN num
aaa 100 oooo NaN 200
bbb 100 ssss NaN 200
ccc 100 dddd NaN 200
In [19]: df
Out[19]:
age states NAN num
aaa 100 oooo NaN 200
bbb 100 ssss NaN 200
ccc 100 dddd NaN 200
In [20]: sr=df['age']
In [21]: sr['aaa']=3000
/home/zelin/anaconda3/bin/ipython:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
#!/home/zelin/anaconda3/bin/python
In [22]: sr
Out[22]:
aaa 3000
bbb 100
ccc 100
Name: age, dtype: int64
In [23]: df
Out[23]:
age states NAN num
aaa 3000 oooo NaN 200
bbb 100 ssss NaN 200
ccc 100 dddd NaN 200
In [25]: df=pd.DataFrame({'ooo':{1:1,2:2,3:3},'ppp':{11:11,22:22,33:33,44:44}})
In [27]: df
Out[27]:
ooo ppp
1 1.0 NaN
2 2.0 NaN
3 3.0 NaN
11 NaN 11.0
22 NaN 22.0
33 NaN 33.0
44 NaN 44.0
In [32]: df2=pd.DataFrame(df,index=(1,2,3,11,33))
In [33]: df2
Out[33]:
ooo ppp
1 1.0 NaN
2 2.0 NaN
3 3.0 NaN
11 NaN 11.0
33 NaN 33.0
可以输入给DataFrame构造器的数据
类型
说明
二维ndarray
数据矩阵，还可以传入行标和列标
由数组、列表、元组组成的字典
每个序列会变成DataFrame的一列，所有序列的长度必须相同
由Series组成的字典
每个Series会成一列。如果没有显式指定索引，则各Series的索引会被合并成结果的行索引
由字典组成的字典
各内层字典会成为一列，键会被合并成结果的行索引，跟‘由Series组成的字典’的情况一样
字典或Series的列表
各项会成为DataFrame的一行，字典键或Series的索引的并集会成为DataFrame的列标
由列表或元组组成的列表
类似二维ndarray
另一个DataFrame
该DataFrame的索引将会被沿用，除非显式指定了其他索引
numpy的maskedArray
类似二维ndarray的情况，只是掩码值在结果DataFrame会变成NA缺失值
如果设置了index和columns的name属性，则这些信息会被显式出来
In [34]: df=pd.DataFrame({'ooo':{1:1,2:2,3:3},'ppp':{11:11,22:22,33:33,44:44}})
In [35]: df.index.name='year'
In [36]: df.columns.name='state'
In [37]: df
Out[37]:
state ooo ppp
year
1 1.0 NaN
2 2.0 NaN
3 3.0 NaN
11 NaN 11.0
22 NaN 22.0
33 NaN 33.0
44 NaN 44.0
In [48]: df=pd.DataFrame({'a':1,'b':'b','c':3.44},index=['a','b','c'])
In [49]: df.values
Out[49]:
array([[1, 'b', 3.44],
[1, 'b', 3.44],
[1, 'b', 3.44]], dtype=object)
如果字典的键的值只有一个，DateFrame的行数，会根据显式指定的index的长度而变化，值重复
In [39]: df=pd.DataFrame({'a':1,'b':'b','c':3.44},index=['a'])
In [40]: df
Out[40]:
a b c
a 1 b 3.44
In [41]: df=pd.DataFrame({'a':1,'b':'b','c':3.44},index=['a','b'])
In [42]: df
Out[42]:
a b c
a 1 b 3.44
b 1 b 3.44
In [43]: df=pd.DataFrame({'a':1,'b':'b','c':3.44},index=['a','b','c'])
In [44]: df
Out[44]:
a b c
a 1 b 3.44
b 1 b 3.44
c 1 b 3.44
In [50]: df=pd.DataFrame({'a':1,'b':'b','c':3.44},index=['a','b','c'])
In [51]: df
Out[51]:
a b c
a 1 b 3.44
b 1 b 3.44
c 1 b 3.44
In [52]: index=df.index
In [53]: index
Out[53]: Index(['a', 'b', 'c'], dtype='object')
In [54]: index[1]
Out[54]: 'b'
In [55]: index[1]='a'
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
----> 1 index[1]='a'
~/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py in __setitem__(self, key, value)
2063
2064 def __setitem__(self, key, value):
-> 2065 raise TypeError("Index does not support mutable operations")
2066
2067 def __getitem__(self, key):
TypeError: Index does not support mutable operations
In [57]: index=pd.Index(np.arange(3))
In [58]: obj2=pd.Series([111,222,333],index=index)
In [59]: obj2.index is index
Out[59]: True
Index长的像数组，功能也类似一个固定大小的集合
In [58]: obj2=pd.Series([111,222,333],index=index)
In [59]: obj2.index is index
Out[59]: True
In [60]: 1 in obj2.index
Out[60]: True
In [61]: 222 in obj2.index
Out[61]: False
In [62]: obj=pd.Series([1,2,3,4],index=['d','b','a','c'])
In [64]: obj
Out[64]:
d 1
b 2
a 3
c 4
dtype: int64
In [65]: obj.reindex(['a','b','c','d','e'])#重新排序，及某个索引值当前不存在，则引入缺失值
Out[65]:
a 3.0
b 2.0
c 4.0
d 1.0
e NaN
dtype: float64
In [66]: obj.reindex(['a','b','c','d','e','f'],fill_value=1.1)#fill_value替换缺失值
Out[66]:
a 3.0
b 2.0
c 4.0
d 1.0
e 1.1
f 1.1
dtype: float64
In [67]: obj=pd.Series(['blue','yellow','orange'],index=[0,2,4])
In [68]: obj.reindex(np.arange(6))
Out[68]:
0 blue
1 NaN
2 yellow
3 NaN
4 orange
5 NaN
dtype: object
In [69]: obj.reindex(np.arange(6),method='ffill')
Out[69]:
0 blue
1 blue
2 yellow
3 yellow
4 orange
5 orange
dtype: object
In [70]: obj=pd.Series(['blue','yellow','orange'],index=[0,2,4])
In [71]: obj.reindex(np.arange(6),method='bfill')#后面没有数据，则仍然为NaN值
Out[71]:
0 blue
1 yellow
2 yellow
3 orange
4 orange
5 NaN
dtype: object
In [73]: df=pd.DataFrame(np.arange(9).reshape(3,3),index=['a','b','c'],columns=[
...: 'va','vb','vc'])
In [74]: df
Out[74]:
va vb vc
a 0 1 2
b 3 4 5
c 6 7 8
In [75]: df.reindex(['a','b','c','d'])
Out[75]:
va vb vc
a 0.0 1.0 2.0
b 3.0 4.0 5.0
c 6.0 7.0 8.0
d NaN NaN NaN
In [76]: df.reindex(columns=['vvva','vvvb','vvvc'])
Out[76]:
vvva vvvb vvvc
a NaN NaN NaN
b NaN NaN NaN
c NaN NaN NaN
利用ix的标签索引功能，重新索引更简洁
In [77]: df=pd.DataFrame(np.arange(9).reshape(3,3),index=['a','b','c'],columns=[
...: 'va','vb','vc'])
In [78]: df
Out[78]:
va vb vc
a 0 1 2
b 3 4 5
c 6 7 8
In [79]: df.ix[['a','b','c','d'],['vva','vvb','vc']]
Out[79]:
vva vvb vc
a NaN NaN 2.0
b NaN NaN 5.0
c NaN NaN 8.0
d NaN NaN NaN
In [3]: ser=pd.Series(np.arange(5.),index=['a','b','c','d','e'])
In [4]: ser
Out[4]:
a 0.0
b 1.0
c 2.0
d 3.0
e 4.0
dtype: float64
In [5]: ser.drop('c')#获取到的是新对象，而不是删除源数据
Out[5]:
a 0.0
b 1.0
d 3.0
e 4.0
dtype: float64
In [6]: ser
Out[6]:
a 0.0
b 1.0
c 2.0
d 3.0
e 4.0
dtype: float64
In [12]: obj=pd.Series(np.arange(4.),index=('a','b','c','d'))
In [13]: obj
Out[13]:
a 0.0
b 1.0
c 2.0
d 3.0
dtype: float64
In [14]: obj['a':'c']
Out[14]:
a 0.0
b 1.0
c 2.0
dtype: float64
索引方式几种特殊的情况，通过切片或布尔型数组选取行
In [15]: obj=pd.Series(np.arange(4.),index=('a','b','c','d'))
In [16]: obj>2
Out[16]:
a False
b False
c False
d True
dtype: bool
In [18]: obj[obj>2]
Out[18]:
d 3.0
dtype: float64
索引字段ix
可以通过Numpy式的标记法以及轴标签从DataFrame中选取行和列的子集
In [31]: df=pd.DataFrame(np.arange(16).reshape(4,4),index=['o','p','q','r'],colu
...: mns=['one','two','three','four'])
In [37]: df
Out[37]:
one two three four
o 0 1 2 3
p 4 5 6 7
q 8 9 10 11
r 12 13 14 15
In [32]: df.ix['o',['one','two']]#[行索引，列标]
Out[32]:
one 0
two 1
Name: o, dtype: int64
In [34]: df.ix[['o','p'],['one','two']]#位置参数，无需指定
Out[34]:
one two
o 0 1
p 4 5
In [36]: df.ix[df.one>3,:3]#one列大于3的所有数据
Out[36]:
one two three
p 4 5 6
q 8 9 10
r 12 13 14
In [39]: s1=pd.Series([1,2,3,4],index=['a','c','d','e'])
In [42]: s2=pd.Series([3,3,3,3,3],index=['a','c','e','f','g'])
In [43]: s1+s2
Out[43]:
a 4.0
c 5.0
d NaN
e 7.0
f NaN
g NaN
dtype: float64
In [45]: df1=pd.DataFrame(np.arange(9).reshape(3,3),columns=list('bcd'),index=li
...: st('xyz'))
In [48]: df2=pd.DataFrame(np.arange(12.).reshape(4,3),columns=list('dbe'),index=
...: list('xymn'))
In [49]: df1+df2
Out[49]:
b c d e
m NaN NaN NaN NaN
n NaN NaN NaN NaN
x 1.0 NaN 2.0 NaN
y 7.0 NaN 8.0 NaN
z NaN NaN NaN NaN
In [11]: import numpy as np
In [12]: import pandas as pd
In [13]: df1=pd.DataFrame(np.arange(12).reshape((3,4)),columns=list('abcd'))
In [14]: df2=pd.DataFrame(np.arange(20).reshape((4,5)),columns=list('abcde'))
In [15]: df1+df2
Out[15]:
a b c d e
0 0.0 2.0 4.0 6.0 NaN
1 9.0 11.0 13.0 15.0 NaN
2 18.0 20.0 22.0 24.0 NaN
3 NaN NaN NaN NaN NaN
In [16]: df1.add(df2)
Out[16]:
a b c d e
0 0.0 2.0 4.0 6.0 NaN
1 9.0 11.0 13.0 15.0 NaN
2 18.0 20.0 22.0 24.0 NaN
3 NaN NaN NaN NaN NaN
In [17]: df1.add(df2,fill_value=0)#并非在结果中填充，而是将不重叠部分填充为0值
Out[17]:
a b c d e
0 0.0 2.0 4.0 6.0 4.0
1 9.0 11.0 13.0 15.0 9.0
2 18.0 20.0 22.0 24.0 14.0
3 15.0 16.0 17.0 18.0 19.0
In [20]: df1.reindex(list('abcde'),fill_value=0)
Out[20]:
a b c d
a 0 0 0 0
b 0 0 0 0
c 0 0 0 0
d 0 0 0 0
e 0 0 0 0
In [21]: df1.reindex(columns=list('abcde'),fill_value=0)#重新索引产生的是新数据，不会影响源数据
Out[21]:
a b c d e
0 0 1 2 3 0
1 4 5 6 7 0
2 8 9 10 11 0
In [25]: df=pd.DataFrame(np.arange(12).reshape((4,3)),index=list('abcd'),columns
...: =list('opq'))
In [26]: df
Out[26]:
o p q
a 0 1 2
b 3 4 5
c 6 7 8
d 9 10 11
In [27]: ser=df.ix['a']#只能以行索引为参数，不然会报错
In [28]: ser
Out[28]:
o 0
p 1
q 2
Name: a, dtype: int64
In [30]: df.add(ser)
Out[30]:
o p q
a 0 2 4
b 3 5 7
c 6 8 10
d 9 11 13
In [31]: df+ser
Out[31]:
o p q
a 0 2 4
b 3 5 7
c 6 8 10
d 9 11 13
如果你希望匹配到行且在列上广播，则必须使用算术运算方法
In [33]: df.add(ser,axis=0)#匹配行，即将Series的列标成为索引与DataFrame 并集
Out[33]:
o p q
a NaN NaN NaN
b NaN NaN NaN
c NaN NaN NaN
d NaN NaN NaN
o NaN NaN NaN
p NaN NaN NaN
q NaN NaN NaN
In [10]: df=pd.DataFrame(np.arange(12).reshape((4,3)),columns=list('xyz'),index=
...: list('adcd'))
In [11]: f=lambda x: x.max()-x.min()
In [12]: df.apply(f)#使用作为参数传入的函数功能
Out[12]:
x 9
y 9
z 9
dtype: int64
In [13]: df.apply(f,axis=1)
Out[13]:
a 2
d 2
c 2
d 2
dtype: int64
applymap 函数应用
元素级的python函数也是可以用的，使用applymap即可
In [15]: df=pd.DataFrame(np.random.randn(4,3),columns=list('xyz'),index=list('ab
...: cd'))
In [16]: f=lambda x:'%.2f'%x
In [17]: df.applymap(f)
Out[17]:
x y z
a 0.33 1.56 -1.21
b 1.18 2.26 -2.17
c 0.59 -0.59 -0.26
d -0.43 -0.41 -1.09
map，Series的应用函数
之所以用applymap，因为Series也有一个应用函数map方法
In [19]: df['x'].map(f)
Out[19]:
a 0.33
b 1.18
c 0.59
d -0.43
Name: x, dtype: object
In [20]: df=pd.DataFrame(np.random.randn(4,3),columns=list('xyz'),index=list('cd
...: ab'))
In [21]: df
Out[21]:
x y z
c 0.574269 1.004828 -1.653797
d -1.153132 -2.733118 0.119374
a -1.616412 -1.232352 -0.053378
b 0.882328 -0.613165 2.238244
In [22]: df.sort_index()
Out[22]:
x y z
a -1.616412 -1.232352 -0.053378
b 0.882328 -0.613165 2.238244
c 0.574269 1.004828 -1.653797
d -1.153132 -2.733118 0.119374
In [28]: df
Out[28]:
x y z
c 0.574269 1.004828 -1.653797
d -1.153132 -2.733118 0.119374
a -1.616412 -1.232352 -0.053378
b 0.882328 -0.613165 2.238244
In [29]: df.sort_index(by='x')
/home/zelin/anaconda3/bin/ipython:1: FutureWarning: by argument to sort_index is deprecated, please use .sort_values(by=...)
#!/home/zelin/anaconda3/bin/python
Out[29]:
x y z
a -1.616412 -1.232352 -0.053378
d -1.153132 -2.733118 0.119374
c 0.574269 1.004828 -1.653797
b 0.882328 -0.613165 2.238244
In [30]: df.sort_index(by=['x','z'])
/home/zelin/anaconda3/bin/ipython:1: FutureWarning: by argument to sort_index is deprecated, please use .sort_values(by=...)
#!/home/zelin/anaconda3/bin/python
Out[30]:
x y z
a -1.616412 -1.232352 -0.053378
d -1.153132 -2.733118 0.119374
c 0.574269 1.004828 -1.653797
b 0.882328 -0.613165 2.238244
In [37]: df.rank()
Out[37]:
a b
0 3.0 1.0
1 2.0 3.0
2 1.0 2.0
3 4.0 4.0
In [38]: df.rank(axis=1)
Out[38]:
a b
0 1.0 2.0
1 1.0 2.0
2 1.0 2.0
3 1.0 2.0
rank的破坏平级method选项
method
说明
average
默认：在相等分组中，为各个值分配平均排名
min
使用整个分组的最小排名
max
使用整个分组的最大排名
first
按值在原始数据种的出现顺序分配排名
带由重复值的轴索引
某个索引值对应多个值，则返回一个Series，而对应单个值的，则返回一个标量值
In [39]: ser=pd.Series([7,-5,7,4,2,0,4],index=['a','b','a','b','c','a','d'])
In [40]: ser['a']
Out[40]:
a 7
a 7
a 0
dtype: int64
In [42]: ser['d']
Out[42]: 4
In [44]: df=pd.DataFrame([[1.4,np.nan],[7.1,-4.5],[np.nan,np.nan],[0.75,-1.3]],i
...: ndex=list('abcd'),columns=['one','two'])
In [45]: df
Out[45]:
one two
a 1.40 NaN
b 7.10 -4.5
c NaN NaN
d 0.75 -1.3
In [46]: df.sum()
Out[46]:
one 9.25
two -5.80
dtype: float64
In [47]: df.sum(axis=1)#默认等于0列相加，axis=1行相加
Out[47]:
a 1.40
b 2.60
c 0.00
d -0.55
dtype: float64
In [48]: df.sum(axis=1,skipna=False)#默认排除NaN值，所以全NaN值则为0
Out[48]:
a NaN
b 2.60
c NaN
d -0.55
dtype: float64
约简方法的选项和方法
axis
DataFrame的行用0,列用1
skipna
排除缺失值，默认值为True
level
如果轴是层次化索引的(即MultilIndex)，则根据level分组约简
idxmin、idxmax达到最小值和最大值的索引，间接统计
In [51]: df.idxmax()
Out[51]:
one b
two d
dtype: object
另一些方法是累计型的：cumsum
In [53]: df
Out[53]:
one two
a 1.40 NaN
b 7.10 -4.5
c NaN NaN
d 0.75 -1.3
In [52]: df.cumsum()
Out[52]:
one two
a 1.40 NaN
b 8.50 -4.5
c NaN NaN
d 9.25 -5.8
一次性汇总多个统计信息：describe(),数值型和非数值型产生不同汇总信息
In [53]: df1#数值型
Out[53]:
one two
a 1.40 NaN
b 7.10 -4.5
c NaN NaN
d 0.75 -1.3
In [54]: df.describe()
Out[54]:
one two
count 3.000000 2.000000
mean 3.083333 -2.900000
std 3.493685 2.262742
min 0.750000 -4.500000
25% 1.075000 -3.700000
50% 1.400000 -2.900000
75% 4.250000 -2.100000
max 7.100000 -1.300000
In [56]: df2#非数值型
Out[56]:
a b
1 aa aa
2 bb bbb
3 cc ddd
4 dd ccc
In [57]: df.describe()
Out[57]:
a b
count 4 4
unique 4 4
top bb ccc
freq 1 1
描述和汇总统计
count
非NA值的数量
describe
针对Series或各DataFrame列计算汇总统计
min、max
计算最小值和最大值
argmin、argmax
计算能够获取到最小值和最大值的索引位置(整数)
idxmin、inxmax
计算能够获取到最小值和最大值的索引值
quantile
计算样本的分位数(0到1)
sum
值的总和
mean
值的平均数
median
值的中位数
mad
根据平均值计算平均绝对离差
var
样本值的方差
std
样本值的标准差
skew
样本值的偏度(三阶矩)
kurt
样本值的峰度(四阶矩)
cumsum
样本值的累计和
cummin、cummax
样本值的leis最小值和累计最大值
cumprod
样本值的累计积
diff
计算一阶差分(多用于时间序列)
pct_change
计算百分数变化
分组运算：ＧroupBy技术
拆分－－应用－－合并
In [1]: import pandas as pd
In [2]: import numpy as np
＃构建ＤataFrame
In [3]: df=pd.DataFrame({'key1':['a','a','b','b','a'],'key2':['one','two','one',
...: 'two','one'],'data1':np.random.randn(5),'data2':np.random.randn(5)})
In [4]: df
Out[4]:
key1 key2 data1 data2
0 a one -0.769863 0.321557
1 a two 0.163815 -0.635989
2 b one -1.048893 1.988060
3 b two -0.427548 0.322831
4 a one -1.115058 0.663252
#以键key1分组，获取data1的GroupBy对象
In [5]: grouped=df['data1'].groupby(df['key1'])
In [6]: grouped
Out[6]:
#GroupBy对象可以调用mean,sum等方法求平均值，和等值
In [7]: grouped.mean()＃返回一个Ｓeries
Out[7]:
key1
a -0.573702
b -0.738221
Name: data1, dtype: float64
在数据库系统中存储过程是必不可少的利器，存储过程是预先编译好的为实现一个复杂功能的一段Sql语句集合。它的优点我就不多说了，说一下我碰到的问题吧。我在项目开发的过程中需要用存储过程来实现一个功能，其中涉及到判断一张表是否已经建立，没有建立就由存储过程来建立这张表。
CREATE OR REPLACE PROCEDURE TestProc
IS
fla
使用jsonp不能发起POST请求。
It is not possible to make a JSONP POST request.
JSONP works by creating a <script> tag that executes Javascript from a different domain; it is not pos
（一）Keepalived
（1）安装
# cd /usr/local/src
# wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gz
# tar zxvf keepalived-1.2.15.tar.gz
# cd keepalived-1.2.15
# ./configure
# make &a
