python3 评估程序性能，复杂度，timeit， profile/cprofile

1. 评估复杂度

手动估算代码行数和计算复杂度。
迭代次数 * 循环中代码行数 * 一行的复杂度 O（n）
复杂度一般看级数, O(n), O(n^2)

2. 用ipython的timeit 算出平均运行时间

#uses ipynb

import time
strings = ['foo','foobar','bax','ssss','python'] * 10000

%timeit [x for x in strings if x[:3] == 'foo']
%timeit [x for x in strings if x.startswith('foo')]

#output 

5.79 ms ± 316 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
9.7 ms ± 520 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

3. python3的Profile分析性能

1. 分析某个函数性能%prun add_and_sum(x,y)

%prun(cProfile)做宏观的性能分析，lprun做微观每一行的分析。

#uses ipython

In [1]: from numpy.random import randn
   ...:
   ...: def add_and_sum(x,y):
   ...:     added = x + y
   ...:     summed = added.sum(axis=1)
   ...:     return summed
   ...:
   ...: def call_fun():
   ...:     x = randn(1000,1000)
   ...:     y = randn(1000,1000)
   ...:     return add_and_sum(x,y)
   ...:

In [14]: %prun add_and_sum(x,y)
         7 function calls in 0.075 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.052    0.052    0.067    0.067 :3(a
dd_and_sum)
        1    0.015    0.015    0.015    0.015 {method 'reduce' of 'numpy.ufunc'
objects}
        1    0.008    0.008    0.075    0.075 :1()
        1    0.000    0.000    0.075    0.075 {built-in method builtins.exec}
        1    0.000    0.000    0.015    0.015 {method 'sum' of 'numpy.ndarray' o
bjects}
        1    0.000    0.000    0.015    0.015 _methods.py:31(_sum)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Prof
iler' objects}

2. 常用语不想看那个模块的源码，简单测试/比较性能

# IN[1]
import cProfile
import re
cProfile.run('re.compile("foo|bar")')

#OUT[1]
 5 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 :1()
        1    0.000    0.000    0.000    0.000 re.py:231(compile)
        1    0.000    0.000    0.000    0.000 re.py:286(_compile)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

#test code Python3.6 
import numpy as np
from numpy.linalg import eigvals

def run_experiment(niter = 100):
    K =100
    results = []
    for _ in xrange(niter):
        mat = np.random.rand(K,K)
        max_eigenvalue = np.abs(eigvals(mat)).max()
        return results
    some_results = run_experiment()
    print('largest one %s'% np.max(some_results))

4. 建议的代码格式

扁平结构比嵌套结构好。编写函数和类时注意低耦合和结构化，容易测试和调用。

---

参考

1. 《利用python进行数据分析》
2. 官网 27.4. The Python Profilers
3. https://pymotw.com/3/profile/

2018.5.18 目前搞不出来的，一直到2018.7.26也并没有想法的问题...............以后想到再回来。

In [19]: %run -p -s cumulative n2.py
ERROR:root:Line magic function `%lprun` not found.

写作时间
2018.5.18 草稿
2018.7.26 V1.0