python程序性能分析
这篇博客简单介绍一些python性能分析的常用工具， 性能分析主要是代码运行的时间和内存分析，希望能给大家提供帮助
通过time模块
import time
def test(num_iterations):
a = 1
for i in range(num_iterations):
a *= -1
return a
num_iterations = 1_000_000_0
t1 = time.time()
res = test(num_iterations)
t2 = time.time()
print(t2 - t1)
这可能是最常用的方法， 我们可以用这个方法测试某一行代码或者某一个函数的运行时间
定义一个装饰器
def time_wrapper(func):
def measure_time(*args, **kwargs):
t1 = time.time()
result = func(*args, **kwargs)
t2 = time.time()
print(f"{func.__name__} took {t2 - t1} seconds")
return result
return measure_time
@time_wrapper
def test(num_iterations):
a = 1
for i in range(num_iterations):
a *= -1
return a
res = test(num_iterations)
# test took 0.4167180061340332 seconds
通过装饰器time_wrapper， 我们可以在任意的函数上加上@time_wrapper，非常方便
用timeit模块测量执行速度
在juputer中，我们甚至还可以使用%timeit来测量速度
# 循环次数（-n 2）和重复次数（-r 2）
# timeit会对语句循环执行n次并计算平均值作为一个结果，重复r次选择最好的那个
%timeit -n 2 -r 2 test(num_iterations)
"""
test took 0.4153749942779541 seconds
test took 0.4038848876953125 seconds
test took 0.40583109855651855 seconds
test took 0.40905189514160156 seconds
409 ms ± 1.09 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)
"""
当然也可以在terminal中进行测试
$python -m timeit -n 2 -r 2 -s "import numpy as np" "np.array(10)"
# 2 loops, best of 2: 2.52 usec per loop
使用unix time命令进行简单的计时
%usr/bin/time -p python test.py
- real记录了整体的耗时
- user记录了CPU花在任务上的时间，不包括内核函数耗费的时间
- sys记录了内核函数耗费的时间
使用cProfile模块
cProfile是一个标准的内建分析工具，它钩入CPython虚拟机来测量其每一个函数运行所花费的时间，这会引入巨大开销，但会获得更多的信息
def test1(value, num_iterations):
res = 1
for i in range(num_iterations):
res += res * 2 % 10
return value
def test2(num_iterations):
res = 0
for i in range(num_iterations):
res += i*i % 10
test1(1, num_iterations)
return res
if __name__ == '__main__':
#-s cumulative对每个函数累计花费时间进行排序
"""python -m cProfile -s cumulative test_cprofiler.py"""
# 生成一个统计文件
"""python -m cProfile -o profile.stats test_cprofiler.py"""
print('hello world')
num_iterations = 1_000_000_0
res1 = test1(1, num_iterations)
res2 = test2(num_iterations)
"""
Sun May 10 19:30:19 2020 profile.stats
5 function calls in 2.221 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 2.221 2.221 {built-in method builtins.exec}
1 0.000 0.000 2.221 2.221 test_cprofiler.py:1()
1 1.303 1.303 1.303 1.303 test_cprofiler.py:8(test2)
1 0.918 0.918 0.918 0.918 test_cprofiler.py:1(test1)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
"""
- for the total time spent in the given function (and excluding time made in calls to sub-functions)
- cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions.
用line_profiler进行逐行分析
line_profiler是调查CPU密集型问题的最强大工具， 它可以对函数进行逐行的分析。可以先用cProfiler找到需要分析的函数，然后用line_profiler对函数进行分析
def test1(value, num_iterations):
res = 1
for i in range(num_iterations):
res += res * 2 % 10
return value
@profile
def test2(num_iterations):
res = 0
for i in range(num_iterations):
res += i*i % 10
return res
if __name__ == '__main__':
# 用装饰器@profile标记选中的函数，用kernprof.py来运行代码
# -l代表逐行分析，-v用于显示输出
"""kernprof -l -v test_lineprofiler.py"""
num_iterations = 1_000_000_0
res1 = test1(1, num_iterations)
res2 = test2(num_iterations)
"""
Wrote profile results to test_lineprofiler.py.lprof
Timer unit: 1e-06 s
Total time: 6.1838 s
File: test_lineprofiler.py
Function: test2 at line 7
Line # Hits Time Per Hit % Time Line Contents
==============================================================
7 @profile
8 def test2(num_iterations):
9 1 231.0 231.0 0.0 res = 0
10 10000001 2488600.0 0.2 40.2 for i in range(num_iterations):
11 10000000 3694972.0 0.4 59.8 res += i*i % 10
12 1 1.0 1.0 0.0 return res
"""
- Timer unit: 1e-06 s：时间单位；
- Total time: 0.004891 s：总时间；
- Hit：代码运行次数；
- %Time：代码占了它所在函数的消耗的时间百分比，通常直接看这一列。
用memory_profiler诊断内存用量
memory_profiler的操作和line_profiler十分类似， 但是运行速度要慢的多. 内存分析可以轻易让你的代码慢上10倍到100倍。所以实际操作可能只是偶尔使用memory_profiler而更多的使用line_profiler来进行CPU分析
import numpy as np
from memory_profiler import profile
@profile
def test2(num_iterations):
res = 0
for i in range(num_iterations):
res += i*i % 10
b = np.random.rand(1000, 1000)
a = [0] * 1000000
return res, b, a
if __name__ == '__main__':
# 在函数前添加 @profile
"""python -m memory_profiler test_memoryprofiler.py"""
num_iterations = 1_000
res2 = test2(num_iterations)
"""
Line # Mem usage Increment Line Contents
================================================
5 55.6 MiB 55.6 MiB @profile
6 def test2(num_iterations):
7 55.6 MiB 0.0 MiB res = 0
8 55.6 MiB 0.0 MiB for i in range(num_iterations):
9 55.6 MiB 0.0 MiB res += i*i % 10
10 63.2 MiB 7.7 MiB b = np.random.rand(1000, 1000)
11 70.9 MiB 7.6 MiB a = [0] * 1000000
12 70.9 MiB 0.0 MiB return res, b, a
"""
- Mem usage : 运行到当前位置使用内存
- Increment : 运行当前代码后，增加的内存