用Cython编译写出更快的Python代码
原文地址： http://www.behnel.de/cython200910/talk.html以下为原文
About myself
Passionate Python developer since 2002
after Basic, Logo, Pascal, Prolog, Scheme, Java, C, ...
CS studies in Germany, Ireland, France
PhD in distributed systems in 2007
Language design for self-organising systems
Darmstadt University of Technologies, Germany
Current occupations:
http://codespeak.net/lxml/
IT transformations, SOA design, Java-Development, ...
Employed by Senacor Technologies AG, Germany
»lxml« OpenSource XML toolkit for Python
»Cython«
Part 1: Intro to Cython
Part 1: Intro to Cython
Part 2: Building Cython modules
Part 3: Writing fast code
Part 4: Talking to other extensions
What is Cython?
Cython is the missing link
between the simplicity of Python
and the speed of C / C++ / Fortran.
What is Cython?
Cython is the missing link
between the simplicity of Python
and the speed of C / C++ / Fortran.
What is Cython?
Cython is
an Open-Source project
http://cython.org
http://pypi.python.org/pypi/Cython
a Python compiler (almost)
an enhanced, optimising fork of Pyrex
an extended Python language for
writing fast Python extension modules
interfacing Python with C libraries
Major Cython Core Developers
Robert Bradshaw, Stefan Behnel, Dag Sverre Seljebotn
lead developers
Lisandro Dalcín
C/C++ portability and various feature patches
Kurt Smith, Danilo Freitas
Google Summer of Code 2009: Fortran/C++ integration
Greg Ewing
main developer and maintainer of Pyrex
many, many others - see
http://cython.org/
the mailing list archives of Cython and Pyrex
How to use Cython
you write Python code
Cython translates it into C code
your C compiler builds a shared library for CPython
you import your module into CPython
Cython has support for
optionally compile Python code from setup.py!
Cython does that for its own modules :-)
distutils
embedding the CPython runtime in an executable
Example: compiling Python code
# file: worker.pyclass HardWorker(object): u"Almost Sisyphos" def __init__(self, task): self.task = task def work_hard(self, repeat=100): for i in range(repeat): self.task()def add_simple_stuff(): x = 1+1HardWorker(add_simple_stuff).work_hard()
Example: compiling Python code
compile with
$ cython worker.py
translates to ~1500 line .c file (Cython 0.11.3)
helps tracing your own code in generated sources
different C compilers, Python versions, ...
lots of portability #define's
tons of helpful C comments with Python code snippets
a lot of code that you don't want to write yourself
Portable Code
Cython compiler generates C code that compiles
with all major compilers (C and C++)
on all major platforms
in Python 2.3 through 3.1
Cython language syntax follows Python 2.6
get involved to get it quicker!
optional Python 3 syntax support is on TODO list
... the fastest way to port Python 2 code to Py3 ;-)
Python language feature support
most of Python 2 syntax is supported
top-level classes and functions
control structures: loops, with, try-except/finally, ...
object operations, arithmetic, ...
plus many Py3 features:
list/set/dict comprehensions
keyword-only arguments
extended iterable unpacking (a,b,*c,d = some_list)
Python features in work
Inner functions with closures
def factory(a,b): def closure_function(c): return a+b+c return closure_function
status: (hopefully) to be merged for 0.12
Planned Cython features
improved C++ integration (GSoC 2009)
e.g. function/operator overloading support
status: mostly there, to be finished and integrated
improved Fortran integration (GSoC 2009)
talking to Fortan code directly
status: mostly there, to be finished and integrated
native array data type with SIMD behaviour
status: large interest, implementation pending
... as usual: great ideas, little time
Currently unsupported
local/inner classes (~open)
lambda expressions (~easy)
generators (~needs work)
generator expressions (~easy)
with obvious optimisations, e.g.
set( x.a for x in some_list )== { x.a for x in some_list }
... all certainly on the TODO list for 1.0.
Speed
Cython generates very efficient C code:
PyBench: most benchmarks run 20-80% faster
conditions and loops run 5-8x faster than in Py2.6.2
overall about 30% faster for plain Python benchmark
obviously, real applications are different
PyPy's richards.py benchmark:
heavily class based scheduler
20% faster than CPython 2.6.2
Type declarations
Cython supports optional type declarations that
can be employed exactly where performance matters
let Cython generate plain C instead of C-API calls
make richards.py benchmark 5x faster than CPython
without Python code modifications :)
can make code 100 - 1000x faster than CPython
expect several 100 times in calculation loops
Part 2: Building Cython modules
Part 1: Intro to Cython
Part 2: Building Cython modules
Part 3: Writing fast code
Part 4: Talking to other extensions
Ways to build Cython code
To compile Python code (.py) or Cython code (.pyx)
you need:
Cython, Python and a C compiler
you can use:
web app that supports writing and running Cython code
on-the-fly build + import (for experiments)
setup.py script (likely required anyway)
distutils
pyximport
Sage notebook
cython source.pyx + manual C compilation
Example: distutils
A minimal setup.py script:
from distutils.core import setupfrom distutils.extension import Extensionfrom Cython.Distutils import build_ext ext_modules = [Extension("worker", ["worker.py"])] setup( name = 'stupid little app', cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules )Run with
$ python setup.py build_ext --inplace
Example: pyximport
Build and import Cython code files (.pyx) on the fly
$ ls worker.pyx$ PYTHONPATH=. python
Python 2.6.2 (r262:71600, Apr 17 2009, 11:29:30)[GCC 4.3.2] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import pyximport>>> pyximport.install()>>> import worker>>> worker<module 'worker' from '~/.pyxbld/.../worker.so'>>>> worker.HardWorker<class 'worker.HardWorker'>>>> worker.HardWorker(worker.add_simple_stuff).work_hard()
pyximporting Python modules
pyximport can also compile Python modules:
>>> import pyximport>>> pyximport.install(pyimport = True)>>> import shlex[lots of compiler errors from different modules ...]>>> help(shlex)
currently works for a few stdlib modules
falls back to normal Python import automatically
not production ready, but nice for testing :)
Writing executable programs
# file: hw.pydef hello_world(): import sys print "Welcome to Python %d.%d!" % sys.version_info[:2]if __name__ == '__main__': hello_world()
Writing executable programs
# file: hw.pydef hello_world(): import sys print "Welcome to Python %d.%d!" % sys.version_info[:2]if __name__ == '__main__': hello_world()
Compile, link and run:
$ cython --embed hw.py # <- embed a main() function$ gcc $CFLAGS -I/usr/include/python2.6 \ -o hw hw.c -lpython2.6 -lpthread -lm -lutil -ldl$ ./hw Welcome to Python 2.6!
Part 3: Writing fast code
Part 1: Intro to Cython
Part 2: Building Cython modules
Part 3: Writing fast code
Part 4: Talking to other extensions
A simple example
Plain Python code:
# integrate_py.pyfrom math import sindef f(x): return sin(x**2)def integrate_f(a, b, N): dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx
Type declarations in Cython
Function arguments are easy
Python:
def f(x): return sin(x**2)
Cython:
def f(double x): return sin(x**2)
Type declarations in Cython
»cdef« keyword declares
variables with C or builtin types
cdef double dx, s
functions with C signatures
cdef double f(double x): return sin(x**2)
classes as 'builtin' extension types
cdef class MyType: cdef int field
Functions: def vs. cdef vs. cpdef
def func(int x):
part of the Python module API
Python call semantics
cdef int func(int x):
C signature
C call semantics
cpdef int func(int x):
Python wrapper around cdef function
C calls cdef function, Python calls wrapper
note: modified C signature!
Typed arguments and return values
def func(int x):
caller passes Python objects for x
function converts to int on entry
implicit return type always object
cdef int func(int x):
caller converts arguments as required
function receives C int for x
arbitrary return type, defaults to object
cpdef int func(int x):
wrapper converts
C callers convert arguments as required
Python callers pass and receive objects
A simple example: Python
# integrate_py.pyfrom math import sindef f(x): return sin(x**2)def integrate_f(a, b, N): dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx
A simple example: Cython
# integrate_cy.pyxcdef extern from "math.h": double sin(double x)cdef double f(double x): return sin(x**2)cpdef double integrate_f(double a, double b, int N): cdef double dx, s cdef int i dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx
Overriding declarations in .pxd
Plain Python code:
# integrate_py.pyfrom math import sindef f(x): return sin(x**2)def integrate_f(a, b, N): dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx
Overriding declarations in .pxd
| Python integrate_py.py | Cython integrate_py.pxd |
# integrate_py.pyfrom math import sindef f(x): return sin(x**2)def integrate_f(a, b, N): dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx |
# integrate_py.pxdcimport cythoncpdef double f(double x)@cython.locals( dx=double, s=double, i=int)cpdef integrate_f( double a, double b, int N) |
The .pxd file used
# integrate_py.pxdcimport cythoncpdef double f(double x): return sin(x**2)cpdef double integrate_f(double a, double b, int N)
Overriding declarations in .pxd
advantage:
Eclipse, pylint, 2to3, ...
runs unchanged in Python interpreter
plain Python code
complete Python tool-chain available
drawback:
cannot override from math import sin
no access to C functions
Typing in Python syntax
Plain Python code:
# integrate_py.pyfrom math import sindef f(x): return sin(x**2)def integrate_f(a, b, N): dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx
Typing in Python syntax
http://wiki.cython.org/pure
from math import sinimport [email protected](x=cython.double)def f(x): return sin(x**2)@cython.locals(a=cython.double, b=cython.double, N=cython.Py_ssize_t, dx=cython.double, s=cython.double, i=cython.Py_ssize_t)def integrate_f(a, b, N): dx = (b-a)/N s = 0 for i in range(N): s += f(a+i*dx) return s * dx
Declaring Python types
Access to Python's builtins is heavily optimised
for ... in range()/list/tuple/dict
list.append(), list.reverse()
set([...]), tuple([...])
Further improvements in Cython 0.12
replacements for enumerate(), type()
dict([...]), unicode.encode(), list.sort()
Declaring Python types is often worth it!
Easy to add new optimisations
don't write prematurely optimised code, fix Cython!
Declaring Python types: dict
example: dict iteration
def filter_a(d): return { key : value for key, value in d.iteritems() if 'a' not in value }import stringd = { s:s for s in string.ascii_letters }print filter_a(d)
Declaring Python types: dict
simple change, ~30% faster:
def filter_a(dict d): # <==== return { key : value for key, value in d.iteritems() if 'a' not in value }import stringd = { s:s for s in string.ascii_letters }print filter_a(d)
Declaring Python types: dict
simple change, ~30% faster:
def filter_a(dict d): # <==== return { key : value for key, value in d.iteritems() if 'a' not in value }import stringd = { s:s for s in string.ascii_letters }print filter_a(d)
drawback:
non-dict mapping arguments raise a TypeError
Think twice before you type
benchmark code before adding static types!
Classes
class MyClass(object):
Python class with __dict__
multiple inheritance
arbitrary Python attributes
Python methods
monkey-patcheable etc.
cdef class MyClass(SomeSuperClass):
C-only access by default, or readonly/public
only from other extension types!
"builtin" extension type
single inheritance
fixed, typed fields
Python + C methods
cdef classes - when to use them?
Use cdef classes
e.g. whenever wrapping C structs/pointers/etc.
when C attribute types are used
when the need for speed beats Python's generality
Use Python classes
for bytes/tuple subtypes (PyVarObject)
for exceptions if Py<2.5 compatibility is required
when multiple inheritance is required
when users are allowed to monkey-patch
Part 4: Talking to other extensions
Part 1: Intro to Cython
Part 2: Building Cython modules
Part 3: Writing fast code
Part 4: Talking to other extensions
Talking to other extensions
Python 3 buffer protocol (available in Py2.6)
external C-APIs
Python 3 buffer protocol
Native support for new Python buffer protocol
PEP 3118
def inplace_invert_2D_buffer( object[unsigned char, 2] image): cdef int i, j for i in range(image.shape[0]): for j in range(image.shape[1]): image[i, j] = 255 - image[i, j]
can be supported for extension types in Py2.x
declared through .pxd files
Cython ships with numpy.pxd
array.pxd available (stdlib's array)
Conclusion
Cython is a tool for
translating Python code to efficient C
easily interfacing to external C/C++/Fortran code
Use it to
concentrate on the mapping, not the glue!
don't change the language just to get fast code!
concentrate on optimisations, not rewrites!
speed up existing Python modules
write C extensions for CPython
wrap C libraries in Python
... but Cython is also
a great project
a very open playground for great ideas!
Cython
Cython
C-Extensions in Python
... use it, and join the project!
http://cython.org/