python multiprocessing.Pool - PicklingError: Can't pickle : attribute lookup __builtin__.instancemethod failed

示例1：

import multiprocessing



def run(msg):

    print '%s, it works!' % msg



def start_process():

    print 'Starting',multiprocessing.current_process().name



if __name__=='__main__':

    pool = multiprocessing.Pool(processes=2, initializer=start_process,)

    pool.apply_async(run('congbo'))

    pool.close()

    pool.join()

运行结果：

Starting PoolWorker-1
Starting PoolWorker-2
congbo, it works!

pool.apply_async(run('congbo')) 这种使用方式也行？见示例2。

 

示例2：

import multiprocessing



def run(msg):

    print '%s, it works!' % msg

    while True: pass



def start_process():

    print 'Starting',multiprocessing.current_process().name



if __name__=='__main__':

    pool = multiprocessing.Pool(processes=2, initializer=start_process,)

    pool.apply_async(run('congbo'))



　  print "It's my turn!"



    pool.close()

    pool.join()

运行结果：

Starting PoolWorker-1

Starting PoolWorker-2

congbo, it works!
...
...

没有打印出“It's my turn!”，程序没有切换到主进程接着执行，子进程死循环。

说明这种使用方式是不对的，只能像 apply_async 原型那样使用。见示例3。

 

示例3：

import multiprocessing



def run(msg):

    print '%s, it works!' % msg



def start_process():

    print 'Starting',multiprocessing.current_process().name



if __name__=='__main__':

    pool = multiprocessing.Pool(processes=2, initializer=start_process,)

    pool.apply_async(run, args=('congbo',))

    pool.close()

    pool.join()

运行结果：

Starting PoolWorker-1
Starting PoolWorker-2
congbo, it works!

tips：apply_async 中 args 应为元组，故单个参数后面需加逗号。如果参数错误，不成功执行，不报异常！

 

示例4：

import multiprocessing



class Worker:

    def run(self, msg):

        print '%s, it works!' % msg



def start_process():

    print 'Starting',multiprocessing.current_process().name



if __name__=='__main__':

    pool = multiprocessing.Pool(processes=2, initializer=start_process,)

    pool.apply_async(Worker().run, args=('congbo',))

    pool.close()

    pool.join()

运行结果：

Starting PoolWorker-1

Starting PoolWorker-2

Exception in thread Thread-2:

Traceback (most recent call last):

  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner

    self.run()

  File "/usr/lib/python2.7/threading.py", line 504, in run

    self.__target(*self.__args, **self.__kwargs)

  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks

    put(task)

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

 

示例5：

import multiprocessing



class Worker:

    def run(self, msg):

        print '%s, it works!' % msg



def wrap(worker, msg): worker.run(msg) def start_process():

    print 'Starting',multiprocessing.current_process().name



if __name__=='__main__':

    pool = multiprocessing.Pool(processes=2, initializer=start_process,)

 pool.apply_async(wrap, args=(Worker(), 'congbo'))

    pool.close()

    pool.join()

不直接传入类实例的方法，用另一个函数包装一下，将类的实例作为参数传入即可。

为什么要这样？有没有更好的办法？ 

 

参考：http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma

 

原文：http://www.cnblogs.com/congbo/archive/2012/08/23/2652322.html