这个问题与theseother的帖子有关,但其中提出的解决方案似乎不适合我的情况。简而言之,我的问题可以通过下面的例子来说明。我有一个Algebra
类,其中通过方法triPower
我的目标是计算三项式的幂,即(a+b+c)**n
对于许多具有固定a, b, c
的n
值。为此,我考虑创建一个方法_triPower(a,b,c,n)
,并通过functools.partial(_triPower,...)
将它传递给我的pool.map()
函数,在这里我修复了a, b, c
,并将n
作为唯一的参数,因为我使用的是Python 2.7,而multiprocessing
模块中的map
只需要一个参数函数(请参见this post)。代码如下:
from __future__ import division
import numpy as np
import functools as fntls
import multiprocessing as mp
import multiprocessing.pool as mppl
# A couple of classes introduced to allow multiple processes to have their own daemons (parallelization)
class NoDaemonProcess(mp.Process):
# make 'daemon' attribute always return False
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(mppl.Pool):
Process = NoDaemonProcess
# Sample class where I want a method to run a parallel loop
class Algebra(object):
def __init__(self,offset):
self.offset = offset
def trinomial(self,a,b,c):
return a+b+c
def _triPower(self,a,b,c,n):
"""This is the method that I want to run in parallel from the next method"""
return self.offset + self.trinomial(a,b,c)**n
def triPower(self,n):
pls = MyPool(4)
vals = pls.map(fntls.partial(self._triPower,a=1.,b=0.,c=1.),n)
print vals
# Testing
if __name__ == "__main__":
A = Algebra(0.)
A.triPower(np.arange(0.,10.,0.01))
上述操作不起作用,并产生(如this post所预期的)错误:
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
因此,在同一篇文章之后,我试图将_triPower
定义为一个全局函数,即
def _triPower(alg,a,b,c,n):
"""This is the method that I want to run in parallel from the next method"""
return alg.trinomial(a,b,c)**n
然后根据以下内容编辑Algebra.triPower(...)
:
def triPower(self,n):
pls = MyPool(4)
vals = pls.map(fntls.partial(_triPower, alg=self, a=1., b=0., c=1.), n)
print vals
后者会产生一些奇怪的类型错误,比如:
TypeError: _triPower() got multiple values for keyword argument 'alg'
另一方面,尝试使方法可由VeryPicklableObject序列化(如this other post)的建议似乎也不起作用,而且这个包现在似乎已经死了(截至2019年5月)。那么,我做错了什么,如何使我的计算并行运行呢
目前没有回答
相关问题 更多 >
编程相关推荐