类方法内的并行循环调用另一个类方法

2024-04-29 02:08:22 发布

您现在位置:Python中文网/ 问答频道 /正文

这个问题与theseother的帖子有关,但其中提出的解决方案似乎不适合我的情况。简而言之,我的问题可以通过下面的例子来说明。我有一个Algebra类,其中通过方法triPower我的目标是计算三项式的幂,即(a+b+c)**n对于许多具有固定a, b, cn值。为此,我考虑创建一个方法_triPower(a,b,c,n),并通过functools.partial(_triPower,...)将它传递给我的pool.map()函数,在这里我修复了a, b, c,并将n作为唯一的参数,因为我使用的是Python 2.7,而multiprocessing模块中的map只需要一个参数函数(请参见this post)。代码如下:

from __future__ import division
import numpy as np
import functools as fntls
import multiprocessing as mp
import multiprocessing.pool as mppl

# A couple of classes introduced to allow multiple processes to have their own daemons (parallelization)
class NoDaemonProcess(mp.Process):
   # make 'daemon' attribute always return False
   def _get_daemon(self):
      return False
   def _set_daemon(self, value):
      pass
   daemon = property(_get_daemon, _set_daemon)

# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(mppl.Pool):
    Process = NoDaemonProcess

# Sample class where I want a method to run a parallel loop
class Algebra(object):
    def __init__(self,offset):
        self.offset = offset

    def trinomial(self,a,b,c):
        return a+b+c

    def _triPower(self,a,b,c,n):
    """This is the method that I want to run in parallel from the next method"""
        return self.offset + self.trinomial(a,b,c)**n

    def triPower(self,n):
        pls = MyPool(4)
        vals = pls.map(fntls.partial(self._triPower,a=1.,b=0.,c=1.),n)
        print vals

# Testing
if __name__ == "__main__":
    A = Algebra(0.)
    A.triPower(np.arange(0.,10.,0.01))

上述操作不起作用,并产生(如this post所预期的)错误:

cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

因此,在同一篇文章之后,我试图将_triPower定义为一个全局函数,即

def _triPower(alg,a,b,c,n):
"""This is the method that I want to run in parallel from the next method"""
    return alg.trinomial(a,b,c)**n 

然后根据以下内容编辑Algebra.triPower(...)

    def triPower(self,n):
    pls = MyPool(4)
    vals = pls.map(fntls.partial(_triPower, alg=self, a=1., b=0., c=1.), n)
    print vals 

后者会产生一些奇怪的类型错误,比如:

TypeError: _triPower() got multiple values for keyword argument 'alg' 

另一方面,尝试使方法可由VeryPicklableObject序列化(如this other post)的建议似乎也不起作用,而且这个包现在似乎已经死了(截至2019年5月)。那么,我做错了什么,如何使我的计算并行运行呢


Tags: thetoimportselfmapreturndefas