多重处理映射无序?

2024-04-29 23:14:50 发布

您现在位置:Python中文网/ 问答频道 /正文

我有一个使用字典的操作,我想把它并行化,但是多处理.map让我头疼

def dict_segmentor(dictionnary,n_processes):
    print("len dictionnary")    
    print(len(dictionnary))
    if len(dictionnary) < n_processes:
        seg_size = len(dictionnary)
    else:
        seg_size = len(dictionnary) // n_processes
    print("segmenting dictionnary")
    print("seg_size "+str(seg_size))
    print("len(dictionnary) "+str(len(dictionnary)))
    itemlist=list(dictionnary.items())
    seg_ranges = [dict(itemlist[s:s+seg_size]) for s in range(1,  len(dictionnary)+1, seg_size)]
    print("finished")
    return seg_ranges

def multiprocess_calc(n_processes, dictionnary,struc):
    dictionnary=dictionnary
    struc=struc

    seg_ranges1 = dict_segmentor(dictionnary,n_processes)
    #this is invoked to break the dict to be passed into dicts into a list. Works as expected
    print("seg_range_check")    
    print("seg_ranges1 {}".format(type(seg_ranges1)))#Returns a dict as expected
    print("seg_ranges1 {}".format(type(seg_ranges1[0])))#Returns a list as expected
    print("seg_ranges1 {}".format(len(seg_ranges1)))    #Returns expected len=1
    print("seg_ranges1 {}".format(len(seg_ranges1[0]))) #Returns expected len

    processes = multiprocessing.Pool(n_processes)
    print("Mapping Building")

    processes.map(Builder, seg_ranges1,1)

def main():
    file_multiprocess  = 'pref_multiprocess.csv'
    n_CPUs = multiprocessing.cpu_count()
    n_processes  = n_CPUs-1
    print("\nNumber of CPUs detected:", n_CPUs)
    multiprocess_calc(n_processes, file_multiprocess,struc)

if __name__ == '__main__':
    main()

以下是完整的回溯:

^{pr2}$

我不明白,即使仔细阅读(https://docs.python.org/3.5/library/multiprocessing.html#module-multiprocessing)。
每个块都是dict,因此应该通过map发送给构建器。
但是我得到了那个愚蠢的错误,回溯也没用。我查了的密码池.py但没有运气,
我的构建器没有参与,因为它的第一个操作(控制打印)甚至没有显示。builder函数似乎完全被忽略了(甚至没有语法错误)
所以我断定这是地图的问题。
万一我误解了多处理.map函数,它将首先生成一个块,然后迭代它在每个子元素上应用映射,我可以使用什么样的多处理函数?只使用一根线。那就意味着我应该手动操作?
请随时更正我的代码,并给我一些见解。提前谢谢

编辑:以下是构建器函数:

def Builder(dictionary,struc=struc):
#def Builder(keys, dictionary=dictionary,struc=struc):  #Alternative
#Note, I even tried to use only the keys, passing the dictionary from a global variable but it didn't work
    print("Building Tree")    #Not even displayed
    print("type dictionary"+str(type(dictionary)))
    frags=0
    try:
        if True:
            print("Building")  
            #for id in keys: #Alternative
            for id in dictionary:
                seq=dictionary[id]
                for i in range(3):
                    frags+=1
                    if len(seq)<=3:
                        break
                    seq=seq[i:-i]
                    struc.append(seq)
                print("Number of frags found {}".format(frags))
    except TypeError as e:
        print (e)
        print ("error in Builder")

Tags: informatsizedictionarylendefdictprocesses