多处理池未映射

2024-04-19 12:31:42 发布

您现在位置:Python中文网/ 问答频道 /正文

我试图创建Pool()对象,以便分解大型数组。不过,每次在我第一次运行下面的代码之后,映射都不会运行。似乎只有第一个过程进入函数,尽管参数大小相同,即使使用完全相同的参数运行函数时也是如此-只有第一个过程

job.map(...)

似乎正在运行。下面是我痛苦的根源(不是文件中的所有代码):

def iterCount(): 
    #m is not in shared memory, as intended.
    global m  
    m = m + 1  
    return m 

def thread_search(pair): 
    divisor_lower = pair[0] 
    divisor_upper = pair[1] 
    for i in range(divisor_lower, divisor_upper,window_size): 
        current_section = np.array(x[i: i +  window_size]) 
        for row in current_section: 
            if (row[2].startswith('NP') ) and checkPep(row[0]): #checkPep is a simple unique-in-array checking function. 

            #shared_list is a multiprocessing.Manager list. 
            shared_list.append(row[[0,1,2]]) 
            m = iterCount() 
            if not m%1000000: 
                print(f'Encountered m = {m}', flush = True) 


def poolMap(pairs, group): 
    job = Pool(3) 
    print(f'Pool Created') 
    print(len(pairs)) 
    job.map(thread_search,pairs) 
    print('Pool Closed') 
    job.close() 

if __name__ == '__main__': 
    for group in [1,2,3]: #Example times to be run...   
        x = None 
        lower_bound = int((group - 1)*group_step) 
        upper_bound = int(group*group_step) 
        x = list(csv.reader(open(pa_table_name,"rt", encoding = "utf-8"), delimiter = "\t"))[lower_bound:upper_bound] 
        print(len(x))     
        divisor_pairs = [ [int(lower_bound + (i - 1)*chunk_size) , int(lower_bound + i*chunk_size)] for i in range(1,6143) ]  
        poolMap(divisor_pairs, group)

此函数的输出为:

Program started: 03/09/19, 12:41:25 (Machine Time)
11008256 - Length of the file read in (in the group)
Pool Created
6142 - len(pairs)
Encountered m = 1000000
Encountered m = 1000000
Encountered m = 1000000
Encountered m = 2000000
Encountered m = 2000000
Encountered m = 2000000
Encountered m = 3000000
Encountered m = 3000000
Encountered m = 3000000 (Total size is ~ 9 million per set)
Pool Closed
11008256 (this is the number of lines read, correct value)
Pool Created
6142 (Number of pairs is correct, though map appears to never run...) 
Pool Closed
11008256
Pool Created
6142
Pool Closed

此时,共享的线程列表被保存,并且只有第一个线程结果出现。你知道吗

我真的对这里发生的事情感到茫然,我试着去寻找虫子(?)或者类似的例子。你知道吗

Ubuntu 18.04版 Python 3.6版


Tags: inforsizeisgroupjobupperlower