我正在使用多进程模块处理一个巨大的文件。我遍历每一行,做一些处理,然后将其打印/标准输出到另一个文件中。问题是,我有时得到两个不同的行打印在一行。我有锁,即使我看到同样的问题。我该怎么解决这个问题?你知道吗
例如
output received:
0,0.784207850012,0.215792149988,1,B5E326C7,0.257956186525,0.742043813475,1,UP87859
expected output:
0,0.784207850012,0.215792149988,1,B5E326C7
0.257956186525,0.742043813475,1,UP87859
代码:
#global lock
lock = multiprocessing.Lock()
def pred(line):
u,seg,c,g,lab = line.strip().split(None,4)
< #### DO SOMETHING ##### and generate p,pp0,pp1>
if('null' not in lab):
lock.acquire()
sys.stdout.write(','.join(map(str, [p,pp0,pp1,1,u])) + '\n')
lock.release()
else:
lock.acquire()
sys.stdout.write(','.join(map(str, [p,pp0,pp1,1,u])) + '\n')
lock.release()
if __name__ == "__main__":
pool = multiprocessing.Pool(25,initializer=init, initargs=(l,))
with open('file.csv') as source_file:
results = pool.map(pred, source_file,1000)
目前没有回答
相关问题 更多 >
编程相关推荐