python多处理打印/stdou到单文件格式issu

2024-04-19 18:48:02 发布

您现在位置:Python中文网/ 问答频道 /正文

我正在使用多进程模块处理一个巨大的文件。我遍历每一行,做一些处理,然后将其打印/标准输出到另一个文件中。问题是,我有时得到两个不同的行打印在一行。我有锁,即使我看到同样的问题。我该怎么解决这个问题?你知道吗

例如

output received:
0,0.784207850012,0.215792149988,1,B5E326C7,0.257956186525,0.742043813475,1,UP87859

expected output:
0,0.784207850012,0.215792149988,1,B5E326C7
0.257956186525,0.742043813475,1,UP87859

代码:

#global lock
lock = multiprocessing.Lock()

def pred(line):
    u,seg,c,g,lab = line.strip().split(None,4)
    < #### DO SOMETHING ##### and generate p,pp0,pp1>
    if('null' not in lab):
        lock.acquire()
        sys.stdout.write(','.join(map(str, [p,pp0,pp1,1,u])) + '\n')
        lock.release()
    else:
        lock.acquire()
        sys.stdout.write(','.join(map(str, [p,pp0,pp1,1,u])) + '\n')
        lock.release()


if __name__ == "__main__":
    pool = multiprocessing.Pool(25,initializer=init, initargs=(l,))
    with open('file.csv') as source_file:
         results = pool.map(pred, source_file,1000)

Tags: 文件lockmapoutputiflinelabmultiprocessing