是否可以在完成时获取Python池的结果？

Question

在使用Python的多进程池时，能不能在结果准备好时就获取结果，而不是等所有结果都准备好再一起拿到呢？

比如说，像这样，当results准备好一个结果时就能立刻得到，而不是等所有结果都准备好后再一起返回。

args = [ ... ]
def foo():
    pass

pool = multiprocessing.Pool()
results = pool.map_async(foo, l)

for result in results:
    save_result(result)

我能想到的唯一方法是

args = [ ... ]
results = []
def foo():
    pass

pool = multiprocessing.Pool()

for arg in args:
    results.append(pool.apply_async(foo, arg))

while results:
    for result in results:
        if result.ready():
            save_result(result.get())

我在Windows 7 x64上使用Python 3.4

Answer 1

其他的回答推荐使用 concurrent.futures，但对于某些应用来说，进程不是线程安全的，因此需要使用多进程来完成。对于这个应用，我找到了一种方法可以满足你的要求。

pool = multiprocessing.Pool()
pool_results = []
pool_results.append(pool.apply_async(<function>, (<args>)))
# ... add more function calls. possibly using loop ...

pool.close()
while len(pool_results) > 0:
    to_remove = [] #avoid removing objects during for_loop
    for r in pool_results:
        # check if process is finished
        if r.ready(): 
            # print result (or do any operation with result)
            print(r.get())
            to_remove.append(r)
    for remove in to_remove:
        pool_results.remove(remove)
    time.sleep(1) # ensures that this thread doesn't consume too much memory
pool.join() # make sure all processes are completed

Answer 2

既然你在使用 Python 3.2 及以上版本，最简单的方法就是用 concurrent.futures，而不是 multiprocessing.Pool。使用 concurrent.futures.as_completed 可以让你得到一个迭代器，它会在任务完成时返回结果。

基本用法结构：

with concurrent.futures.ProcessPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(foo, arg) for arg in some_args]
    for future in concurrent.futures.as_completed(futures):
        #do something with completed result

可以查看链接中的文档，里面有更多的例子可以帮助你入门。