等待多个并行任务的子进程结束

Question

我在用Python同时运行一些子进程。我想等到每个子进程都完成后再继续。现在我用的方法不太优雅：

runcodes = ["script1.C", "script2.C"]
ps = []
for script in runcodes:
  args = ["root", "-l", "-q", script]
  p = subprocess.Popen(args)
  ps.append(p)
while True:
  ps_status = [p.poll() for p in ps]
  if all([x is not None for x in ps_status]):
    break

有没有什么类可以处理多个子进程？问题是，wait这个方法会让我的程序停下来。

更新：我想在计算过程中显示进度，比如“4/7个子进程完成...”。

如果你感兴趣，root可以编译这个C++脚本并执行它。

Answer 1

你可以这样做：

runcodes = ["script1.C", "script2.C"]

ps = []
for script in runcodes:
    args = ["root", "-l", "-q", script]
    p = subprocess.Popen(args)
    ps.append(p)

for p in ps:
    p.wait()

这些过程会同时运行，最后你会等所有的过程都完成。

Answer 2

这样怎么样

import os, subprocess
runcodes = ["script1.C", "script2.C"]
ps = {}
for script in runcodes:
    args = ["root", "-l", "-q", script]
    p = subprocess.Popen(args)
    ps[p.pid] = p
print "Waiting for %d processes..." % len(ps)
while ps:
    pid, status = os.wait()
    if pid in ps:
        del ps[pid]
        print "Waiting for %d processes..." % len(ps)

Answer 3

如果你的系统不是Windows，你可以选择监控你子进程的标准输出管道。这样的话，你的应用程序会一直等待，直到发生以下情况之一：

• 注册的文件描述符中有输入输出事件（在这里，我们关注的是子进程的标准输出管道出现挂起的情况）
• 轮询超时

下面是一个使用Linux 2.6.xx的epoll的简单示例：

import subprocess
import select

poller = select.epoll()
subprocs = {} #map stdout pipe's file descriptor to the Popen object

#spawn some processes
for i in xrange(5):
    subproc = subprocess.Popen(["mylongrunningproc"], stdout=subprocess.PIPE)
    subprocs[subproc.stdout.fileno()] = subproc
    poller.register(subproc.stdout, select.EPOLLHUP)

#loop that polls until completion
while True:
    for fd, flags in poller.poll(timeout=1): #never more than a second without a UI update
        done_proc = subprocs[fd]
        poller.unregister(fd)
        print "this proc is done! blah blah blah"
        ...  #do whatever
    #print a reassuring spinning progress widget
    ...
    #don't forget to break when all are done