多个subprocess.Popen实例

Question

你好，我刚开始学习用Python编程，想用subprocess.Popen来同时运行多个我用“make”编译的程序。不过在我执行“make”之前，我需要先处理一些文本，生成一组“make”会用到的文件。现在我想用不同的生成文件同时运行同一个程序，并把所有程序的输出写到同一个文件里。根据我运行的实例数量，我还需要生成相应数量的文本文件。简单来说，我想在第一个for循环下面的所有操作同时进行，假设要进行'n'次。如果有人能提供帮助，我将非常感激 :).

for mC in range(monteCarlo):
    print "Simulation Number",str(mC+1),"of",str(monteCarlo)
    L = numpy.zeros((1,4),float)
    W = numpy.zeros((1,4),float)
    i = 0
    j = 0
    with open("1t.sp", "r") as inFile:
        with open("2t.sp","w") as outFile:
            line = inFile.readline()
            while (line != ""):
                newLine = []
                for words in line.split():
                    if words.startswith("W="):
                        W[0,i] = float(words[2:].replace('n',''))*random.normalvariate(1,widthDeviation)
                        #print i,words,str('W='+str(W[i]).strip('[]')+'n').replace(" ","")
                        words = str('W='+str(W[0,i]).strip('[]')+'n').replace(" ","")
                        i = i+1
                    elif words.startswith("L="):
                        L[0,j] = float(words[2:].replace('n',''))*random.normalvariate(1,lengthDeviation)
                        #print j,words,str('L='+str(L[j]).strip('[]')+'n').replace(" ","")
                        words = str('L='+str(L[0,j]).strip('[]')+'n').replace(" ","")
                        j = j+1
                    newLine.append(words)
            #print newLine
                outFile.write(" ".join(newLine))
                outFile.write("\n")
                line = inFile.readline()
    outFile.close()
    inFile.close()
    openWrite.write(str(W).strip('[]'))
    openWrite.write(str(L).strip('[]'))
    call(["make"])
    fRate = (open("tf.log","r").readlines()[34]).split()[-2]
    cSect = (open("tf.log","r").readlines()[35]).split()[-2]
    openWrite.write("\t")
    openWrite.write(fRate)
    openWrite.write(" ") 
    openWrite.write(cSect)
    openWrite.write("\n")
openWrite.close()   

Answer 1

如果你的系统有多个处理器或核心，你可以利用这个优势，通过使用multiprocessing模块来同时运行Python函数：

import multiprocessing as mp

def run_mc(mC):
    print "Simulation Number", str(mC+1), "of", str(monteCarlo)
    ...
    call(["make"])
    fRate = (open("tf.log", "r").readlines()[34]).split()[-2]
    cSect = (open("tf.log", "r").readlines()[35]).split()[-2]
    return fRate, cSect

def log_result(result):
    # This is called whenever run_mc returns a result.
    # result is modified only by the main process, not the pool workers.
    fRate, cSect = result
    with open(..., 'a') as openWrite:
        openWrite.write('\t{f} {c}\n'.format(f = fRate, c = cSect))

def main():
    # mp.Pool creates a pool of worker processes. By default it creates as many
    # workers as the system has processors. When the problem is CPU-bound, there
    # is no point in making more.
    pool = mp.Pool()
    for mC in range(monteCarlo):
        # This will call run_mc(mC) in a worker process.
        pool.apply_async(run_mc, args = (mC), callback = log_result)

if __name__ == '__main__':
    main()