如何在Python中使用多进程进行网格搜索（参数优化）

Question

我正在尝试找出如何通过使用多进程来加速一个网格搜索，这个搜索需要两个输入（一个是下限值，一个是上限值），我是在Python中进行的。我查看了这个链接，并尝试使用以下代码：

import multiprocessing

lower_threshold = range(0,100)
upper_threshold = range(100,200)

def worker():
    bestAccuracy = 0
    bestLowerThreshold = 0
    bestUpperThreshold = 0    
    for lowerThreshold in lower_threshold:
        for upperThreshold in upper_threshold:
            accuracy = someOtherFunction(lower_threshold,upper_threshold)
            if accuracy > bestAccuracy:
                bestAccuracy = accuracy
                bestLowerTheshold = lowerThreshold
                bestUpperThreshold = upperThreshold
    return bestAccuracy, bestLowerThreshold, bestUpperThreshold

if __name__ == '__main__':
    jobs = []
    for i in range(8):
        p = multiprocessing.Process(target=worker)
        jobs.append(p)
        p.start()

当我运行这段代码时，我发现我的8个核心进程都在使用相同的参数/输入来运行我的函数。有没有什么方法可以让我指定这些核心进程使用不同的参数呢？提前感谢你的帮助。

Answer 1

一般来说，多进程处理最有效的方式是有几个工人（也就是处理任务的程序），以及一长串需要处理的输入数据。每一批输入数据会同时被处理，处理完的结果会返回给调用者。然后，这些工人会继续处理下一批数据，直到所有的输入数据都被处理完。

下面的代码传入了 lower_threshold 的范围，每次都重复使用 upper_threshold：

源代码

import multiprocessing, random

upper_threshold = range(10,20)

def someOtherFunction(a,b):
    print 'ding:',a,b
    return random.randint(10, 100)

def worker(thresh):
    bestAccuracy = 0
    bestLowerThreshold = 0
    bestUpperThreshold = 0    
    for upperThreshold in upper_threshold:
        accuracy = someOtherFunction(thresh, upper_threshold)
        if accuracy > bestAccuracy:
            bestAccuracy = accuracy
            bestLowerTheshold = thresh
            bestUpperThreshold = upperThreshold
    return bestAccuracy, bestLowerThreshold, bestUpperThreshold

if __name__ == '__main__':
    lower_threshold = range(0,10)
    pool = multiprocessing.Pool()
    print pool.map( worker, [[th] for th in lower_threshold] )
    pool.close()
    pool.join()

输出结果

ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[(80, 0, 11), (98, 0, 17), (96, 0, 17), (89, 0, 17), (96, 0, 17), (95, 0, 10), (95, 0, 18), (100, 0, 18), (90, 0, 18), (97, 0, 17)]

Answer 2

你可以使用 itertools.product 来创建一个迭代器，这个迭代器会生成所有的 (lowerThreshHold, upperThreshold) 组合，组合的数量是你在嵌套的循环中产生的。然后，你可以把这个迭代器分成八个部分，每个部分传给你池中的一个进程。最后，你只需要对每个工作者返回的结果进行排序，从中挑选出最好的结果。我还建议使用 multiprocessing.Pool，而不是 multiprocessing.Process，这样会更简单一些。

import multiprocessing
import itertools

def get_chunks(iterable, chunks=1):
    # This is from http://stackoverflow.com/a/2136090/2073595
    lst = list(iterable)
    return [lst[i::chunks] for i in xrange(chunks)]

def someOtherFunction(lower, upper):
    return (lower + upper) / 2

def worker(pairs):
    best_accuracy = best_lower = best_upper = 0
    for lower_threshold, upper_threshold in pairs:
        accuracy = someOtherFunction(lower_threshold, upper_threshold)
        if accuracy > best_accuracy:
            best_accuracy = accuracy
            best_lower = lower_threshold
            best_upper = upper_threshold
    return best_accuracy, best_lower, best_upper

if __name__ == '__main__':
    jobs = []
    pairs = itertools.product(xrange(0, 100), xrange(100, 200))
    chunked_pairs = get_chunks(pairs, chunks=multiprocessing.cpu_count())
    pool = multiprocessing.Pool()
    results = pool.map(worker, chunked_pairs)
    pool.close()
    pool.join()
    print results
    # Now combine the results
    sorted_results = reversed(sorted(results, key=lambda x: x[0]))
    print next(sorted_results)  # Winner

输出：

[(147, 98, 196), (148, 99, 197), (148, 98, 198), (149, 99, 199)]
(149, 99, 199)