使嵌套forloops的cpu使用率达到最大的最简单方法是什么？

Question

我有一些代码，可以对元素进行独特的组合。有6种类型，每种大约有100种。所以有100^6个组合。每个组合都必须经过计算、检查相关性，然后要么被丢弃要么被保存。在

代码的相关部分如下所示：

def modconffactory():
for transmitter in totaltransmitterdict.values():
    for reciever in totalrecieverdict.values():
        for processor in totalprocessordict.values():
            for holoarray in totalholoarraydict.values():
                for databus in totaldatabusdict.values():
                    for multiplexer in totalmultiplexerdict.values():
                        newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
                        data_I_need = dosomethingwith(newconfiguration)
                        saveforlateruse_if_useful(data_I_need)

现在这需要很长时间，这很好，但是现在我意识到这个过程（进行配置，然后计算以备以后使用）一次只使用我8个处理器核中的一个。在

我一直在阅读多线程和多处理，但我只看到不同进程的例子，没有看到如何多线程一个进程。在我的代码中，我调用了两个函数：“dosomethingwith（）”和“saveforlateruse\u if\u usivery（）”。我可以把它们分成单独的进程，并让它们同时运行到for循环中，对吗？在

但是for循环本身呢？我能加快这个过程吗？因为这就是时间消耗的地方。（<；--这是我的主要问题）

有没有作弊？例如编译成C然后操作系统自动多线程？在

Answer 1

我只看到不同进程的示例，没有看到如何多线程一个进程

Python中有多线程，但由于GIL（globalinterprelatorlock）的存在，它的效率很低。因此，如果您想要使用所有的处理器核心，如果您想要并发，除了使用多个进程之外别无选择，这可以通过multiprocessing模块来实现（好吧，您也可以使用另一种语言而不存在此类问题）

案例中多处理用法的近似示例：

import multiprocessing

WORKERS_NUMBER = 8

def modconffactoryProcess(generator, step, offset, conn):
    """
    Function to be invoked by every worker process.

    generator: iterable object, the very top one of all you are iterating over, 
    in your case, totalrecieverdict.values()

    We are passing a whole iterable object to every worker, they all will iterate 
    over it. To ensure they will not waste time by doing the same things 
    concurrently, we will assume this: each worker will process only each stepTH 
    item, starting with offsetTH one. step must be equal to the WORKERS_NUMBER, 
    and offset must be a unique number for each worker, varying from 0 to 
    WORKERS_NUMBER - 1

    conn: a multiprocessing.Connection object, allowing the worker to communicate 
    with the main process
    """
    for i, transmitter in enumerate(generator):
        if i % step == offset:
            for reciever in totalrecieverdict.values():
                for processor in totalprocessordict.values():
                    for holoarray in totalholoarraydict.values():
                        for databus in totaldatabusdict.values():
                            for multiplexer in totalmultiplexerdict.values():
                                newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
                                data_I_need = dosomethingwith(newconfiguration)
                                saveforlateruse_if_useful(data_I_need)
    conn.send('done')


def modconffactory():
    """
    Function to launch all the worker processes and wait until they all complete 
    their tasks
    """
    processes = []
    generator = totaltransmitterdict.values()
    for i in range(WORKERS_NUMBER):
        conn, childConn = multiprocessing.Pipe()
        process = multiprocessing.Process(target=modconffactoryProcess, args=(generator, WORKERS_NUMBER, i, childConn))
        process.start()
        processes.append((process, conn))
    # Here we have created, started and saved to a list all the worker processes
    working = True
    finishedProcessesNumber = 0
    try:
        while working:
            for process, conn in processes:
                if conn.poll():  # Check if any messages have arrived from a worker
                    message = conn.recv()
                    if message == 'done':
                        finishedProcessesNumber += 1
            if finishedProcessesNumber == WORKERS_NUMBER:
                working = False
    except KeyboardInterrupt:
        print('Aborted')

您可以根据需要调整WORKERS_NUMBER。在

与multiprocessing.Pool相同：

^{pr2}$

您可能希望使用.map_async而不是.map

这两个代码段的作用是一样的，但我想说的是，在第一个代码段中，您可以更好地控制程序。在

不过，我想第二个是最简单的：）

但是第一个应该让你知道第二个发生了什么

multiprocessing文档：https://docs.python.org/3/library/multiprocessing.html

Answer 2

您可以这样运行函数：

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
   p = Pool(5)
   print(p.map(f, [1, 2, 3]))

https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers