如何修复Python多进程错误？

Question

我正在开始学习Python的多进程编程。现在我从一段简单的代码入手。我使用的是Anaconda和Spyder这个开发环境来运行这段代码，操作系统是Windows 10专业版。

import time
import multiprocessing

start = time.perf_counter()

def do_something():
    print('Sleeping for 1 second ...')
    time.sleep(1)
    print('Done Sleeping ...')


p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)

p1.start()
p2.start()

p1.join()
p2.join()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} in seconds')

当我运行上面的代码时，出现了以下错误信息：

Finished in 0.15 in seconds

Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>

我注意到最后的打印语句是有输出的，但接着就出现了上面的错误。为什么会出现这个错误呢？我需要对代码做什么才能修复这个错误？

Answer 1

几个月前我也遇到过你这样的情况，最后我发现用concurrent.future比用multiprocessing更有效。

import time
import concurrent.futures

def some_cpu_bound_task(num):
    print(f"in cpu task {num}")
    range_to_sum = list(range(int(10000000)))  # create a big list to sum
    result = sum(range_to_sum)  # sum the list (CPU-intensive)
    return num ** 3  # return result of power operation

def some_io_bound_task(num):
    print(f"in io task {num}")
    time.sleep(10)  # simulated I/O delay
    return num * num

def start_calculations():
    with concurrent.futures.ProcessPoolExecutor(16) as executor:
        tasks = []
        for i in range(16):
            tasks.append(executor.submit(some_cpu_bound_task, i))
            tasks.append(executor.submit(some_io_bound_task, i))
    return tasks

def wait_all_calculations(tasks):
    for future in concurrent.futures.as_completed(tasks):
        result = future.result()
        print(f"Result: {result}")

if __name__ == "__main__":
    all_tasks = start_calculations()
    wait_all_calculations(all_tasks)

我觉得你的代码可能不太好用，因为你需要在主程序里运行，就像文档里的例子那样。