Python 多进程：示例未引发 PicklingError

Question

有没有人能告诉我，为什么下面这个页面中的最后一个例子（示例 3. Net-SNMP 的多进程包装器）不会出现 PicklingError 错误？

我用自己的绑定方法进行了尝试，这个方法更新并返回一个实例属性（类似于那个更新并返回实例属性的例子），但是却出现了以下错误：

 Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

这是我的代码：

from multiprocessing import Process, Queue
import requests

class MyClass(object):

    def do_request(self, url):
        try:
            self.response = requests.get(url)
        except:
            self.response = None
        return self.response

def make_request(url):
    s = MyClass()
    return s.do_request(url)

# Function run by worker processes
def worker(input, output):
    for func in iter(input.get, 'STOP'):
        result = make_request(func)
        output.put(result)

def main():
    """Runs everything"""

    #clients
    urls = ['http://www.google.com', 'http://www.amazon.com']
    NUMBER_OF_PROCESSES = len(urls)

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    #submit tasks
    for url in urls:
        task_queue.put(url)

    #Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

     # Get and print results
    print 'Unordered results:'
    for i in range(len(urls)):
        print '\t', done_queue.get()

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')
        print "Stopping Process #%s" % i

if __name__ == "__main__":
    main()

Answer 1

问题在于，requests.get() 返回的结果不是一个可以被“打包”的对象。你需要提取你想要的信息，然后把这些信息返回给父对象。个人来说，我喜欢使用简单的数据类型，比如列表和字典，这样可以减少出错的机会。

这里有一个实验，在我的 Linux + Python 2.7 的机器上抛出了一个更复杂的错误，但可以让你理解这个问题：

>>> import requests
>>> import pickle
>>> r=requests.get('http://google.com')
>>> pickle.dumps(r)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)

  ... many lines removed

  File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

你可以通过把 self.response = requests.get(url) 替换成 self.response = "hello" 来测试我说的是否正确。