如何加快pickle.dump（）的写入速度

Question

目前，我想将生成的对象数组保存为文件。我使用pickle.dump（）方法如下：

    if mode == 'local' or mode == 'hpc':
        graph = toolbox.init_regular_network()
        all_record_list = simulation.run_simulation_show_all_process(
            graph, single_iterator_list,
            list_name.split('_')[0])
        if mode == 'hpc':
            f = open('result\Record_list', 'wb')
            pickle.dump(all_record_list, f)     

由于我的对象数组非常大，因此生成的文件为>；4G，而且写起来需要很长时间，我应该如何优化它？ 期待您的帮助

Answer 1

您可以尝试将协议更改为尽可能高的值

pickle.dump(all_record_list, f, protocol=pickle.HIGHEST_PROTOCOL)

从文档中：

Changed in version 3.0: The default protocol is 3.

Changed in version 3.8: The default protocol is 4.

...

• Protocol version 0 is the original “human-readable” protocol and is backwards compatible with earlier versions of Python.

• Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.

• Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes. Refer to PEP 307 for information about improvements brought by protocol 2.

• Protocol version 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. This was the default protocol in Python 3.0–3.7.

• Protocol version 4 was added in Python 3.4. It adds support for very large objects, pickling more kinds of objects, and some data format optimizations. It is the default protocol starting with Python 3.8. Refer to PEP 3154 for information about improvements brought by protocol 4.

• Protocol version 5 was added in Python 3.8. It adds support for out-of-band data and speedup for in-band data. Refer to PEP 574 for information about improvements brought by protocol 5.

可以在此处找到文档：https://docs.python.org/3/library/pickle.html