在Python中使用小于128KB的字符串时内存泄漏？问题的回答

在Python中使用小于128KB的字符串时内存泄漏？

回答此问题可获得 20 贡献值，回答如果被采纳可获得 50 分。

<h2>原标题：内存泄漏打开文件&lt；128KB在Python中？在</h2> <h2>原始问题</h2> 在运行Python脚本时，我看到了内存泄漏。这是我的剧本： <pre><code>import sys import time class MyObj(object): def __init__(self, filename): with open(filename) as f: self.att = f.read() def myfunc(filename): mylist = [MyObj(filename) for x in xrange(100)] len(mylist) return [] def main(): filename = sys.argv[1] myfunc(filename) time.sleep(3600) if __name__ == '__main__': main() </code></pre> main函数调用<code>myfunc()</code>，它创建一个由100个对象组成的列表，每个对象都打开并读一个文件。从<code>myfunc()</code>返回后，我希望从100项列表和从读取要释放的文件，因为它们不再被引用。但是，当我使用<code>ps</code>命令检查内存使用情况，Python进程使用大约10000 KB 内存比从第12行和第13行注释掉的脚本运行的Python进程多。在 奇怪的是，内存泄漏（如果是这样的话）似乎只会发生对于128KB大小的文件。我创建了一个bash脚本来运行这个脚本，文件范围大小从1KB到200KB，当文件大小达到128KB时，内存增长停止。下面是bash脚本： ^{pr2}$ 下面是bash脚本的输出： <pre><code>PID RSS S TTY TIME COMMAND 28471 5552 S pts/16 00:00:00 python debug_memory.py data/stuff_1K.txt 28477 5656 S pts/16 00:00:00 python debug_memory.py data/stuff_2K.txt 28483 5756 S pts/16 00:00:00 python debug_memory.py data/stuff_3K.txt 28488 5852 S pts/16 00:00:00 python debug_memory.py data/stuff_4K.txt 28494 5952 S pts/16 00:00:00 python debug_memory.py data/stuff_5K.txt 28499 6052 S pts/16 00:00:00 python debug_memory.py data/stuff_6K.txt 28505 6156 S pts/16 00:00:00 python debug_memory.py data/stuff_7K.txt 28511 6256 S pts/16 00:00:00 python debug_memory.py data/stuff_8K.txt 28516 6356 S pts/16 00:00:00 python debug_memory.py data/stuff_9K.txt 28522 6452 S pts/16 00:00:00 python debug_memory.py data/stuff_10K.txt 28527 6552 S pts/16 00:00:00 python debug_memory.py data/stuff_11K.txt 28533 6656 S pts/16 00:00:00 python debug_memory.py data/stuff_12K.txt 28539 6756 S pts/16 00:00:00 python debug_memory.py data/stuff_13K.txt 28544 6852 S pts/16 00:00:00 python debug_memory.py data/stuff_14K.txt 28550 6952 S pts/16 00:00:00 python debug_memory.py data/stuff_15K.txt 28555 7056 S pts/16 00:00:00 python debug_memory.py data/stuff_16K.txt 28561 7156 S pts/16 00:00:00 python debug_memory.py data/stuff_17K.txt 28567 7252 S pts/16 00:00:00 python debug_memory.py data/stuff_18K.txt 28572 7356 S pts/16 00:00:00 python debug_memory.py data/stuff_19K.txt 28578 7452 S pts/16 00:00:00 python debug_memory.py data/stuff_20K.txt 28584 7556 S pts/16 00:00:00 python debug_memory.py data/stuff_21K.txt 28589 7652 S pts/16 00:00:00 python debug_memory.py data/stuff_22K.txt 28595 7756 S pts/16 00:00:00 python debug_memory.py data/stuff_23K.txt 28600 7852 S pts/16 00:00:00 python debug_memory.py data/stuff_24K.txt 28606 7952 S pts/16 00:00:00 python debug_memory.py data/stuff_25K.txt 28612 8052 S pts/16 00:00:00 python debug_memory.py data/stuff_26K.txt 28617 8152 S pts/16 00:00:00 python debug_memory.py data/stuff_27K.txt 28623 8252 S pts/16 00:00:00 python debug_memory.py data/stuff_28K.txt 28629 8356 S pts/16 00:00:00 python debug_memory.py data/stuff_29K.txt 28634 8452 S pts/16 00:00:00 python debug_memory.py data/stuff_30K.txt 28640 8556 S pts/16 00:00:00 python debug_memory.py data/stuff_31K.txt 28645 8656 S pts/16 00:00:00 python debug_memory.py data/stuff_32K.txt 28651 8756 S pts/16 00:00:00 python debug_memory.py data/stuff_33K.txt 28657 8856 S pts/16 00:00:00 python debug_memory.py data/stuff_34K.txt 28662 8956 S pts/16 00:00:00 python debug_memory.py data/stuff_35K.txt 28668 9056 S pts/16 00:00:00 python debug_memory.py data/stuff_36K.txt 28674 9156 S pts/16 00:00:00 python debug_memory.py data/stuff_37K.txt 28679 9256 S pts/16 00:00:00 python debug_memory.py data/stuff_38K.txt 28685 9352 S pts/16 00:00:00 python debug_memory.py data/stuff_39K.txt 28691 9452 S pts/16 00:00:00 python debug_memory.py data/stuff_40K.txt 28696 9552 S pts/16 00:00:00 python debug_memory.py data/stuff_41K.txt 28702 9656 S pts/16 00:00:00 python debug_memory.py data/stuff_42K.txt 28707 9756 S pts/16 00:00:00 python debug_memory.py data/stuff_43K.txt 28713 9852 S pts/16 00:00:00 python debug_memory.py data/stuff_44K.txt 28719 9952 S pts/16 00:00:00 python debug_memory.py data/stuff_45K.txt 28724 10052 S pts/16 00:00:00 python debug_memory.py data/stuff_46K.txt 28730 10156 S pts/16 00:00:00 python debug_memory.py data/stuff_47K.txt 28739 10256 S pts/16 00:00:00 python debug_memory.py data/stuff_48K.txt 28746 10352 S pts/16 00:00:00 python debug_memory.py data/stuff_49K.txt 28752 10452 S pts/16 00:00:00 python debug_memory.py data/stuff_50K.txt 28757 10556 S pts/16 00:00:00 python debug_memory.py data/stuff_51K.txt 28763 10656 S pts/16 00:00:00 python debug_memory.py data/stuff_52K.txt 28769 10752 S pts/16 00:00:00 python debug_memory.py data/stuff_53K.txt 28774 10852 S pts/16 00:00:00 python debug_memory.py data/stuff_54K.txt 28780 10952 S pts/16 00:00:00 python debug_memory.py data/stuff_55K.txt 28786 11052 S pts/16 00:00:00 python debug_memory.py data/stuff_56K.txt 28791 11152 S pts/16 00:00:00 python debug_memory.py data/stuff_57K.txt 28797 11256 S pts/16 00:00:00 python debug_memory.py data/stuff_58K.txt 28802 11356 S pts/16 00:00:00 python debug_memory.py data/stuff_59K.txt 28808 11452 S pts/16 00:00:00 python debug_memory.py data/stuff_60K.txt 28814 11556 S pts/16 00:00:00 python debug_memory.py data/stuff_61K.txt 28819 11656 S pts/16 00:00:00 python debug_memory.py data/stuff_62K.txt 28825 11752 S pts/16 00:00:00 python debug_memory.py data/stuff_63K.txt 28831 11852 S pts/16 00:00:00 python debug_memory.py data/stuff_64K.txt 28836 11956 S pts/16 00:00:00 python debug_memory.py data/stuff_65K.txt 28842 12052 S pts/16 00:00:00 python debug_memory.py data/stuff_66K.txt 28847 12152 S pts/16 00:00:00 python debug_memory.py data/stuff_67K.txt 28853 12256 S pts/16 00:00:00 python debug_memory.py data/stuff_68K.txt 28859 12356 S pts/16 00:00:00 python debug_memory.py data/stuff_69K.txt 28864 12452 S pts/16 00:00:00 python debug_memory.py data/stuff_70K.txt 28871 12556 S pts/16 00:00:00 python debug_memory.py data/stuff_71K.txt 28877 12652 S pts/16 00:00:00 python debug_memory.py data/stuff_72K.txt 28883 12756 S pts/16 00:00:00 python debug_memory.py data/stuff_73K.txt 28889 12856 S pts/16 00:00:00 python debug_memory.py data/stuff_74K.txt 28894 12952 S pts/16 00:00:00 python debug_memory.py data/stuff_75K.txt 28900 13056 S pts/16 00:00:00 python debug_memory.py data/stuff_76K.txt 28906 13156 S pts/16 00:00:00 python debug_memory.py data/stuff_77K.txt 28911 13256 S pts/16 00:00:00 python debug_memory.py data/stuff_78K.txt 28917 13352 S pts/16 00:00:00 python debug_memory.py data/stuff_79K.txt 28922 13452 S pts/16 00:00:00 python debug_memory.py data/stuff_80K.txt 28928 13556 S pts/16 00:00:00 python debug_memory.py data/stuff_81K.txt 28934 13652 S pts/16 00:00:00 python debug_memory.py data/stuff_82K.txt 28939 13752 S pts/16 00:00:00 python debug_memory.py data/stuff_83K.txt 28945 13852 S pts/16 00:00:00 python debug_memory.py data/stuff_84K.txt 28951 13952 S pts/16 00:00:00 python debug_memory.py data/stuff_85K.txt 28956 14052 S pts/16 00:00:00 python debug_memory.py data/stuff_86K.txt 28962 14152 S pts/16 00:00:00 python debug_memory.py data/stuff_87K.txt 28967 14256 S pts/16 00:00:00 python debug_memory.py data/stuff_88K.txt 28973 14352 S pts/16 00:00:00 python debug_memory.py data/stuff_89K.txt 28979 14456 S pts/16 00:00:00 python debug_memory.py data/stuff_90K.txt 28984 14552 S pts/16 00:00:00 python debug_memory.py data/stuff_91K.txt 28990 14652 S pts/16 00:00:00 python debug_memory.py data/stuff_92K.txt 28996 14756 S pts/16 00:00:00 python debug_memory.py data/stuff_93K.txt 29001 14852 S pts/16 00:00:00 python debug_memory.py data/stuff_94K.txt 29007 14956 S pts/16 00:00:00 python debug_memory.py data/stuff_95K.txt 29012 15052 S pts/16 00:00:00 python debug_memory.py data/stuff_96K.txt 29018 15156 S pts/16 00:00:00 python debug_memory.py data/stuff_97K.txt 29024 15252 S pts/16 00:00:00 python debug_memory.py data/stuff_98K.txt 29029 15360 S pts/16 00:00:00 python debug_memory.py data/stuff_99K.txt 29035 15456 S pts/16 00:00:00 python debug_memory.py data/stuff_100K.txt 29040 15556 S pts/16 00:00:00 python debug_memory.py data/stuff_101K.txt 29046 15652 S pts/16 00:00:00 python debug_memory.py data/stuff_102K.txt 29052 15756 S pts/16 00:00:00 python debug_memory.py data/stuff_103K.txt 29057 15852 S pts/16 00:00:00 python debug_memory.py data/stuff_104K.txt 29063 15952 S pts/16 00:00:00 python debug_memory.py data/stuff_105K.txt 29069 16056 S pts/16 00:00:00 python debug_memory.py data/stuff_106K.txt 29074 16152 S pts/16 00:00:00 python debug_memory.py data/stuff_107K.txt 29080 16256 S pts/16 00:00:00 python debug_memory.py data/stuff_108K.txt 29085 16356 S pts/16 00:00:00 python debug_memory.py data/stuff_109K.txt 29091 16452 S pts/16 00:00:00 python debug_memory.py data/stuff_110K.txt 29097 16552 S pts/16 00:00:00 python debug_memory.py data/stuff_111K.txt 29102 16652 S pts/16 00:00:00 python debug_memory.py data/stuff_112K.txt 29108 16756 S pts/16 00:00:00 python debug_memory.py data/stuff_113K.txt 29113 16852 S pts/16 00:00:00 python debug_memory.py data/stuff_114K.txt 29119 16952 S pts/16 00:00:00 python debug_memory.py data/stuff_115K.txt 29125 17056 S pts/16 00:00:00 python debug_memory.py data/stuff_116K.txt 29130 17156 S pts/16 00:00:00 python debug_memory.py data/stuff_117K.txt 29136 17256 S pts/16 00:00:00 python debug_memory.py data/stuff_118K.txt 29141 17356 S pts/16 00:00:00 python debug_memory.py data/stuff_119K.txt 29147 17452 S pts/16 00:00:00 python debug_memory.py data/stuff_120K.txt 29153 17556 S pts/16 00:00:00 python debug_memory.py data/stuff_121K.txt 29158 17656 S pts/16 00:00:00 python debug_memory.py data/stuff_122K.txt 29164 17756 S pts/16 00:00:00 python debug_memory.py data/stuff_123K.txt 29170 17856 S pts/16 00:00:00 python debug_memory.py data/stuff_124K.txt 29175 17952 S pts/16 00:00:00 python debug_memory.py data/stuff_125K.txt 29181 18056 S pts/16 00:00:00 python debug_memory.py data/stuff_126K.txt 29186 18152 S pts/16 00:00:00 python debug_memory.py data/stuff_127K.txt 29192 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_128K.txt 29198 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_129K.txt 29203 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_130K.txt 29209 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_131K.txt 29215 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_132K.txt 29220 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_133K.txt 29226 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_134K.txt 29231 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_135K.txt 29237 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_136K.txt 29243 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_137K.txt 29248 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_138K.txt 29254 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_139K.txt 29260 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_140K.txt 29265 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_141K.txt 29271 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_142K.txt 29276 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_143K.txt 29282 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_144K.txt 29288 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_145K.txt 29293 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_146K.txt 29299 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_147K.txt 29305 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_148K.txt 29310 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_149K.txt 29316 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_150K.txt 29321 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_151K.txt 29327 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_152K.txt 29333 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_153K.txt 29338 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_154K.txt 29344 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_155K.txt 29349 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_156K.txt 29355 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_157K.txt 29361 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_158K.txt 29366 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_159K.txt 29372 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_160K.txt 29378 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_161K.txt 29383 5460 S pts/16 00:00:00 python debug_memory.py data/stuff_162K.txt 29389 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_163K.txt 29394 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_164K.txt 29400 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_165K.txt 29406 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_166K.txt 29411 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_167K.txt 29417 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_168K.txt 29423 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_169K.txt 29428 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_170K.txt 29434 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_171K.txt 29439 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_172K.txt 29445 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_173K.txt 29451 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_174K.txt 29456 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_175K.txt 29463 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_176K.txt 29483 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_177K.txt 29489 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_178K.txt 29496 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_179K.txt 29501 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_180K.txt 29507 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_181K.txt 29512 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_182K.txt 29518 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_183K.txt 29524 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_184K.txt 29529 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_185K.txt 29535 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_186K.txt 29541 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_187K.txt 29546 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_188K.txt 29552 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_189K.txt 29557 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_190K.txt 29563 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_191K.txt 29569 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_192K.txt 29574 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_193K.txt 29580 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_194K.txt 29586 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_195K.txt 29591 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_196K.txt 29597 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_197K.txt 29602 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_198K.txt 29608 5456 S pts/16 00:00:00 python debug_memory.py data/stuff_199K.txt 29614 5452 S pts/16 00:00:00 python debug_memory.py data/stuff_200K.txt </code></pre> 有人能解释一下发生了什么吗？为什么我看到内存使用量增加了使用文件时&lt；128KB？在 我的完整测试环境位于： <a href="https://github.com/saltycrane/debugging-python-memory-usage/tree/50f73358c7a84a504333ce9c4071b0f3537bbc0f" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/tree/50f73358c7a84a504333ce9c4071b0f3537bbc0f</a> 我在Ubuntu 12.04上运行Python2.7.3。在 <h2>更新1</h2> 此问题并非特定于处理大小为128K的文件。我也一样结果将对象属性设置为与从中读取的大小相同的值文件。以下是更新后的代码： <pre><code>import sys import time class MyObj(object): def __init__(self, size_kb): self.att = ' ' * int(size_kb) * 1024 def myfunc(size_kb): mylist = [MyObj(size_kb) for x in xrange(100)] len(mylist) return [] def main(): size_kb = sys.argv[1] myfunc(size_kb) time.sleep(3600) if __name__ == '__main__': main() </code></pre> 运行这个脚本会得到类似的结果。更新的测试环境位于： <a href="https://github.com/saltycrane/debugging-python-memory-usage/tree/59b7ff61134dfc11c4195e9201b2c1728ed4fcce" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/tree/59b7ff61134dfc11c4195e9201b2c1728ed4fcce</a> <h2>更新2</h2> 我进一步简化了我的测试脚本：1。删除类并简单地创建一个字符串列表2。删除<code>myfunc()</code>并使用<code>del</code>删除{<cd6>}对象 <pre><code>import sys import time def main(): size_kb = sys.argv[1] mylist = [] for x in xrange(100): mystr = ' ' * int(size_kb) * 1024 mylist.<a href="https://www.cnpython.com/list/append" class="inner-link">append</a>(mystr) del mylist time.sleep(3600) if __name__ == '__main__': main() </code></pre> 我的简化脚本也给出了与原始脚本相似的结果。但是，如果不创建单独的字符串变量，我看不到记忆的增加。下面是一个脚本，它不创建增加内存： <pre><code>import sys import time def main(): size_kb = sys.argv[1] mylist = [] for x in xrange(100): mylist.append(' ' * int(size_kb) * 1024) del mylist time.sleep(3600) if __name__ == '__main__': main() </code></pre> 更新的测试环境位于： <a href="https://github.com/saltycrane/debugging-python-memory-usage/tree/423ca6a50dccbe32572a9d0dea1068ddcb06663b" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/tree/423ca6a50dccbe32572a9d0dea1068ddcb06663b</a> 更多问题： <ul> <li>其他人能复制我的结果吗？在</li> <li>是否期望<code>ps</code>看到内存增加？在</li> </ul> <h3>关于正在发生的事情的提示</h3> 我发现了一些关于“免费列表”的有趣信息它们可能与这个问题有关： <ul> <li><a href="http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm" rel="nofollow noreferrer">Why doesn't Python release the memory when I delete a large object?</a></li> <li><a href="https://stackoverflow.com/a/1316799/101911">How can I explicitly free memory in Python?</a></li> <li><a href="http://deeplearning.net/software/theano/tutorial/python-memory-management.html#internal-memory-management" rel="nofollow noreferrer">Python Memory Management — Theano v0.6rc3 documentation</a></li> </ul> 从最后一个链接： <blockquote> To speed-up memory allocation (and reuse) Python uses a number of lists for small objects. Each list will contain objects of similar size Indeed: if an item (of size x) is deallocated (freed by lack of reference) its location is not returned to Python’s global memory pool (and even less to the system), but merely marked as free and added to the free list of items of size x. If small objects memory is never freed, then the inescapable conclusion is that, like goldfishes, these small object lists only keep growing, never shrinking, and that the memory footprint of your application is dominated by the largest number of small objects allocated at any given point. </blockquote> <h2>更新3</h2> 我在更新2中过度简化了代码。在末尾添加<code>del mystr</code>行释放了内存。（参见：<a href="https://github.com/saltycrane/debugging-python-memory-usage/blob/dd058e4774802cae7cbfca520fb835ea46b645e8/debug_memory_leaks.py" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/blob/dd058e4774802cae7cbfca520fb835ea46b645e8/debug_memory_leaks.py</a>） 我更新了脚本，使之足够复杂来演示这个问题。以下代码中仍然存在该问题。最新的代码/环境位于此处：<a href="https://github.com/saltycrane/debugging-python-memory-usage/tree/fc0c8ce9ba621cb86b6abb93adf1b297a7c0230b" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/tree/fc0c8ce9ba621cb86b6abb93adf1b297a7c0230b</a> <pre><code>import gc import sys import time def main(): size_kb = sys.argv[1] mylist = [] for x in xrange(100): mystr = ' ' * int(size_kb) * 1024 mydict = {'mykey': mystr} mylist.append(mydict) del mystr del mydict del mylist gc.collect() time.sleep(3600) if __name__ == '__main__': main() </code></pre> 我还运行了一些其他环境的脚本。奇怪的结果是在一个干净的虚拟环境中运行。在这种情况下，内存会下降发生在260KB而不是128KB。见<a href="https://github.com/saltycrane/debugging-python-memory-usage/tree/52fbd5d57ff45affdcd70623ddb74fa1f1ffbbc2" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/tree/52fbd5d57ff45affdcd70623ddb74fa1f1ffbbc2</a> 环境： <ul> <li>Ubuntu 12.04 64位，系统Python 2.7.3：原始运行</li> <li>Ubuntu12.04 64位，Python3.3.0编译自源代码：相似的结果</li> <li>Scientific Linux6 64位，Python2.6.6：相似的结果</li> <li>Ubuntu12.04 64位，来自virtualenv的Python2.7.3：内存衰减发生在260KB而不是128KB</li> </ul> 更多参考资料： <ul> <li><a href="http://revista.python.org.ar/2/en/html/memory-fragmentation.html" rel="nofollow noreferrer">http://revista.python.org.ar/2/en/html/memory-fragmentation.html</a></li> <li><a href="http://www.evanjones.ca/python-memory.html" rel="nofollow noreferrer">http://www.evanjones.ca/python-memory.html</a></li> <li><a href="http://mail.python.org/pipermail/python-dev/2004-October/049480.html" rel="nofollow noreferrer">http://mail.python.org/pipermail/python-dev/2004-October/049480.html</a>（注：这是2004年的照片）</li> <li><a href="http://mail.python.org/pipermail/python-dev/2006-March/061991.html" rel="nofollow noreferrer">http://mail.python.org/pipermail/python-dev/2006-March/061991.html</a></li> <li><a href="http://www.evanjones.ca/memoryallocator/" rel="nofollow noreferrer">http://www.evanjones.ca/memoryallocator/</a></li> <li><a href="http://www.evanjones.ca/memory-allocator.pdf" rel="nofollow noreferrer">http://www.evanjones.ca/memory-allocator.pdf</a></li> <li><a href="http://hg.python.org/releasing/2.7.3/file/7bb96963d067/Objects/obmalloc.c" rel="nofollow noreferrer">http://hg.python.org/releasing/2.7.3/file/7bb96963d067/Objects/obmalloc.c</a> 在阅读了其中的一些之后，我看到了一个256KB的“竞技场大小”的引用。也许这是相关的？</li> </ul> <h2>更新4（基本解决）</h2> {a17}。 128KB是“内存分配函数”（malloc？）使用mmap而不是使用sbrk增加程序中断。有趣的是，阈值可以通过一个环境变量来改变。我运行了一个测试集将<code>MALLOC_MMAP_THRESHOLD_</code>环境变量不同的值和内存使用量的下降与该值匹配。有关结果，请参见此处： <a href="https://github.com/saltycrane/debugging-python-memory-usage/blob/97d93cd165a139a6b6f96720de63a92561dd2f05/output_debug_memory_leaks.py.txt" rel="nofollow noreferrer">https://github.com/saltycrane/debugging-python-memory-usage/blob/97d93cd165a139a6b6f96720de63a92561dd2f05/output_debug_memory_leaks.py.txt</a> 我仍然想知道它是否期望我的脚本行为为字符串值泄漏内存&lt；128KB。在 更多链接： <ul> <li><a href="http://man7.org/linux/man-pages/man3/mallopt.3.html" rel="nofollow noreferrer">mallopt(3) - Linux manual page</a>（来自施伦克）</li> <li><a href="http://pushingtheweb.com/2010/06/python-and-tcmalloc/" rel="nofollow noreferrer">Python memory management and TCMalloc | Pushing the Web</a></li> <li><a href="http://code.activestate.com/lists/python-list/637995/" rel="nofollow noreferrer">Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) « python-list « ActiveState List Archives</a></li> <li><a href="http://bugs.python.org/issue3526" rel="nofollow noreferrer">Issue 3526: Customized malloc implementation on SunOS and AIX - Python tracker</a></li> <li><a href="http://www.sourceware.org/ml/libc-alpha/2006-03/msg00033.html" rel="nofollow noreferrer">Make the mmap/brk threshold in malloc dynamic to improve performance</a></li> </ul> 注意：根据最后两个链接，有一个性能（速度）命中用mmap代替sbrk。在

0 条评论
分类：Python问答

默认排序时间排序

1 个回答

匿名 1天前

　擅长：python、mysql、java

我会调查垃圾收集。较大的文件可能会更频繁地触发垃圾回收，但小文件会被释放，但总体上会保持在某个阈值上。具体来说，打电话gc.收集（）然后打电话gc.get_引用程序（）来显示实例的存在。请参阅此处的Python文档： <a href="http://docs.python.org/2/library/gc.html?highlight=gc#gc.get_referrers" rel="nofollow">http://docs.python.org/2/library/gc.html?highlight=gc#gc.get_referrers</a> <h2>更新：</h2> 该问题与垃圾收集、命名空间和引用计数有关。您发布的bash脚本对垃圾收集器的行为提供了一个相当狭窄的视图。尝试一个更大的范围，你会看到模式在多少内存特定的范围将采取。例如，将bash For循环更改为更大的范围，例如：<code>seq 0 16 2056</code>。在 您注意到，如果<code>del mystr</code>，内存使用量会减少，因为您正在删除对它的任何引用。如果将mystr变量限制为它自己的函数，可能会出现类似的结果： <pre><code>def loopy(): mylist = [] for x in xrange(100): mystr = ' ' * int(size_kb) * 1024 mydict = {x: mystr} mylist.append(mydict) return mylist </code></pre> 与使用bash脚本相比，我认为使用内存分析器可以获得更多有用的信息。下面是几个使用<a href="http://pythonhosted.org/Pympler/index.html" rel="nofollow">Pympler</a>的示例。第一个版本与更新3中的代码类似： ^{pr2}$ 以及输出： <pre><code>$ python mem_test.py 256 begin: types | # objects | total size ======================= | =========== | ============= list | 957 | 97.44 KB str | 951 | 53.65 KB int | 118 | 2.77 KB wrapper_descriptor | 8 | 640 B weakref | 3 | 264 B member_descriptor | 2 | 144 B getset_descriptor | 2 | 144 B function (store_info) | 1 | 120 B cell | 2 | 112 B instancemethod | -1 | -80 B _sre.SRE_Pattern | -2 | -176 B tuple | -1 | -216 B dict | 2 | -1744 B empty list & dict: types | # objects | total size ======= | =========== | ============ list | 2 | 168 B str | 2 | 97 B int | 1 | 24 B after for loop: types | # objects | total size ======= | =========== | ============ str | 1 | 256.04 KB list | 0 | 848 B after deleting stuff: types | # objects | total size ======= | =========== | =============== list | -1 | -920 B str | -1 | -262181 B after garbage collection (collected: 0): types | # objects | total size ======= | =========== | ============ took a short nap after all that work: types | # objects | total size ======= | =========== | ============ create an empty list for some reason: types | # objects | total size ======= | =========== | ============ list | 1 | 72 B </code></pre> 注意，在for循环之后，str类的总大小为256kb，基本上与我传递给它的参数相同。在<code>del mystr</code>中显式删除对mystr的引用后，内存将被释放。在这之后，垃圾已经被捡走了，所以<code>gc.collect()</code>之后就没有进一步的减少了。在 下一个版本使用函数为字符串创建不同的命名空间。在 <pre><code>import gc import sys import time from pympler import tracker def loopy(): mylist = [] for x in xrange(100): mystr = ' ' * int(size_kb) * 1024 mydict = {x: mystr} mylist.append(mydict) return mylist tr = tracker.SummaryTracker() print 'begin:' tr.print_diff() size_kb = sys.argv[1] mylist = loopy() print 'after for loop:' tr.print_diff() del mylist print 'after deleting stuff:' tr.print_diff() collected = gc.collect() print 'after garbage collection (collected: %d):' % collected tr.print_diff() time.sleep(2) print 'took a short nap after all that work:' tr.print_diff() mylist = [] print 'create an empty list for some reason:' tr.print_diff() </code></pre> 最后，这个版本的输出： <pre><code>$ python mem_test_2.py 256 begin: types | # objects | total size ======================= | =========== | ============= list | 958 | 97.53 KB str | 952 | 53.70 KB int | 118 | 2.77 KB wrapper_descriptor | 8 | 640 B weakref | 3 | 264 B member_descriptor | 2 | 144 B getset_descriptor | 2 | 144 B function (store_info) | 1 | 120 B cell | 2 | 112 B instancemethod | -1 | -80 B _sre.SRE_Pattern | -2 | -176 B tuple | -1 | -216 B dict | 2 | -1744 B after for loop: types | # objects | total size ======= | =========== | ============ list | 2 | 1016 B str | 2 | 97 B int | 1 | 24 B after deleting stuff: types | # objects | total size ======= | =========== | ============ list | -1 | -920 B after garbage collection (collected: 0): types | # objects | total size ======= | =========== | ============ took a short nap after all that work: types | # objects | total size ======= | =========== | ============ create an empty list for some reason: types | # objects | total size ======= | =========== | ============ list | 1 | 72 B </code></pre> 现在，我们不必清理str，我想这个例子说明了为什么使用函数是个好主意。在一个命名空间中有一个大块的地方生成代码实际上是在阻止垃圾回收器完成它的工作。它不会进入你的房子，并开始假设这些东西是垃圾：）它必须知道这些东西是安全的收集。在 顺便说一句，埃文·琼斯很有趣

在Python中使用小于128KB的字符串时内存泄漏？

1 个回答

相关Python问题