在Python中使用小于128KB的字符串时内存泄漏?

2024-05-10 01:28:26 发布

您现在位置:Python中文网/ 问答频道 /正文

原标题:内存泄漏打开文件<;128KB在Python中?在

原始问题

在运行Python脚本时,我看到了内存泄漏。这是我的剧本:

import sys
import time


class MyObj(object):
    def __init__(self, filename):
        with open(filename) as f:
            self.att = f.read()


def myfunc(filename):
    mylist = [MyObj(filename) for x in xrange(100)]
    len(mylist)
    return []


def main():
    filename = sys.argv[1]
    myfunc(filename)
    time.sleep(3600)


if __name__ == '__main__':
    main()

main函数调用myfunc(),它创建一个由100个对象组成的列表,每个对象都打开并 读一个文件。从myfunc()返回后,我希望从100项列表和 从读取要释放的文件,因为它们不再被引用。但是,当我 使用ps命令检查内存使用情况,Python进程使用大约10000 KB 内存比从第12行和第13行注释掉的脚本运行的Python进程多。在

奇怪的是,内存泄漏(如果是这样的话)似乎只会发生 对于128KB大小的文件。我创建了一个bash脚本来运行这个脚本,文件范围 大小从1KB到200KB,当文件大小达到128KB时,内存增长停止。 下面是bash脚本:

^{pr2}$

下面是bash脚本的输出:

PID RSS S TTY TIME COMMAND
28471  5552 S pts/16   00:00:00 python debug_memory.py data/stuff_1K.txt
28477  5656 S pts/16   00:00:00 python debug_memory.py data/stuff_2K.txt
28483  5756 S pts/16   00:00:00 python debug_memory.py data/stuff_3K.txt
28488  5852 S pts/16   00:00:00 python debug_memory.py data/stuff_4K.txt
28494  5952 S pts/16   00:00:00 python debug_memory.py data/stuff_5K.txt
28499  6052 S pts/16   00:00:00 python debug_memory.py data/stuff_6K.txt
28505  6156 S pts/16   00:00:00 python debug_memory.py data/stuff_7K.txt
28511  6256 S pts/16   00:00:00 python debug_memory.py data/stuff_8K.txt
28516  6356 S pts/16   00:00:00 python debug_memory.py data/stuff_9K.txt
28522  6452 S pts/16   00:00:00 python debug_memory.py data/stuff_10K.txt
28527  6552 S pts/16   00:00:00 python debug_memory.py data/stuff_11K.txt
28533  6656 S pts/16   00:00:00 python debug_memory.py data/stuff_12K.txt
28539  6756 S pts/16   00:00:00 python debug_memory.py data/stuff_13K.txt
28544  6852 S pts/16   00:00:00 python debug_memory.py data/stuff_14K.txt
28550  6952 S pts/16   00:00:00 python debug_memory.py data/stuff_15K.txt
28555  7056 S pts/16   00:00:00 python debug_memory.py data/stuff_16K.txt
28561  7156 S pts/16   00:00:00 python debug_memory.py data/stuff_17K.txt
28567  7252 S pts/16   00:00:00 python debug_memory.py data/stuff_18K.txt
28572  7356 S pts/16   00:00:00 python debug_memory.py data/stuff_19K.txt
28578  7452 S pts/16   00:00:00 python debug_memory.py data/stuff_20K.txt
28584  7556 S pts/16   00:00:00 python debug_memory.py data/stuff_21K.txt
28589  7652 S pts/16   00:00:00 python debug_memory.py data/stuff_22K.txt
28595  7756 S pts/16   00:00:00 python debug_memory.py data/stuff_23K.txt
28600  7852 S pts/16   00:00:00 python debug_memory.py data/stuff_24K.txt
28606  7952 S pts/16   00:00:00 python debug_memory.py data/stuff_25K.txt
28612  8052 S pts/16   00:00:00 python debug_memory.py data/stuff_26K.txt
28617  8152 S pts/16   00:00:00 python debug_memory.py data/stuff_27K.txt
28623  8252 S pts/16   00:00:00 python debug_memory.py data/stuff_28K.txt
28629  8356 S pts/16   00:00:00 python debug_memory.py data/stuff_29K.txt
28634  8452 S pts/16   00:00:00 python debug_memory.py data/stuff_30K.txt
28640  8556 S pts/16   00:00:00 python debug_memory.py data/stuff_31K.txt
28645  8656 S pts/16   00:00:00 python debug_memory.py data/stuff_32K.txt
28651  8756 S pts/16   00:00:00 python debug_memory.py data/stuff_33K.txt
28657  8856 S pts/16   00:00:00 python debug_memory.py data/stuff_34K.txt
28662  8956 S pts/16   00:00:00 python debug_memory.py data/stuff_35K.txt
28668  9056 S pts/16   00:00:00 python debug_memory.py data/stuff_36K.txt
28674  9156 S pts/16   00:00:00 python debug_memory.py data/stuff_37K.txt
28679  9256 S pts/16   00:00:00 python debug_memory.py data/stuff_38K.txt
28685  9352 S pts/16   00:00:00 python debug_memory.py data/stuff_39K.txt
28691  9452 S pts/16   00:00:00 python debug_memory.py data/stuff_40K.txt
28696  9552 S pts/16   00:00:00 python debug_memory.py data/stuff_41K.txt
28702  9656 S pts/16   00:00:00 python debug_memory.py data/stuff_42K.txt
28707  9756 S pts/16   00:00:00 python debug_memory.py data/stuff_43K.txt
28713  9852 S pts/16   00:00:00 python debug_memory.py data/stuff_44K.txt
28719  9952 S pts/16   00:00:00 python debug_memory.py data/stuff_45K.txt
28724 10052 S pts/16   00:00:00 python debug_memory.py data/stuff_46K.txt
28730 10156 S pts/16   00:00:00 python debug_memory.py data/stuff_47K.txt
28739 10256 S pts/16   00:00:00 python debug_memory.py data/stuff_48K.txt
28746 10352 S pts/16   00:00:00 python debug_memory.py data/stuff_49K.txt
28752 10452 S pts/16   00:00:00 python debug_memory.py data/stuff_50K.txt
28757 10556 S pts/16   00:00:00 python debug_memory.py data/stuff_51K.txt
28763 10656 S pts/16   00:00:00 python debug_memory.py data/stuff_52K.txt
28769 10752 S pts/16   00:00:00 python debug_memory.py data/stuff_53K.txt
28774 10852 S pts/16   00:00:00 python debug_memory.py data/stuff_54K.txt
28780 10952 S pts/16   00:00:00 python debug_memory.py data/stuff_55K.txt
28786 11052 S pts/16   00:00:00 python debug_memory.py data/stuff_56K.txt
28791 11152 S pts/16   00:00:00 python debug_memory.py data/stuff_57K.txt
28797 11256 S pts/16   00:00:00 python debug_memory.py data/stuff_58K.txt
28802 11356 S pts/16   00:00:00 python debug_memory.py data/stuff_59K.txt
28808 11452 S pts/16   00:00:00 python debug_memory.py data/stuff_60K.txt
28814 11556 S pts/16   00:00:00 python debug_memory.py data/stuff_61K.txt
28819 11656 S pts/16   00:00:00 python debug_memory.py data/stuff_62K.txt
28825 11752 S pts/16   00:00:00 python debug_memory.py data/stuff_63K.txt
28831 11852 S pts/16   00:00:00 python debug_memory.py data/stuff_64K.txt
28836 11956 S pts/16   00:00:00 python debug_memory.py data/stuff_65K.txt
28842 12052 S pts/16   00:00:00 python debug_memory.py data/stuff_66K.txt
28847 12152 S pts/16   00:00:00 python debug_memory.py data/stuff_67K.txt
28853 12256 S pts/16   00:00:00 python debug_memory.py data/stuff_68K.txt
28859 12356 S pts/16   00:00:00 python debug_memory.py data/stuff_69K.txt
28864 12452 S pts/16   00:00:00 python debug_memory.py data/stuff_70K.txt
28871 12556 S pts/16   00:00:00 python debug_memory.py data/stuff_71K.txt
28877 12652 S pts/16   00:00:00 python debug_memory.py data/stuff_72K.txt
28883 12756 S pts/16   00:00:00 python debug_memory.py data/stuff_73K.txt
28889 12856 S pts/16   00:00:00 python debug_memory.py data/stuff_74K.txt
28894 12952 S pts/16   00:00:00 python debug_memory.py data/stuff_75K.txt
28900 13056 S pts/16   00:00:00 python debug_memory.py data/stuff_76K.txt
28906 13156 S pts/16   00:00:00 python debug_memory.py data/stuff_77K.txt
28911 13256 S pts/16   00:00:00 python debug_memory.py data/stuff_78K.txt
28917 13352 S pts/16   00:00:00 python debug_memory.py data/stuff_79K.txt
28922 13452 S pts/16   00:00:00 python debug_memory.py data/stuff_80K.txt
28928 13556 S pts/16   00:00:00 python debug_memory.py data/stuff_81K.txt
28934 13652 S pts/16   00:00:00 python debug_memory.py data/stuff_82K.txt
28939 13752 S pts/16   00:00:00 python debug_memory.py data/stuff_83K.txt
28945 13852 S pts/16   00:00:00 python debug_memory.py data/stuff_84K.txt
28951 13952 S pts/16   00:00:00 python debug_memory.py data/stuff_85K.txt
28956 14052 S pts/16   00:00:00 python debug_memory.py data/stuff_86K.txt
28962 14152 S pts/16   00:00:00 python debug_memory.py data/stuff_87K.txt
28967 14256 S pts/16   00:00:00 python debug_memory.py data/stuff_88K.txt
28973 14352 S pts/16   00:00:00 python debug_memory.py data/stuff_89K.txt
28979 14456 S pts/16   00:00:00 python debug_memory.py data/stuff_90K.txt
28984 14552 S pts/16   00:00:00 python debug_memory.py data/stuff_91K.txt
28990 14652 S pts/16   00:00:00 python debug_memory.py data/stuff_92K.txt
28996 14756 S pts/16   00:00:00 python debug_memory.py data/stuff_93K.txt
29001 14852 S pts/16   00:00:00 python debug_memory.py data/stuff_94K.txt
29007 14956 S pts/16   00:00:00 python debug_memory.py data/stuff_95K.txt
29012 15052 S pts/16   00:00:00 python debug_memory.py data/stuff_96K.txt
29018 15156 S pts/16   00:00:00 python debug_memory.py data/stuff_97K.txt
29024 15252 S pts/16   00:00:00 python debug_memory.py data/stuff_98K.txt
29029 15360 S pts/16   00:00:00 python debug_memory.py data/stuff_99K.txt
29035 15456 S pts/16   00:00:00 python debug_memory.py data/stuff_100K.txt
29040 15556 S pts/16   00:00:00 python debug_memory.py data/stuff_101K.txt
29046 15652 S pts/16   00:00:00 python debug_memory.py data/stuff_102K.txt
29052 15756 S pts/16   00:00:00 python debug_memory.py data/stuff_103K.txt
29057 15852 S pts/16   00:00:00 python debug_memory.py data/stuff_104K.txt
29063 15952 S pts/16   00:00:00 python debug_memory.py data/stuff_105K.txt
29069 16056 S pts/16   00:00:00 python debug_memory.py data/stuff_106K.txt
29074 16152 S pts/16   00:00:00 python debug_memory.py data/stuff_107K.txt
29080 16256 S pts/16   00:00:00 python debug_memory.py data/stuff_108K.txt
29085 16356 S pts/16   00:00:00 python debug_memory.py data/stuff_109K.txt
29091 16452 S pts/16   00:00:00 python debug_memory.py data/stuff_110K.txt
29097 16552 S pts/16   00:00:00 python debug_memory.py data/stuff_111K.txt
29102 16652 S pts/16   00:00:00 python debug_memory.py data/stuff_112K.txt
29108 16756 S pts/16   00:00:00 python debug_memory.py data/stuff_113K.txt
29113 16852 S pts/16   00:00:00 python debug_memory.py data/stuff_114K.txt
29119 16952 S pts/16   00:00:00 python debug_memory.py data/stuff_115K.txt
29125 17056 S pts/16   00:00:00 python debug_memory.py data/stuff_116K.txt
29130 17156 S pts/16   00:00:00 python debug_memory.py data/stuff_117K.txt
29136 17256 S pts/16   00:00:00 python debug_memory.py data/stuff_118K.txt
29141 17356 S pts/16   00:00:00 python debug_memory.py data/stuff_119K.txt
29147 17452 S pts/16   00:00:00 python debug_memory.py data/stuff_120K.txt
29153 17556 S pts/16   00:00:00 python debug_memory.py data/stuff_121K.txt
29158 17656 S pts/16   00:00:00 python debug_memory.py data/stuff_122K.txt
29164 17756 S pts/16   00:00:00 python debug_memory.py data/stuff_123K.txt
29170 17856 S pts/16   00:00:00 python debug_memory.py data/stuff_124K.txt
29175 17952 S pts/16   00:00:00 python debug_memory.py data/stuff_125K.txt
29181 18056 S pts/16   00:00:00 python debug_memory.py data/stuff_126K.txt
29186 18152 S pts/16   00:00:00 python debug_memory.py data/stuff_127K.txt
29192  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_128K.txt
29198  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_129K.txt
29203  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_130K.txt
29209  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_131K.txt
29215  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_132K.txt
29220  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_133K.txt
29226  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_134K.txt
29231  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_135K.txt
29237  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_136K.txt
29243  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_137K.txt
29248  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_138K.txt
29254  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_139K.txt
29260  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_140K.txt
29265  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_141K.txt
29271  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_142K.txt
29276  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_143K.txt
29282  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_144K.txt
29288  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_145K.txt
29293  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_146K.txt
29299  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_147K.txt
29305  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_148K.txt
29310  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_149K.txt
29316  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_150K.txt
29321  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_151K.txt
29327  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_152K.txt
29333  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_153K.txt
29338  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_154K.txt
29344  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_155K.txt
29349  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_156K.txt
29355  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_157K.txt
29361  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_158K.txt
29366  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_159K.txt
29372  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_160K.txt
29378  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_161K.txt
29383  5460 S pts/16   00:00:00 python debug_memory.py data/stuff_162K.txt
29389  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_163K.txt
29394  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_164K.txt
29400  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_165K.txt
29406  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_166K.txt
29411  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_167K.txt
29417  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_168K.txt
29423  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_169K.txt
29428  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_170K.txt
29434  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_171K.txt
29439  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_172K.txt
29445  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_173K.txt
29451  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_174K.txt
29456  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_175K.txt
29463  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_176K.txt
29483  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_177K.txt
29489  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_178K.txt
29496  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_179K.txt
29501  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_180K.txt
29507  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_181K.txt
29512  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_182K.txt
29518  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_183K.txt
29524  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_184K.txt
29529  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_185K.txt
29535  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_186K.txt
29541  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_187K.txt
29546  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_188K.txt
29552  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_189K.txt
29557  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_190K.txt
29563  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_191K.txt
29569  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_192K.txt
29574  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_193K.txt
29580  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_194K.txt
29586  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_195K.txt
29591  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_196K.txt
29597  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_197K.txt
29602  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_198K.txt
29608  5456 S pts/16   00:00:00 python debug_memory.py data/stuff_199K.txt
29614  5452 S pts/16   00:00:00 python debug_memory.py data/stuff_200K.txt

有人能解释一下发生了什么吗?为什么我看到内存使用量增加了 使用文件时<;128KB?在

我的完整测试环境位于: https://github.com/saltycrane/debugging-python-memory-usage/tree/50f73358c7a84a504333ce9c4071b0f3537bbc0f

我在Ubuntu 12.04上运行Python2.7.3。在

更新1

此问题并非特定于处理大小为128K的文件。我也一样 结果将对象属性设置为与从中读取的大小相同的值 文件。以下是更新后的代码:

import sys
import time


class MyObj(object):
    def __init__(self, size_kb):
        self.att = ' ' * int(size_kb) * 1024


def myfunc(size_kb):
    mylist = [MyObj(size_kb) for x in xrange(100)]
    len(mylist)
    return []


def main():
    size_kb = sys.argv[1]
    myfunc(size_kb)
    time.sleep(3600)


if __name__ == '__main__':
    main()

运行这个脚本会得到类似的结果。更新的测试环境位于: https://github.com/saltycrane/debugging-python-memory-usage/tree/59b7ff61134dfc11c4195e9201b2c1728ed4fcce

更新2

我进一步简化了我的测试脚本:1。删除类并简单地创建一个字符串列表2。删除myfunc()并使用del删除{}对象

import sys
import time

def main():
    size_kb = sys.argv[1]

    mylist = []
    for x in xrange(100):
        mystr = ' ' * int(size_kb) * 1024
        mylist.append(mystr)

    del mylist

    time.sleep(3600)

if __name__ == '__main__':
    main()

我的简化脚本也给出了与原始脚本相似的结果。 但是,如果不创建单独的字符串变量,我看不到 记忆的增加。下面是一个脚本,它不创建 增加内存:

import sys
import time

def main():
    size_kb = sys.argv[1]

    mylist = []
    for x in xrange(100):
        mylist.append(' ' * int(size_kb) * 1024)

    del mylist

    time.sleep(3600)

if __name__ == '__main__':
    main()

更新的测试环境位于: https://github.com/saltycrane/debugging-python-memory-usage/tree/423ca6a50dccbe32572a9d0dea1068ddcb06663b

更多问题:

  • 其他人能复制我的结果吗?在
  • 是否期望ps看到内存增加?在

关于正在发生的事情的提示

我发现了一些关于“免费列表”的有趣信息 它们可能与这个问题有关:

从最后一个链接:

To speed-up memory allocation (and reuse) Python uses a number of lists for small objects. Each list will contain objects of similar size

Indeed: if an item (of size x) is deallocated (freed by lack of reference) its location is not returned to Python’s global memory pool (and even less to the system), but merely marked as free and added to the free list of items of size x.

If small objects memory is never freed, then the inescapable conclusion is that, like goldfishes, these small object lists only keep growing, never shrinking, and that the memory footprint of your application is dominated by the largest number of small objects allocated at any given point.

更新3

我在更新2中过度简化了代码。在末尾添加del mystr行 释放了内存。 (参见:https://github.com/saltycrane/debugging-python-memory-usage/blob/dd058e4774802cae7cbfca520fb835ea46b645e8/debug_memory_leaks.py

我更新了脚本,使之足够复杂来演示这个问题。 以下代码中仍然存在该问题。 最新的代码/环境位于此处:https://github.com/saltycrane/debugging-python-memory-usage/tree/fc0c8ce9ba621cb86b6abb93adf1b297a7c0230b

import gc
import sys
import time


def main():
    size_kb = sys.argv[1]

    mylist = []
    for x in xrange(100):
        mystr = ' ' * int(size_kb) * 1024
        mydict = {'mykey': mystr}
        mylist.append(mydict)

    del mystr
    del mydict
    del mylist

    gc.collect()

    time.sleep(3600)


if __name__ == '__main__':
    main()

我还运行了一些其他环境的脚本。奇怪的结果是 在一个干净的虚拟环境中运行。在这种情况下,内存会下降 发生在260KB而不是128KB。见https://github.com/saltycrane/debugging-python-memory-usage/tree/52fbd5d57ff45affdcd70623ddb74fa1f1ffbbc2

环境:

  • Ubuntu 12.04 64位,系统Python 2.7.3:原始运行
  • Ubuntu12.04 64位,Python3.3.0编译自源代码:相似的结果
  • Scientific Linux6 64位,Python2.6.6:相似的结果
  • Ubuntu12.04 64位,来自virtualenv的Python2.7.3:内存衰减发生在260KB而不是128KB

更多参考资料:

更新4(基本解决)

{a17}。 128KB是“内存分配函数”(malloc?) 使用mmap而不是使用sbrk增加程序中断。 有趣的是,阈值可以通过一个环境变量来改变。 我运行了一个测试集将MALLOC_MMAP_THRESHOLD_环境变量 不同的值和内存使用量的下降与该值匹配。 有关结果,请参见此处: https://github.com/saltycrane/debugging-python-memory-usage/blob/97d93cd165a139a6b6f96720de63a92561dd2f05/output_debug_memory_leaks.py.txt

我仍然想知道它是否期望我的脚本行为 为字符串值泄漏内存<;128KB。在

更多链接:

注意:根据最后两个链接,有一个性能(速度)命中 用mmap代替sbrk。在


Tags: 内存pydebugimporttxt脚本datasize
2条回答

我会调查垃圾收集。较大的文件可能会更频繁地触发垃圾回收,但小文件会被释放,但总体上会保持在某个阈值上。具体来说,打电话gc.收集()然后打电话gc.get_引用程序()来显示实例的存在。请参阅此处的Python文档:

http://docs.python.org/2/library/gc.html?highlight=gc#gc.get_referrers

更新:

该问题与垃圾收集、命名空间和引用计数有关。您发布的bash脚本对垃圾收集器的行为提供了一个相当狭窄的视图。尝试一个更大的范围,你会看到模式在多少内存特定的范围将采取。例如,将bash For循环更改为更大的范围,例如:seq 0 16 2056。在

您注意到,如果del mystr,内存使用量会减少,因为您正在删除对它的任何引用。如果将mystr变量限制为它自己的函数,可能会出现类似的结果:

def loopy():
    mylist = []
    for x in xrange(100):
        mystr = ' ' * int(size_kb) * 1024
        mydict = {x: mystr}
        mylist.append(mydict)
    return mylist

与使用bash脚本相比,我认为使用内存分析器可以获得更多有用的信息。下面是几个使用Pympler的示例。第一个版本与更新3中的代码类似:

^{pr2}$

以及输出:

$ python mem_test.py 256
begin:
                  types |   # objects |    total size
======================= | =========== | =============
                   list |         957 |      97.44 KB
                    str |         951 |      53.65 KB
                    int |         118 |       2.77 KB
     wrapper_descriptor |           8 |     640     B
                weakref |           3 |     264     B
      member_descriptor |           2 |     144     B
      getset_descriptor |           2 |     144     B
  function (store_info) |           1 |     120     B
                   cell |           2 |     112     B
         instancemethod |          -1 |     -80     B
       _sre.SRE_Pattern |          -2 |    -176     B
                  tuple |          -1 |    -216     B
                   dict |           2 |   -1744     B
empty list & dict:
  types |   # objects |   total size
======= | =========== | ============
   list |           2 |    168     B
    str |           2 |     97     B
    int |           1 |     24     B
after for loop:
  types |   # objects |   total size
======= | =========== | ============
    str |           1 |    256.04 KB
   list |           0 |    848     B
after deleting stuff:
  types |   # objects |      total size
======= | =========== | ===============
   list |          -1 |      -920     B
    str |          -1 |   -262181     B
after garbage collection (collected: 0):
  types |   # objects |   total size
======= | =========== | ============
took a short nap after all that work:
  types |   # objects |   total size
======= | =========== | ============
create an empty list for some reason:
  types |   # objects |   total size
======= | =========== | ============
   list |           1 |     72     B

注意,在for循环之后,str类的总大小为256kb,基本上与我传递给它的参数相同。在del mystr中显式删除对mystr的引用后,内存将被释放。在这之后,垃圾已经被捡走了,所以gc.collect()之后就没有进一步的减少了。在

下一个版本使用函数为字符串创建不同的命名空间。在

import gc
import sys
import time
from pympler import tracker

def loopy():
    mylist = []
    for x in xrange(100):
        mystr = ' ' * int(size_kb) * 1024
        mydict = {x: mystr}
        mylist.append(mydict)
    return mylist


tr = tracker.SummaryTracker()
print 'begin:'
tr.print_diff()

size_kb = sys.argv[1]

mylist = loopy()

print 'after for loop:'
tr.print_diff()

del mylist

print 'after deleting stuff:'
tr.print_diff()

collected = gc.collect()
print 'after garbage collection (collected: %d):' % collected
tr.print_diff()

time.sleep(2)
print 'took a short nap after all that work:'
tr.print_diff()

mylist = []
print 'create an empty list for some reason:'
tr.print_diff()

最后,这个版本的输出:

$ python mem_test_2.py 256
begin:
                  types |   # objects |    total size
======================= | =========== | =============
                   list |         958 |      97.53 KB
                    str |         952 |      53.70 KB
                    int |         118 |       2.77 KB
     wrapper_descriptor |           8 |     640     B
                weakref |           3 |     264     B
      member_descriptor |           2 |     144     B
      getset_descriptor |           2 |     144     B
  function (store_info) |           1 |     120     B
                   cell |           2 |     112     B
         instancemethod |          -1 |     -80     B
       _sre.SRE_Pattern |          -2 |    -176     B
                  tuple |          -1 |    -216     B
                   dict |           2 |   -1744     B
after for loop:
  types |   # objects |   total size
======= | =========== | ============
   list |           2 |   1016     B
    str |           2 |     97     B
    int |           1 |     24     B
after deleting stuff:
  types |   # objects |   total size
======= | =========== | ============
   list |          -1 |   -920     B
after garbage collection (collected: 0):
  types |   # objects |   total size
======= | =========== | ============
took a short nap after all that work:
  types |   # objects |   total size
======= | =========== | ============
create an empty list for some reason:
  types |   # objects |   total size
======= | =========== | ============
   list |           1 |     72     B

现在,我们不必清理str,我想这个例子说明了为什么使用函数是个好主意。在一个命名空间中有一个大块的地方生成代码实际上是在阻止垃圾回收器完成它的工作。它不会进入你的房子,并开始假设这些东西是垃圾:)它必须知道这些东西是安全的收集。在

顺便说一句,埃文·琼斯很有趣

您可能只需点击linux内存分配器的默认行为。在

基本上Linux有两种分配策略,sbrk()用于小内存块,mmap()用于较大的内存块。sbrk()分配的内存块不容易返回到系统,而基于mmap()的内存块则可以(只需取消页面映射)。在

因此,如果分配的内存块大于libc中malloc()分配器决定在sbrk()和mmap()之间切换的值,就会看到这种效果。请参见mallopt()调用,尤其是MMAP_阈值(http://man7.org/linux/man-pages/man3/mallopt.3.html)。在

更新 回答您的额外问题:是的,如果内存分配器的工作方式与Linux上的libc分配器类似,那么您可能会以这种方式泄漏内存。如果改用Windows LowFragmentationHeap,它可能不会泄漏,这在AIX上类似,这取决于配置了哪个malloc。也许其他分配器(tcmalloc等)也可以解决这些问题。sbrk()速度非常快,但存在内存碎片问题。CPython对此无能为力,因为它没有压缩垃圾收集器,而是简单的引用计数。在

Python提供了一些减少缓冲区分配的方法,例如请参阅以下博客文章:http://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews/

相关问题 更多 >