Django内存泄漏

Question

我正在开发一个使用 Django ORM 的 Python 应用程序，主要是用它来管理数据库，但我遇到了一个很大的内存问题。我发现导致这个问题的部分是：

ports_list_save = []
    for host in results['hosts']:
        for protocol in results['hosts'][host]['protocols']:
            for port in results['hosts'][host]['protocols'][protocol]:
                current_port = history.Port(number=int(port), 
                                            protocol=protocol, 
                                            state=results['hosts'][host]['protocols'][protocol][port]['state'], 
                                            service='', 
                                            version='', 
                                            address=history.Ip.objects.get(scan=self.scan, address=host))
                ports_list_save.append(current_port)
    history.Port.objects.bulk_create(ports_list_save)

这部分在处理154个主机和每个主机150个端口（总共23000个对象）时运行得很好，但现在我尝试用1000个端口时，我的电脑内存每次都撑爆。

还有一点，我并没有在调试模式下运行 Django，所以内存问题并不是来自于 django.db.backends.postgresql_psycopg2.base.DatabaseWrapper。

Answer 1

我也遇到了同样的问题，最后找到了解决办法：

class BulkCreateManager(object):

    model = None
    chunk_size = None
    instances = None

    def __init__(self, model, chunk_size=None, *args):
        self.model = model
        self.chunk_size = chunk_size
        self.instances = []

    def append(self, instance):
        if self.chunk_size and len(self.instances) > self.chunk_size:
            self.create()
            self.instances = []

        self.instances.append(instance)

    def create(self):
        self.model.objects.bulk_create(self.instances)



ports_list_save = BulkCreateManager(history.Port, 23000)
for host in results['hosts']:
    for protocol in results['hosts'][host]['protocols']:
        for port in results['hosts'][host]['protocols'][protocol]:
            current_port = history.Port(number=int(port), 
                                        protocol=protocol, 
                                        state=results['hosts'][host]['protocols'][protocol][port]['state'], 
                                        service='', 
                                        version='', 
                                        address=history.Ip.objects.get(scan=self.scan, address=host))
            ports_list_save.append(current_port)

ports_list_save.create()

Answer 2

如果你有很多数据，可能还是需要分批加载和处理这些数据，可以试试这个方法：

CHUNK_SIZE = 23000
ports_list_save = []
for host in results['hosts']:
    for protocol in results['hosts'][host]['protocols']:
        for port in results['hosts'][host]['protocols'][protocol]:
            current_port = history.Port(number=int(port), 
                                        protocol=protocol, 
                                        state=results['hosts'][host]['protocols'][protocol][port]['state'], 
                                        service='', 
                                        version='', 
                                        address=history.Ip.objects.get(scan=self.scan, address=host))
            ports_list_save.append(current_port)
            if len(ports_list_save) > CHUNK_SIZE:
                history.Port.objects.bulk_create(ports_list_save)
                ports_list_save = []
if ports_list_save:   
    history.Port.objects.bulk_create(ports_list_save)