如何在数百万Django模型对象之间循环，而不会出现超出范围或其他错误

Question

我在一个Postgres数据库中有数百万个对象，我需要一次将其中200个对象的数据发送到一个API，这将为我提供额外的信息（API一次最多只能处理200个元素）。我试过几种策略。第一个策略最终导致我的脚本被杀死，因为它使用了太多内存。下面的尝试效果更好，但我得到了以下错误：django.db.utils.DataError:bigint超出范围。此错误发生在“start”变量达到42000时。完成这项任务更有效的方法是什么？多谢各位

articles_to_process = Article.objects.all() # This will be in the millions
dois = articles_to_process.values_list('doi', flat=True) # These are IDs of articles

start = 0
end = 200 # The API to which I will send IDs can only return up to 200 records at a time.
number_of_dois = dois.count()
times_to_loop = (number_of_dois / 200) + 1

while times_to_loop > 0:
     times_to_loop = times_to_loop - 1
     chunk = dois[start:end]
     doi_string = ', '.join(chunk)
     start = start + 200
     end = end + 200

     [DO API CALL, GET DATA FOR EACH ARTICLE, SAVE THAT DATA TO ARTICLE]

Answer 1

考虑使用{a1}：

chunk_size = 200
counter = 0
idx = []
for article_id in dois.iterator(chunk_size):
    counter += 1
    idx.append(str(article_id))
    if counter >= chunk_size:
        doi_string = ', '.join(idx)
        idx = []
        counter = 0
        # DO API CALL, GET DATA FOR EACH ARTICLE, SAVE THAT DATA TO ARTICLE