擅长:python、mysql、java
<p>第一点:python<code>db-api.cursor</code>是一个迭代器,因此除非您真的需要</strong>一次将整个批加载到内存中,否则您可以从使用此功能开始,即不要:</p>
<pre><code>cursor.execute("SELECT * FROM mytable")
rows = cursor.fetchall()
for row in rows:
do_something_with(row)
</code></pre>
<p>你可以:</p>
<pre><code>cursor.execute("SELECT * FROM mytable")
for row in cursor:
do_something_with(row)
</code></pre>
<p>如果您的db connector的实现仍然不能正确地使用这个特性,那么现在就应该为这个组合添加LIMIT和OFFSET:</p>
<pre><code># py2 / py3 compat
try:
# xrange is defined in py2 only
xrange
except NameError:
# py3 range is actually p2 xrange
xrange = range
cursor.execute("SELECT count(*) FROM mytable")
count = cursor.fetchone()[0]
batch_size = 42 # whatever
for offset in xrange(0, count, batch_size):
cursor.execute(
"SELECT * FROM mytable LIMIT %s OFFSET %s",
(batch_size, offset))
for row in cursor:
do_something_with(row)
</code></pre>