使用cxu Oracle和multiprocessing并发查询数据

Question

所有人

我试图从Oracle数据库访问和处理大量数据。所以我使用多处理模块生成50个进程来访问数据库。为了避免打开50个物理连接，我尝试使用cx_Oracle的会话池。代码如下所示。不过，我总是犯一个令人讨厌的错误。我知道cxu Oracle有酸洗问题，但我想我可以用一个全局变量来解决它。谁能帮忙吗。在

import sys
import cx_Oracle
import os
from multiprocessing import Pool

 # Read a list of ids from the input file
 def ReadList(inputFile):
        ............


def GetText(applId):
        global sPool
        connection = sPool.acquire()
        cur = connection.cursor()
        cur.prepare('Some Query')
        cur.execute(None, appl_id = applId)
        result = cur.fetchone()
        title = result[0]
        abstract = result[2].read()
        sa = result[3].read()
        cur.close()
        sPool.release(connection)
        return (title, abstract, sa)
if __name__=='__main__':
        inputFile = sys.argv[1]
        ids = ReadList(inputFile)
        dsn = cx_Oracle.makedsn('xxx', ...)
        sPool=cx_Oracle.SessionPool(....., min=1, max=10, increment=1)
        pool = Pool(10)
        results = pool.map(GetText, ids)


Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 282, in _handle_results
task = get()
UnpicklingError: NEWOBJ class argument has NULL tp_new

Answer 1

首先，您的代码会导致错误“NameError:global name‘sPool’is not defined”，因此sPool=cx_Oracle.SessionPool(....., min=1, max=10, increment=1)必须位于def GetText(applId):之上

对我来说，在将from multiprocessing import Pool更改为from multiprocessing.dummy import Pool并将参数threaded=True添加到cx调用之后，这段代码开始正常工作_Oracle.SessionPool作为sPool=cx_Oracle.SessionPool(....., min=1, max=10, increment=1, threaded=True)

Answer 2

你希望50个进程如何使用相同的进程内管理数据库连接（池）？！在