使用multiprocessing.Managers构建简单的远程调度器
考虑以下代码：
服务器：
import sys
from multiprocessing.managers import BaseManager, BaseProxy, Process
def baz(aa) :
l = []
for i in range(3) :
l.append(aa)
return l
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
manager.register('solver', callable=baz, proxytype=MyProxy)
def serve_forever(server):
try :
server.serve_forever()
except KeyboardInterrupt:
pass
def runpool(n):
server = manager.get_server()
workers = []
for i in range(int(n)):
Process(target=serve_forever, args=(server,)).start()
if __name__ == '__main__':
runpool(sys.argv[1])
客户端：
import sys
from multiprocessing.managers import BaseManager, BaseProxy
import multiprocessing, logging
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
def main(args) :
SolverManager.register('solver')
m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
m.connect()
print m.solver(args[1])._getvalue()
if __name__ == '__main__':
sys.exit(main(sys.argv))
如果我只用一个进程运行服务器，比如说 python server.py 1
那么客户端就能正常工作。但是如果我启动两个进程（python server.py 2）来监听连接，就会出现一个很糟糕的错误：
$python client.py ping
Traceback (most recent call last):
File "client.py", line 24, in <module>
sys.exit(main(sys.argv))
File "client.py", line 21, in main
print m.solver(args[1])._getvalue()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 637, in temp
authkey=self._authkey, exposed=exp
File "/usr/lib/python2.6/multiprocessing/managers.py", line 894, in AutoProxy
incref=incref)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 700, in __init__
self._incref()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 750, in _incref
dispatch(conn, None, 'incref', (self._id,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 181, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 402, in incref
self.id_to_refcount[ident] += 1
KeyError: '7fb51084c518'
---------------------------------------------------------------------------
我的想法其实很简单。我想创建一个服务器，能够启动多个工作进程，这些进程共享同一个套接字，并独立处理请求。也许我用错工具了？
我的目标是建立一个三层结构，所有请求都通过一个http服务器处理，然后再分发到集群中的节点，最后从节点分发到工作进程，使用多进程管理器...
这里有一个公共服务器，每台机器上有一个节点，以及根据核心数量不同的多个工作进程... 我知道我可以使用更复杂的库，但对于这么简单的任务（我只是想做个原型），我只想用多进程库... 这样做可行吗，还是我应该直接考虑其他解决方案？我觉得我离成功很近了... 谢谢。
1 个回答
0