Python多进程：子进程向父进程发送变量
我有一个程序池，它会对一组IP地址运行一个函数，并返回一个0到99之间的整数。子进程的数量是根据IP地址的数量来决定的。这个程序的想法是，父进程会创建一个字典，字典的键是IP地址，值是对应的整数。
我需要子进程把这个数字报告回父进程，这样父进程就可以把它保存到一个序列化的文件里。把这个整数变量设为全局变量并没有成功。
从一个序列化的文件中读取为字典：
infile = open(server_file, 'rb')
servers = pickle.load(infile)
infile.close()
结果：
servers = {'10.116.35.141': ''}
这个函数：
def check(server):
response = urllib2.urlopen("http://"+server+"/avicapture.html")
tall = response.read() # puts the data into a string
html = tall.rstrip()
match = re.search('.*In Progress \((.*)%\).*', html)
if match:
global temp
global results
temp = match.group(1)
results = temp
servers[server] = temp # not setting key to temp in parent process like I want.
程序池：
p = Pool(len(servers))
p.map(check, servers)
p.close()
p.join()
我希望在父进程中得到的结果：
servers = {'10.116.35.141': 'integer from child'}
dano：
现在它在工作了。算是吧。以下是当前的状态：
从一个字典开始：
servers = { '10.116.35.141': '' }
def check(server):
response = urllib2.urlopen("http://"+server+"/avicapture.html")
tall = response.read() # puts the data into a string
html = tall.rstrip()
match = re.search('.*In Progress \((.*)%\).*', html)
global temp
if match:
global temp
global results
temp = match.group(1)
results = temp
servers[server] = temp
print servers
return str(temp)
print("Results: " +results)
这是程序池的情况：
if __name__ == "__main__":
print "Print Servers:\n"+str(servers)
print "Starting pool"
p = Pool(len(servers))
print "p.map(check, servers)"
results = p.map(check, servers)
p.close()
p.join()
d = dict(ent for ent in zip(servers, results))
print "d = dict(ent for ent in zip(servers, results))\nResults in.."
print(d)
servers = d
print "Translation into servers dict:"
print(servers)
这得到了我想要的结果：
Print Servers:
{'10.116.35.141': ''}
Starting pool
p.map(check, servers)
{'10.116.35.141': '96'}
Server: 10.116.35.141 ... 96%
d = dict(ent for ent in zip(servers, results))
Results in..
{'10.116.35.141': '96'}
Translation into servers dict:
{'10.116.35.141': '96'}
问题出现在我有一个包含多个键的字典时。例如：
servers = { '10.116.35.141': '', '10.116.35.1': '' }
添加额外键的方式是：
input = raw_input("Add IP: ")
if input in servers:
print "Server already in servers list."
elif input == "":
print "No input."
else:
global servers
servers[input] = ""
我遇到了StringIO错误：
Print Servers:
{'192.168.91.236': '', '10.116.35.141': ''}
Starting pool
p.map(check, servers)
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 71, in worker
put((job, i, result))
File "/usr/lib64/python2.6/multiprocessing/queues.py", line 366, in put
return send(obj)
PicklingError: Can't pickle <type 'cStringIO.StringO'>: attribute lookup cStringIO.StringO failed
{'192.168.91.236': '', '10.116.35.141': '1'}
Traceback (most recent call last):
File "./bsdae", line 233, in <module>
results = p.map(check, servers)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
UnboundLocalError: local variable 'message' referenced before assignment
1 个回答
2
pool.map 实际上允许你直接返回结果：
def check(val):
return val + 5
if __name__ == "__main__":
servers = [1,3,4,5]
p = Pool(len(servers))
results = p.map(check, servers)
p.close()
p.join()
print(results)
输出：
[6, 8, 9, 10]
你可以让你的 check 函数 return 结果，这样结果就会出现在 map 调用返回的列表里。
所以如果你有：
servers = ['1.1.1.1', '2.2.2.2', '3.3.3.3']
results = p.map(check, servers) # Let's say this returns [1,2,3]
你可以像这样得到你想要的字典：
d = dict(ent for ent in zip(servers, results))
print(d)
输出：
{'3.3.3.3': 3, '1.1.1.1': 1, '2.2.2.2': 2}