出于某种原因,我找不到在Queue
任何地方执行此操作的一般示例(即使Python的doc示例也不会生成多个进程),所以这是经过10次尝试后我才开始工作的内容:
def add_helper(queue, arg1, arg2): # the func called in child processes
ret = arg1 + arg2
queue.put(ret)
def multi_add(): # spawns child processes
q = Queue()
processes = []
rets = []
for _ in range(0, 100):
p = Process(target=add_helper, args=(q, 1, 2))
processes.append(p)
p.start()
for p in processes:
ret = q.get() # will block
rets.append(ret)
for p in processes:
p.join()
return rets
Queue
是一个线程安全的阻塞队列,可用于存储子进程的返回值。因此,您必须将队列传递给每个进程。一些不太明显的是,你们必须get()
从队列你之前join
的Process
ES否则队列已满,并且块一切。
针对那些面向对象的人的更新(在Python 3.4中测试):
from multiprocessing import Process, Queue
class Multiprocessor():
def __init__(self):
self.processes = []
self.queue = Queue()
@staticmethod
def _wrapper(func, queue, args, kwargs):
ret = func(*args, **kwargs)
queue.put(ret)
def run(self, func, *args, **kwargs):
args2 = [func, self.queue, args, kwargs]
p = Process(target=self._wrapper, args=args2)
self.processes.append(p)
p.start()
def wait(self):
rets = []
for p in self.processes:
ret = self.queue.get()
rets.append(ret)
for p in self.processes:
p.join()
return rets
# tester
if __name__ == "__main__":
mp = Multiprocessor()
num_proc = 64
for _ in range(num_proc): # queue up multiple tasks running `sum`
mp.run(sum, [1, 2, 3, 4, 5])
ret = mp.wait() # get all results
print(ret)
assert len(ret) == num_proc and all(r == 15 for r in ret)
multiprocessing.Queue
,而不是Manager
这里。使用aManager
需要产生一个全新的过程,而当aQueue
这样做时,这是过大的。