任务失败后如何将其放回队列？

Question

我有一个脚本，大概长这样：

#!/usr/bin/env python
# encoding: utf-8

import time, random, os, multiprocessing

def main():
    NPROCESSES = 5
    pool = multiprocessing.Pool(processes=NPROCESSES)

    a = [1,2,3,4,5,6,7,8,9,0]
    for _ in pool.imap_unordered(do_task, a):
        pass

def do_task(n):
    try:
        might_crash(n)
    except Hell, e:
        print e, " crashed."

def might_crash(n):
    time.sleep(3*random.random())
    if random.randrange( 3 ) == 0:
        raise Hell(n)
    print n

class Hell(Exception):
    pass  

if __name__=="__main__":    
    main()

这个脚本通常会打印出'a'的值，但might_crash()这个函数会随机抛出一个错误。

我想要捕捉这些错误，并把当前的do_task()任务放回队列，以便稍后再试。

如果这个任务失败了，我该怎么把它放回队列呢？

Answer 1

你可以从 do_task 收集结果，检查哪些结果是 Hell 的实例，把这些任务放进一个列表 new_tasks 里，然后循环执行，直到没有 new_tasks 为止：

import time
import random
import os
import multiprocessing as mp

def main():
    NPROCESSES = 5
    pool=mp.Pool(NPROCESSES)
    a = [1,2,3,4,5,6,7,8,9,0]
    new_tasks=a
    while new_tasks:
        a=new_tasks
        new_tasks=[]
        for result in pool.imap_unordered(do_task, a):
            if isinstance(result,Hell):
                new_tasks.append(result.args[0])
            else:
                print(result)

def do_task(n):
    try:
        result=might_crash(n)
    except Hell as e:        
        print("{0} crashed.".format(e.args[0]))
        result=e
    return result

def might_crash(n):
    time.sleep(3*random.random())
    if random.randrange( 3 ) == 0:
        raise Hell(n)
    return '{0} done'.format(n)

class Hell(Exception):
    pass  

if __name__=="__main__":    
    main()

会产生

1 done
6 crashed.
4 done
7 crashed.
5 done
9 done
3 done
2 crashed.
8 done
0 crashed.
0 crashed.
2 done
7 crashed.
6 done
0 done
7 done