我有一个带有luigi
的数据管道,如果我把一个worker放到任务中,它就可以正常工作。但是,如果我放置了>;1个worker,那么它将在一个包含2个依赖项的阶段中死亡(意外地退出代码为-11)。代码相当复杂,因此很难给出最小的示例。问题的要点是我正在用gensim
做以下事情:
由于某些原因,每次我放置多个工人时,步骤(3)都会崩溃,即使(1)和(2)已经完成。。。在
任何帮助将不胜感激!在
编辑:这里是一个日志记录信息的示例。TrainLDA是任务(3)。之后还有两项任务需要TrainLDA。所有先前的任务都正确完成。我用TrainLDA的参数替换了...
,这样输出就更可读了。附加信息只是print
我们用来帮助我们知道发生了什么的声明。在
德布
UG: Pending tasks: 3
DEBUG: Asking scheduler for work...
INFO: [pid 28851] Worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825) running TrainLDA(...)
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
==============================
Corriendo LDA de spanish con nivel de limpieza stopwords
==============================
Número de tópicos: 40
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
INFO: Worker task TrainLDA(...) died unexpectedly with exit code -11
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: There are 2 pending tasks possibly being run by other workers
INFO: There are 2 pending tasks unique to this worker
INFO: Worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825) was stopped. Shutting down Keep-Alive thread
目前没有回答
相关问题 更多 >
编程相关推荐