pythonluigi意外死亡,退出代码为11

2024-04-25 05:48:59 发布

您现在位置:Python中文网/ 问答频道 /正文

我有一个带有luigi的数据管道,如果我把一个worker放到任务中,它就可以正常工作。但是,如果我放置了>;1个worker,那么它将在一个包含2个依赖项的阶段中死亡(意外地退出代码为-11)。代码相当复杂,因此很难给出最小的示例。问题的要点是我正在用gensim做以下事情:

  1. 从一些文本中建立词典。在
  2. 根据所述文本和词典建立语料库(需要(1))。在
  3. 从语料库和词典中训练LDA模型(需要(1)和(2))。在

由于某些原因,每次我放置多个工人时,步骤(3)都会崩溃,即使(1)和(2)已经完成。。。在

任何帮助将不胜感激!在

编辑:这里是一个日志记录信息的示例。TrainLDA是任务(3)。之后还有两项任务需要TrainLDA。所有先前的任务都正确完成。我用TrainLDA的参数替换了...,这样输出就更可读了。附加信息只是print我们用来帮助我们知道发生了什么的声明。在

德布

UG: Pending tasks: 3
DEBUG: Asking scheduler for work...
INFO: [pid 28851] Worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825) running   TrainLDA(...)
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
==============================
Corriendo LDA de spanish con nivel de limpieza stopwords
==============================
Número de tópicos: 40
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
INFO: Worker task TrainLDA(...) died unexpectedly with exit code -11
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: There are 2 pending tasks possibly being run by other workers
INFO: There are 2 pending tasks unique to this worker
INFO: Worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825) was stopped. Shutting down Keep-Alive thread

Tags: runinfohostlocalusernamepidaretasks