我尝试使用Python的多处理模块对多个样本并行运行分析。我使用pool.map_async
在一个元组上为参数(zipped_args
)生成函数(称为crispr_analysis
)。zipped_args
中的每个元组都不是空的,因为这可能导致多处理挂起。池完成后,它将挂起并无法继续执行脚本的其余部分。我知道crispr_analysis
在创建输出文件(用with
语句生成的)时结束;我可以浏览这些文件,它们是完整的。我从未看到对结果排序的调试消息,程序也从未终止。在
try:
# Use map_async and get with a large timeout
# to allow for keyboardInterrupts to be caught
# and handled with the try/except
timeout = max((9999, 600 * len(fastq_list)))
logging.debug("Setting timeout to %s seconds", timeout)
res = pool.map_async(crispr_analysis, zipped_args) # type: multiprocessing.pool.MapResult
pool.close()
results = res.get(timeout)
except (KeyboardInterrupt, ExitPool) as error: # Handle ctrl+c or custom ExitPool
pool.terminate()
# pool.join()
if isinstance(error, KeyboardInterrupt): # ctrl+c
sys.exit('\nkilled')
elif isinstance(error, ExitPool): # My way of handling SystemExits
sys.exit(error.msg)
else: # Shouldn't happen, but you know...
raise
except:
pool.terminate(); pool.join()
raise
else:
pool.join()
try:
logging.debug("Sorting results into alignments and summaries")
sort_start = time.time() # type: float
alignments, summaries = zip(*results) # type: Tuple[Tuple[alignment.Alignment]], Tuple[Dict[str, Any]]
logging.debug("Sorting results took %s seconds", round(time.time() - sort_start, 3))
except ExitPool as error: # Handle ExitPool calls for single-threaded map
sys.exit(error.msg)
有人知道为什么多处理挂起了吗?我该怎么解决它?在
额外信息:
crispr_analysis
返回一个元组和字典,它们要么是空的,要么根据输入有一定的长度pool.join()
语句,但没有用ExitPool
是一个错误,我抛出它来代替SystemExit
s来停止整个池;多处理通常会吞没{main
)这个分析程序从easy install入口脚本调用,其中入口点是启动多处理池的main
函数
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT:
'EdiTyper==1.0.0','console_scripts','EdiTyper'
__requires__ = 'EdiTyper==1.0.0'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('EdiTyper==1.0.0', 'console_scripts', 'EdiTyper')()
)
目前没有回答
相关问题 更多 >
编程相关推荐