作为multiprocessing
进程的一部分运行以下df.to_csv()
行时:
filtered_chunk.to_csv(os.path.join(output_dir, "filtered_{}".format(csv_filename)), mode='a+')
我得到以下错误。我不知道问题出在哪里(下面的代码片段是针对一个multiprocessing
进程的):
Process ForkPoolWorker-13:
Traceback (most recent call last):
File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/multiprocessing/process.py", line 108, in run
File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/multiprocessing/pool.py", line 109, in worker
<insert stacktrace for code I wrote...>
filtered_chunk.to_csv(os.path.join(output_dir, "filtered_{}".format(csv_filename)), mode='a+')
File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/pandas/core/generic.py", line 3170, in to_csv
File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/pandas/io/formats/csvs.py", line 185, in save
File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/pandas/io/common.py", line 418, in get_handle
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 914, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1342, in find_spec
File "<frozen importlib._bootstrap_external>", line 1314, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1443, in find_spec
File "<frozen importlib._bootstrap_external>", line 1483, in _fill_cache
OSError: [Errno 127] Key has expired: '/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8'
在上面的stacktrace的每个进程都失败之后,我会得到一个Bus error (core dumped)
打印输出
“密钥已过期”让我想到一些需要验证或提供有时间限制的访问(文件指针?访问服务器的密钥/票据?)。我在服务器上的tmux
会话中运行了脚本,因此我认为访问服务器不会是一个问题(一些错误127的谷歌搜索指出了装载问题,但我不认为这是一个问题)。由于脚本在get_handle
方法中失败,基本上返回一个文件句柄,所以文件指针似乎是问题所在,但我不确定原因
如果有帮助的话,我正在使用multiprocessing
来并行化一个函数,但是不同的进程写入不同的csv文件,所以这在这里不应该是一个问题
目前没有回答
相关问题 更多 >
编程相关推荐