我试着用urllib.request.urlretrieve与多处理模块一起下载一些文件并对其进行处理。但是,每次我尝试运行我的程序时,都会出现错误:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "./thumb.py", line 13, in download_and_convert
filename, headers = urlretrieve(url)
File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 463, in open
response = self._open(req, data)
File "/usr/lib/python3.4/urllib/request.py", line 486, in _open
'unknown_open', req)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 1252, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: http>
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./thumb.py", line 27, in <module>
pool.map(download_and_convert, enumerate(csvr))
File "/usr/lib/python3.4/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
urllib.error.URLError: <urlopen error unknown url type: http>
它似乎阻塞的url是http://phytoimages.siu.edu/users/vitt/10_27_06_2/Equisetumarvense.JPG
。这是我的代码:
我不知道为什么会出现这个错误。可能是因为我正在使用多重处理吗?如果有人能帮助我,我将不胜感激。在
编辑:这是它尝试处理的第一个url,如果我更改它,它不会更改错误。在
检查此代码段
如您所见,urllib奇怪地解释了“:”字符。巧合的是你的程序挂在你身上。在
尝试一下
urllib.parse.urlencode()
这会让你走上正轨。在在一些评论的帮助下,我找到了解决办法。问题似乎是csv模块在字节顺序标记(BOM)上跳闸。我可以通过按建议使用
encoding='utf-8-sig'
打开文件来修复它。在相关问题 更多 >
编程相关推荐