从URL下载.html文件时出现超时错误

Question

在从url下载html页面时，我遇到了以下错误。你知道吗

Error: raise URLError(err) urllib2.URLError: <urlopen error [Errno
 10060] A connection attempt failed because the connected party did not
 properly respond after a period of time or established connection
 failed because connected host has failed to respond>

代码：

import urllib2 
hdr = {'User-Agent': 'Mozilla/5.0'}

for i,site in enumerate(urls[index]):
    print (site)
    req = urllib2.Request(site, headers=hdr)
    page = urllib2.build_opener(urllib2.HTTPCookieProcessor).open(req)
    page_content = page.read()
    with open(path_current+'/'+str(i)+'.html', 'w') as fid:
        fid.write(page_content)

我想这可能是由于一些代理设置或更改超时，但我不确定。请帮助，我手动检查的网址似乎打开得很好。你知道吗

Answer 1

好吧，既然你大部分时间都不这样，我可以推断你的人际网络可能很慢。 尝试按以下方式设置超时：

req = urllib2.Request(site, headers=hdr)
timeout_in_sec = 360
page = urllib2.build_opener(urllib2.HTTPCookieProcessor).open(req, timeout=timeout_in_sec)
page_content = page.read()