我正在尝试获取App Store > Business的内容:
import requests
from lxml import html
page = requests.get("https://itunes.apple.com/in/genre/ios-business/id6000?mt=8")
tree = html.fromstring(page.text)
flist = []
plist = []
for i in range(0, 100):
app = tree.xpath("//div[@class='column first']/ul/li/a/@href")
ap = app[0]
page1 = requests.get(ap)
当我用(0,2)
尝试range
时,它起作用,但是当我将range
放入100
s时,它显示以下错误:
Traceback (most recent call last):
File "/home/preetham/Desktop/eg.py", line 17, in <module>
page1 = requests.get(ap)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max retries exceeded with url: /in/app/adobe-reader/id469337564?mt=8 (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)
这里发生的是itunes服务器拒绝您的连接(您在短时间内从同一ip地址发送了太多请求)
错误跟踪具有误导性,它应该类似于“无法建立连接,因为目标计算机主动拒绝它”。
Github上的about python.requests lib存在问题,请查看here
要解决此问题(与其说是一个问题,不如说是一个误导调试跟踪的问题),您应该捕获与连接相关的异常,如下所示:
另一种解决此问题的方法是,如果使用足够的时间间隔向服务器发送请求,则可以通过python中的
sleep(timeinsec)
函数来实现(不要忘记导入sleep)总之请求是很棒的python库,希望能解决你的问题。
只需使用
requests'
功能:这将
GET
URL并在requests.exceptions.ConnectionError
的情况下重试3次。backoff_factor
将有助于在尝试之间应用延迟,以避免在周期性请求配额的情况下再次失败。看看^{} ,它有许多选项可以简化重试。
就这么做
粘贴以下代码代替
page = requests.get(url)
:不客气:)
相关问题 更多 >
编程相关推荐