Scrapy:错误：蜘蛛错误处理

Question

我是python&scrapy的新手。 我试图运行现有代码，但每个地址都出现此错误：

>     2015-07-02 01:52:19 [scrapy] DEBUG: Crawled (200) <GET http://www.tripadvisor.com/ShowUserReviews-g187147-d197524-r281927613-Hotel_Mirific_Opera-Paris_Ile_de_France.html>
> (referer:
> http://www.tripadvisor.com/Hotel_Review-g187147-d197524-Reviews-Hotel_Mirific_Opera-Paris_Ile_de_France.html)2015-07-02
> 01:52:19 
>     [scrapy] ERROR: Spider error processing <GET http://www.tripadvisor.com/ShowUserReviews-g187147-d197524-r281927613-Hotel_Mirific_Opera-Paris_Ile_de_France.html>
> (referer:
> http://www.tripadvisor.com/Hotel_Review-g187147-d197524-Reviews-Hotel_Mirific_Opera-Paris_Ile_de_France.html)
> 
    > Traceback (most recent call last):   File
    > "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line
    > 102, in iter_errback
    >     yield next(it)   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py",
    > line 28, in process_spider_output
    >     for x in result:   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py",
    > line 22, in <genexpr>
    >     return (_set_referer(r) for r in result or ())   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py",
    > line 37, in <genexpr>
    >     return (r for r in result or () if _filter(r))   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py",
    > line 54, in <genexpr>
    >     return (r for r in result or () if _filter(r))   File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/crawl.py", line
    > 67, in _parse_response
    >     cb_res = callback(response, **cb_kwargs) or ()   File "/home/talmosko/Documents/scrapy/tripAdvisor/spiders/tripAdvisor.py",
    > line 30, in parse_item
    >      item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()[0].encode('ascii',
    > errors='ignore')
    > 
    > IndexError: list index out of range

这是我的代码： http://pastebin.com/XzM5DrDD

怎么了？好像斯皮德没有得到答案。。

谢谢！

Answer 1

您试图访问一个不存在的元素，错误在这行

item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()[0].encode('ascii', errors='ignore')

可能的

item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()

是空的，您正在尝试访问第一个元素。你有两个选择：

• 修改选择器，使其始终返回一些数据，最好使用Scrapy shell测试它
• Try and catch the IndexError