将referer中的响应对象带入parse\u item callb

Question

问题

我正在尝试爬网一个像YouTube这样的网站，它有一个包含大量视频的列表和一个指向单个视频的链接。我要做的是在使用parse\u item（）进入特定视频之前获取视频的缩略图。你知道吗

问题是我不知道如何将“列表视图”的响应对象引入parse\u item（）函数。我知道您可以使用process\u request截获请求，并向请求对象插入一个meta，但我不知道如何获得列表视图响应。你知道吗

这个问题有不同的解决方法吗？你知道吗

我的代码：

import re
import datetime

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import Selector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

from ..items import ExampleItem


class ExampleSpider(CrawlSpider):
    """
    Crawler for: www.example.com
    """

    name = "example"

    allowed_domains = ['www.example.com']

    start_urls = ['http://www.example.com']

    rules = (
        Rule(SgmlLinkExtractor(
            restrict_xpaths=["//div[@class='pagination']"]
        )),

        Rule(SgmlLinkExtractor(
            restrict_xpaths=["//ul[@class='list']"],
            deny=['/user/'],
        ), callback='parse_item', process_request='parent_url')
    )

    def parent_url(self, request):

        request.meta['parent_page'] = '' #  Get the parent response somehow?
        return request

    def parse_item(self, response):

        sel = Selector(response)

        item = ExampleItem()

        duration = sel.css('.video span::text')[0].extract()

        item['title'] = sel.css('.title::text')[0].extract()
        item['description'] = sel.xpath('//div[@class="description"]/text()').extract()
        item['duration'] = self._parse_duration(duration)
        item['link'] = response.url

        return item

    def _parse_duration(self, string):
        """
        Parse the duration field for times
        return Datetime object
        """

        if len(string) > 20:
            return datetime.datetime.strptime(string, '%H hours %M min %S sec').time()

        if '60 min' in string:
            string.replace('60 min', '01 hours 00 min')
            return datetime.datetime.strptime(string, '%H hours %M min %S sec')

        return datetime.datetime.strptime(string, '%M min %S sec').time()

Answer 1

我假设您想知道从中提取链接（请求）的URL。你知道吗

您可以重写方法_requests_to_follow，以便传递请求的源页面：

    def _requests_to_follow(self, response):
        for req in super(ExampleSpider, self)._requests_to_follow(response):
            req.meta['parent_page'] = response.url
            yield req