如何从scrapy中的后续函数中获取数据

Question

我很难按照自己的意愿构建零碎的数据。我的蜘蛛从一个页面获取一些数据，然后按照该页面上的链接列表获取下一个页面的链接

    def parse_page(self, response):
 
        links = response.css(LINK_SELECTOR).extract()

        data = {
            'name': response.css(NAME_SELECTOR).extract_first(),
            'date': response.css(DATE_SELECTOR).extract(),
        }

        for link in links:
            next_link = response.urljoin(link)
            yield scrapy.Request(next_link, callback=self.parse_url, meta={'data': data})

    def parse_url(self, response):
        data = response.meta['data']
        data['url'] = response.css(a::attr(href)').get()
        yield data

我希望获得具有以下结构的数据：

{'name': name, 'date': date, 'url': [url1, url2, url3, url4]}

而不是

{'name': name, 'date': date, 'url': url1}
{'name': name, 'date': date, 'url': url2}
{'name': name, 'date': date, 'url': url3}
{'name': name, 'date': date, 'url': url4}

我尝试使用items，但不知道如何将数据从parse_url传递到parse_page函数。我该怎么做

提前谢谢

Answer 1

以下是实现这一目标的方法之一。有一个库inline_requests，它将帮助您获得预期的输出

import scrapy
from scrapy.crawler import CrawlerProcess
from inline_requests import inline_requests

class YellowpagesSpider(scrapy.Spider):
    name = "yellowpages"
    start_urls = ["https://www.yellowpages.com/san-francisco-ca/mip/honey-honey-cafe-crepery-4752771"]

    @inline_requests
    def parse(self, response):
        data = {
            'name':response.css(".sales-info > h1::text").get(),
            'phone':response.css(".contact > p.phone::text").get(),
            'target_link':[]
        }
        for item_link in response.css(".review-info > a.author[href]::attr(href)").getall():
            resp = yield scrapy.Request(response.urljoin(item_link), meta={'handle_httpstatus_all': True})
            target_link = resp.css("a.review-business-name::attr(href)").get()
            data['target_link'].append(target_link)

        print(data)

if __name__ == "__main__":
    c = CrawlerProcess({
        'USER_AGENT':'Mozilla/5.0',
        'LOG_LEVEL':'ERROR',
    })
    c.crawl(YellowpagesSpider)
    c.start()

它产生的输出：

{'name': 'Honey Honey Cafe & Crepery', 'phone': '(415) 351-2423', 'target_link': ['/san-francisco-ca/mip/honey-honey-cafe-crepery-4752771', '/walnut-ca/mip/akasaka-japanese-cuisine-455476824', '/san-francisco-ca/mip/honey-honey-cafe-crepery-4752771']}

Answer 2

您可以使用scrapy的coroutine support非常容易地完成这项工作

代码如下所示：

async def parse_page(self, response):
    ...
    for link in links:
        request = response.follow(link)
        response = await self.crawler.engine.download(request, self)
        urls.append(response.css('a::attr(href)').get())