如何在不再次激活的情况下从管道执行特定的spider

Question

导言

我正在撤销的网站有两个URL：

• /top列出了顶级玩家
• /player/{name}显示名为{name}info的播放器

从第一个URL，我得到玩家的名字和位置，然后我可以使用给定的名字调用第二个URL。我目前的目标是将所有数据存储在数据库中

问题

我创造了两个蜘蛛。第一个是为第一个蜘蛛找到的每个玩家抓取/top，第二个是为每个玩家抓取/player/{name}。但是，为了能够将第一个spider数据插入数据库，我需要调用profile spider，因为它是一个外键，如以下查询中所述：

INSERT INTO top_players (player_id, position) values (1, 1)

INSERT INTO players (name) values ('John Doe')

问题:

是否可以从管道中执行spider以获得spider结果？我的意思是，被调用的spider不应该再次激活管道

Answer 1

我建议你对刮削过程有更多的控制。特别是从第一页和详细页抓取名称、位置。 试试这个：

# -*- coding: utf-8 -*-
import scrapy

class MyItem(scrapy.Item):
    name = scrapy.Field()
    position= scrapy.Field()
    detail=scrapy.Field() 
class MySpider(scrapy.Spider):

    name = '<name of spider>'
    allowed_domains = ['mywebsite.org']
    start_urls = ['http://mywebsite.org/<path to the page>']

    def parse(self, response):

        rows = response.xpath('//a[contains(@href,"<div id or class>")]')

        #loop over all links to stories
        for row in rows:
            myItem = MyItem() # Create a new item
            myItem['name'] = row.xpath('./text()').extract() # assign name from link
            myItem['position']=row.xpath('./text()').extract() # assign position from link
            detail_url = response.urljoin(row.xpath('./@href').extract()[0]) # extract url from link
            request = scrapy.Request(url = detail_url, callback = self.parse_detail) # create request for detail page with story
            request.meta['myItem'] = myItem # pass the item with the request
            yield request

    def parse_detail(self, response):
        myItem = response.meta['myItem'] # extract the item (with the name) from the response
        text_raw = response.xpath('//font[@size=3]//text()').extract() # extract the detail (text)
        myItem['detail'] = ' '.join(map(unicode.strip, text_raw)) # clean up the text and assign to item
        yield myItem # return the item