链状的有数据依赖关系的蜘蛛

Question

事实上，那些破破烂烂的医生解释了如何把两个斯派德锁在一起

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

configure_logging()
runner = CrawlerRunner()


@defer.inlineCallbacks
def crawl():
    yield runner.crawl(MySpider1)
    yield runner.crawl(MySpider2)
    reactor.stop()

crawl()
reactor.run() # the script will block here until the last crawl call is finished

但是在我的用例中，MySpider2需要在转换后使用一些transformFunction()由{}检索的信息。在

所以我想要这样的东西：

^{pr2}$

我想安排的是：

1. MySpider1启动，在磁盘上写入data，然后退出
2. transformFunction()转换data到{}
3. MySpider2开始使用newData

那么我如何使用扭曲反应器和废料来管理这种行为呢？在

Answer 1

^{}返回一个^{}，这样就可以将回调链接到它。小调整将不得不做你的代码。在

from twisted.internet import task
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging

configure_logging()

def crawl(reactor):
    runner = CrawlerRunner()
    d = runner.crawl(MySpider1)
    d.addCallback(transformFunction)
    d.addCallback(crawl2, runner)
    return d

def transformFunction(result):
    # crawl doesn't usually return any results if successful so ignore result var here
    # ...
    return newdata

def crawl2(result, runner):
    # result == newdata from transformFunction
    # runner is passed in from crawl()
    return runner.crawl(MySpider2, data=result)

task.react(crawl)

主函数是crawl()，它由{a3}执行，它将为您启动和停止反应堆。Deferred从runner.crawl()返回，transformFunction+crawl2函数链接到它，这样当一个函数完成时，下一个函数就会启动。在