我想在我的父爬虫类设置以下内容,因为这应该是每个孩子都一样,我该怎么做?你知道吗
scrapy crawl spiderX -a full >> FEED_URI = /xx/spiderX_full
scrapy crawl spiderX -a quick >> FEED_URI = /xx/spiderX_quick
到目前为止,我的情况是:
@classmethod
def update_settings(cls, settings):
settings_dict = cls.custom_settings or {}
feed_uri = path.join(settings.get('FEED_DIR'), '%s' % cls.name)
settings_dict['FEED_URI'] = feed_uri
settings.setdict(settings_dict, priority='spider')
如何从此函数访问快速/完整参数? 我试着这样做:
def __new__(cls, full=False, quick=False, *a, **kw):
cls.full = full
cls.quick = quick
return super(MyCrawlSpider, cls).__new__(cls, *a, **kw)
但很明显,更新设置在它运行之前
尝试使用
-s
参数。你知道吗scrapy crawl spiderX -s FEED_URI=s3://mybucket/path/to/export.csv
相关问题 更多 >
编程相关推荐