我正在抓取Dmoz网站,我抓取了about页面,但是当我创建了另一个名为parse_editor
的函数并试图抓取时,它没有给我结果。你知道吗
from ..items import DmoztutorialItem
import scrapy
class DmozSpiderSpider(scrapy.Spider):
name = 'Dmoz'
start_urls = ['http://dmoz-odp.org/']
about_page = 'http://dmoz-odp.org/docs/en/about.html'
editor = 'http://dmoz-odp.org/docs/en/help/become.html'
def parse(self, response):
# collect data on first page
items = {
'Navbar': response.css('#main-nav a::text').extract(),
'Category_names': response.css('.top-cat a::text').extract(),
'Subcategories': response.css('.sub-cat a::text').extract(),
'About_page': self.about_page,
'Become_an_editor': self.editor
}
# save and call request to another page
yield response.follow(self.about_page, self.parse_about, self.editor, self.parse_editor, meta={'items': items})
def parse_about(self, response):
# do your stuff on second page
items = response.meta['items']
items['Headings'] = response.css('h2::text , #mainContent h1::text').extract() # add your logics
items['Paragraphs'] = response.css('p::text').extract()
items['3 Projects'] = response.css('li~ li+ li b a::text , li:nth-child(1) b a::text').extract()
items['About Dmoz'] = response.css('.nav ul a::text , li:nth-child(2) b a::text').extract()
items['Languages'] = response.css('.nav~ .nav a::text').extract()
items['You can make a difference'] = response.css('dd::text , #about-contribute::text').extract()
items['Further information'] = response.css('li::text , #about-more-info a::text').extract()
yield items
def parse_editor(self, response):
# do your stuff on third page
editor_items = response.meta['items']
editor_items['Heading'] = response.css('#mainContent h1::text').extract()
yield editor_items
你把一切都写在一个
response.follow
里,那就错了。它需要一对url回调。所以把它们写在两个不同的函数中:不正确的变体:
正确的变体:
您可以在
parse
函数中编写第一个follow
;调用parse_about
,然后在parse_editor
函数中生成第二个follow
,并生成最终项。你知道吗相关问题 更多 >
编程相关推荐