我定义了一个scrapy spider,它可以从https://www.cancercarenorthwest.com/survivor-stories抓取所有的名字和一些故事,而定义的xpath不能捕获故事
# -*- coding: utf-8 -*-
import scrapy
from scrapy.contrib.loader import ItemLoader
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.selector import XmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from cancerstories.items import CancerstoriesItem
class LungcancerSpider(CrawlSpider):
name = "lungcancer"
allowed_domains = ["coloncancercoalition.org"]
start_urls = (
'http://www.coloncancercoalition.org/community/stories/survivor-stories/',
)
rules = (
Rule(SgmlLinkExtractor(allow=[r'http://www.coloncancercoalition.org/\d+/\d+/\d+/\w+']),callback='parse_page',follow=True),
)
def parse_page(self, response):
Li = ItemLoader(item=CancerstoriesItem(),response=response)
Li.add_xpath('name', '/html/body/div[4]/div[1]/div[1]/div/h1/text()')
Li.add_xpath('story','//../div/div/p/text()')
yield Li.load_item()
我想你需要把所有段落的文字加入到帖子内容下:
其中^{} 是作为以下格式导入的输出处理器:
相关问题 更多 >
编程相关推荐