用Scrapy模拟JavaScript按钮点击

Question

我的目的是在这个网页上运行一个小爬虫：http://visit.rio/en/o-que-fazer/outdoors/。但是，在^{id1}上有一些资源$

Answer 1

如果你读对了，你最好的选择是使用Firefox浏览器的scrapy+selenium，或者像PhantomJS这样的无头浏览器，以加快抓取速度。

示例改编自https://stackoverflow.com/a/17979285/2781701

import scrapy
from selenium import webdriver

class ProductSpider(scrapy.Spider):
    name = "product_spider"
    allowed_domains = ['visit.rio']
    start_urls = ['http://visit.rio/en/o-que-fazer/outdoors']

    def __init__(self):
        self.driver = webdriver.Firefox()
    def parse(self, response):
        self.driver.get(response.url)

        while True:
            next = self.driver.find_element_by_xpath('//div[@id="show_more"]/a')

            try:
                next.click()

                # get the data and write it to scrapy items
            except:
                break

        self.driver.close()