在不发送请求的情况下，无法在方法之间传递参数

Question

我用python创建了一个脚本，使用scrapy和selenium来解析主页上不同餐厅的链接，然后从它们的内部页面解析每个餐厅的名称。在

当我将scrapy与selenium结合使用时，如何在不发送请求的情况下回调（或在方法之间传递参数）工作？

下面的脚本使用self.driver.get(response.url)重写回调，我无法摆脱它：

import scrapy
from selenium import webdriver
from scrapy.crawler import CrawlerProcess
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC

class YPageSpider(scrapy.Spider):
    name = "yellowpages"
    link = 'https://www.yellowpages.com/search?search_terms=Pizza+Hut&geo_location_terms=San+Francisco%2C+CA'

    def start_requests(self):
        self.driver = webdriver.Chrome()
        self.wait = WebDriverWait(self.driver, 10)
        yield scrapy.Request(self.link,callback=self.parse)

    def parse(self,response):
        self.driver.get(response.url)
        for elem in self.wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".v-card .info a.business-name"))):
            yield scrapy.Request(elem.get_attribute("href"),callback=self.parse_info)

    def parse_info(self,response):
        self.driver.get(response.url)
        elem = self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".sales-info > h1"))).text
        yield {"title":elem}

if __name__ == '__main__':
    c = CrawlerProcess()
    c.crawl(YPageSpider)
    c.start()

Answer 1

@vezunchik已经指出的关联答案几乎能让你找到答案。唯一的问题是，当您使用完全相同的代码时，您将有多个chromedriver实例。要重复使用同一个驱动程序多次，您可以尝试如下。在

在你的废项目中创建一个文件middleware.py并粘贴以下代码：

from scrapy.http import HtmlResponse
from selenium import webdriver

class SeleniumMiddleware(object):
    def __init__(self):
        chromeOptions = webdriver.ChromeOptions()
        chromeOptions.add_argument(" headless")
        self.driver = webdriver.Chrome(options=chromeOptions)

    def process_request(self, request, spider):
        self.driver.get(request.url)
        body = self.driver.page_source
        return HtmlResponse(self.driver.current_url, body=body, encoding='utf-8', request=request)

想出来一个更新，以防你想看看chmoedriver是如何在可见模式下遍历的。要让浏览器浏览得更清晰，请尝试以下操作：

^{pr2}$

使用以下脚本获取所需的内容。对于通过中间件使用selenium的每个url只有一个请求（导航）。现在，您可以在spider中使用Selector()来获取数据，如下面所示。在

import sys
# The hardcoded address leads to your project location which ensures that
# you can add middleware reference within crawlerprocess
sys.path.append(r'C:\Users\WCS\Desktop\yourproject')
import scrapy
from scrapy import Selector
from scrapy.crawler import CrawlerProcess

class YPageSpider(scrapy.Spider):
    name = "yellowpages"
    start_urls = ['https://www.yellowpages.com/search?search_terms=Pizza+Hut&geo_location_terms=San+Francisco%2C+CA']

    def parse(self,response):
        items = Selector(response)
        for elem in items.css(".v-card .info a.business-name::attr(href)").getall():
            yield scrapy.Request(url=response.urljoin(elem),callback=self.parse_info)

    def parse_info(self,response):
        items = Selector(response)
        title = items.css(".sales-info > h1::text").get()
        yield {"title":title}

if __name__ == '__main__':
    c = CrawlerProcess({
            'DOWNLOADER_MIDDLEWARES':{'yourspider.middleware.SeleniumMiddleware': 200},
        })
    c.crawl(YPageSpider)
    c.start()

Answer 2

你是说把变量从一个函数传递到另一个函数？为什么不使用meta来完成这个呢？不管有没有硒，它都能起作用。我使用的代码和你一样，只是两个小的更新：

def parse(self,response):
    self.driver.get(response.url)
    for elem in self.wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".v-card .info a.business-name"))):
        yield scrapy.Request(elem.get_attribute("href"),
                             callback=self.parse_info,
                             meta={'test': 'test'})  # <- pass anything here

def parse_info(self,response):
    self.driver.get(response.url)
    elem = self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".sales-info > h1"))).text
    yield {"title": elem, 'data': response.meta['test']}  # <- getting it here

因此，它输出：

^{pr2}$