使用scrapy模拟POST的FormRequest不可用

2024-06-16 12:13:10 发布

您现在位置:Python中文网/ 问答频道 /正文

我正在试图获取有关此site上产品规格的数据。默认视图是价格,但我想进入规格视图。因为按钮执行POST请求,所以我尝试使用FormRequest模拟它,但什么都没有发生

以下是我正在使用的代码:

import scrapy
from scrapy import Spider
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
from Demo.items import DemoItem, Distributor

class ProductSpider(scrapy.Spider):
    name='nameCode'

    start_urls = ['https://octopart.com/electronic-parts/integrated-circuits-ics' ]

    def parse(self,response):
        token = response.xpath('//button[@name="serp-view"]/@value').extract_first()
        return FormRequest.from_response(response, url="https://octopart.com/electronic-parts/integrated-circuits-ics", formdata={'serp-view':token}, callback=self.scrape_pages)

    def scrape_pages(self,response):
        open_in_browser(response)        

输出:

[scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics?q=> (referer: https://octopart.com/electronic-parts/integrated-circuits-ics)
[scrapy.core.engine] INFO: Closing spider (finished)

Tags: fromhttpscoreimportselfcomresponseparts