我正在试图获取有关此site上产品规格的数据。默认视图是价格,但我想进入规格视图。因为按钮执行POST请求,所以我尝试使用FormRequest模拟它,但什么都没有发生
以下是我正在使用的代码:
import scrapy
from scrapy import Spider
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
from Demo.items import DemoItem, Distributor
class ProductSpider(scrapy.Spider):
name='nameCode'
start_urls = ['https://octopart.com/electronic-parts/integrated-circuits-ics' ]
def parse(self,response):
token = response.xpath('//button[@name="serp-view"]/@value').extract_first()
return FormRequest.from_response(response, url="https://octopart.com/electronic-parts/integrated-circuits-ics", formdata={'serp-view':token}, callback=self.scrape_pages)
def scrape_pages(self,response):
open_in_browser(response)
输出:
[scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics?q=> (referer: https://octopart.com/electronic-parts/integrated-circuits-ics)
[scrapy.core.engine] INFO: Closing spider (finished)
目前没有回答
相关问题 更多 >
编程相关推荐