使用scrapy获取表单请求的响应

Question

我正在使用scrapy发送表单请求。数据检索在浏览器中以两种方式工作：

1. 此表格填写在page
2. 表单请求被发送到此link

响应用于在填写表单的同一页面中显示数据表和图表

这是我的蜘蛛的片段

class NamisSpider(scrapy.Spider):
    name = "namis-spider"
    form_url = 'http://namis.gov.np/pages/market/marketPriceForm/'
    request_url = 'http://namis.gov.np/site/pages/ajax_priceList/'

    def start_requests(self):
        yield scrapy.Request(url=self.form_url, callback=self.parse)

    def parse(self, response):
        token = response.css('input[name="csrf_appcore"]::attr(value)').get()
        data = {
            'csrf_appcore': token,
            'market': '1',
            'category': '1',
            'commodity': '104',
            'date_from': '2075-01-01',
            'date_to': '2076-11-21',
        }

        yield scrapy.FormRequest(url=self.request_url, formdata=data, callback=self.parse_market_price)

    def parse_market_price(self, response):
        print(response)

将显示提交表单后在浏览器中收到的响应 here。但我无法用scrapy复制这个。我用scrapy得到的反应是

<200 http://namis.gov.np/site/pages/ajax_priceList/>

但是我想接收浏览器接收到的json响应

注：日期以Vikram Samvat为基础

Answer 1

您缺少X-Requested-With的标头

`

import scrapy
import json

class NamisSpider(scrapy.Spider):
    name = "namis-spider"
    form_url = 'http://namis.gov.np/pages/market/marketPriceForm/'
    request_url = 'http://namis.gov.np/site/pages/ajax_priceList/'

    def start_requests(self):
        yield scrapy.Request(url=self.form_url, callback=self.parse)

    def parse(self, response):
        token = response.css('input[name="csrf_appcore"]::attr(value)').get()

        data = {
            'csrf_appcore': token,
            'market': '1',
            'category': '1',
            'commodity': '104',
            'date_from': '2076-11-21',
            'date_to': '2076-11-21',
        }
        headers = {
            "X-Requested-With":"XMLHttpRequest",
        }

        yield scrapy.FormRequest(url=self.request_url, headers=headers, formdata=data, callback=self.parse_market_price)

    def parse_market_price(self, response):
        unicoded_response = response.body_as_unicode()
        json_response = json.loads(unicoded_response)
        print('json  >',json_response)
        print(json_response['status'])
        print(json_response['data'])

`