抓取需要多个请求才能访问特定表的网站（连接的下拉菜单）

Question

我正试图从这个网站https://pigeon-ndb.com/races/搜集数据。一开始，我认为如果我知道如何从下拉菜单中选择元素，这个问题将很容易解决，但结果比预期的要复杂。你知道吗

理想情况下，我希望遍历所有年份和季节（2010-2019），然后遍历所有组织和比赛的所有记录。总之，使用scrapy（无硒）从网站中的所有表中刮取数据。你知道吗

我知道这个问题涉及到对下拉菜单（总共3个）使用GET请求，如下所示：

1. https://pigeon-ndb.com/api/?request=get_databases（为下一个请求选择year和season的json元素）

2. https://pigeon-ndb.com/api/?request=get_organizations&database=2010%20OB&_=1557098607652（需要上一个工作请求中的年份和季节）

3. https://pigeon-ndb.com/api/?request=get_races&organization=&_=1557098607653（需要上一个请求（#2）中的组织名称才能工作）

以下代码是我计划使用的scrapy spider的基本概要，可能会有所更改：

from scrapy import Spider
from scrapy.http import Request


class PigeonSpider(Spider):
    name = 'pigeonspider'
    allowed_domains = ['pigeon-ndb.com']
    start_urls = ['https://pigeon-ndb.com/races/']

    def parse(self, response):
        pass
    def parse2(self,response):
        pass
    def parse3(self,response):
        pass

因为这是一个GET请求，所以我希望多次使用它（或一些变体）：

yield Request(url,callback=self.parse2)

我想我需要将json合并到刮削过程的动态部分中，但不确定这是否是最好的方法

在胶状壳中：

import json
jsonresponse = json.loads(response.body)

这是第一个请求（https://pigeon-ndb.com/api/?request=get_databases）的json输出：

{'data': [{'year': '2010', 'season': 'OB'}, {'year': '2010', 'season': 'YB'}, {'year': '2011', 'season': 'OB'}, {'year': '2011', 'season': 'YB'}, {'year': '2012', 'season': 'OB'}, {'year': '2012', 'season': 'YB'}, {'year': '2013', 'season': 'OB'}, {'year': '2013', 'season': 'YB'}, {'year': '2014', 'season': 'OB'}, {'year': '2014', 'season': 'YB'}, {'year': '2015', 'season': 'OB'}, {'year': '2015', 'season': 'YB'}, {'year': '2016', 'season': 'OB'}, {'year': '2016', 'season': 'YB'}, {'year': '2017', 'season': 'OB'}, {'year': '2017', 'season': 'YB'}, {'year': '2018', 'season': 'OB'}, {'year': '2018', 'season': 'YB'}, {'year': '2019', 'season': 'OB'}], 'jsonapi': {'version': 2.2, 'db': 'pigeon-ndb'}, 'meta': {'copyright': 'Copyright 2019 Craig Vander Galien', 'authors': ['Craig Vander Galien']}}

我仍然在学习scrapy，所以将感激如何处理这个问题的示例代码。谢谢！你知道吗

编辑：

因此，我尝试实现以下代码，但遇到了错误：

from scrapy import Spider
from scrapy.http import Request
import json


class PigeonSpider(Spider):
    name = 'pigeonspider'
    allowed_domains = ['pigeon-ndb.com']
    start_urls = ['https://pigeon-ndb.com/races/']

    def parse(self, response):
        result = json.loads(response.body)

        for node in result['data']:
            yield Request(
                url = 'https://pigeon-ndb.com/api/?request=get_organizations&database={year}%20{season}'.format(year=node["year"], season=node["season"]),
                callback = self.parse_organizations,
                cookies = {'database':'{year} {season}'.format(year=node['year'],season=node['season'])},
                meta = {
                'year': node['year'],
                'season': node['season'],
                }
            )
    def parse_organizations(self,response):
        result = json.loads(response.body)

        for node in result['data']:
            org_num = node['orgNum']
            if node['orgNum'] is None:
                org_num = 'null'

            yield Request(
                url='https://pigeon-ndb.com/api/?request=get_races&organization={org_name}&orgNum={org_num}'.format(org_name=node["Sys"], org_num=org_num),
                callback=self.parse_races,
                headers={'x-requested-with': 'XMLHttpRequest'},
                cookies={'database':'{year} {season}'.format(year=response.meta["year"], season=response.meta["season"])}
            )
    def parse_races(self,response):
        result = json.loads(response.body)
        for node in result['clockings']['data']:
            yield {
            'race':node['racename'],
            'season':node['season'],
            'date':node['date'],
            'year':node['year'],
            'time':node['Time'],
            'complevel':node['CompLevel'],
            'class': node['class'],
            'city': node['City'],
            'zip': node['Zip'],
            'state': node['State'],
            'entry': node['entry'],
            'first_name':node['FirstName'],
            'last_name':node['LastName'],
            'line_num':node['LineNum'],
            'band_num':node['band_no'],
            'color': node['BB'],
            'sex': node['sex'],
            'arrival_time':node['arri_time'],
            'distance':node['distance'],
            'speed':node['speed'],
            'reg_points':node['reg_points'],
            'std_points':node['std_points'],
            'unirate':node['unirate'],
            'place': node['Place'],
            }

运行spider时（错误）：

Traceback (most recent call last):
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output
    for x in result:
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/glenn/Projects/pigeonscraper/pigeonscraper/spiders/pigeonspider.py", line 13, in parse
    result = json.loads(response.body)
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/glenn/anaconda3/envs/scraperenv/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Answer 1

首先，您需要使用cookies设置两个参数（database和season）。之后，您可以迭代JSON结果：

from scrapy import Spider
from scrapy.http import Request
import json


class PigeonSpider(Spider):
    name = 'pigeonspider'
    allowed_domains = ['pigeon-ndb.com']
    start_urls = ['https://pigeon-ndb.com/api/?request=get_databases']

    def parse(self, response):
        result = json.loads(response.body)

        for node in result["data"]:
            yield Request(
                url="https://pigeon-ndb.com/api/?request=get_organizations&database={year}%20{season}".format(year=node["year"], season=node["season"]),
                callback=self.parse_organizations,
                # headers={'x-requested-with': "XMLHttpRequest", 'referer': "https://pigeon-ndb.com/races/"},
                cookies={'database':'{year} {season}'.format(year=node["year"], season=node["season"])},
                meta={
                    "year": node["year"],
                    "season": node["season"],
                }
            )

        pass
    def parse_organizations(self,response):
        result = json.loads(response.body)

        for node in result["data"]:

            org_num = node["orgNum"]
            if node["orgNum"] is None:
                org_num = "null"

            yield Request(
                url="https://pigeon-ndb.com/api/?request=get_races&organization={org_name}&orgNum={org_num}".format(org_name=node["Sys"], org_num=org_num),
                callback=self.parse_races,
                headers={'x-requested-with': "XMLHttpRequest"},
                cookies={'database':'{year} {season}'.format(year=response.meta["year"], season=response.meta["season"])}
            )

        pass
    def parse_races(self,response):
        result = json.loads(response.body)

        for race_key in result["data"].keys():

            race_date = result["data"][race_key]["date"]
            race_release_time = result["data"][race_key]["release_time"]
            race_bird_attend = result["data"][race_key]["bird_attend"]
            # etc.
        pass

更新你完全忽略了我的评论。parse_race_details根本没有在代码中实现！你知道吗

from scrapy import Spider
from scrapy.http import Request
import json


class PigeonSpider(Spider):
    name = 'pigeonspider'
    allowed_domains = ['pigeon-ndb.com']
    start_urls = ['https://pigeon-ndb.com/api/?request=get_databases']
    debug = False

    def parse(self, response):
        result = json.loads(response.body)

        for node in result["data"]:
            yield Request(
                url="https://pigeon-ndb.com/api/?request=get_organizations&database={year}%20{season}".format(
                    year=node["year"], season=node["season"]),
                callback=self.parse_organizations,
                # headers={'x-requested-with': "XMLHttpRequest", 'referer': "https://pigeon-ndb.com/races/"},
                cookies={
                    'database': '{year} {season}'.format(
                        year=node["year"],
                        season=node["season"])},
                meta={
                    "year": node["year"],
                    "season": node["season"],
                },
                dont_filter=True,
            )

            # Debug
            if self.debug:
                break

        pass

    def parse_organizations(self, response):
        result = json.loads(response.body)

        for node in result["data"]:

            org_num = node["orgNum"]
            if node["orgNum"] is None:
                org_num = "null"

            yield Request(
                url="https://pigeon-ndb.com/api/?request=get_races&organization={org_name}&orgNum={org_num}".format(org_name=node["Sys"], org_num=org_num),
                callback=self.parse_races,
                headers={'x-requested-with': "XMLHttpRequest"},
                cookies={'database': '{year} {season}'.format(year=response.meta["year"], season=response.meta["season"])},
                dont_filter=True,
                # meta={
                #     "year": response.meta["year"],
                #     "season": response.meta["season"],
                # },
            )

            # Debug
            if self.debug:
                break


        pass

    def parse_races(self, response):
        result = json.loads(response.body)

        if result["response"] == "failed":
            print("Failed response!")

        for race_key in result["data"].keys():
            race_name = result["data"][race_key]["racename"]
            race_date = result["data"][race_key]["date"].replace("/", "%2F")
            race_time = result["data"][race_key]["Time"]
            yield Request(
                url="https://pigeon-ndb.com/api/?request=get_race_details&racename={race_name}&date={race_date}&time={race_time}".format(race_name=race_name, race_date=race_date, race_time=race_time),
                callback=self.parse_race_details,
                headers={'x-requested-with': "XMLHttpRequest"},
                # cookies={'database': '{year} {season}'.format(year=response.meta["year"], season=response.meta["season"])},
                dont_filter=True,
            )

            # Debug
            if self.debug:
                break

        pass

    def parse_race_details(self, response):
        result = json.loads(response.body)

        if result["response"] == "failed":
            print("Failed response!")

        for node in result['data']['clockings']['data']:
            yield {
            'race':node['racename'],
            'season':node['season'],
            'date':node['date'],
            'year':node['year'],
            'time':node['Time'],
            'complevel':node['CompLevel'],
            'class': node['Class'],
            'city': node['City'],
            'zip': node['Zip'],
            'state': node['State'],
            'entry': node['entry'],
            'first_name':node['FirstName'],
            'last_name':node['LastName'],
            'line_num':node['LineNum'],
            'band_num':node['band_no'],
            'color': node['BB'],
            'sex': node['sex'],
            'arrival_time':node['arri_time'],
            'distance':node['distance'],
            'speed':node['speed'],
            'reg_points':node['reg_points'],
            'std_points':node['std_points'],
            'unirate':node['unirate'],
            'place': node['Place'],
            }

        pass