Python请求模块得到相同的结果，尽管增加了pagenumb

Question

URL中唯一更改的是页码，每次请求后页码都会递增

除了Selenium或相关工具之外，我不确定可以使用什么方法遍历页面。我的直觉是，可能有一些头/查询组合来直接获取数据，但我不知道在哪里可以找到它

url = 'http://therunningbug.co.uk/events/find-races.aspx?EventName=&AddressRegion=&AddressCounty=&Date=&Surface=#Sort=Date&page='

page = 1

while True:

    pageData = BeautifulSoup(requests.get(url + str(page)).content)

    articles = pageData.find('div', {'class':"items-content"})

    for a in articles.find_all('article'):
        name = a.find('span', {'itemprop':"name"}).text
        d, t = a.find('time').get('datetime').split('T')

        timeData = t[:-3]

        dateData = d.split('-')
        date = (dateData[1] + '/' + dateData[2] + '/' + dateData[0][2:]).strip()
        description = a.find('p', {'itemprop':"description"}).text.strip()
        webLink = 'http://therunningbug.co.uk' + a.find('a', {'itemprop':"url"}).get('href')
        category = a.find('span', {'class':"surface"}).text
        location = a.find('span', {'class':"region"}).text + ', ' + a.find('span', {'class':"county"}).text

        print name, ' -- name'
        print date, ', ', timeData, ' -- date, time'
        print description, ' -- description'
        print webLink, ' -- website link'
        print category, ' -- category'
        print location, ' -- location\n'

    page += 1

Answer 1

问题可能是URL编码。您可以进行URL编码：

url = 'http://therunningbug.co.uk/events/find-races.aspx'
payload = {'page': page}
pageData = BeautifulSoup(requests.get(url, params = payload).content)

这也是因为URI中没有复杂的字符需要进行URL编码

url = 'http://therunningbug.co.uk/events/find-races.aspx'
pageData = BeautifulSoup(requests.get(url + '?page=' + str(page)).content)

有关url编码，请参阅请求文档http://docs.python-requests.org/en/master/user/quickstart/

完整代码：

#!/usr/bin/env python

import requests
from bs4 import BeautifulSoup

page = 1
while True:

    url = 'http://therunningbug.co.uk/events/find-races.aspx'
    payload = {'page': page}
    pageData = BeautifulSoup(requests.get(url, params = payload).content)

    articles = pageData.find('div', {'class':"items-content"})

    for a in articles.find_all('article'):
        name = a.find('span', {'itemprop':"name"}).text
        d, t = a.find('time').get('datetime').split('T')

        timeData = t[:-3]

        dateData = d.split('-')
        date = (dateData[1] + '/' + dateData[2] + '/' + dateData[0][2:]).strip()
        description = a.find('p', {'itemprop':"description"}).text.strip()
        webLink = 'http://therunningbug.co.uk' + a.find('a', {'itemprop':"url"}).get('href')
        category = a.find('span', {'class':"surface"}).text
        location = a.find('span', {'class':"region"}).text + ', ' + a.find('span', {'class':"county"}).text

        print name, '   name'
        print date, ', ', timeData, '   date, time'
        print description, '   description'
        print webLink, '   website link'
        print category, '   category'
        print location, '   location\n'

    page += 1