通过“page=”u中段“刮分页

Question

我正在尝试从这个网页以及后面的900多个网页中搜集数据：https://hansard.parliament.uk/search/Contributions?endDate=2019-07-11&page=1&searchTerm=%22climate+change%22&startDate=1800-01-01&partial=True

重要的是scraper不以分页链接为目标，而是遍历url中的“page=”数字。这是因为当前的数据动态加载到原始网页中，分页链接指向原始网页

我试过写一些东西，通过pagination ul的“last”类在url中的页码中循环，以找到最终的页面，但我不确定如何定位url的特定部分，同时保持每个结果的搜索查询相同

r = requests.get(url_pagination)
soup = BeautifulSoup(r.content, "html.parser")

page_url = "https://hansard.parliament.uk/search/Contributions?endDate=2019-07-11&page={}" + "&searchTerm=%22climate+change%22&startDate=1800-01-01&partial=True"
last_page = soup.find('ul', class_='pagination').find('li', class_='last').a['href'].split('=')[1]
dept_page_url = [page_url.format(i) for i in range(1, int(last_page)+1)]

print(dept_page_url)

理想情况下，我只想从类“secondaryTitle”中刮取名称，以及每行包含日期的第二个未命名div

我不断得到一个错误：ValueError:invalid literal for int（）with base 10:'2019-07-11&；'搜索术语'

Answer 1

你的错误是因为你使用了错误的号码从你的分裂。你想要-1.观察：

last_page = soup.find('ul', class_='pagination').find('li', class_='last').a['href']
print(last_page)
print(last_page.split('=')[1])
print(last_page.split('=')[-1])

提供：

/search/Contributions?endDate=2019-07-11&searchTerm=%22climate+change%22&startDate=1800-01-01&page=966

拆分时使用1

2019-07-11&searchTerm

对-1

966

为了从你想要的每一个页面获得信息，我会像其他答案一样使用css选择器和压缩。下面是一些其他的循环构造，在给定请求数的情况下使用会话来提高效率

---

您可以发出初始请求并提取页数，然后循环这些页数。使用会话对象以提高连接重用的效率

import requests
from bs4 import BeautifulSoup as bs

def make_soup(s, page):
    page_url = "https://hansard.parliament.uk/search/Contributions?endDate=2019-07-11&page={}&searchTerm=%22climate+change%22&startDate=1800-01-01&partial=True"
    r = s.get(page_url.format(page))
    soup = bs(r.content, 'lxml')
    return soup

with requests.Session() as s:
    soup = make_soup(s, 1)
    pages = int(soup.select_one('.last a')['href'].split('page=')[1])
    for page in range(2, pages + 1):
        soup = make_soup(s, page)
        #do something with soup 

---

可以循环直到类last停止出现

import requests
from bs4 import BeautifulSoup as bs

present = True
page = 1
#results = {}

def make_soup(s, page):
    page_url = "https://hansard.parliament.uk/search/Contributions?endDate=2019-07-11&page={}&searchTerm=%22climate+change%22&startDate=1800-01-01&partial=True"
    r = s.get(page_url.format(page))
    soup = bs(r.content, 'lxml')
    return soup

with requests.Session() as s:
    while present:
        soup = make_soup(s, page)
        present = len(soup.select('.last')) > 0
        #results[page] = soup.select_one('.pagination-total').text
        #extract info
        page+=1