Python/Beautiful soup find_all（）找不到

Question

当我使用find_all()时，我应该得到100个结果，但我只得到25个结果。在

编码

下面是我抓取tweakers并尝试返回类等于largethumb的每个元素的代码。在

一旦我做了，我过滤出名称和价格。在

my_url = 'https://tweakers.net/categorie/545/geheugen-intern/producten/'
uReq(my_url)
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")

#should be 100 tr object's
products = page_soup.find_all("tr", attrs={"class": "largethumb"})
for product in products:
    title = product.p.text
    price_container = product.find_all("p", {"class": "price"})
    price = price_container[0].text
    lijst = title, price
    print(lijst)

结果

结果是25倍。在

^{pr2}$

Answer 1

试试看。它将为您获取所有25个结果：

from bs4 import BeautifulSoup
import requests

res =requests.get('https://tweakers.net/categorie/545/geheugen-intern/producten/')
soup = BeautifulSoup(res.text, "lxml")
for product in soup.find_all(class_="largethumb"):
    title = product.find_all(class_="editionName")[0]['title']
    price = product.find_all(class_="price")[0].text
    print(title,price)

顺便说一句，你提供的链接在每页显示25个结果。在

Answer 2

该网站默认显示25个搜索结果。如果您的web浏览器中存在差异，那是因为您的浏览器有来自相关站点的cookies。如果要获得100个结果，请按如下方式编辑my_url：

my_url = 'https://tweakers.net/categorie/545/geheugen-intern/producten/?pageSize=100&page=1'
uReq(my_url)
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")

# WILL be 100 tr object's
products = page_soup.find_all("tr", attrs={"class": "largethumb"})
for product in products:
    title = product.p.text
    price_container = product.find_all("p", {"class": "price"})
    price = price_container[0].text
    lijst = title, price
    print(lijst)

有效性证明：

^{pr2}$

如果您将鼠标悬停在左下角的“100个结果”按钮上，您会看到这个url正是它们将您重定向到的位置。刮痧快乐！在