如何通过data-testid找到按钮并在Selenium中点击它？

Question

在一个研究项目中，我想要抓取这个网址的数据：https://charts.spotify.com/charts/view/citytoptrack-barcelona-weekly/2024-02-29。但是，当我用Selenium访问这个页面时，出现了一个登录页面。

为了登录并访问原始页面，我想用以下代码模拟人类点击“登录”按钮：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

website = 'https://charts.spotify.com/charts/view/citytoptrack-barcelona-weekly/2024-02-29'
path = '/home/isaac/chromedriver/chromedriver'
service = Service(executable_path=path)
driver = webdriver.Chrome(service=service)
driver.get(website)
driver.find_element(By.XPATH, value='//button[@class="Button-qlcn5g-0 jdERo"]').click()

这是按钮的HTML代码：

<div class="ChartsHomeHeader__HeaderLinks-vilr39-2 hQLnwL">
  <a data-testid="charts-login" href="https://accounts.spotify.com/login?continue=https%3A%2F%2Fcharts.spotify.com/login" style="margin-left:12px" class="Button-qlcn5g-0 jdERo">
    <div class="ButtonInner-sc-14ud5tc-0 iMWZgy encore-bright-accent-set">Log in</div>
    <div class="ButtonFocus-sc-2hq6ey-0 iPAQn"></div>
  </a>
 </div>

但是，当程序尝试找到“登录”按钮时，它崩溃了，并给我返回了这个错误：

Traceback (most recent call last):

  File "/home/isaac/scrapingTest.py", line 10, in <module>

    driver.find_element(By.XPATH, value='//button[@class="Button-qlcn5g-0 jdERo"]').click()

  File "/home/isaac/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 741, in find_element

    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]

  File "/home/isaac/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute

    self.error_handler.check_response(response)

  File "/home/isaac/.local/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response

    raise exception_class(message, screen, stacktrace)

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@class="Button-qlcn5g-0 jdERo"]"}

  (Session info: chrome=122.0.6261.111); For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception

Stacktrace:

#0 0x56474bb1fe93 <unknown>

#1 0x56474b817ce6 <unknown>

#2 0x56474b862e48 <unknown>

#3 0x56474b862f01 <unknown>

#4 0x56474b8a63f4 <unknown>

#5 0x56474b884edd <unknown>

#6 0x56474b8a3899 <unknown>

#7 0x56474b884c53 <unknown>

#8 0x56474b855db3 <unknown>

#9 0x56474b85677e <unknown>

#10 0x56474bae57cb <unknown>

#11 0x56474bae97e5 <unknown>

#12 0x56474bad30e1 <unknown>

#13 0x56474baea372 <unknown>

#14 0x56474bab71bf <unknown>

#15 0x56474bb0e488 <unknown>

#16 0x56474bb0e683 <unknown>

#17 0x56474bb1f044 <unknown>

#18 0x7f45c5294ac3 <unknown>

我花了一些时间在网上查找，但找不到解决办法。我该如何成功找到并点击登录按钮呢？

我对网页抓取还很陌生，所以如果这个问题很基础，请见谅。

另外，有没有办法在不使用Selenium模拟登录操作的情况下，直接访问这个网址：https://charts.spotify.com/charts/view/citytoptrack-barcelona-weekly/2024-02-29来抓取数据？也许我走了弯路……

提前谢谢你们 :)

Answer 1

把解决方案的代码放在这里，以防有人需要它！:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

website = 'https://charts.spotify.com/charts/view/citytoptrack-barcelona-weekly/2024-02-29'
path = '/home/isaac/chromedriver/chromedriver'
service = Service(executable_path=path)
driver = webdriver.Chrome(service=service)
driver.get(website)
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='Button-qlcn5g-0 jdERo']")))
element.click();