点击“更多”按钮后如何报废网站数据

Question

我正在尝试学习使用BS4+selenium进行web清理。网站链接是tripadvisor

review文本有更大的范围，在这里单击一些更多的文本使用AJAX加载到同一个div中。在

但是我的代码在selenium单击More按钮之前输出了评论文本。在

我怎么能等到更多的按钮点击使用硒

    from selenium import webdriver
from bs4 import BeautifulSoup


def openUrl(link):
    driver = webdriver.Firefox()
    driver.get(link)

    elem1 = driver.find_element_by_xpath("//span[@class='taLnk ulBlueLinks']")
    elem1.click()
    html_source = driver.page_source
    driver.quit()

    soup = BeautifulSoup(html_source, 'lxml')
    foundDiv = soup.findAll("div", {"class": "review-container"})
    for reviewContainer in foundDiv:

        ratingText = reviewContainer.select_one(".partial_entry").text
        print(ratingText)

openUrl("https://www.tripadvisor.in/Hotel_Review-g1010231-d1065009-Reviews-Radisson_Blu_Resort_Spa_Alibaug-Alibaug_Raigad_District_Maharashtra.html")

但是BS4不用等待更多的按钮点击就可以丢弃数据。在

请帮忙

Answer 1

请参阅下面的WebDriverWait示例。在

driver.get('https://www.tripadvisor.in/Hotel_Review-g1010231-d1065009-Reviews-Radisson_Blu_Resort_Spa_Alibaug-Alibaug_Raigad_District_Maharashtra.html')
moreButton = driver.find_element_by_css_selector("span.taLnk.ulBlueLinks")
moreButton.click()

wait = WebDriverWait(driver, 10)
element = wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "div[data-reviewid='493434022'] div.loadingShade")))

html_source = driver.page_source
print(html_source)