当尝试使用BeautifulSoup查找div时，Webscraping不会返回任何结果

Question

我正试图建立一个程序，在香港到新德里到大阪的“最佳”航班价格下降时通知我。我把Skyscanner的搜索结果页作为我程序的基础。你知道吗

import requests
from bs4 import BeautifulSoup

URL = 'https://www.skyscanner.com.hk/transport/d/hkga/2019-12-28/del/del/2020-01-05/osaa?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&ref=home&locale=en-GB#/'

page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')
pricebox = soup.find('div', class_= 'Price_mainPriceContainer__1dqsw')
price = pricebox.find('span', class_= 'BpkText_bpk-text__2NHsO BpkText_bpk-text--base__2vfTl BpkText_bpk-text--bold__4yauk').text

我想找到第一个包含span的div，其中包含“Best Flight”的价格，所以我使用find（）方法找到类属性为“price\u mainPriceContainer\uu1dqsw”的div。然而，它什么也没回来。我不知道出了什么问题。你知道吗

编辑：首先，感谢大家的帮助。真的很感激。万一你还没意识到，我是个超级初学者。不管怎样，我安装了ChromeDriver和Selenium，并尝试运行@QHarr提供的代码，它返回了如下错误

Traceback (most recent call last):
  File "testflight.py", line 6, in <module>
     d = webdriver.Chrome(r'/usr/local/bin/chromedriver')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
 selenium.common.exceptions.WebDriverException: Message: unknown error: Failed to create a Chrome process.

我不知道为什么会这样。我已经将ChromeDriver重新定位到/usr/local/bin，安装了对应于我的Google Chrome版本的ChromeDriver，并且Google Chrome是以正确的默认路径安装的。你知道吗

Answer 1

当页面向不同的合作伙伴发出请求以找到最佳价格时，将会有大量的流量，并且可能没有一个xhr可以始终以最佳价格解决（理论未经测试）。我可能会去硒和有一个等待条件，所有的结果在页面上，然后提取最佳价格

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

d = webdriver.Chrome(r'path\chromedriver.exe')
d.get('https://www.skyscanner.com.hk/transport/d/hkga/2019-12-28/del/del/2020-01-05/osaa?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&ref=home&locale=en-GB#/')
WebDriverWait(d,20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "[class^=Results_dayViewItems]")))
d.find_element_by_css_selector('[class^=Price_mainPriceContainer] span').text