我不能用漂亮的汤来刮谷歌新闻。我得到一个错误：TypeError:“NoneType”对象不可调用

Question

我已经做了一段时间的网页抓取练习，它似乎一直进展顺利，直到我遇到谷歌新闻。我正在使用Beautiful soup搜索这些网站，但每次我尝试搜索google新闻时，都会出现错误TypeError: 'NoneType' object is not callable，尽管我很有信心我使用了正确的标签

经过多次尝试，我决定在我的文本编辑器（beautiful soup正在接收的编辑器）中打印出该页面的源代码，我发现它不包含任何标记（这可能解释为什么我会出现该错误）

这是我的密码

from bs4 import BeautifulSoup
from urllib.request import urlopen

page_info=urlopen('https://news.google.com')
soup=BeautifulSoup(page_info,'html.parser')
headlines=soup.findall('div',{'jscontroller':'d0DtYd'})
for head in headlines:
    headline=head.find('h3').find('a').get_text()
    print(headline)

有没有什么特别的原因，我有这个问题，或者我缺少了什么重要的东西。我需要一些帮助

Answer 1

该网站是使用javascript动态加载的，因此您不能使用requests或urllib来刮取它。最好的方法是使用selenium。除此之外，你的陈述是错误的。它应该是find_all，而不是findall

因此，以下是完整的代码：

from bs4 import BeautifulSoup
from selenium import webdriver
import time

driver = webdriver.Chrome()

driver.get('https://news.google.com')

time.sleep(4)

page_info = driver.page_source

driver.close()

soup=BeautifulSoup(page_info,'html5lib')
headlines=soup.find_all('div',{'jscontroller':'d0DtYd'})
for head in headlines:
    headline=head.find('h3').find('a').get_text()
    print(headline)

输出：

US presidential debate LIVE: Trump making friends with ‘thugs’ like Putin and Kim, says Biden
To buy or not to buy: Why BJP’s Bihar move comes as a surprise to states
Two terrorists surrender after families brought to encounter site in Jammu and Kashmir`s Baramulla
Fire breaks out at City Centre mall in Mumbai
After teacher’s killing, French Muslims fear rising Islamophobia
To buy or not to buy: Why BJP’s Bihar move comes as a surprise to states
Delhi's air quality dips, morning walkers irked
55,839 Fresh Coronavirus Cases In India, Total Cases At 77.06 Lakh: 10 Points
After Bihar, Tamil Nadu, Madhya Pradesh Promise Free Vaccine: 10 Points
US presidential debate LIVE: Trump making friends with ‘thugs’ like Putin and Kim, says Biden
COVID-19 vaccine may be ready by December. Trials to safety — a look at India's vaccine journey
US Elections 2020 HIGHLIGHTS: Intelligence agencies say Iran, Russia have tried to meddle in polls
Trump-allied groups pour $30 million into Barrett’s confirmation to Supreme Court

注意：您的输出可能不同，因为我们都生活在世界的不同地区。