Rottentomotos的Web抓取出现错误

Question

我想从rottentomatoes上刮下一页

页面截图为

如图所示span class= descriptor是a class的父类div class = info director是Directed By的父类

我想刮去董事们的名字

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
url= 'https://editorial.rottentomatoes.com/guide/best-sci-fi-movies-of-all-time/'
r = requests.get(url, headers=headers)#, proxies=proxies)
content = r.content
soup = BeautifulSoup(content)
director = []
people1 = soup.find_all('div',{'class':'info director'})
for d in people1:
    Dir = d.find('a').text
    director.append(Dir)

我犯了这个错误

AttributeError: 'NoneType' object has no attribute 'text'

Answer 1

使用“info director”类将div作为目标，并使用一行程序将所有href文本转储到列表中

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
url = 'https://editorial.rottentomatoes.com/guide/best-sci-fi-movies-of-all-time/'
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html5lib')
directors = [a.text for a in (d.find('a') for d in soup.find_all('div', attrs={'class': 'info director'})) if a]
for x in range(len(directors)):
    print(directors[x])  # output directors

# alternative loop
directors = []
for d in soup.find_all('div', attrs={'class': 'info director'}):
    for a in d.find('a'):
        directors.append(a)
        print(a)