如何遍历隐藏的div和废弃文本？

Question

我试图刮一个网站与可扩展的div，其中隐藏的文本，我试图刮。我只能在第一个可展开的div中刮取文本。但是，我可以单击所有div。如何从所有div中刮取文本？你知道吗

关闭的HTML:

<li class="views-row views-row-1 pub1 default-on clk" tabindex="150">  
          <div class="teaser Speeches col-xs-12 col-sm-12 col-md-12 col-lg-12 crop2" data-nid="50849" data-tid="6971" aria-hidden="false">
  <div class="thumb" style="padding-top: 0px; padding-bottom: 0px;">
  <img class="img-responsive" src="/sites/pm/files/styles/news_listing_square/public/default_news/20180501_default_news2.jpg?itok=a1pfZTOA" alt="" title=""></div>
  <div class="news-teaser">
    <div class="title">TITLE</div>
    <div class="body">TEASER TEXT</div>
    <div class="category">Speeches<br>PLACE <span class="date-display-single" property="dc:date" datatype="xsd:dateTime" content="2019-06-10T18:15:00-04:00">June 10, 2019</span></div>
  </div>
</div>
<div class="sticky0"></div>
<div class="full-article" aria-hidden="true"></div>  
</li>
<li class="views-row views-row-2 pub1 default-on clk" tabindex="150"> </li>
<li class="views-row views-row-3 pub1 default-on clk" tabindex="150"> </li>

单击项目时，可以看到完整的讲话：

<li class="views-row views-row-1 pub1 default-on clk active" tabindex="150">     
          <div class="news-article-body-fields">    
          <h1 class="field-content">TITLE</h1>    

          <div class="image col-xs-12 col-sm-12 col-md-12 col-lg-12 news-image-caption">
<span class="caption"></span>
</div>    
          <span class="field-content Speeches-news-article-date"><div class="inline-date">
  PLACE <span class="date-display-single" property="dc:date" datatype="xsd:dateTime" content="2019-06-10T18:15:00-04:00">June 10, 2019</span>
</div></span>    
  <div class="views-field views-field-body">        <p><span lang="EN-CA" xml:lang="EN-CA">CHECK AGAINST DELIVERY</span></p><p><span lang="EN-CA" xml:lang="EN-CA">Good morning, everyone. </span></p><p><span lang="EN-CA" xml:lang="EN-CA">Before we get into things, I want to take a second to thank ____ – for his introduction, yes, but more importantly, for his leadership. </p> SPEECHES CONTINUE IN <P> TAGS. 

下面是我的Python脚本：

# Libraries
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import requests
import time

# Opening up connection and grabbing HTML file via Chrome
url = 'https://pm.gc.ca/eng/news/speeches'
browser = webdriver.Chrome()
browser.get(url)

# Delaying scrapper to prevent scrapper from closing too soon
browser.implicitly_wait(2)

# Creating loop to open up all divs with same class name
article_list = browser.find_elements_by_css_selector(".views-row.pub1.default-on.clk")

# All titles for expanded divs printed. Works!
for article in article_list:
    print(article.text)


# Only works for first article in list
for article in article_list:
    article.click()

    time.sleep(3)

    # Getting title
    title = browser.find_element_by_xpath("//h1[@class = 'field-content']")
    print(title.text)   

    # Getting date
    date = browser.find_element_by_class_name("date-display-single")
    print(date.text)

    # Getting place
    place = browser.find_element_by_xpath("//div[@class = 'inline-date']")
    print(place.text)

    # Getting speech
    speech_div = browser.find_elements_by_xpath("//span[@lang = 'EN-CA']")

    for p in speech_div:
        print(p.text)

现在，我可以把第一篇文章的整个演讲删掉。然后，驱动程序单击下一个可扩展div中的第二个语音，输出一大堆空格，并以与第二个语音相同的方式继续下几个语音（一大堆空格）。你知道吗

任何帮助都将不胜感激！你知道吗

Answer 1

语音细节是使用AJAX请求加载的。这意味着您甚至不必为此使用硒，requests就足够了，这大大加快了速度：

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent':  'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0'
}


def make_soup(url: str) -> BeautifulSoup:
    res = requests.get(url, headers=headers)
    res.raise_for_status()
    return BeautifulSoup(res.text, 'html.parser')


def fetch_speech_details(speech_id: str) -> str:
    url = f'https://pm.gc.ca/eng/views/ajax?view_name=news_article&view_display_id=block&view_args={speech_id}'
    res = requests.get(url, headers=headers)
    res.raise_for_status()
    data = res.json()
    html = data[1]['data']
    soup = BeautifulSoup(html, 'html.parser')
    body = soup.select_one('.views-field-body')
    return str(body)


def scrape_speeches(soup: BeautifulSoup) -> dict:
    speeches = []
    for teaser in soup.select('.teaser'):
        title = teaser.select_one('.title').text.strip()
        speech_id = teaser['data-nid']
        speech_html = fetch_speech_details(speech_id)
        s = {
            'title': title,
            'details': speech_html
        }
        speeches.append(s)


if __name__ == "__main__":
    url = 'https://pm.gc.ca/eng/news/speeches'
    soup = make_soup(url)
    speeches = scrape_speeches(soup)
    from pprint import pprint
    pprint(speeches)

输出：

[
    {'title': 'PM remarks for Lunar Gateway', 'details': '<div class="views-field views-field-body"> <p>CHECK AGAINST DELIVERY</p><p>Hello everyone!</p><p>I’m delighted to be here at the Canadian Space Agency to share some great news with Canadians.</p><p>I’d like to start by thanking the President of the Agency, Sylvain Laporte ... },
    {...},
    ....
]

Answer 2

您需要将搜索范围限定到当前div，而不是整个文档。对当前元素调用find*（article而不是browser）：

title = article.find_element_by_xpath("//h1[@class = 'field-content']")
speech_div = article.find_elements_by_xpath("//span[@lang = 'EN-CA']")