如何使用Python和Beautiful Soup从flexbox元素/容器中刮取数据

Question

我正在尝试使用python、beautiful soup和selenium从一个实用程序网站上获取数据。我试图收集的数据是：时间、原因、状态等。当我运行一个典型的页面请求时，解析页面，并解析我正在寻找的数据（数据在 id="OutageListTable"), and print it, the divs and strings are nowhere to be found. When I inspect the page element, the data is there, but it is in a flex container.

这是我正在使用的代码：

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import urllib3
from selenium import webdriver

my_url = 'https://www.pse.com/outage/outage-map'

browser = webdriver.Firefox()
browser.get(my_url)

html = browser.page_source
page_soup = soup(html, features='lxml')

outage_list = page_soup.find(id='OutageListTable')
print(outage_list)

browser.quit()

如何检索flex/flexbox容器中的信息？我在网上找不到任何资源来帮助我解决这个问题

Answer 1

数据通过Javascript动态加载。您可以使用requests模块来获取数据

例如：

import json
import requests

url = 'https://www.pse.com/api/sitecore/OutageMap/AnonymoussMapListView'

data = requests.get(url).json()

# uncomment this to print all data:
#print(json.dumps(data, indent=4))

for d in data['PseMap']:
    print('{} - {}'.format(d['DataProvider']['PointOfInterest']['Title'], d['DataProvider']['PointOfInterest']['MapType']))
    for info in d['DataProvider']['Attributes']:
        print(info['Name'], info['Value'])
    print('-' * 80)

印刷品：

Bellingham - Outage
Start time 06/02 06:09 PM
Est. restoration time 06/03 06:30 AM
Customers impacted 1
Cause Trees/Vegetation
Status Crew assigned
Last updated 06/02 11:50 PM
                                        
Deming - Outage
Start time 06/02 07:10 PM
Est. restoration time 06/03 03:30 AM
Customers impacted 568
Cause Accident
Status Repair crew onsite
Last updated 06/02 11:50 PM
                                        
Everest - Outage
Start time 06/02 10:42 AM
Customers impacted 4
Cause Scheduled Outage
Status Repair crew onsite
Last updated 06/02 10:50 AM
                                        
Kenmore - Outage
Start time 06/02 09:59 PM
Est. restoration time 05/29 01:00 AM
Customers impacted 2
Cause Scheduled Outage
Status Repair crew onsite
Last updated 06/02 10:05 PM
                                        
Kent - Outage
Start time 06/02 06:43 PM
Est. restoration time To Be Determined
Customers impacted 26
Cause Car/Equip Accident
Status Waiting for repairs
Last updated 06/02 10:15 PM
                                        
Kent - Outage
Start time 06/02 10:09 PM
Est. restoration time To Be Determined
Customers impacted 13
Cause Under Investigation
Status Repair crew onsite
Last updated 06/02 10:15 PM
                                        
Northwest Bellevue - Outage
Start time 06/02 11:28 PM
Est. restoration time To Be Determined
Customers impacted 14
Cause Under Investigation
Status Repair crew onsite
Last updated 06/02 11:30 PM
                                        
Pacific - Outage
Start time 06/02 06:19 PM
Est. restoration time 06/03 02:30 AM
Customers impacted 3
Cause Accident
Status Crew assigned
Last updated 06/02 11:00 PM
                                        
Woodinville - Outage
Start time 06/02 08:29 PM
Est. restoration time 06/03 03:30 AM
Customers impacted 2
Cause Under Investigation
Status Crew assigned
Last updated 06/03 12:15 AM