在抓取html表之前单击按钮

Question

您好，我正在尝试scape一个HTML表，我有工作代码

然而，一个URL包含两个html表。第一个表包含“季度”数字，默认情况下加载url。单击表格上方的按钮时，可以切换到第二个带有“年度”编号的表格

我的代码只拾取加载url时出现的第一个默认（季度）表

如何让我的python代码刮取第二个“年度”表？硒能做到这一点吗？如果有，谁能提供指导

#!/usr/local/bin/python3

import requests
import pandas as pd

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0'}
r = requests.get("https://www.investing.com/equities/exxon-mobil-income-statement", headers=headers)
df = pd.read_html(r.content)[1]
print(df)

非常感谢

Answer 1

在谷歌搜索和其他一些堆栈帖子之后，终于实现了这一点：

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

browser = webdriver.Firefox(executable_path=r'/Users/xxxxxx/Documents/python/web_drivers/geckodriver')
browser.get('https://www.investing.com/equities/exxon-mobil-income-statement')
linkElem = browser.find_element_by_link_text('Annual')
linkElem.click()

r = browser.find_element_by_css_selector("#rrtable > table").get_attribute('outerHTML')
browser.quit()

soup = BeautifulSoup(r, 'html.parser')

df = pd.read_html(str(soup))[0]

print(df)

Answer 2

请尝试以下操作：

Sub Web_Table()
    Dim HTMLDoc As New HTMLDocument
    Dim objTable As Object
    Dim lRow As Long
    Dim lngTable As Long
    Dim lngRow As Long
    Dim lngCol As Long
    Dim ActRw As Long
    Dim objIE As InternetExplorer
    Set objIE = New InternetExplorer
    objIE.Navigate "https://www.investing.com/equities/exxon-mobil-income-statement"

    Do Until objIE.ReadyState = 4 And Not objIE.Busy
        DoEvents
    Loop
    Application.Wait (Now + TimeValue("0:00:03")) 'wait for java script to load
    HTMLDoc.body.innerHTML = objIE.Document.body.innerHTML
    With HTMLDoc.body
        Set objTable = .getElementsByTagName("table")
        For lngTable = 0 To objTable.Length - 1
            For lngRow = 0 To objTable(lngTable).Rows.Length - 1
                For lngCol = 0 To objTable(lngTable).Rows(lngRow).Cells.Length - 1
                    ThisWorkbook.Sheets("Sheet1").Cells(ActRw + lngRow + 1, lngCol + 1) = objTable(lngTable).Rows(lngRow).Cells(lngCol).innerText
                Next lngCol
            Next lngRow
            ActRw = ActRw + objTable(lngTable).Rows.Length + 1
        Next lngTable
    End With
    objIE.Quit
End Sub

Answer 3

对

你可以用硒来做

driver.get("https://www.investing.com/equities/exxon-mobil-income-statement")
annual_button = driver.find_element_by_css_selector("#leftColumn > div.alignBottom > div.float_lang_base_1 > a:nth-child(1)")
annual_button.click()
print(driver.find_element_by_css_selector("#rrtable > table").get_attribute('innerHTML'))

这里有一个python代码

它是干什么的？它进入页面，通过css选择器查找年度按钮元素，然后单击它。然后，它通过css选择器查找表并打印其HTML

希望能有帮助