当HTML没有改变的时候，我怎么能用Python进行webscrape呢？

Question

我目前正在使用Selenium和beauthoulsoup尝试从googlefinance获取财务报表数据。例如：

http://www.google.com/finance?q=GOOG&fstype=ii

打开谷歌损益表。当我让Selenium点击页面顶部的“Balance Statement”和“Cash Flow”按钮时，页面上的图表和表格会发生变化，但url不变，当我拉页面源时，它是带有损益表表的原始页面。我的代码发布在下面：

driver = webdriver.Firefox()
driver.get("http://www.google.com/finance?q=" + ticker[0] + "&fstype=ii")

url1 = driver.page_source
soup1 = BeautifulSoup(url1)

element = driver.find_element_by_xpath('//*[@id=":1"]/a/b/b')
element.click()

driver.implicity_wait(3.0)
url2 = driver.page_source
soup2 = BeautifulSoup(url2)

element = driver.find_element_by_xpath('//*[@id=":2"]/a/b/b')
element.click()

driver.implicity_wait(3.0)
url3 = driver.page_source
soup3 = BeautifulSoup(url3)

driver.quit()

感谢任何帮助。谢谢。在

Answer 1

这里不需要BeautifulSoupHTML解析器。Selenium本身在navigating on the page and getting elements by almost everything you can imagine中足够强大。在

您需要的表数据位于具有不同的div元素中。激活每个选项卡并从适当的div获取数据

下面是一个在所有选项卡中打印出表标题的示例：

from selenium import webdriver

def print_header(element):
    table = element.find_element_by_id('fs-table')
    for row in table.find_elements_by_tag_name('th'):
        print row.text


driver = webdriver.Firefox()
driver.get('http://www.google.com/finance?q=GOOG&fstype=ii')

print_header(driver.find_element_by_id('incinterimdiv'))
print "  "

# activate Balance Sheet
element = driver.find_element_by_xpath('//*[@id=":1"]/a/b/b')
element.click()

print_header(driver.find_element_by_id('balinterimdiv'))
print "  "

# activate Cash Flow
element = driver.find_element_by_xpath('//*[@id=":2"]/a/b/b')
element.click()

print_header(driver.find_element_by_id('casinterimdiv'))

driver.quit()

印刷品：

^{pr2}$