Python webscraping非对象失败破坏HTML？

Question

我在python中解析脚本时遇到问题。我已经在另一个页面（雅虎财经）尝试过了，效果很好。但在晨星上它却不起作用。 我在table变量的末端“NoneObject”中得到错误。我想这和晨星遗址的结构有关，但我不确定。也许索姆尼能告诉我出了什么问题。 或者是因为晨星网站的网站结构无法使用我的简单脚本？在

一个简单的csv直接从晨星导出不是一个解决方案，因为我想把这个脚本用于其他没有这个功能的站点。在

import requests
import csv
from bs4 import BeautifulSoup
from lxml import html

url = 'http://financials.morningstar.com/ratios/r.html?t=SBUX&region=USA&culture=en_US'
response = requests.get(url)
html = response.content

soup = BeautifulSoup(html)
table = soup.find('table', attrs={'class': 'r_table1 text2'})

print table.prettify() #debugging

list_of_rows = []
for row in table.findAll('tr'):
   list_of_cells =[]

   for cell in row.findAll(['th','td']):
        text = cell.text.replace(' ', '')
        list_of_cells.append(text)
   list_of_rows.append(list_of_cells)
print list_of_rows #debugging

outfile = open("./test.csv", "wb")
writer = csv.writer(outfile)
writer.writerows(list_of_rows)

Answer 1

该表是通过对端点的单独XHR调用动态加载的，该端点将返回JSONP响应。模拟该请求，从JSONP响应中提取JSON字符串，用json加载它，从componentData键提取HTML并用BeautifulSoup加载：

import json
import re

import requests
from bs4 import BeautifulSoup

# make a request
url = 'http://financials.morningstar.com/financials/getFinancePart.html?&callback=jsonp1450279445504&t=XNAS:SBUX&region=usa&culture=en-US&cur=&order=asc&_=1450279445578'
response = requests.get(url)

# extract the HTML under the "componentData"
data = json.loads(re.sub(r'([a-zA-Z_0-9\.]*\()|(\);?$)', '', response.content))["componentData"]

# parse HTML
soup = BeautifulSoup(data, "html.parser")
table = soup.find('table', attrs={'class': 'r_table1 text2'})
print(table.prettify())