无法使用BeautifulSoup刮表

Question

从下面的代码中：我只得到了1行数据

url = 'http://investmentmoats.com/DividendScreener/DividendScreener.php'
res = requests.get(url)
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0] 
df = pd.read_html(str(table))[0]

有人能帮忙吗？在

Answer 1

尝试以下操作：

url = 'http://investmentmoats.com/DividendScreener/DividendScreener.php'
res = requests.get(url)
soup = BeautifulSoup(res.text, 'lxml')
for row in soup.find('table').find_all('tr'):
    print(' '.join([x.text for x in row.find_all('td')]))
    # Or just use '[x.text for x in row.find_all('td')]' in your data frame.

部分输出：

^{pr2}$

编辑：如上述程序中的注释所述，要将数据存储在数据框中，只需创建列表并将其添加到数据框中即可。在

^{3}$

Answer 2

您可以尝试以下方法：

>>> from urllib.request import Request, urlopen
>>> from bs4 import BeautifulSoup
>>> url = 'http://investmentmoats.com/DividendScreener/DividendScreener.php'
>>> req = Request(url,headers={'User-Agent': 'Mozilla/5.0'})
>>> webpage = urlopen(req).read()
>>> soup = BeautifulSoup(webpage, "html.parser")
>>> required = soup.find_all("table", {"class":"securityTable"})
>>> x = []
>>> for i in required:
...     x.append(i.get_text())
>>> for i in x:
...     print(i)