使用PythonBS4从网页中刮取所有表

Question

我尝试使用bs4从网页获取表，并使用pandas将它们转换为csv。在

网页有两个表，我可以得到第一个表，但只有第二个表的标题被刮伤。在

我试过下面的代码。在

from urllib2 import Request, urlopen
from bs4 import BeautifulSoup
from scrapelib import table_to_2d
import pandas as pd

ehurl = 'https://www.fpi.nsdl.co.in/web/Reports/Latest.aspx'
hd = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1;WOW64;rv:46.0) Gecko/46.0 Firefox/46.0'}

raq = Request(ehurl, headers=hd)
resp = urlopen(raq)
eh_page = resp.read()

soup = BeautifulSoup(eh_page, "html.parser")

i=1
for qeros in soup.findAll("table"):
    x = table_to_2d(qeros)
    df = pd.DataFrame(x)
    df.to_csv("fpi" + str(i) + ".csv", sep=",", header=False, index=False)
    i += 1

函数table_to_2d取自https://stackoverflow.com/a/48451104/2724299

Answer 1

对于第二个表，实际的tr、th和{}元素在table标记下没有结构化。因此，刮掉所有tr、th、和{}标记将生成所需的数据，并且通过应用itertools.groupby，可以获得原始的表结构。在

import requests, itertools
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.fpi.nsdl.co.in/web/Reports/Latest.aspx').text, 'html.parser')
table_data = [[j.text for j in (lambda x:i.find_all('td') if not x else x)(i.find_all('th'))] for i in d.find_all('tr')] 
final_table = [list(b) for _, b in itertools.groupby(table_data, key=lambda x:x[0].startswith('Daily Trends'))]
table1, table2 = [final_table[i]+final_table[i+1] for i in range(0, len(final_table), 2)]

输出：

table：

^{pr2}$

table2：

[['Daily Trends in FPI Derivative Trades on 08-Aug-2018'], ['Reporting Date', 'Derivative Products', 'Buy', 'Sell', 'Open Interest at the'], ['Open Interest at the'], ['No. of Contracts', 'Amount in Crore', 'No. of Contracts', 'Amount in Crore', 'No. of Contracts', 'Amount in Crore'], ['08-Aug-2018', 'Index Futures', '18797.00', '1732.24', '16696.00', '1600.94', '303684.00', '26636.51'], ['Index Options', '495820.00', '50403.69', '512765.00', '52075.29', '673371.00', '60394.18'], ['Stock Futures', '176472.00', '11999.53', '178301.00', '12020.70', '1116162.00', '83275.79'], ['Stock Options', '98471.00', '6949.88', '101906.00', '7204.18', '116286.00', '8824.33'], ['Interest Rate Futures', '0.00', '0.00', '0.00', '0.00', '2530.00', '47.57'], ['The above report is compiled on the basis of reports submitted to depositories by NSE and BSE on 08-Aug-2018 and constitutes  FPIs/FIIs trading / position of the previous trading day.']]