我试图点击展开按钮，然后刮桌子

Question

我正在抓取一个网站表格https://csr.gov.in/companyprofile.php?year=FY+2015-16&CIN=L00000CH1990PLC010573，但我没有得到我想要的确切结果。我想从这个链接11列，“公司名称”，“类”，“国家”，“公司类型”，“中华民国”，“子类别”，“上市状态”。这是7列，在此之后，您可以看到一个扩展按钮“2017-18财年的企业社会责任详细信息”，当您点击该按钮时，您将看到另外4列“平均净利润”、“企业社会责任规定支出”、“企业社会责任支出”、“本地区支出”。我想在csv文件中的所有这些列。我写了一个代码，但它不能正常工作。我附加了一个图像的结果，供参考。这是我的密码。请帮助获取这些数据

from selenium import webdriver
import pandas as pd
from bs4 import BeautifulSoup
from urllib.request import urlopen
import requests
import csv

driver = webdriver.Chrome()

url_file = "csrdata.txt"

with open(url_file, "r") as url:
    url_pages = url.read()
# we need to split each urls into lists to make it iterable
pages = url_pages.split("\n") # Split by lines using \n
data = []

# now we run a for loop to visit the urls one by one
for single_page in pages:
    driver.get(single_page)
    r = requests.get(single_page)
    soup = BeautifulSoup(r.content, 'html5lib')
    driver.find_element_by_link_text("CSR Details of FY 2017-18").click()
    table = driver.find_elements_by_xpath("//*[contains(@id,'colfy4')]")
    about = table.__getitem__(0).text
    x = about.split('\n')

    print(x)

    data.append(x)
    df = pd.DataFrame(data)
    print(df)

    # write to csv
    df.to_csv('csr.csv')

---

Answer 1

您不需要使用selenium，因为所有信息都在html代码中。您还可以使用pandas inbuild函数pd_read_html()将html表直接转换为数据帧

data = []
for single_page in pages:
    r = requests.get(single_page)
    soup = BeautifulSoup(r.content, 'html5lib')

    table = soup.find_all('table')               #finds all tables
    table_top = pd.read_html(str(table))[0]      #the top table

    try:                                         #try to get the other table if exists
        table_extra = pd.read_html(str(table))[7]
    except:
        table_extra = pd.DataFrame()            
    
    result = pd.concat([table_top, table_extra])
    data.append(result)

pd.concat(data).to_csv('test.csv')

输出：

                            0                          1
0                       Class                     Public
1                       State                 Chandigarh
2                Company Type           Other than Govt.
3                         RoC             RoC-Chandigarh
4                Sub Category  Company limited by shares
5              Listing Status                     Listed
0          Average Net Profit                          0
1  CSR Prescribed Expenditure                          0
2                   CSR Spent                          0
3            Local Area Spent                          0