从多个链接抓取文本数据并将抓取的数据存储在csv文件中

Question

到目前为止我所做的：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(r"C:\Users\Documents\chromedriver")
cursor.execute("select TOP (10) Url from TABLE_NAME ORDER BY ID DESC ")
data = cursor.fetchall()
result_data = cursor.fetchall()

for link in result_data:
urllink = link.Url
driver.get(urllink)

如何获取url循环的文本数据，并将url和爬网文本数据保存到CSV文件中
我也试过用这种又脏又漂亮的汤

由于有多个Url，所以甚至不能使用css选择器获取数据

Answer 1

创建一个数据框，并将所有数据推入其中，当您完成刮取时。将其导出到csv文件是保存表格数据的最佳方法

Answer 2

查看下面的示例代码格式

import scrapy
from selenium import webdriver
from selenium.webdriver.common.keys import Keys


cursor.execute("select TOP (10) Url from TABLE_NAME ORDER BY ID DESC ")
result_data = cursor.fetchall()
start_urls = []
for link in result_data:
    start_urls.append(link.Url)


class ToScrapeSpider(scrapy.Spider):
    name = 'toscrape'
    start_urls = start_urls

    def parse(self, response):

        # Write the code to scrape data, If the URLs are different, use different cases
        # Also add the items to a pandas data frame, dataframe_name.to_csv('path/to/filename.csv') does the work.