如何将新列添加到csv的Scrapy输出中？

Question

我分析网站，它工作得很好，但我需要添加新的列与id的输出。该列以csv格式保存，URL为：

https://www.ceneo.pl/48523541, 1362
https://www.ceneo.pl/46374217, 2457

我的蜘蛛代号：

^{pr2}$

在结果中，我得到了错误的数据，因为（zip函数？）ID交替使用：

urlid,price,seller
1362,109,eMAG
1457,116,electro.pl
1362,597,apollo.pl
1457,597,allegro.pl

它应该输出：

urlid,price,seller
1362,109,eMAG
1362,116,electro.pl
1457,597,apollo.pl
1457,597,allegro.pl

Answer 1

您可以在start_requests中获得{}，并使用meta={'id': id_}分配给请求，稍后在{}中可以使用response.meta['id']获得{}。在

这样您将在parse中得到正确的ID。在

我使用字符串data而不是file来创建工作示例。在

#!/usr/bin/env python3

import scrapy

data = '''https://www.ceneo.pl/48523541, 1362
https://www.ceneo.pl/46374217, 2457'''

class QuotesSpider(scrapy.Spider):

    name = "quotes" 

    def start_requests(self):
        #f = open('urls.csv', 'r')

        f = data.split('\n')

        for row in f:
            url, id_ = row.split(',')

            url = url.strip()
            id_ = id_.strip()

            #print(url, id_)

            # use meta to assign value 
            yield scrapy.Request(url=url, callback=self.parse, meta={'id': id_})

    def parse(self, response):
        # use meta to receive value
        id_ = response.meta["id"]

        all_prices = response.xpath('(//td[@class="cell-price"] /a/span/span/span[@class="value"]/text())[position() <= 10]').extract()
        all_sellers = response.xpath('(//tr/td/div/ul/li/a[@class="js_product-offer-link"]/text())[position()<=10]').extract()

        all_sellers = [ item.replace('Opinie o ', '') for item in all_sellers ]

        for price, seller in zip(all_prices, all_sellers):
            yield {'urlid': id_, 'price': price.strip(), 'seller': seller.strip()}

#  - it runs without project and saves in `output.csv`  -

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',
    'FEED_FORMAT': 'csv',
    'FEED_URI': 'output.csv', 
})
c.crawl(QuotesSpider)
c.start()

顺便说一句：有标准函数id()，所以我使用变量id_而不是{}