我可以在scrapy上的web爬虫中创建多个URL吗？

Question

我曾尝试在start_url中创建多个url，但在许多参考文献中都不存在如何在crapy中创建多个url

多个URL类似于tokopedia.com、olx.co.id等

Answer 1

试试这个

from simplified_scrapy import Spider, SimplifiedDoc, SimplifiedMain

class WebsiteSpider(Spider):
    name = 'my_spider'
    allowed_domains = ['example1.com','example2.com']
    start_urls = ['https://www.example1.com','https://www.example2.com']

    # refresh_urls = True  # For debug. If efresh_urls = True, start_urls will be crawled again.

    def extract(self, url, html, models, modelNames):
        doc = SimplifiedDoc(html)
        lstA = doc.listA(
            url=url["url"])  # Get link data for subsequent crawling
        data = [{"title": doc.title.text}]  # Get target data

        return {"Urls": lstA, "Data": data}  # Return data to framework


SimplifiedMain.startThread(WebsiteSpider())