我在从scrapy spider下载/刮取图像时遇到一个值错误我正在使用图像管道

Question

我得到一个值错误：

raise ValueError('Missing scheme in request url: %s' % self._url)

ValueError: Missing scheme in request url: h

我的项目.py代码为：

class Brand(scrapy.Item):
    name = scrapy.Field()
    url = scrapy.Field()
    brand_image = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()

我的设置.py是：

BOT_NAME = 'scraper'    
SPIDER_MODULES = ['scraper.spiders']
NEWSPIDER_MODULE = 'scraper.spiders'
ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
IMAGES_STORE = 'images'

我的蜘蛛代码：

import scrapy
import json
from scraper.items import Brand


class QuotesSpider(scrapy.Spider):
    name = "brandDetails"
    allowed_domains = ["ozhat-turkiye.com"]
    with open('brands.json') as data_file:
        data_item = json.load(data_file)
        start_urls = list()
    for item in data_item:
        start_urls.append(item["url"])

    def parse(self, response):
        item = Brand()
        name = response.css("div.th::text").extract_first()
        name = name.replace('Products of ', '')
        item['name'] = name
        item['url'] = response.url
        urls = response.css("div.productimage img::attr(src)").extract_first()
        urls = response.urljoin(urls)
        item['image_urls'] = urls
        yield item

Answer 1

Missing scheme in request url

总是意味着你的URL无效，它缺少http://和https://

所以在你拥有的图片url之前加上https://或http://

`https://` + response.css("div.productimage img::attr(src)").extract_first()