扫描图像的URL模式？

Question

我想扫描一个网站，下载其中的图像。 例如，对于这样的网站URL：a.example.com/2vZBkE.jpg，我需要一个bot来扫描从a.example.com/aaaaaa.jpg到a.example.com/AAAAAA.jpg到a.example.com/999999.jpg，如果有图像，保存URL或下载图像。你知道吗

我试过使用Python和Scrapy，但我对它非常陌生。 我只能做到这一点：

import scrapy

from scrapy.contrib.spiders import Rule, CrawlSpider
from scrapy.contrib.linkextractors import LinkExtractor
from example.items import ExampleItem

class exampleSpider(CrawlSpider):
    name = 'example'
    allowed_domains = ['example.com']
    start_urls = ['http://a.example.com/2vZBkE']
    #rules = [Rule(LinkExtractor(allow=['/.*']),'parse_example')]
     rules = (Rule(SgmlLinkExtractor(allow=('\/%s\/.*',)), callback='parse_example'),
    )

    def parse_example(self,response):
        image = ExampleItem()
        image['title']=response.xpath(\
            "//h5[@id='image-title']/text()").extract()

        rel = response.xpath("//img/@src").extract()
        image ['image_urls'] = ['http:'+rel[0]]
        return image

我想我需要改变这一行：

rules = (Rule(SgmlLinkExtractor(allow=('\/%s\/.*',)), callback='parse_example'),
        )

要想把%s限制在6个字符以内，并使之成为可能的组合。有什么想法吗？你知道吗

Answer 1

提取链接，如 href=“a。example.com/123456.jpg“

使用以下正则表达式：

“=\”（\S+/[\w\d]{6}.jpg）

Answer 2

我不知道刮痧。但是你可以用requests和itertools来做

from string import ascii_letters, digits
from itertools import product
import requests

# You need to implement this function to download images
# check this http://stackoverflow.com/questions/13137817
def download_image(url):
    print url  

def check_link(url):
    r = requests.head(url)
    if r.status_code == 200:
        return True
    return False

# Check all possible combinations and check them
def generate_links():
    base_url = "http://a.example.com/{}.jpg"
    for combination in product(ascii_letters+digits, repeat=5):
        url = base_url.format("".join(combination))
        if check_link(url):
            download_image(url)