我怎样才能不出错地抓取这些图像？

Question

我正在尝试抓取这个论坛（http://www.xossip.com/showthread.php?t=1384077）的图片（或图片链接）。我试过美丽的汤4，下面是我尝试过的代码：

import requests
from bs4 import BeautifulSoup

def spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.xossip.com/showthread.php?t=1384077&page=' + str(page)
        sourcecode= requests.get(url)
        plaintext = sourcecode.text
        soup = BeautifulSoup(plaintext)
        for link in soup.findAll('a',{'class': 'alt1'}):
            src = link.get('src')
            print(src)


        page += 1
spider(1)

我应该如何更正它以获得像pzy.be/example这样的图像链接？在

Answer 1

最简单的方法是请求每个页面并过滤img标记：

from bs4 import BeautifulSoup
from requests import get
import re

def get_wp():
    start_url = "http://www.xossip.com/showthread.php?t=1384077&page={}"
    for i in range(73):
        r = get(start_url.format(i))
        soup = BeautifulSoup(r.content)
        for img in (i["src"] for i in  soup.find_all("img", src=re.compile("http://pzy.be.*.jpg"))):
           yield img

Answer 2

好的，我通过获取所有的#post_message_*div，然后从每个div中获取图像。在

import requests
from bs4 import BeautifulSoup

def spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.xossip.com/showthread.php?t=1384077&page=' + str(page)
        sourcecode= requests.get(url)
        plaintext = sourcecode.text
        soup = BeautifulSoup(plaintext)
        divs = soup.findAll('div', id=lambda d: d and d.startswith('post_message_'))
        for div in divs:
            src = div.find('img')['src']
            if src.startswith('http'): # b/c it could be a smilie or something like that 
                print(src)

        page += 1

spider(1)