Python爬虫 - 如果目标返回404，套接字错误会中断脚本

Question

在构建一个网络爬虫，目的是收集数据并输出成XLS格式时，我遇到了一个错误。当我再次测试我想要抓取的域名列表时，程序在收到一个套接字错误时出现了问题。我希望能找到一个'if'语句，这样可以在遇到坏网站时跳过解析，继续我的循环。有没有什么好的建议？

workingList = xlrd.open_workbook(listSelection)
workingSheet = workingList.sheet_by_index(0)
destinationList = xlwt.Workbook()
destinationSheet = destinationList.add_sheet('Gathered')
startX = 1
startY = 0
while startX != 21:
    workingCell = workingSheet.cell(startX,startY).value
    print ''
    print ''
    print ''
    print workingCell
    #Setup
    preSite = 'http://www.'+workingCell
    theSite = urlopen(preSite).read()
    currentSite = BeautifulSoup(theSite)
    destinationSheet.write(startX,0,workingCell)

这是错误信息：

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    homeMenu()
  File "C:\Python27\farming.py", line 31, in homeMenu
    openList()
  File "C:\Python27\farming.py", line 79, in openList
    openList()
  File "C:\Python27\farming.py", line 83, in openList
    openList()
  File "C:\Python27\farming.py", line 86, in openList
    homeMenu()
  File "C:\Python27\farming.py", line 34, in homeMenu
    startScrape()
  File "C:\Python27\farming.py", line 112, in startScrape
    theSite = urlopen(preSite).read()
  File "C:\Python27\lib\urllib.py", line 84, in urlopen
    return opener.open(url)
  File "C:\Python27\lib\urllib.py", line 205, in open
    return getattr(self, name)(url)
  File "C:\Python27\lib\urllib.py", line 342, in open_http
    h.endheaders(data)
  File "C:\Python27\lib\httplib.py", line 951, in endheaders
    self._send_output(message_body)
  File "C:\Python27\lib\httplib.py", line 811, in _send_output
    self.send(msg)
  File "C:\Python27\lib\httplib.py", line 773, in send
    self.connect()
  File "C:\Python27\lib\httplib.py", line 754, in connect
    self.timeout, self.source_address)
  File "C:\Python27\lib\socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11004] getaddrinfo failed

Answer 1

嗯，这看起来像是我在网络连接断了的时候遇到的错误。HTTP 404 错误是当你有网络连接，但你输入的网址找不到时出现的错误。

这里没有用到条件语句来处理异常；你需要使用 try/except 结构 来“捕捉”这些异常。

更新：这里有个演示：

import urllib

def getconn(url):
    try:
        conn = urllib.urlopen(url)
        return conn, None
    except IOError as e:
        return None, e

urls = """
    qwerty
    http://www.foo.bar.net
    http://www.google.com
    http://www.google.com/nonesuch
    """
for url in urls.split():
    print
    print url
    conn, exc = getconn(url)
    if conn:
        print "connected; HTTP response is", conn.getcode()
    else:
        print "failed"
        print exc.__class__.__name__
        print str(exc)
        print exc.args

输出：

qwerty
failed
IOError
[Errno 2] The system cannot find the file specified: 'qwerty'
(2, 'The system cannot find the file specified')

http://www.foo.bar.net
failed
IOError
[Errno socket error] [Errno 11004] getaddrinfo failed
('socket error', gaierror(11004, 'getaddrinfo failed'))

http://www.google.com
connected; HTTP response is 200

http://www.google.com/nonesuch
connected; HTTP response is 404

请注意，到目前为止我们只是打开了连接。接下来你需要做的是检查 HTTP 响应代码，然后决定是否有值得获取的内容，使用 conn.read()。