多进程 + 请求出现异常：AttributeError: 'file'对象没有'out'属性

Question

我正在尝试创建一个类，使用多进程和请求库来并行发送多个请求。不过，我遇到了一个问题，程序总是卡住，并且给我一个很难理解的错误信息，我也不知道为什么。

下面是我的代码，基本上是使用一个池（Pool）和一个回调函数，把结果放进一个列表里。我有一个要求，就是每个网址都需要有一个“硬性超时”，也就是说，如果某个网址下载内容的时间超过几秒钟，我就想跳过它。所以我使用了池的超时功能，并对尝试过的网址和返回内容的网址做了对比，尝试过但没有返回的那些网址就认为是失败了。以下是我的代码：

import time
import json
import requests
import sys
from urlparse import parse_qs
from urlparse import urlparse
from urlparse import urlunparse
from urllib import urlencode
from multiprocessing import Process, Pool, Queue, current_process
from multiprocessing.pool import ThreadPool
from multiprocessing import TimeoutError
import traceback
from sets import Set
from massweb.pnk_net.pnk_request import pnk_request_raw
from massweb.targets.fuzzy_target import FuzzyTarget
from massweb.payloads.payload import Payload

class MassRequest(object):

    def __init__(self, num_threads = 10, time_per_url = 10, request_timeout = 10, proxy_list = [{}]):

        self.num_threads = num_threads
        self.time_per_url = time_per_url
        self.request_timeout = request_timeout
        self.proxy_list = proxy_list

        self.results = []
        self.urls_finished = []
        self.urls_attempted = []

        self.targets_results = []
        self.targets_finished = []
        self.targets_attempted = []

    def add_to_finished(self, x):

        self.urls_finished.append(x[0])
        self.results.append(x)

    def add_to_finished_targets(self, x):

        self.targets_finished.append(x[0])
        self.targets_results.append(x)

    def get_urls(self, urls):

        timeout = float(self.time_per_url * len(urls))
        pool = Pool(processes = self.num_threads)
        proc_results = []

        for url in urls:
            self.urls_attempted.append(url)
            proc_result = pool.apply_async(func = pnk_request_raw, args = (url, self.request_timeout, self.proxy_list), callback = self.add_to_finished)
            proc_results.append(proc_result)

        for pr in proc_results:

            try:
                pr.get(timeout = timeout)

            except:
                pool.terminate()
                pool.join()

        pool.terminate()
        pool.join()
        list_diff = Set(self.urls_attempted).difference(Set(self.urls_finished))

        for url in list_diff:
            sys.stderr.write("URL %s got timeout" % url)
            self.results.append((url, "__PNK_GET_THREAD_TIMEOUT"))

if __name__ == "__main__":

    f = open("out_urls_to_fuzz_1mil")
    urls_to_request = []
    for line in f:
        url = line.strip()
        urls_to_request.append(url)

    mr = MassRequest()
    mr.get_urls(urls_to_request)

这是被线程调用的函数：

def pnk_request_raw(url_or_target, req_timeout = 5, proxy_list = [{}]):

    if proxy_list[0]:
        proxy = get_random_proxy(proxy_list)
    else:
        proxy = {}

    try:
        if isinstance(url_or_target, str):

            sys.stderr.write("Requesting: %s with proxy %s\n" % (str(url_or_target), str(proxy)))
            r = requests.get(url_or_target, proxies = proxy, timeout = req_timeout)
            return (url_or_target, r.text)

        if isinstance(url_or_target, FuzzyTarget):

            sys.stderr.write("Requesting: %s with proxy %s\n" % (str(url_or_target), str(proxy)))
            r = requests.get(url_or_target.url, proxies = proxy, timeout = req_timeout)
            return (url_or_target, r.text)

    except:
        #use this to mark failure on exception
        traceback.print_exc()
        #edit: this is the line that was breaking it all
        sys.stderr.out("A request failed to URL %s\n" % url_or_target)
        return (url_or_target, "__PNK_REQ_FAILED")

对于较小的网址集合，这个方法似乎效果不错，但这是输出结果：

Requesting: http://www.sportspix.co.za/ with proxy {}
Requesting: http://www.sportspool.co.za/ with proxy {}
Requesting: http://www.sportspredict.co.za/ with proxy {}
Requesting: http://www.sportspro.co.za/ with proxy {}
Requesting: http://www.sportsrun.co.za/ with proxy {}
Requesting: http://www.sportsstuff.co.za/ with proxy {}
Requesting: http://sportsstuff.co.za/2011-rugby-world-cup with proxy {}
Requesting: http://www.sportstar.co.za/4-stroke-racing with proxy {}
Requesting: http://www.sportstats.co.za/ with proxy {}
Requesting: http://www.sportsteam.co.za/ with proxy {}
Requesting: http://www.sportstec.co.za/ with proxy {}
Requesting: http://www.sportstours.co.za/ with proxy {}
Requesting: http://www.sportstrader.co.za/ with proxy {}
Requesting: http://www.sportstravel.co.za/ with proxy {}
Requesting: http://www.sportsturf.co.za/ with proxy {}
Requesting: http://reimo.sportsvans.co.za/ with proxy {}
Requesting: http://www.sportsvans.co.za/4x4andmoreWindhoek.html with proxy {}
Handled exception:Traceback (most recent call last):
  File "mass_request.py", line 87, in get_fuzzy_targets
    pr.get(timeout = timeout)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
AttributeError: 'file' object has no attribute 'out'

在最后一个异常出现时，程序就卡住了，我不得不完全结束它。根据我所知，我从来没有尝试访问一个带有“out”属性的文件对象。我的问题是……怎么解决这个问题！？我是不是做错了什么明显的事情？为什么没有更清晰的异常信息呢？

异常处理回调函数超时处理并行计算错误调试多进程请求库线程池

多进程 + 请求出现异常：AttributeError: 'file'对象没有'out'属性

1 个回答

撰写回答