urllib3:抛出MaxRetryError时如何获取响应?

2024-03-28 19:04:47 发布

您现在位置:Python中文网/ 问答频道 /正文

我使用Pythonurllib3+requests中的Retry模块来处理第三方API出现零星错误的情况。我遇到的一个问题是,如果重试不断失败,我会得到一个exceptions.MaxRetryError,永远看不到响应是什么。如果有来自服务器的有价值的调试数据呢?你知道吗

有没有一种方法仍然可以在抛出MaxRetryError的情况下得到反应?你知道吗

下面是我的代码

from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def req_with_retry(retries=3, backoff_factor=0.5, status_forcelist=(400, 404, 500, 502, 504,), method_whitelist=frozenset(['POST', 'HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE']), session=None,):
    ''' 
    this returns a session that functions like the requests module but with retries built it for certain status codes
    '''
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

r = req_with_retry().get(url="https://www.google.com")

Tags: adaptersessionstatuswith情况requestsmethodretry
1条回答
网友
1楼 · 发布于 2024-03-28 19:04:47

首先,我想提一下from requests.packages.urllib3.util.retry import Retry已经不再需要了,因为urllib3已经从请求中取消了背书。考虑改用from urllib3.util.retry import Retry形式,因为它不涉及黑客(参见https://github.com/psf/requests/blob/v2.22.0/requests/packages.py)。你知道吗

您无法同时获得MaxRetryError和urllib3的响应。但是您可以从urllib3获得响应,并在请求级别使用raise_for_status。但是,raise_for_status()不会受到status_forcelist的影响,因此您可能需要自己重新实现它。你知道吗

下面是对代码的一个修改,1/重试状态2/打印最终响应3/引发异常。你知道吗

import logging

import urllib3
import requests
import requests.adapters

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)


def req_with_retry(
    retries=3,
    backoff_factor=0.5,
    status_forcelist=(400, 404, 500, 502, 504),
    method_whitelist=frozenset(
        ["POST", "HEAD", "TRACE", "GET", "PUT", "OPTIONS", "DELETE"]
    ),
    session=None,
):
    """Returns a session that functions like the requests module but with retries
    built it for certain status codes
    """
    session = session or requests.Session()
    retry = urllib3.Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist,
        raise_on_status=False,
    )
    adapter = requests.adapters.HTTPAdapter(max_retries=retry)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session


r = req_with_retry().get(url="https://httpbin.org/status/404")
print(r.status_code, r.text)
r.raise_for_status()

相关问题 更多 >