网站存在，但request.head/get超时

Question

我已经编写了一个Python脚本来检查网站是否存在。除了检查http://www.dhl.com-请求超时外，所有操作都正常。我试过GET和HEAD两种方法。我用https://httpstatus.io/和https://app.urlcheckr.com/检查DHL网站，结果是错误的。DHL网站确实存在！这是我的密码：

import requests
a ='http://www.dhl.com'
def check(url):
    try:
        header = {'User-Agent':'Mozilla/5.0 (X11; CrOS x86_64 8172.45.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.64 Safari/537.36'}
        request = requests.head(url, headers = header , timeout = 60)
        code = request.status_code
        if code < 400:
            return "Exist",str(code)
        else:
            return "Not exist", str(code)
    except Exception as e:
        return "Not Exist",str(type(e).__name__)

print(check(a))

如何解决此错误

Answer 1

使用curl进行测试表明，DHL站点还需要几个其他标题

import requests

url = 'http://www.dhl.com'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9,fil;q=0.8',
    }
request = requests.head(url, headers=headers, timeout=60, allow_redirects=True)
print(request.status_code, request.reason)
print(request.history)

没有这些头，curl永远不会得到响应