我不能用靓汤刮网页

Question

我试图用python3中的BeautifulSoup来废弃https://www.crowdcube.com/investments?sector=technology。你知道吗

Traceback (most recent call last):

      File "D:\DataVisualization\lib\urllib\request.py", line 163, in urlopen
        return opener.open(url, data, timeout)
      File "D:\DataVisualization\lib\urllib\request.py", line 472, in open
        response = meth(req, response)
      File "D:\DataVisualization\lib\urllib\request.py", line 582, in http_response
        'http', request, response, code, msg, hdrs)
      File "D:\DataVisualization\lib\urllib\request.py", line 510, in error
        return self._call_chain(*args)
      File "D:\DataVisualization\lib\urllib\request.py", line 444, in _call_chain
        result = func(*args)
      File "D:\DataVisualization\lib\urllib\request.py", line 590, in http_error_default
        raise HTTPError(req.full_url, code, msg, hdrs, fp)
    urllib.error.HTTPError: HTTP Error 403: Forbidden

Answer 1

使用请求，此站点不需要UA:

In [23]: import requests

In [24]: r = requests.get('https://www.crowdcube.com/investments?sector=technology')

In [25]: r.status_code
Out[25]: 200