请求响应留下了一些数据

Question

我正试图使用请求从谷歌获取一些数据，但它无法返回网页中的所有内容

代码：

import requests
from bs4 import BeautifulSoup

headers = {'user-agent':'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Mobile Safari/537.36'}
url = 'https://www.google.com/search?num=50&q="potato+is+good"'

r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')

for idx, val in enumerate(soup.find_all('em'), 1):
    print('{} = {}'.format(idx, val))

输出：

1 = <em>potato is good</em>
2 = <em>potato is good</em>
3 = <em>potato is good</em>
4 = <em>Potato is good</em>
5 = <em>potato is good</em>
6 = <em>potato is good</em>
7 = <em>Potato is good</em>
8 = <em>potato is good</em>
9 = <em>potato is good</em>
10 = <em>potato is good</em>
11 = <em>potato.is.good</em>

它只显示了11个结果，但当我在谷歌上手动搜索时，有35个以上的结果

我的代码可能有什么问题

Answer 1

它会不会像你通过移动设备搜索一样返回结果？ 我刚刚试过，在iPhone上的谷歌首页上只得到了11个结果。也许一个不同的用户代理，如下面的，会这样做吗

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36

编辑：

我运行了这个：

import requests
from bs4 import BeautifulSoup

headers = {'user-agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'}
url = 'https://www.google.com/search?num=50&q="potato+is+good"'

r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')

for idx, val in enumerate(soup.find_all('em'), 1):
    print('{} = {}'.format(idx, val))

并且得到：

1 = <em>potato is good</em>
2 = <em>potato is good</em>
3 = <em>potato is good</em>
4 = <em>potato is good</em>
5 = <em>potatoÔÇØ is good</em>
6 = <em>potato is good</em>
7 = <em>potato-is good</em>
8 = <em>potato is good</em>
9 = <em>Potato is good</em>
10 = <em>potato is good</em>
11 = <em>potato is good</em>
12 = <em>potato is good</em>
13 = <em>potato is good</em>
14 = <em>potato is good</em>
15 = <em>potato is good</em>
16 = <em>potato is good</em>
17 = <em>Potato is good</em>
18 = <em>potato is good</em>
19 = <em>potato is good</em>
20 = <em>potato is good</em>
21 = <em>potato is good</em>
22 = <em>potato is good</em>
23 = <em>potato is good</em>
24 = <em>potato is good</em>
25 = <em>potato is good</em>
26 = <em>potato is good</em>
27 = <em>potato is good</em>
28 = <em>potato is good</em>
29 = <em>potato is good</em>
30 = <em>potato is good</em>
31 = <em>potato is good</em>
32 = <em>potato is good</em>
33 = <em>potato is good</em>
34 = <em>potato is good</em>
35 = <em>potato is good</em>
36 = <em>potato is good</em>