Python请求返回不同的d

Question

我尝试使用以下代码检索历史天气数据：

url = 'https://www.wunderground.com/history/airport/KDCA/2017/05/07/DailyHistory.html'
querystring = {'format': '1'}
headers = {'cache-control': 'no-cache', 
           "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8"}
response = requests.get(url, headers=headers, params=querystring)
print(response.text)

我从请求中得到的结果如下：

TimeEDT,TemperatureF,Dew PointF,Humidity,Sea Level PressureIn,VisibilityMPH,Wind Direction,Wind SpeedMPH,Gust SpeedMPH,PrecipitationIn,Events,Conditions,WindDirDegrees,DateUTC<br />
12:52 AM,50.0,43.0,77,29.63,10.0,WSW,6.9,-,N/A,,Partly Cloudy,240,2017-05-07 04:52:00<br />
1:52 AM,51.1,42.1,71,29.64,10.0,WSW,10.4,-,N/A,,Scattered Clouds,250,2017-05-07 05:52:00<br />
2:52 AM,50.0,41.0,71,29.65,10.0,WSW,10.4,-,N/A,,Partly Cloudy,240,2017-05-07 06:52:00<br />

但是，如果我在浏览器（Safari）中使用相同的url，我会得到以下结果：

TimeEDT,TemperatureF,Dew PointF,Humidity,Sea Level PressureIn,VisibilityMPH,Wind Direction,Wind SpeedMPH,Gust SpeedMPH,PrecipitationIn,Events,Conditions,FullMetar,WindDirDegrees,DateUTC
12:52 AM,50.0,43.0,77,29.63,10.0,WSW,6.9,-,N/A,,Partly Cloudy,METAR KDCA 070452Z 24006KT 10SM FEW050 10/06 A2963 RMK AO2 SLP034 T01000061 401830100,240,2017-05-07 04:52:00
1:52 AM,51.1,42.1,71,29.64,10.0,WSW,10.4,-,N/A,,Scattered Clouds,METAR KDCA 070552Z 25009KT 10SM SCT080 11/06 A2964 RMK AO2 SLP037 T01060056 10128 20100 53012,250,2017-05-07 05:52:00
2:52 AM,50.0,41.0,71,29.65,10.0,WSW,10.4,-,N/A,,Partly Cloudy,METAR KDCA 070652Z 24009KT 10SM FEW050 10/05 A2965 RMK AO2 SLP040 T01000050,240,2017-05-07 06:52:00

请注意，“FullMetar”列在Safari中返回，但在请求输出中丢失。（有趣的是，Chrome还省略了“FullMetar”列。）

我想使用python检索数据，包括“FullMetar”列。你知道吗

（这是一个非常简单的页面，没有auth、css、javascript等，根据我读到的其他问题，这似乎是一个典型的问题。）

Answer 1

在浏览浏览器开发人员检查器之后，我发现Chrome和Safari之间的Prefscookie是不同的：

铬： FAVS:1|WXSN:1|PWSOBS:1|WPHO:1|PHOT:1|RADC:0|RADALL:0|HIST0:NULL|GIFT:1|PHOTOTHUMBS:50|HISTICAO:KDCA*NULL|EXPFCT:1|

野生动物园： FAVS:1|WXSN:1|PWSOBS:1|WPHO:1|PHOT:1|RADC:0|RADALL:0|HIST0:NULL|GIFT:1|PHOTOTHUMBS:50|HISTICAO:KDCA*NULL|EXPFCT:1|SHOWMETAR:1|

因此，将带有SHOWMETAR:1的Prefscookie添加到我的请求可以解决我的问题：

url = 'https://www.wunderground.com/history/airport/KDCA/2017/05/07/DailyHistory.html'
cookies = {'Prefs':'FAVS:1|WXSN:1|PWSOBS:1|WPHO:1|PHOT:1|RADC:0|RADALL:0|HIST0:NULL|GIFT:1|PHOTOTHUMBS:50|HISTICAO:NULL|EXPFCT:1|SHOWMETAR:1|'}
querystring = {'format': '1'}
response = requests.get(url, params=querystring, cookies=cookies)   
print(response.text)