无限滚动使用请求.getpython方法

Question

是否可以使用just get方法实现无限量的读取废料。例如

http://www.justdial.com/Ahmedabad/Bearing-Dealers/ct-302676

当我们向下滚动时，为每个页面提供以下链接

^{pr2}$

到目前为止，我的代码看起来像：

import requests

def readJustDial(c):
    hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
           'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
           'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
           'Accept-Encoding': 'none',
           'Accept-Language': 'en-US,en;q=0.8',
           'Connection': 'keep-alive'}    
    for i in range(1,10):
        url = 'http://www.justdial.com/functions/ajxsearch.php?national_search=0&act=pagination&city='+str(c)+'&search=Bearing+Dealers&where=&catid=302676&psearch=&prid=&page='+str(i)+'&SID=&mntypgrp=0&toknbkt=&bookDate=&jdsrc='
        page = requests.get(url,hdr)

def main(): #this is main function of this program
    allCities=["Ahmedabad","Hyderabad","Bangalore","Kolkata","Chennai","Mumbai","Delhi-NCR","Pune"]
    for city in allCities:
        readJustDial(city)
        #print(city)

if __name__ == "__main__":
    main()    

也请建议我可以对我的现有代码进行任何更改。我只是在学习python，所以任何建议都是好的。在

Answer 1

尝试模仿正常工作的浏览器xhr请求附带的标题。您可以使用浏览器的开发工具（我使用chrome的）查看这些标题。 当我查看请求时，我看到它是用以下标题发送的：

Accept:application/json, text/javascript, */*; q=0.01
Accept-Encoding:gzip, deflate, sdch
Accept-Language:he-IL,he;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Cookie:f5avrbbbbbbbbbbbbbbbb=BEKIPCFANCEHKADKNPJJJLHGCDKJOEEGKIIEPAAPHGEDJDNKFFBPCKEGMMIAECHOLECIMLJDAICKIFECEPMNKJNMKIDIMHPCOMHNNHMANENHHKEGMABPKFGKBAPGCHCJ; ppc=; PHPSESSID=bh34mlv2ba4gmgbntjtsjtt753; www=1712105664.20480.0000; _gat=1; scity=Ahmedabad; sarea=; dealBackCity=Ahmedabad; inweb_city=Ahmedabad; profbd=0; bdcheck=1; _ga=GA1.2.338746795.1480258713; tab=toprs; BDprofile=1; prevcatid=302676; view=lst_v; main_city=Ahmedabad
Host:www.justdial.com
Referer:http://www.justdial.com/Ahmedabad/Bearing-Dealers/ct-302676
User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
X-Requested-With:XMLHttpRequest

尝试用这些头发送一个请求，除了cookies（通常它们只是临时工作）。在

如果那也不管用，你就需要饼干了。您可以使用浏览器（例如使用selenium），也可以对网页或Cookie进行反向工程，并尝试编写一个获取工作cookie的方法。在