使用scrapy刮除带有javascript按钮和ajax请求的asp.net网站

Question

我一直试图从asp.net网站上删除一些日期，起始页应该是以下页面： http://www.e3050.com/Items.aspx?cat=SON

首先，我希望每页显示50个项目（来自select元素） 第二，我想翻页。

我尝试了以下代码，每页50个项目，但没有成功：

start_urls = ["http://www.e3050.com/Items.aspx?cat=SON"]    
def parse(self, response):
        requests = []
        hxs = HtmlXPathSelector(response)

        # Check if there's more than 1 page         
        if len(hxs.select('//span[@id="ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_lbl_PageSize"]/text()').extract()) > 0:
            # Get last page number
            last_page = hxs.select('//span[@id="ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_lbl_PageSize"]/text()').extract()[0]
            i = 1

            # preparing requests for each page
            while i < (int(last_page) / 5)  + 1:
                requests.append(Request("http://www.e3050.com/Items.aspx?cat=SON", callback=self.parse_product))
                i +=1 

            # posting form date (50 items and next page button)
            requests.append(FormRequest.from_response(
                    response,
                    formdata={'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pagesddl':'50',
                              '__EVENTTARGET':'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pager1$ctl00$ctl01'},
                    callback=self.parse_product,
                    dont_click=True
                )
            )

            for request in requests:
                yield request

Answer 1

我没有深入研究过你的代码，但我看到了一些奇怪的东西：

        # Get last page number
        last_page = hxs.select('//span[@id="ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_lbl_PageSize"]/text()').extract()[0]
        i = 1

        # preparing requests for each page
        while i < (int(last_page) / 5)  + 1:
            requests.append(Request("http://www.e3050.com/Items.aspx?cat=SON", callback=self.parse_product))
            i +=1 

首先，与使用i进行这些操作不同，您可以执行以下操作：

        for i in xrange(1, last_page // 5 + 1):

然后你会：

            requests.append(Request("http://www.e3050.com/Items.aspx?cat=SON", callback=self.parse_product))

您是否正在创建对同一URL的多个请求？

Answer 2

看看这是一个精确的解决方案。。

在parse方法中，每页选择50个产品

在第50页中处理分页

start_urls = ['http://www.e3050.com/Items.aspx?cat=SON']
pro_urls = [] # all product Urls

def parse(self, response): # select 50 products on each page
    yield FormRequest.from_response(response,
        formdata={'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pagesddl': '50',
                  'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$sortddl': 'Price(ASC)'},
        meta={'curr': 1, 'total': 0, 'flag': True},
        dont_click=True,
        callback=self.page_rs_50)

def page_rs_50(self, response): # paginate the pages
    hxs = HtmlXPathSelector(response)
    curr = int(response.request.meta['curr'])
    total = int(response.request.meta['total'])
    flag = response.request.meta['flag']
    self.pro_urls.extend(hxs.select(
        "//td[@class='name']//a[contains(@id,'ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_itemslv_ctrl')]/@href"
    ).extract())
    if flag:
        total = hxs.select(
            "//span[@id='ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_lbl_pagesizeBtm']/text()").re('\d+')[0]
    if curr < total:
        curr += 1
        yield FormRequest.from_response(response,
            formdata={'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pagesddl': '50',
                      'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$sortddl': 'Price(ASC)',
                      'ctl00$ctl00$ScriptManager1': 'ctl00$ctl00$ScriptManager1|ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pager1$ctl00$ctl01'
                , '__EVENTTARGET': 'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$pager1$ctl00$ctl01',
                      'ctl00$ctl00$ContentPlaceHolder1$ItemListPlaceHolder$hfVSFileName': hxs.select(
                          ".//input[@id='ctl00_ctl00_ContentPlaceHolder1_ItemListPlaceHolder_hfVSFileName']/@value").extract()[
                                                                                          0]},
            meta={'curr': curr, 'total': total, 'flag': False},
            dont_click=True,
            callback=self.page_rs_50
        )
    else:
        for pro in self.pro_urls:
            yield Request("http://www.e3050.com/%s" % pro,
                callback=self.parse_product)


def parse_product(self, response):
    pass
    #TODO Implementation Required For Parsing