如何在selenium中获取结束标记

Question

我正在处理分页，并希望我的脚本能够刮取一个表，单击“下一步”按钮，刮取下一个表，然后单击“下一步”，直到无法再单击为止

可点击和不可点击的唯一区别似乎是disabled>结束标记

我的想法是创建一个while循环并单击按钮，直到禁用的标记消失，但我不确定首先如何获取该标记

即使按钮被禁用，Selenium也不会抛出“Element not Interactiable”错误，因此我认为我不能继续这样做

airport_list = []
fees_list = []

airports = ["https://www.aopa.org/destinations/business/13035#fees", "https://www.aopa.org/destinations/business/35555#fees"]

for a in airports:
    driver.get(a)
    time.sleep(3)

    # Click dropdown
    driver.find_element_by_xpath('//div[@class = "mat-select-arrow"]').click()
    time.sleep(1)
    # Select "All aircraft"
    driver.find_elements_by_xpath('//span[@class = "mat-option-text"]')[8].click()
    time.sleep(2)

    try:        

        # Check if fees are available
        driver.find_element_by_xpath('//mat-row[@class = "mat-row ng-star-inserted"]')

        #Scrape each row
        fees_table = driver.find_elements_by_xpath('//mat-row[@class = "mat-row ng-star-inserted"]')
            for fee in fees_table:
            fees_list.append(fee.text)
            airport_list.append(a)

        #Click on "Next" button

        driver.find_elements_by_xpath('//span[@class = "mat-button-wrapper"]')[4].click()
        time.sleep(2)

    except:
        fees_list.append("This location has not disclosed fees or does not charge fees.")
        airport_list.append(a)           

driver.close()

Answer 1

不要继续下一页，而是使用下面代码中所示的最大行限制。除此之外，您真的不需要使用try-except-block-

for a in airports:
    driver.get(a)
    time.sleep(3)

    # Click dropdown
    driver.find_element_by_xpath('//div[@class = "mat-select-arrow"]').click()
    time.sleep(1)
    # Select "All aircraft"
    driver.find_elements_by_xpath('//span[@class = "mat-option-text"]')[8].click()
    time.sleep(3)

    # select 100 Items per page if items are present
    if len(driver.find_elements_by_xpath(".//mat-select[@aria-label='Items per page:']")) > 0 :
        driver.find_element_by_xpath(".//mat-select[@aria-label='Items per page:']").click()
        time.sleep(3)
        driver.find_element_by_xpath(".//span[@class='mat-option-text' and text()='100']/parent::mat-option").click()

    # Scrape each row
    fees_table = driver.find_elements_by_xpath('//mat-row[@class = "mat-row ng-star-inserted"]')
    for fee in fees_table:
        fees_list.append(fee.text)
    print(fees_list)

        # if needed then Click on "Next" button using this xpath and apply same for loop as above
        #driver.find_elements_by_xpath(".//button[@aria-label='Next page']").click()
driver.close()

Answer 2

我能够从表的底部提取项目的最大数量，将该数字除以10，然后四舍五入到最接近的数字。然后我用这个数字迭代一个范围

airport_list = []
fees_list = []

airports = ["https://www.aopa.org/destinations/business/13035#fees"]

for a in airports:
    driver.get(a)
    time.sleep(3)

    # Click dropdown
    driver.find_element_by_xpath('//div[@class = "mat-select-arrow"]').click()
    time.sleep(1)
    # Select "All aircraft"
    driver.find_elements_by_xpath('//span[@class = "mat-option-text"]')[8].click()
    time.sleep(2)

    try:        
        # Check if fees are available
        driver.find_element_by_xpath('//mat-row[@class = "mat-row ng-star-inserted"]')

        # Get number of items
        number_of_items = driver.find_element_by_xpath('//div[@class = "mat-paginator-range-label"]').text.split()[-1]
        #print(number_of_items)

        if float(number_of_items) >= 11:
            number_of_button_clicks = math.ceil(float(number_of_items)/10)
        else:
            number_of_button_clicks = 0
        #print(number_of_button_clicks)

        for click in range(0, number_of_button_clicks):
            #Scrape each row
            fees_table = driver.find_elements_by_xpath('//mat-row[@class = "mat-row ng-star-inserted"]')
            for fee in fees_table:
                fees_list.append(fee.text)
                airport_list.append(a)

            #Click on "Next" button

            driver.find_elements_by_xpath('//span[@class = "mat-button-wrapper"]')[4].click()
            time.sleep(2)

    except:
        fees_list.append("This location has not disclosed fees or does not charge fees.")
        airport_list.append(a)


#print(fee_list)
#print(airport_list)            

driver.close()