刮LinkedIn

Question

我正试图从LinkedIn上获取特定的搜索结果。我可以使用selenium登录并获得页面的HTML代码。然而，我只得到每一页的前5条记录。 我附上了代码脚本。 我试过用靓汤，但没用。 用于抓取的url:https://www.linkedin.com/search/results/all/?keywords=director%20supply%20chain&origin=GLOBAL_SEARCH_HEADER&page=1

names=[]


for link in search_urls:

    url = link
    driver.get(url)
    html = driver.page_source
    index= html.find("<span class=\"name actor-name\"")

    html2=html[index:]
    while html2.find("<span class=\"name actor-name\"") != -1:

        index=html2.find("<span class=\"name actor-name\"")
        remaining=html2[index:]

        start=remaining.find(">")

        end=remaining.find("</")


        names.append(remaining[start+1:end])

        remaining=remaining[end:]

        index=remaining.find("</span>")
        html2=remaining[index:]

Answer 1

它使用scroll事件从ajax获得更多结果，使用.location_once_scrolled_into_view来滚动。在

names=[]

for link in search_urls:
    url = link
    driver.get(url)
    hasScroll = True
    while hasScroll:
        try:
            hasScroll = driver.find_element_by_css_selector('.search-result__occlusion-hint')
            hasScroll.location_once_scrolled_into_view
            # wait ajax request, increase if needed
            time.sleep(2)
        except:
            break

    # no more scrollable content, extract it
    actorNames = driver.find_elements_by_css_selector('.name.actor-name')
    names = [name.text for name in actorNames]