执行某些步骤后，无法从网页中获取动态填充的数字

Question

我已经创建了一个脚本，使用请求模块和BeautifulSoup库从网页中获取一些表格内容。要生成表，必须手动执行我在所附图像中显示的步骤。我在下面粘贴的代码可以正常工作，但我试图解决的主要问题是以编程方式获取title编号，在本例中，它是628086906，附加到我在这里硬编码的table_link上

在第6步中单击工具按钮后，当您将光标悬停在地图上时，您可以看到此选项Multiple，当您单击此选项时，它将引导您找到包含标题号的url

start page

这正是脚本所遵循的the steps

这是linc编号0030278592，需要在步骤6的输入框中输入

我尝试过（在table_link中使用硬编码的标题号时使用了一个）：

import requests
from bs4 import BeautifulSoup

link = 'https://alta.registries.gov.ab.ca/spinii/logon.aspx'
lnotice = 'https://alta.registries.gov.ab.ca/spinii/legalnotice.aspx'
search_page = 'https://alta.registries.gov.ab.ca/SpinII/SearchSelectType.aspx'
map_page = 'http://alta.registries.gov.ab.ca/SpinII/mapindex.aspx'
map_find = 'http://alta.registries.gov.ab.ca/SpinII/mapfinds.aspx'
table_link = 'https://alta.registries.gov.ab.ca/SpinII/popupTitleSearch.aspx?title=628086906'

def get_content(s,link):   
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['uctrlLogon:cmdLogonGuest.x'] = '80'
    payload['uctrlLogon:cmdLogonGuest.y'] = '20'

    r = s.post(link,data=payload)
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['cmdYES.x'] = '52'
    payload['cmdYES.y'] = '8'

    s.post(lnotice,data=payload)
    s.headers['Referer'] = 'https://alta.registries.gov.ab.ca/spinii/welcomeguest.aspx'
    
    s.get(search_page)
    s.headers['Referer'] = 'https://alta.registries.gov.ab.ca/SpinII/SearchSelectType.aspx'
    
    s.get(map_page)
    
    r = s.get(map_find)
    s.headers['Referer'] = 'http://alta.registries.gov.ab.ca/SpinII/mapfinds.aspx'
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['__EVENTTARGET'] = 'Finds$lstFindTypes'
    payload['Finds:lstFindTypes'] = 'Linc'
    payload['Finds:ctlLincNumber:txtLincNumber'] = '0030278592'
    
    r = s.post(map_find,data=payload)
    
    r = s.get(table_link)
    print(r.text)


if __name__ == "__main__":
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'
        get_content(s,link)

How can I grab the title number from the url?

或

How can I fetch all the linc numbers from that site so that I don't need to use map at all?

The only problem with this site is that it is unavailable in daytime for maintenance.

Answer 1

有两个选项可以获取您正在寻找的信息，其中一个是您可能已经知道的selenium

打开“网络”选项卡，并在您将鼠标悬停在地图上时监视浏览器正在传递的请求（无论是否向服务器发出请求）。对于请求和BS4，您最好的选择是，如果已经加载了数据，则下面的解决方案可能无法工作

import re 
print(re.findall(r’628086906’, r.text) )

如果它打印出数字，则意味着数据以json格式提供，并通过页面加载，您可以加载json或使用正则表达式查找。否则，您唯一的选择就是硒