使用bs4进行网页抓取时不返回数值

Question

我试图从中获取的页面是https://www.investagrams.com/Stock/ac，我试图获取价格值（779.00），但我的代码只返回：{{ViewStockPage.Data.Stock.LatestStockHistory.Last | numberPriceFormat}}

我的代码：

from bs4 import BeautifulSoup
import requests

r = requests.get('https://www.investagrams.com/Stock/ac')
soup = BeautifulSoup(r.text, "lxml")

main = soup.find('div', class_= 'd-flex flex-row justify-content-between')

header = main.find('h4', class_= 'mb-0')

price = header.find('span', class_= 'mr-2').string

print(price)

网站HTML:

<h4 class="mb-0"> 
<small class="ng-binding">Ayala Corporation (PSE:AC) </small> <br>
<strong> 
 <span class="mr-2 ng-binding" data-ng-class="ViewStockPage.Data.Stock.LatestStockHistory.LastClass">779.00 </span> 
 <span data-ng-class="{'stock-up-caret stockprice-up' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage > 0, 'stock-down-caret stockprice-down' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage < 0, 'glyphicon glyphicon-minus stockprice-flat': ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage == 0}" class="stock-down-caret stockprice-down" style=""> </span> 
 <span style="font-size: 13px; vertical-align: middle;" data-ng-class="{'stockprice-up' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage > 0, 'stockprice-down' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage < 0, 'stockprice-flat': ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage == 0}" class="stockprice-down"> 
   <span class="ng-binding">-21.00 </span> 
   <span class="ml-1 ng-binding">-2.62% </span> 
 </span> 
</strong> 
</h4>

Answer 1

您试图从中获取的页面正在使用JavaScript异步填充DOM。您可以期望BeautifulSoup不适用于这样的页面，因为BeautifulSoup只能看到在服务器向您提供文档时直接烘焙到HTML中的内容

如果在浏览器中查看页面并记录网络流量，您将看到对各种REST API端点发出的多个请求，其中一个端点/InvestaApi/Stock/ViewStock，并将“股票代码”作为查询字符串参数。该端点的响应是JSON，包含您试图获取的信息。您只需模拟HTTP GET请求：

def get_price(stock_code):
    import requests

    url = "https://webapi.investagrams.com/InvestaApi/Stock/ViewStock"

    params = {
        "stockCode": stock_code,
        "defaultExchangeType": "1",
        "cv": "1622292000-0-"
    }

    headers = {
        "accept": "application/json",
        "accept-encoding": "gzip, deflate",
        "referer": "https://www.investagrams.com/",
        "user-agent": "Mozilla/5.0"
    }

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()

    return response.json()["LatestStockHistory"]["Last"]

def main():

    print(get_price("ac"))
    
    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

输出：

779
>>>