Python BeautifulSoup web刮片返回无

Question

from bs4 import BeautifulSoup
import requests

url = 'https://www.facilitron.com/searchfacility/u:facilitron/lat:37.8271784/lng:-122.2913078/dist:20/activities:-1/types:-1/namelike:grass%20field'

source = requests.get(url)
soup = BeautifulSoup(source.content, 'lxml')

find = soup.find('div', class_="search-results-body")
print(find)

我正在尝试获取提供网站的img、标题和位置，但BeautifulSoup无法获取类为“search results body”的div中的元素。当我尝试查找类位于该div中的div时，它将不返回任何内容

Answer 1

尝试将您的解析器更改为

soup = BeautifulSoup(source.content, 'html.parser')

Answer 2

试试这个

find = soup.find('div', {'class': 'clearfix'})

Answer 3

您查找的内容是高度动态的，不存在于页面源（Ctrl+U）中。但是，使用适当的参数发出post http请求，您可以从一些json响应中获取它们。我只分析了每个线索的名称和地址。您可以从回复中找到与每个潜在客户相关的许多其他信息

import requests

url = 'https://www.facilitron.com/api/search/facilities/facilitron'

payload = {
    'search_option[startLoc][]': ['-122.2913078','37.8271784'],
    'search_option[distance]': '20',
    'search_option[activities][]': '-1',
    'search_option[start_row]': '0',
    'search_option[limit]': '15',
    'search_option[types][]': '-1',
    'search_option[namelike]': 'grass field'
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
    'Referer': 'https://www.facilitron.com/searchfacility/u:facilitron/lat:37.8271784/lng:-122.2913078/dist:20/activities:-1/types:-1/namelike:grass%20field'
}

res = requests.post(url,data=payload,headers=headers)
for item in res.json()['facilities']:
    print(item['name'],item['address'])