Python 3与Bs4的刮擦

Question

我正在尝试用python3（使用PyCharm）来抓取这个网站的HTML:https://www.idealista.com/venta-viviendas/madrid-madrid/。 我只对房子的价格感兴趣，所以我把搜索范围缩小到某些跨度，比如：

import requests
from bs4 import BeautifulSoup


page = requests.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')

soup = BeautifulSoup(page.text, 'html.parser')


prices=soup.findAll("span", {"class": "item-price h2-simulated"})

print(len(prices))
print(prices)

当我运行它时，我得到这个：0 []

也就是说它什么也没找到。此外，如果我打印所有内容：print（soup）那么大的一个页面，我得到的html内容很少，所以它显然不能获取所有内容。你知道吗

Answer 1

网站是动态的，因此，您需要使用浏览器操作工具，如^{}：

from bs4 import BeautifulSoup as soup
from selenium import webdriver
import re, collections, itertools
d = webdriver.Chrome('/Users/jamespetullo/Downloads/chromedriver')
d.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')
homes = soup(d.page_source, 'html.parser').find_all('div', {'class':'item-info-container'})
results = [i.find('div', {'class':re.compile('price\-row')}).text for i in homes]
price = collections.namedtuple('price', ['original', 'current', 'drop'])
new_prices = [list(filter(None, re.split('\s{2,}', i))) for i in results]
final_prices = [price(a, *[a for a, _ in itertools.zip_longest(b, [None, None])]) for a, *b in new_prices]

输出：

[price(original=' 369.000€', current='395.000 €', drop='7%'), price(original=' 1.250.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido', current='1.650.000 €', drop='4%'), price(original=' 1.750.000€ Garaje incluido', current='1.875.000 €', drop='7%'), price(original=' 1.090.000€ Garaje incluido', current='1.195.000 €', drop='9%'), price(original=' 795.000€ ', current=None, drop=None), price(original=' 1.095.000€ Garaje incluido ', current=None, drop=None), price(original=' 355.000€ Garaje incluido ', current=None, drop=None), price(original=' 995.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.130.000€ Garaje incluido', current='1.190.000 €', drop='5%'), price(original=' 850.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.200.000€ Garaje incluido ', current=None, drop=None), price(original=' 990.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.100.000€ ', current=None, drop=None), price(original=' 830.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.390.000€ Garaje incluido ', current=None, drop=None), price(original=' 685.000€ ', current=None, drop=None), price(original=' 1.150.000€ Garaje incluido', current='1.200.000 €', drop='4%'), price(original=' 915.000€ ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido ', current=None, drop=None), price(original=' 625.000€ Garaje incluido', current='640.000 €', drop='2%'), price(original=' 735.000€', current='760.000 €', drop='3%'), price(original=' 890.000€ Garaje incluido', current='950.000 €', drop='6%'), price(original=' 925.000€', current='999.000 €', drop='7%'), price(original=' 975.000€', current='1.100.000 €', drop='11%'), price(original=' 850.000€ Garaje incluido', current='870.000 €', drop='2%'), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.500.000€ ', current=None, drop=None), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.359.000€ ', current=None, drop=None)]