<p>您可以使用<code>selenium</code>滚动到页面底部。但是,网站仍然需要一段时间才能加载建议。因此,此解决方案通过使用<code>while</code>循环等待产品建议部分出现:</p>
<pre><code>from selenium import webdriver
from bs4 import BeautifulSoup as soup
import time
d = webdriver.Chrome('/Users/jamespetullo/Downloads/chromedriver')
d.get('https://www.michaelkors.com/logo-tape-ribbed-stretch-viscose-sweater/_/R-US_MH86NXK5ZW')
last_height = d.execute_script("return document.body.scrollHeight")
while True:
d.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(0.5)
new_height = d.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
start = soup(d.page_source, 'html.parser')
while start.find('div', {'class':'product-tile-rfk'}) is None:
start = soup(d.page_source, 'html.parser')
products = [i.find_all('li', {'class':'product-name-container'})[0].text for i in start.find_all('div', {'class':'product-tile-rfk'})]
</code></pre>
<p>输出:</p>
<pre><code>['Ribbed Stretch-Viscose Tank', 'Ribbed Stretch-Viscose Tank Top', 'Ribbed Stretch-Viscose Tank Top', 'Stretch-Viscose Tank', 'Striped Ribbed Sweater Tank', 'Tie-Dye Stretch-Viscose Sweater', 'Striped Stretch-Viscose Tank', 'Striped Stretch-Cotton Sweater', 'Rainbow Stretch-Viscose Short-Sleeve Sweater', 'Stretch-Viscose Cropped Tank', 'Neon Striped Stretch-Viscose Tank Top', 'Geometric Grid Stretch-Viscose Top', 'Logo Tape Stretch-Viscose Pullover', 'Logo Tape Stretch-Viscose Cropped T-Shirt', 'Logo Tape Cotton-Jersey Top', 'Logo Tape Viscose Joggers', 'Logo Tape Buttoned Track Pants', 'Contrast Stripe Joggers', 'Contrast Stripe Hooded Jacket', 'Logo Tape Stretch-Viscose Pencil Skirt', 'Logo Tape Stretch-Viscose Zip-Up Hoodie', 'Stretch-Viscose Joggers', 'Cotton Asymmetric Turtleneck', 'Logo Tape Ribbed Knit Dress']
</code></pre>