我的组上没有文本

Question

我想抓取电子商务网站https://www.bhinneka.com/我已经解析了html，但是为什么我的解析没有包含div类上的任何文本

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import pandas

html = urlopen("https://www.bhinneka.com/jual?cari=iphone&catId=3457215").read()
print (type(html))

soup = BeautifulSoup(html, "html.parser")

data = soup.find("div", "col-lg-3 col-md-3 col-xs-6 bt-product-catalog-item")
print(data)

Answer 1

这些产品是使用json资源中的javascript加载的，这就是为什么在请求的响应中找不到任何内容

您需要发出此请求才能获得产品：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0',
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Accept-Language': 'en-US,en;q=0.5',
    'Content-Type': 'application/json; charset=utf-8;',
    'X-Requested-With': 'XMLHttpRequest',
    'Connection': 'keep-alive',
    'Referer': 'https://www.bhinneka.com/jual?cari=iphone&catId=3457215&page=1',
    'Pragma': 'no-cache',
    'Cache-Control': 'no-cache', }

params = (
    ('size', '40'),
    ('brand', ''),
    ('q', 'iphone'),
    ('catid', '3457215'),
    ('catselect', '3457215'),
    ('page', '1'),
    ('_', '1590651955014'), )

response = requests.get('https://www.bhinneka.com/search/data', headers=headers, params=params)
products = response.json()

您可以调整参数以获得所有页面

Answer 2

您在页面上看到的数据是通过JavaScript加载的。您可以使用requests库来模拟此调用

例如：

import requests
from bs4 import BeautifulSoup

url = 'https://www.bhinneka.com/jual?cari=iphone&catId=3457215'
ajax_url = 'https://www.bhinneka.com/search/data?size=40&brand=&q=iphone&catid={cat_id}&catselect={cat_id}'

cat_id = url.rsplit('=', maxsplit=1)[-1]
data = requests.get(ajax_url.format(cat_id=cat_id), headers={'X-Requested-With':'XMLHttpRequest'}).json()
soup = BeautifulSoup(data['data'], 'html.parser')

for prod in soup.select('.col-sm-3.bt-product-catalog-item__price'):
    name = prod.select_one('.bt-product-catalog-item__title').get_text(strip=True)
    price = prod.select_one('p.bt-product-catalog-item__price-normal').get_text(strip=True)

    # convert to int
    price = int( price.rsplit('.', maxsplit=1)[0].split()[-1].replace('.', '') )

    print('{:<50}{}'.format(name, price))

印刷品：

APPLE iPhone 11 128GB - Yellow                    16499
APPLE iPhone 11 128GB - Green                     16499
APPLE iPhone 11 128GB - Red                       16499
APPLE iPhone 11 128GB - Purple                    16499
APPLE iPhone 11 256GB - White                     18499
APPLE iPhone 11 256GB - Purple                    18499
APPLE iPhone 11 256GB - Green                     18499
APPLE iPhone 11 256GB - Yellow                    18499
APPLE iPhone 11 64GB - Purple                     14499
APPLE iPhone 11 64GB - Yellow                     14499
APPLE iPhone 11 64GB - Green                      14499
APPLE iPhone 11 64GB - Red                        14499
APPLE iPhone 11 256GB - Red                       18499
APPLE iPhone 11 128GB - White                     16499
APPLE iPhone 11 64GB - Black                      14499
APPLE iPhone XS 512GB - Gold                      24999
APPLE iPhone XS 64GB - Space Gray                 17999
APPLE iPhone XS 64GB - Gold                       17999
APPLE iPhone XS 256GB - Space Gray                20999
APPLE iPhone XS 64GB - Silver                     20499
APPLE iPhone XS 256GB - Gold                      23499
APPLE iPhone XS 512GB - Space Gray                27499
APPLE iPhone X 256GB - Silver                     18499
APPLE iPhone 11 256GB - Black                     18499
APPLE iPhone 11 64GB - White                      14499
APPLE iPhone 11 128GB - Black                     16499
APPLE iPhone XS Max 64GB - Gold                   21599
APPLE iPhone XS Max 64GB - Silver                 21599
APPLE iPhone XS Max 256GB - Space Gray            24599
APPLE iPhone XS Max 512GB - Gold                  28599
APPLE iPhone XS Max 512GB - Silver                28599
APPLE iPhone XS Max 256GB - Gold                  24599
APPLE iPhone XS Max 256GB - Silver                24599
APPLE iPhone XS Max 64GB - Space Gray             21599
APPLE iPhone 8 Plus 64GB - Red                    12999
APPLE iPhone 8 Plus 256GB - Silver                14999
APPLE iPhone 11 Pro 256GB - Silver                25499
APPLE iPhone 11 Pro 256GB - Midnight Green        25499
APPLE iPhone 11 Pro 64GB - Midnight Green         21499
APPLE iPhone 11 Pro 64GB - Space Grey             21499