如何在HTML中的Div类抓取中嵌套If语句和For循环

Question

下面是一个刮刀，它使用漂亮的汤从这个webpage上刮下医生的信息。从下面的html代码可以看到，每个医生在网页上都有一个单独的配置文件，其中显示了医生的姓名、诊所、职业、分类和城市。你知道吗

<div class="views-field views-field-title practitioner__name" ><a href="/practitioners/41824">Marilyn Adams</a></div>
              <div class="views-field views-field-field-pract-clinic practitioner__clinic" ><a href="/clinic/fortius-sport-health">Fortius Sport &amp; Health</a></div>
              <div class="views-field views-field-field-pract-profession practitioner__profession" >Physiotherapist</div>
              <div class="views-field views-field-taxonomy-vocabulary-5 practitioner__region" >Fraser River Delta</div>
              <div class="views-field views-field-city practitioner__city" ></div>

正如您可以从示例html代码中看到的，医生配置文件偶尔会丢失一些信息。如果出现这种情况，我希望刮板打印'不适用'。我需要刮刀打印'不适用'，因为我最终想把每个div类类别（名称，诊所，专业等）到一个数组中，每个列的长度完全相同，这样我就可以正确地导出数据到一个CSV文件。下面是一个示例，我希望输出与实际显示的内容相比较。你知道吗

Actual            Expected

[Names]            [Names]
Greg               Greg
Bob                Bob

[Clinic]           [Clinic]
Sport/Health       Sport/Health
                   N/A

[Profession]       [Profession]
Physical Therapist  Physical Therapist
Physical Therapist  Physical Therapist

[Taxonomy]          [Taxonomy]
Fraser River        Fraser River
                    N/A

[City]              [City]
Vancouver           Vancouver
Vancouver           Vancouver

我试过在每个for循环中嵌套一个if语句，但是代码似乎没有正确循环，因为“N/A”对于每个div类部分只显示一次。有人知道如何用for循环正确地嵌套if语句，以便在每列中获得适当数量的“N/As”吗？提前谢谢！你知道吗

import requests
import re
from bs4 import BeautifulSoup

page=requests.get('https://sportmedbc.com/practitioners')
soup=BeautifulSoup(page.text, 'html.parser')

#Find Doctor Info

for doctor in soup.find_all('div',attrs={'class':'views-field views-field-title practitioner__name'}):
    for a in doctor.find_all('a'):
        print(a.text)

for clinic_name in soup.find_all('div',attrs={'class':'views-field views-field-field-pract-clinic practitioner__clinic'}):
    for b in clinic_name.find_all('a'):
        if b==(''):
            print('N/A')

profession_links=soup.findAll('div',attrs={'class':'views-field views-field-field-pract-profession practitioner__profession'})
for profession in profession_links:
    if profession.text==(''):
        print('N/A')
    print(profession.text)

taxonomy_links=soup.findAll('div',attrs={'class':'views-field views-field-taxonomy-vocabulary-5 practitioner__region'})
for taxonomy in taxonomy_links:
    if taxonomy.text==(''):
        print('N/A')
    print(taxonomy.text)

city_links=soup.findAll('div',attrs={'class':'views-field views-field-taxonomy-vocabulary-5 practitioner__region'})
for city in city_links:
    if city.text==(''):
        print('N/A')
    print(city.text)

Answer 1

对于这个问题，您可以使用来自collections模块（docs here）的ChainMap。这样您就可以定义默认值，在本例中是'n/a'，并且只获取每个医生的信息：

from bs4 import BeautifulSoup
import requests
from collections import ChainMap

url = 'https://sportmedbc.com/practitioners'
soup = BeautifulSoup(requests.get(url).text, 'lxml')

def get_data(soup):
    default_data = {'name': 'n/a', 'clinic': 'n/a', 'profession': 'n/a', 'region': 'n/a', 'city': 'n/a'}

    for doctor in soup.select('.view-practitioners .practitioner'):
        doctor_data = {}

        if doctor.select_one('.practitioner__name').text.strip():
            doctor_data['name'] = doctor.select_one('.practitioner__name').text

        if doctor.select_one('.practitioner__clinic').text.strip():
            doctor_data['clinic'] = doctor.select_one('.practitioner__clinic').text

        if doctor.select_one('.practitioner__profession').text.strip():
            doctor_data['profession'] = doctor.select_one('.practitioner__profession').text

        if doctor.select_one('.practitioner__region').text.strip():
            doctor_data['region'] = doctor.select_one('.practitioner__region').text

        if doctor.select_one('.practitioner__city').text.strip():
            doctor_data['city'] = doctor.select_one('.practitioner__city').text

        yield ChainMap(doctor_data, default_data)

for doctor in get_data(soup):
    print('name:\t\t', doctor['name'])
    print('clinic:\t\t',doctor['clinic'])
    print('profession:\t',doctor['profession'])
    print('city:\t\t',doctor['city'])
    print('region:\t\t',doctor['region'])
    print('-' * 80)

印刷品：

name:        Jaimie Ackerman
clinic:      n/a
profession:  n/a
city:        n/a
region:      n/a
                                        
name:        Marilyn Adams
clinic:      Fortius Sport & Health
profession:  Physiotherapist
city:        n/a
region:      Fraser River Delta
                                        
name:        Mahsa Ahmadi
clinic:      Wellpoint Acupuncture (Sports Medicine)
profession:  Acupuncturist
city:        Vancouver
region:      Vancouver & Sea to Sky
                                        
name:        Tracie Albisser
clinic:      Pacific Sport Northern BC, Tracie Albisser
profession:  Strength and Conditioning Specialist, Exercise Physiologist
city:        n/a
region:      Cariboo - North East
                                        
name:        Christine Alder
clinic:      n/a
profession:  n/a
city:        Vancouver
region:      Vancouver & Sea to Sky
                                        
name:        Steacy Alexander
clinic:      Go! Physiotherapy Sports and Wellness Centre
profession:  Physiotherapist
city:        Vancouver
region:      Vancouver & Sea to Sky
                                        
name:        Page Allison
clinic:      AET Clinic, .
profession:  Athletic Therapist
city:        Victoria
region:      Vancouver Island - Central Coast
                                        
name:        Dana Alumbaugh
clinic:      n/a
profession:  Podiatrist
city:        Squamish
region:      Vancouver & Sea to Sky
                                        
name:        Manouch Amel
clinic:      Mountainview Kinesiology Ltd.
profession:  Strength and Conditioning Specialist
city:        Anmore
region:      Vancouver & Sea to Sky
                                        
name:        Janet Ames
clinic:      Dr. Janet Ames
profession:  Physician
city:        Prince George
region:      Cariboo - North East
                                        
name:        Sandi Anderson
clinic:      n/a
profession:  n/a
city:        Coquitlam
region:      Fraser Valley
                                        
name:        Greg Anderson
clinic:      University of the Fraser Valley
profession:  Exercise Physiologist
city:        Mission
region:      Fraser Valley
                                        

编辑：

要获取列中的输出，可以使用以下示例：

def print_data(header_text, data, key):
    print(header_text)
    for d in data:
        print(d[key])
    print()

data = list(get_data(soup))
print_data('[Names]', data, 'name')
print_data('[Clinic]', data, 'clinic')
print_data('[Profession]', data, 'profession')
print_data('[Taxonomy]', data, 'region')
print_data('[City]', data, 'city')

这张照片：

[Names]
Jaimie Ackerman
Marilyn Adams
Mahsa Ahmadi
Tracie Albisser
Christine Alder
Steacy Alexander
Page Allison
Dana Alumbaugh
Manouch Amel
Janet Ames
Sandi Anderson
Greg Anderson

[Clinic]
n/a
Fortius Sport & Health
Wellpoint Acupuncture (Sports Medicine)
Pacific Sport Northern BC, Tracie Albisser
n/a
Go! Physiotherapy Sports and Wellness Centre
AET Clinic, .
n/a
Mountainview Kinesiology Ltd.
Dr. Janet Ames
n/a
University of the Fraser Valley

[Profession]
n/a
Physiotherapist
Acupuncturist
Strength and Conditioning Specialist, Exercise Physiologist
n/a
Physiotherapist
Athletic Therapist
Podiatrist
Strength and Conditioning Specialist
Physician
n/a
Exercise Physiologist

[Taxonomy]
n/a
Fraser River Delta
Vancouver & Sea to Sky
Cariboo - North East
Vancouver & Sea to Sky
Vancouver & Sea to Sky
Vancouver Island - Central Coast
Vancouver & Sea to Sky
Vancouver & Sea to Sky
Cariboo - North East
Fraser Valley
Fraser Valley

[City]
n/a
n/a
Vancouver
n/a
Vancouver
Vancouver
Victoria
Squamish
Anmore
Prince George
Coquitlam
Mission