Python靓汤：如何提取标签旁边的文本？

Question

我有以下HTML

<p>
<b>Father:</b> Michael Haughton
<br>
<b>Mother:</b> Diane
<br><b>Brother:</b> 
Rashad Haughton<br>
<b>Husband:</b> <a href="/people/540/000024468/">R. Kelly</a> (m. 1994, annulled that same year)
<br><b>Boyfriend:</b> <a href="/people/420/000109093/">Damon Dash</a> (Roc-a-Fella co-CEO)<br></p>

我必须把标题和文本分开，例如，母亲：黛安。。在

所以最后我会有一个字典列表：

^{pr2}$

我尝试了以下方法，但没有成功：

def parse(u):
    u = u.rstrip('\n')
    r = requests.get(u, headers=headers)
    if r.status_code == 200:
        html = r.text.strip()
        soup = BeautifulSoup(html, 'lxml')
        headings = soup.select('table p')
        for h in headings:
            b = h.find('b')
            if b is not None:
                print(b.text)
                print(h.text + '\n')
                print('=================================')


url = 'http://www.nndb.com/people/742/000024670/'

Answer 1

from bs4 import BeautifulSoup
from urllib.request import urlopen

#html = '''<p>
#<b>Father:</b> Michael Haughton
#<br>
#<b>Mother:</b> Diane
#<br><b>Brother:</b> 
#Rashad Haughton<br>
#<b>Husband:</b> <a href="/people/540/000024468/">R. Kelly</a> (m. 1994, annulled that same year)
#<br><b>Boyfriend:</b> <a href="/people/420/000109093/">Damon Dash</a> (Roc-a-Fella co-CEO)<br></p>'''

page = urlopen('http://www.nndb.com/people/742/000024670/')
source = page.read()

soup = BeautifulSoup(source)

needed_p = soup.find_all('p')[8]

bs = needed_p.find_all('b')

res = {}

for b in bs:
    if b.find_next('a').text:
        res[b.text] = b.find_next('a').text.strip().strip('\n')
    if b.next_sibling != ' ':
        res[b.text] = b.next_sibling.strip().strip('\n')

res

输出：

^{pr2}$

编辑： 有关页面顶部的其他信息：

... (code above) ...
soup = BeautifulSoup(source)

needed_p = soup.find_all('p')[1:4] + [soup.find_all('p')[8]] # here explicitly selecting needed p-tags for further parsing

res = {}

for p in needed_p:
    bs = p.find_all('b')
    for b in bs:
        if b.find_next('a').text:
            res[b.text] = b.find_next('a').text.strip().strip('\n')
        if b.next_sibling != ' ':
            res[b.text] = b.next_sibling.strip().strip('\n')

res

输出：

{'Race or Ethnicity:': 'Black',
 'Husband:': 'R. Kelly',
 'Died:': '25-Aug',
 'Nationality:': 'United States',
 'Executive summary:': 'R&B singer, died in plane crash',
 'Mother:': 'Diane',
 'Birthplace:': 'Brooklyn, NY',
 'Born:': '16-Jan',
 'Boyfriend:': 'Damon Dash',
 'Sexual orientation:': 'Straight',
 'Occupation:': 'Singer',
 'Cause of death:': 'Accident - Airplane',
 'Brother:': 'Rashad Haughton',
 'Remains:': 'Interred,',
 'Gender:': 'Female',
 'Father:': 'Michael Haughton',
 'Location of death:': 'Marsh Harbour, Abaco Island, Bahamas'}

对于正是这个页面，您还可以通过以下方式刮取高中：

res['High School'] = soup.find_all('p')[9].text.split(':')[1].strip()

Answer 2

您正在查找^{}标记属性。 这将给您下一个NavigableString或下一个Tag，这取决于它首先找到的内容。在

以下是如何使用它：

html = """..."""            
soup = BeautifulSoup(html)

bTags = soup.find_all('b')
for it_tag in bTags:
    print(it_tag.string)
    print(it_tag.next_sibling)

输出：

^{pr2}$

这好像有点不对劲。 部分原因是换行和空格，使用str.strip方法可以很容易地消除它们。在

尽管如此，Boyfriend和Husband条目缺少值。 这是因为next_sibling要么是NavigableString（即astr）要么是Tag。 此处<b>标记和<a>标记之间的空白被解释为非空文本：

<b>Boyfriend:</b> <a href="/people/420/000109093/">Damon Dash</a>
                 ^

如果它不存在，<b>Boyfriend:</b>的下一个兄弟将是<a>标记。 既然它存在，你必须检查：

• 下一个兄弟是字符串还是标记
• 如果是字符串，则是否只包含空格。在

如果下一个同级是一个只包含空格的字符串，那么您要查找的信息是NavigableString的下一个同级，它将是一个<a>标记。在

编辑代码：

bTags = soup.find_all('b')

for it_tag in bTags:
    print(it_tag.string)

    nextSibling = it_tag.next_sibling
    if nextSibling is not None:
        if isinstance(nextSibling, str):
            if nextSibling.isspace():
                print(it_tag.next_sibling.next_sibling.string.strip())
            else:
                print(nextSibling.strip())

        elif isinstance(it_tag.next_sibling, bs4.Tag):
            print(it_tag.next_sibling.string)

输出：

Father:
Michael Haughton
Mother:
Diane
Brother:
Rashad Haughton
Husband:
R. Kelly
Boyfriend:
Damon Dash

现在您可以轻松构建字典：

entries = {}
bTags = soup.find_all('b')

for it_tag in bTags:
    key = it_tag.string.replace(':', '')
    value = None

    nextSibling = it_tag.next_sibling
    if nextSibling is not None:
        if isinstance(nextSibling, str):
            if nextSibling.isspace():
                value = it_tag.next_sibling.next_sibling.string.strip()
            else:
                value = nextSibling.strip()

        elif isinstance(it_tag.next_sibling, bs4.Tag):
            value = it_tag.next_sibling.string

    entries[key] = value

输出字典：

{'Father': 'Michael Haughton',
 'Mother': 'Diane',
 'Brother': 'Rashad Haughton',
 'Husband': 'R. Kelly',
 'Boyfriend': 'Damon Dash'}