BeautifulSoup标记是bs4.element.NavigableString和bs4.element.tag类型

Question

我正试图在维基百科的一篇文章中删除一个表，每个表元素的类型似乎都是<class 'bs4.element.Tag'>和<class 'bs4.element.NavigableString'>。

import requests
import bs4
import lxml


resp = requests.get('https://en.wikipedia.org/wiki/List_of_municipalities_in_Massachusetts')

soup = bs4.BeautifulSoup(resp.text, 'lxml')

munis = soup.find(id='mw-content-text')('table')[1]

for muni in munis:
    print type(muni)
    print '============'

生成以下输出：

<class 'bs4.element.Tag'>
============
<class 'bs4.element.NavigableString'>
============
<class 'bs4.element.Tag'>
============
<class 'bs4.element.NavigableString'>
============
<class 'bs4.element.Tag'>
============
<class 'bs4.element.NavigableString'>
...

当我试图检索muni.contents时，我得到了AttributeError: 'NavigableString' object has no attribute 'contents'错误。

我做错什么了？如何获取每个muni的bs4.element.Tag对象？

（使用Python2.7）。

Answer 1

from bs4 import BeautifulSoup
import requests

r = requests.get('https://en.wikipedia.org/wiki/List_of_municipalities_in_Massachusetts')
soup = BeautifulSoup(r.text, 'lxml')
rows = soup.find(class_="wikitable sortable").find_all('tr')[1:]

for row in rows:
    cell = [i.text for i in row.find_all('td')]
    print(cell)

输出：

['Abington', 'Town', 'Plymouth', 'Open town meeting', '15,985', '1712']
['Acton', 'Town', 'Middlesex', 'Open town meeting', '21,924', '1735']
['Acushnet', 'Town', 'Bristol', 'Open town meeting', '10,303', '1860']
['Adams', 'Town', 'Berkshire', 'Representative town meeting', '8,485', '1778']
['Agawam', 'City[4]', 'Hampden', 'Mayor-council', '28,438', '1855']
['Alford', 'Town', 'Berkshire', 'Open town meeting', '494', '1773']
['Amesbury', 'City', 'Essex', 'Mayor-council', '16,283', '1668']
['Amherst', 'Town', 'Hampshire', 'Representative town meeting', '37,819', '1775']
['Andover', 'Town', 'Essex', 'Open town meeting', '33,201', '1646']
['Aquinnah', 'Town', 'Dukes', 'Open town meeting', '311', '1870']
['Arlington', 'Town', 'Middlesex', 'Representative town meeting', '42,844', '1807']
['Ashburnham', 'Town', 'Worcester', 'Open town meeting', '6,081', '1765']
['Ashby', 'Town', 'Middlesex', 'Open town meeting', '3,074', '1767']
['Ashfield', 'Town', 'Franklin', 'Open town meeting', '1,737', '1765']
['Ashland', 'Town', 'Middlesex', 'Open town meeting', '16,593', '1846']

Answer 2

如果在标记中的节点之间有空格，美化组将把它们转换为NavigableString。试一下，看看内容是否像你希望的那样被获取-

for muni in munis:
    #print type(muni)
    try:
        print muni.contents
    except AttributeError:
        pass
    print '============'

Answer 3

#!/usr/bin/env python
# coding:utf-8
'''黄哥Python'''

import requests
import bs4
from bs4 import BeautifulSoup
# from urllib.request import urlopen

html = requests.get('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
soup = BeautifulSoup(html.text, 'lxml')

symbolslist = soup.find('table').tr.next_siblings
for sec in symbolslist:
    # print(type(sec))
    if type(sec) is not bs4.element.NavigableString:
        print(sec.get_text())