使用bs4对div数据进行Python-html解析

Question

现在我想删除html页面的页眉和页脚。我发现页眉和页脚显示为每个div的最后两行。有人能告诉我如何从一个div中提取除最后两行以外的所有数据，如下所示：

<div class="page"><p />
<p></p>
<p>First line required
</p>
<p>Second line required
</p>
<p>Third line required
</p>
<p>Line 1 not required
</p>
<p>Line 2 not required
</p>
<p></p>
</div>
<div class="page"><p />
<p>line required 1
</p>
<p></p>
<p>line required 2
</p>
<p>line required 3
</p>
<p></p>
<p>line required 4
</p>
<p>line required 5
</p>
<p>line required 6
</p>
<p>Line 1 not required
</p>
<p>Line 2 not required
<p />
</div>

现有代码如下：

soup = BeautifulSoup(file_content, 'html.parser')
for num, page in enumerate(soup.select('.page'), 1):
    content = page.get_text(strip=True, separator=' ').replace("\n", " ")

Answer 1

#import packages
from bs4 import BeautifulSoup

with open('test.html', 'r') as f:
    file_content = f.read()
soup = BeautifulSoup(file_content, 'html.parser')

for page in soup.find_all("div", class_="page"):
    page.contents[-3].extract()
    page.contents[-1].extract()

print(soup.prettify())

似乎达到了预期效果。你知道吗

注意事项：

• 你知道吗测试.html这是你的html示例
• 我不得不删除第-1行和第-3行，这可能与您使用的奇怪的html有关（<p>Line 2 not required永远不会结束，<p />标记似乎不是个好主意：Should I use the <p /> tag in markup?）

谨致问候

Answer 2

最新答案：

from bs4 import BeautifulSoup

html_str = """<div class="page"><p />
<p></p>
<p>First line required
</p>
<p>Second line required
</p>
<p>Third line required
</p>
<p>Line 1 not required
</p>
<p>Line 2 not required
</p>
<p></p>
</div>
<div class="page"><p />
<p>line required 1
</p>
<p></p>
<p>line required 2
</p>
<p>line required 3
</p>
<p></p>
<p>line required 4
</p>
<p>line required 5
</p>
<p>line required 6
</p>
<p>Line 1 not required
</p>
<p>Line 2 not required
<p />
</div>"""



#Load the html string into bs4 object
soup = BeautifulSoup(html_str, 'lxml')

#Strip off empty tags. This also removes empty <p> tags
[x.decompose() for x in soup.findAll(lambda tag: not tag.contents and not tag.name == 'br' )]



#Load all divs with classname = 'page'
items = soup.find_all('',{'class':'page'})

final_html=''
#This for loop removes the last 2 tags from every div (as requested)
for item in items:
    last_item = str(item.find_all('p')[-1])
    second_last_item = str(item.find_all('p')[-2])
    current_item = str(item)
    current_item = current_item.replace(last_item,'')
    current_item = current_item.replace(second_last_item,'')
    final_html = final_html + current_item


final_soup = BeautifulSoup(final_html)
final_str = final_soup.text
print(final_str)

输出：

print(final_str)
                
First line required

Second line required

Third line required


line required 1

line required 2

line required 3

line required 4

line required 5

line required 6