从BeautifulSoup对象中删除非BMP字符

2024-03-29 14:25:09 发布

您现在位置:Python中文网/ 问答频道 /正文

我是python的初学者。我正在使用BeautifulSoup从网站中提取数据。但是每当页面的源代码包含表情符号时,我的程序就会停止。在解析时/之前,我应该做些什么,以便删除表情符号/非BMP字符并刮除页面。在

import bs4 as bs
import string
import urllib.request

str = 'http://www.storypick.com/harshad-mehta-scam-web-series/' #myurl
source = urllib.request.urlopen(str);
soup = bs.BeautifulSoup(source,'lxml');

match=soup.find('div',class_='td-post-content');
str=soup.title.text+"\n";
name=soup.title.text;
for paragraph in match.find_all(['p' , 'h4' , 'h3' , 'h2' , 'blockquote']):
    str+=paragraph.text+"\n";
print(str);

输出:

^{2}$

Tags: textimportsourcebstitlerequestmatch页面
2条回答

我改为使用requests,这使事情更简单。这是一个比您所要做的更简单的例子,但是它确实有效。你现在完成你的剧本应该没有问题。在

import requests
from bs4 import BeautifulSoup

requestURL = 'http://www.storypick.com/harshad-mehta-scam-web-series'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36'}

with requests.Session() as session:
    r = session.get(requestURL, headers=headers)
    if r.ok:
        soup = BeautifulSoup(r.content, 'lxml')
        for paragraph in soup.find_all('p'):
            print (paragraph)

完美地为我工作!我修改了一下代码

import bs4 as bs
import string
import urllib

str = 'http://www.storypick.com/harshad-mehta-scam-web-series/' #myurl
source = urllib.urlopen(str);
soup = bs.BeautifulSoup(source);

match=soup.find('div',class_='td-post-content');
str=soup.title.text+"\n";
name=soup.title.text;
for paragraph in match.find_all(['p' , 'h4' , 'h3' , 'h2' , 'blockquote']):
    str+=paragraph.text+"\n";
print(str);

相关问题 更多 >