回答此问题可获得 20 贡献值,回答如果被采纳可获得 50 分。
<p>我试图用feedparser解析RSS提要,并使用SQLAlchemy将其插入到mySQL表中。实际上,我可以很好地运行此功能,但今天提要中有一个项目的描述中有一个省略号字符,我得到以下错误:</p>
<p>UnicodeEncodeError:“latin-1”编解码器无法对位置35中的字符u“\u2026”进行编码:序号不在范围(256)内</p>
<p>如果我将convert_unicode=True选项添加到引擎中,我就可以完成插入操作,但是省略号并没有显示出来,它只是一些奇怪的字符。这似乎有道理,因为据我所知,拉丁语-1中没有横向省略。即使我将编码设置为utf-8,似乎也没什么区别。如果我使用phpmyadmin进行插入并包含省略号,它就可以正常运行。</p>
<p>我想我只是不明白字符编码或者如何让SQLAlchemy使用我指定的编码。有人知道如何让文本不带奇怪的字符进入吗?</p>
<p>更新</p>
<p>我想我已经弄明白了,但我不确定为什么这很重要。。。</p>
<p>代码如下:</p>
<pre><code>import sys
import feedparser
import sqlalchemy
from sqlalchemy import create_engine, MetaData, Table
COMMON_CHANNEL_PROPERTIES = [
('Channel title:','title', None),
('Channel description:', 'description', 100),
('Channel URL:', 'link', None),
]
COMMON_ITEM_PROPERTIES = [
('Item title:', 'title', None),
('Item description:', 'description', 100),
('Item URL:', 'link', None),
]
INDENT = u' '*4
def feedinfo(url, output=sys.stdout):
feed_data = feedparser.parse(url)
channel, items = feed_data.feed, feed_data.entries
#adding charset=utf8 here is what fixed the problem
db = create_engine('mysql://user:pass@localhost/db?charset=utf8')
metadata = MetaData(db)
rssItems = Table('rss_items', metadata,autoload=True)
i = rssItems.insert();
for label, prop, trunc in COMMON_CHANNEL_PROPERTIES:
value = channel[prop]
if trunc:
value = value[:trunc] + u'...'
print >> output, label, value
print >> output
print >> output, "Feed items:"
for item in items:
i.execute({'title':item['title'], 'description': item['description'][:100]})
for label, prop, trunc in COMMON_ITEM_PROPERTIES:
value = item[prop]
if trunc:
value = value[:trunc] + u'...'
print >> output, INDENT, label, value
print >> output, INDENT, u'---'
return
if __name__=="__main__":
url = sys.argv[1]
feedinfo(url)
</code></pre>
<p>下面是在不使用charset选项的情况下运行代码的输出/跟踪:</p>
<pre><code>Channel title: [H]ardOCP News/Article Feed
Channel description: News/Article Feed for [H]ardOCP...
Channel URL: http://www.hardocp.com
Feed items:
Item title: Windows 8 UI is Dropping the 'Start' Button
Item description: After 15 years of occupying a place of honor on the desktop, the "Start" button will disappear from ...
Item URL: http://www.hardocp.com/news/2012/02/05/windows_8_ui_dropping_lsquostartrsquo_button/
---
Item title: Which Crashes More&#63; Apple Apps or Android Apps
Item description: A new study of smartphone apps between Android and Apple conducted over a two month period came up w...
Item URL: http://www.hardocp.com/news/2012/02/05/which_crashes_more63_apple_apps_or_android/
---
Traceback (most recent call last):
File "parse.py", line 47, in <module>
feedinfo(url)
File "parse.py", line 36, in feedinfo
i.execute({'title':item['title'], 'description': item['description'][:100]})
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/expression.py", line 2758, in execute
return e._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2304, in _execute_clauseelement
return connection._execute_clauseelement(elem, multiparams, params)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_context
context)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 330, in do_execute
cursor.execute(statement, parameters)
File "build/bdist.linux-i686/egg/MySQLdb/cursors.py", line 159, in execute
File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 264, in literal
File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 202, in unicode_literal
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2026' in position 35: ordinal not in range(256)
</code></pre>
<p>所以看起来像是在mysql连接字符串中添加了字符集。我想它默认为拉丁语-1?我试过将content_engine上的编码标志设置为utf8,但没有任何效果。有人知道为什么当表和字段设置为utf8 unicode时它会使用拉丁语-1吗?在发送之前,我还尝试过使用.encode('cp1252')对item['description]进行编码,即使没有将charset选项添加到连接字符串中,也同样有效。这不应该和拉丁语-1一起使用,但很明显是这样的?我已经找到了解决办法,但我想得到一个答案:)</p>