Unicode编码错误：'ascii'编解码器无法编码字符u'\u201c'，在使用utf-16将系列对象转换为unicode时出错
我有一个utf-16编码的csv文件，想把它加载到Pandas中。默认情况下，数据会以对象类型的形式出现。我打算对“caption”这一列进行一些建模，所以我想把df['caption']这一列从对象类型转换为unicode字符串。目前，我在执行df['caption']=df['caption'].astype(unicode)时遇到了一个错误，错误信息是'UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 6: ordinal not in range(128)'。
我尝试通过对df['caption']列中的每个值使用encode和decode函数来解决这个问题，但没有成功。
我对Pandas和unicode都很陌生，所以想知道我哪里做错了。
提前谢谢你。
特蕾莎
以下是更多信息：
错误追踪信息如下：
UnicodeEncodeError: Traceback (most recent call last)
<ipython-input-5-aad36f4acf38> in <module>()
10 print df['caption'].head(10)
11
---> 12 df['caption']=df['caption'].astype(unicode)
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/generic.pyc in astype(self, dtype, copy, raise_on_error)
2016
2017 mgr = self._data.astype(
-> 2018 dtype, copy=copy, raise_on_error=raise_on_error)
2019 return self._constructor(mgr).__finalize__(self)
2020
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in astype(self, *args, **kwargs)
2414
2415 def astype(self, *args, **kwargs):
-> 2416 return self.apply('astype', *args, **kwargs)
2417
2418 def convert(self, *args, **kwargs):
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in apply(self, f, *args, **kwargs)
2373
2374 else:
-> 2375 applied = getattr(blk, f)(*args, **kwargs)
2376
2377 if isinstance(applied, list):
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in astype(self, dtype, copy, raise_on_error, values)
425 def astype(self, dtype, copy=False, raise_on_error=True, values=None):
426 return self._astype(dtype, copy=copy, raise_on_error=raise_on_error,
--> 427 values=values)
428
429 def _astype(self, dtype, copy=False, raise_on_error=True, values=None,
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in _astype(self, dtype, copy, raise_on_error, values, klass)
442 # force the copy here
443 if values is None:
--> 444 values = com._astype_nansafe(self.values, dtype, copy=True)
445 newb = make_block(values, self.items, self.ref_items,
446 ndim=self.ndim, placement=self._ref_locs,
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/common.pyc in _astype_nansafe(arr, dtype, copy)
2222 return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape)
2223 elif issubclass(dtype.type, compat.string_types):
-> 2224 return lib.astype_str(arr.ravel()).reshape(arr.shape)
2225
2226 if copy:
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/lib.so in pandas.lib.astype_str (pandas/lib.c:12944)()
/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/lib.so in pandas.lib.astype_str (pandas/lib.c:12862)()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 6: ordinal not in range(128)
我的代码如下：
import pandas as pd
import numpy as np
df = pd.read_csv('Chevrolet_4-7-2014_cvid_data.csv',encoding='utf-16',header=0,na_values=['N/A',''],names=['channel','link','title','posted','views','likes','dislikes','description','category','statdate','statviews','timewatched','averagetw','subsdriven','shares','caption'])
print df.head(5)
print df.dtypes
print df['caption'].head(10)
df['caption']=df['caption'].astype(unicode)
数据看起来像这样：
channel link \
0 Chevrolet http://www.youtube.com/watch?v=dCayKZe6WvI
1 Chevrolet http://www.youtube.com/watch?v=IRXK35dPXbE
2 Chevrolet http://www.youtube.com/watch?v=XXdj4QMw748
3 Chevrolet http://www.youtube.com/watch?v=_ger32ROs94
4 Chevrolet http://www.youtube.com/watch?v=Chfm7Pou49k
5 Chevrolet http://www.youtube.com/watch?v=ySmEJyQ94BI
title posted views \
0 Chevy Open House Event: From Our House to Your... Apr 1 2014 73111
1 Truck Towing Capabilities: 2014 Silverado -- #... Mar 26 2014 11934
2 Potholes at the Milford Proving Grounds: Tips ... Mar 20 2014 8037
3 Diesel Trucks: Heavy Duty Strengths -- 2015 Si... Mar 20 2014 12096
4 Captain America: All in a Day's Work -- 2014 T... Mar 14 2014 93377
5 Media Blasting: Camaro Engineering -- 2014 Cam... Mar 13 2014 109931
likes dislikes description \
0 43 13 In March over 100000 people visited our Chevy ...
1 183 56 Farmer Dewayne Kleman and General Motors engin...
2 58 10 Chevrolet vehicles are carefully designed to w...
3 210 6 Introducing the all-new 2015 Silverado HD. The...
4 1095 35 From saving the world to working on math homew...
category statdate statviews timewatched averagetw subsdriven \
0 Autos & Vehicles NaN NaN NaN NaN NaN
1 Autos & Vehicles NaN NaN NaN NaN NaN
2 Autos & Vehicles NaN NaN NaN NaN NaN
3 Autos & Vehicles NaN NaN NaN NaN NaN
4 Autos & Vehicles NaN NaN NaN NaN NaN
shares caption
0 NaN The Chevy Spring Open House Sale the perfect ...
1 NaN 0:03 A Man And His Truck And An Engineer / To...
2 NaN 0:02 Severe Bump road sign 0:07 Pothole Facil...
3 NaN 0:03 And there's no stronger Silverado than t...
4 NaN 0:03 Are you doing anything fun Saturday nigh...
5 NaN 0:05 Camaro Z/28 logo 0:07 Z/28 Bead Lock 0:0...
[5 rows x 16 columns]
channel object
link object
title object
posted object
views object
likes int64
dislikes int64
description object
category object
statdate object
statviews float64
timewatched object
averagetw object
subsdriven float64
shares float64
caption object
dtype: object
0 The Chevy Spring Open House Sale the perfect ...
1 0:03 A Man And His Truck And An Engineer / To...
2 0:02 Severe Bump road sign 0:07 Pothole Facil...
3 0:03 And there's no stronger Silverado than t...
4 0:03 Are you doing anything fun Saturday nigh...
5 0:05 Camaro Z/28 logo 0:07 Z/28 Bead Lock 0:0...
Name: caption, dtype: object
1 个回答
1
你可以试着在调用 read_csv() 的时候加上 dtype={'caption' : str} 吗？就像这样：
df = pd.read_csv('Chevrolet_4-7-2014_cvid_data.csv',
encoding='utf-16',
header=0,
na_values=['N/A',''],
names=[...],
dtype={'caption' : str})
顺便说一下，pandas 默认情况下会使用 header=0。虽然我看不到你的 CSV 文件，但这可能和你使用 names 这个参数有点重复，因为如果你的 CSV 文件的第一行有列名，pandas 会自动使用这些列名。不过，不管怎样，如果你试试其他的办法有效了，记得告诉我哦。:)