UnicodeDecodeError：'utf8' 编解码器无法解码位置 1266 的字节 0xba：无效的起始字节

Question

我正在尝试用scikit来训练一些文本数据。在其他电脑上用同样的代码没有任何错误，但在我的系统上却出现了错误：

File "/root/Desktop/karim/svn/questo-anso/v5/trials/classify/domain_detection_final/test_classifier_temp.py", line 130, in trainClassifier
    X_train = self.vectorizer.fit_transform(self.data_train.data)
  File "/root/Desktop/karim/software/scikit-learn-0.15.1/sklearn/feature_extraction/text.py", line 1270, in fit_transform
    X = super(TfidfVectorizer, self).fit_transform(raw_documents)
  File "/root/Desktop/karim/software/scikit-learn-0.15.1/sklearn/feature_extraction/text.py", line 808, in fit_transform
    vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
  File "/root/Desktop/karim/software/scikit-learn-0.15.1/sklearn/feature_extraction/text.py", line 741, in _count_vocab
    for feature in analyze(doc):
  File "/root/Desktop/karim/software/scikit-learn-0.15.1/sklearn/feature_extraction/text.py", line 233, in <lambda>
    tokenize(preprocess(self.decode(doc))), stop_words)
  File "/root/Desktop/karim/software/scikit-learn-0.15.1/sklearn/feature_extraction/text.py", line 111, in decode
    doc = doc.decode(self.encoding, self.decode_error)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xba in position 1266: invalid start byte

我已经查看了类似的讨论，但没有找到帮助。

更新：

self.data_train = self.fetch_data(cache, subset='train')
if not os.path.exists(self.root_dir+"/autocreated/vectorizer.txt"):
                self.vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,
                                                 stop_words='english')
                start_time = time()
                print("Transforming the dataset")
                X_train = self.vectorizer.fit_transform(self.data_train.data)  // Error is here
                joblib.dump(self.vectorizer, self.root_dir+"/autocreated/vectorizer.txt")

Answer 1

在处理训练数据时遇到了一些问题。有一个方法解决了我的问题，就是使用 decode_error='ignore' 来 忽略错误，可能还有其他的解决办法。

self.vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,stop_words='english',decode_error='ignore')

Answer 2

你的文件实际上是用 ISO-8869-1 编码的，而不是 UTF-8。你需要先正确解码它，然后才能再编码。

0xBA 在 ISO-8869-1 中表示的是 数字符号 (º)。