UnicodeDecodeError:“ascii”编解码器无法解码位置13中的字节0xe2:序号不在范围（128）内

Question

我使用NLTK对文本文件执行kmeans集群，其中每一行都被视为一个文档。例如，我的文本文件如下：

belong finger death punch <br>
hasty <br>
mike hasty walls jericho <br>
jägermeister rules <br>
rules bands follow performing jägermeister stage <br>
approach 

现在我要运行的演示代码是：

import sys

import numpy
from nltk.cluster import KMeansClusterer, GAAClusterer, euclidean_distance
import nltk.corpus
from nltk import decorators
import nltk.stem

stemmer_func = nltk.stem.EnglishStemmer().stem
stopwords = set(nltk.corpus.stopwords.words('english'))

@decorators.memoize
def normalize_word(word):
    return stemmer_func(word.lower())

def get_words(titles):
    words = set()
    for title in job_titles:
        for word in title.split():
            words.add(normalize_word(word))
    return list(words)

@decorators.memoize
def vectorspaced(title):
    title_components = [normalize_word(word) for word in title.split()]
    return numpy.array([
        word in title_components and not word in stopwords
        for word in words], numpy.short)

if __name__ == '__main__':

    filename = 'example.txt'
    if len(sys.argv) == 2:
        filename = sys.argv[1]

    with open(filename) as title_file:

        job_titles = [line.strip() for line in title_file.readlines()]

        words = get_words(job_titles)

        # cluster = KMeansClusterer(5, euclidean_distance)
        cluster = GAAClusterer(5)
        cluster.cluster([vectorspaced(title) for title in job_titles if title])

        # NOTE: This is inefficient, cluster.classify should really just be
        # called when you are classifying previously unseen examples!
        classified_examples = [
                cluster.classify(vectorspaced(title)) for title in job_titles
            ]

        for cluster_id, title in sorted(zip(classified_examples, job_titles)):
            print cluster_id, title

（也可以找到here）

我收到的错误是：

Traceback (most recent call last):
File "cluster_example.py", line 40, in
words = get_words(job_titles)
File "cluster_example.py", line 20, in get_words
words.add(normalize_word(word))
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/nltk/decorators.py", line 183, in memoize
result = func(*args)
File "cluster_example.py", line 14, in normalize_word
return stemmer_func(word.lower())
File "/usr/local/lib/python2.7/dist-packages/nltk/stem/snowball.py", line 694, in stem
word = (word.replace(u"\u2019", u"\x27")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13: ordinal not in range(128)

这里发生了什么事？

Answer 1

您也可以尝试：

import sys
reload(sys)
sys.setdefaultencoding('utf8')

Answer 2

这对我很有用。

f = open(file_path, 'r+', encoding="utf-8")

您可以添加第三个参数encoding以确保编码类型为“utf-8”

注意：这种方法在Python3中效果很好，我在Python2.7中没有尝试过。

Answer 3

该文件作为一堆strs读取，但应该是unicodes。Python尝试隐式转换，但失败。更改：

job_titles = [line.strip() for line in title_file.readlines()]

要显式地将str解码为unicode（这里假设是UTF-8）：

job_titles = [line.decode('utf-8').strip() for line in title_file.readlines()]

它也可以通过导入the ^{} module并使用^{}而不是内置的^{}来解决。