NLTK词干分析器返回非类型的列表

2024-03-28 07:23:56 发布

您现在位置:Python中文网/ 问答频道 /正文

我正在预处理文本数据(确切地说是twitter数据),但是每当我应用NLTK词干分析器时,我都会得到一个非类型列表。我不明白为什么会这样,我也不知道怎么解决。在

以下是我的文本数据在处理过程中的外观:

处理前:

In [10]:
undefined



import pandas as pd
import numpy as np
import glob
import os
import nltk
dir = "C:\Users\Anonymous\Desktop\KAGA FOLDER\Hashtags"
train = np.array(pd.read_csv(os.path.join(dir,"train.csv")))[:,1]
def clean_the_text(data):
    alist = []
    data = nltk.word_tokenize(data)
    for j in data:
        alist.append(j.rstrip('\n'))
    alist = " ".join(alist)
    return alist

def stemmer(data):
    stemmer = nltk.stem.PorterStemmer()
    new_list = []
    new_list = [new_list.append(stemmer.stem(word)) for word in data]
    return new_list
def loop_data(data):
    for i in range(len(data)):
        data[i] = clean_the_text(data[i])
    return data
train


Out[10]:
array(['Jazz for a Rainy Afternoon:  {link}',
       'RT: @mention: I love rainy days.',
       'Good Morning Chicago! Time to kick the Windy City in the nuts and head back West!',
       ...,
       'OMG #WeatherForecast for tomm 80 degrees & Sunny <=== #NeedThat #Philly #iMustSeeItToBelieveIt yo',
       "@mention Oh no! We had cold weather early in the week, but now it's getting warmer! Hoping the rain holds out to Saturday!",
       'North Cascades Hwy to reopen Wed.: quite late after a long, deep winter. Only had to clear snow 75 ft deep {link}'], dtype=object)

标记和清除文本后:

^{pr2}$

最后在词干处理后:

In [13]:
undefined



train = stemmer(train)
train
Out[13]:
[None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,

Tags: theto数据in文本importnonenew
1条回答
网友
1楼 · 发布于 2024-03-28 07:23:56

问题就在这里:new_list = [new_list.append(stemmer.stem(word)) for word in data]。应该是的

new_list = [stemmer.stem(word) for word in data]
# or 
# new_data = map(stemmer.stem, data) # returns a map object

new_list被追加len(data)次,然后从包含new_list.append的len(data)结果的list comprehension语句中将其设置为一个新列表,该语句为None。在

相关问题 更多 >