我正在预处理文本数据(确切地说是twitter数据),但是每当我应用NLTK词干分析器时,我都会得到一个非类型列表。我不明白为什么会这样,我也不知道怎么解决。在
以下是我的文本数据在处理过程中的外观:
处理前:
In [10]:
undefined
import pandas as pd
import numpy as np
import glob
import os
import nltk
dir = "C:\Users\Anonymous\Desktop\KAGA FOLDER\Hashtags"
train = np.array(pd.read_csv(os.path.join(dir,"train.csv")))[:,1]
def clean_the_text(data):
alist = []
data = nltk.word_tokenize(data)
for j in data:
alist.append(j.rstrip('\n'))
alist = " ".join(alist)
return alist
def stemmer(data):
stemmer = nltk.stem.PorterStemmer()
new_list = []
new_list = [new_list.append(stemmer.stem(word)) for word in data]
return new_list
def loop_data(data):
for i in range(len(data)):
data[i] = clean_the_text(data[i])
return data
train
Out[10]:
array(['Jazz for a Rainy Afternoon: {link}',
'RT: @mention: I love rainy days.',
'Good Morning Chicago! Time to kick the Windy City in the nuts and head back West!',
...,
'OMG #WeatherForecast for tomm 80 degrees & Sunny <=== #NeedThat #Philly #iMustSeeItToBelieveIt yo',
"@mention Oh no! We had cold weather early in the week, but now it's getting warmer! Hoping the rain holds out to Saturday!",
'North Cascades Hwy to reopen Wed.: quite late after a long, deep winter. Only had to clear snow 75 ft deep {link}'], dtype=object)
标记和清除文本后:
^{pr2}$最后在词干处理后:
In [13]:
undefined
train = stemmer(train)
train
Out[13]:
[None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
问题就在这里:
new_list = [new_list.append(stemmer.stem(word)) for word in data]
。应该是的new_list
被追加len(data)次,然后从包含new_list.append
的len(data)结果的list comprehension语句中将其设置为一个新列表,该语句为None。在相关问题 更多 >
编程相关推荐