我正在研究关键词提取问题。考虑一下非常一般的情况
tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')
t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.
"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."
"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and yet you say I am useless! Thus ungratefully, O Jupiter, do men receive their blessings!"
Our best blessings are often the least appreciated."""
tfs = tfidf.fit_transform(t.split(" "))
str = 'tree cat travellers fruit jupiter'
response = tfidf.transform([str])
feature_names = tfidf.get_feature_names()
for col in response.nonzero()[1]:
print(feature_names[col], ' - ', response[0, col])
这给了我
(0, 28) 0.443509712811
(0, 27) 0.517461475101
(0, 8) 0.517461475101
(0, 6) 0.517461475101
tree - 0.443509712811
travellers - 0.517461475101
jupiter - 0.517461475101
fruit - 0.517461475101
这很好。对于任何新文档,是否有办法获得tfidf得分最高的前n项?
你需要做一点歌舞来获得矩阵作为numpy数组,但这应该符合你的要求:
这给了我:
argsort
调用是非常有用的,here are the docs for it。我们必须做[::-1]
,因为argsort
只支持从小到大的排序。我们调用flatten
来将维数减少到1d,这样排序后的索引就可以用于索引1d特征数组。请注意,包含对flatten
的调用只有在一次测试一个文档时才起作用。另外,在另一个问题上,你的意思是
tfs = tfidf.fit_transform(t.split("\n\n"))
?否则,多行字符串中的每个术语都将被视为“文档”。相反,使用\n\n
意味着我们实际上在查看4个文档(每行一个文档),这在考虑tfidf时更有意义。使用稀疏矩阵本身的解决方案(没有
.toarray()
)!相关问题 更多 >
编程相关推荐