使用sciket Learn查找集群关键词

Question

我想对一些文本数据进行聚类。数据包含来自维基百科的关于食物、大脑、篮球和电话的陈述和句子。 我想对数据进行聚类，并为我的插补语句预测聚类，然后打印结果和聚类的关键字。你知道吗

我知道我有4个主题（食物，大脑，篮球和电话），但我不能得到4个集群的结果。你知道吗

如何打印关键字，簇中最重要的单词？ 另外，我应该使用CountVectorizer还是TfidfVectorizer？你知道吗

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.preprocessing import scale
from sklearn.cluster import KMeans, MiniBatchKMeans

import matplotlib.pyplot as plt

from sklearn.metrics import adjusted_rand_score
from sklearn.metrics import silhouette_samples, silhouette_score

x = ['whats the price for this phone', 'what is the price for this cellphone', 'where can i buy this phone', 'how much this cellphone costs',
     'i do not know where can i buy this telephone', 'whats the name that store where you can find good phones', 'i love this phone, it is great', 
     'this phone is priceless', 'the best item i have ever bought', 'this store has great products', 'great item, i m going to buy it next week',
     'basketball is my favourite stport', 'i love basketball', 'basketball is borring', 'how can i learn to play basketball', 'i love sports',
     'let s play basketball', 'i love to watch basketball world cup', 'i m going to be coach when i grow up' ,'i would like to be basketball coach',
     'i want to play tennis', 'i watched tennis all day', 'tennis is my favourite sport', 'tennis is amazing sport, you should try it', 'tennis is so fun to watch',
     'Food is any substance consumed to provide nutritional support for an organism', 'I love to eat healty and tasty food', "this food is amazing",
     "The substance is ingested by an organism and assimilated by the organism's cells to provide energy, maintain life, or stimulate growth"
     "It is usually of plant or animal origin, and contains essential nutrients, such as carbohydrates, fats, proteins, vitamins, or minerals",
     "Food safety and food security are monitored by agencies", "Historically, humans secured food through two methods: hunting and gathering and agriculture",
     "Most food has its origin in plants", "Some food is obtained directly from plants", "Animals are used as food either directly or indirectly by the products they produce",
     "Even animals that are used as food sources are raised by feeding them food derived from plants", "I love eating meat, its very tasty",
     "Meat is an example of a direct product taken from an animal, which comes from muscle systems or from organs", "milk is very healty drink",
     "Food products produced by animals include milk produced by mammary glands, which in many cultures is drunk or processed into dairy products",
     "Some cultures and people do not consume meat or animal food products for cultural, dietary, health, ethical, or ideological reasons",
     "Vegetarians choose to forgo food from animal sources to varying degrees","Vegans do not consume any foods that are or contain ingredients from an animal source", 'Vegans do not eat meat',
     "A functional food is a food given an additional function by adding new ingredients or more of existing ingredients","proteins are very important",
     "A healthy diet is a diet that helps to maintain or improve overall health","A healthy diet provides the body with essential nutrition",
     "fluid, macronutrients, micronutrients, and adequate calories are important for people","if you want to live long you need to eat healty food",
     "There are specialized healthy diets, called medical nutrition therapy, for people with various diseases or conditions", "meat is food",
     "There are also prescientific ideas about such specialized diets, as in dietary therapy in traditional Chinese medicine", "fruits are food",
     "Basketball was originally played with a soccer ball","Everybody loves to play basketball because its amazing sport", "lets play basketball tomorrow",
     "basketball tournaments are held for boys and girls of all age levels","The global popularity of the sport is reflected in the nationalities represented in the NBA",
     "Women's basketball began in 1892 at Smith College", 'women can also play basketball and they are very good at it', 'you need a brain to have a good life',
     "Game of basketball is very complex and it atracts a lot of people"," Ilove to watch olympic games", 'brain is the most important organ',
     "A mobile phone or cell phone, sometimes shortened to simply cell or just phone, is a portable telephone that can make and receive calls over a radio frequency",
     "Feature phone is a term typically used as a retronym to describe mobile phones which are limited in capabilities in contrast to a modern smartphone",
     "Feature phones and basic mobile phones tend to use a proprietary, custom-designed software and user interface", "i love my new cellphone, it s soo amazing","i need to buy new phone",
     "Mobile phones communicate with cell towers that are placed to give coverage across a telephone service area which is divided up into 'cells'",         
    "A brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals", 'im brainstorming',
    "brains exert centralized control over a body's other organs", "You should start using your brain if you want to be smart",
    "They act on the rest of the body both by generating patterns of muscle activity and by driving the secretion of chemicals called hormones",
    "The shape and size of the brain varies greatly between species, and identifying common features is often difficult",
    "The diversity of invertebrate body plans is matched by an equal diversity in brain structures",
    "The most obvious difference between the brains of mammals and other vertebrates is in terms of size",
    "On average, a mammal has a brain roughly twice as large as that of a bird of the same body size, and ten times as large as that of a reptile of the same body size",
    "The brain develops in an intricately orchestrated sequence of stages"]

stop_words_de = get_stop_words('de')

#cv = TfidfVectorizer(analyzer = 'word', max_features = 4000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')
cv = CountVectorizer(analyzer = 'word', max_features = 4000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')  

x = cv.fit_transform(x)

my_list = []
list_of_clusters = []
for i in range(2,20):

    kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
    kmeans.fit(x)
    my_list.append(kmeans.inertia_)

    cluster_labels = kmeans.fit_predict(x)

    silhouette_avg = silhouette_score(x, cluster_labels)
    print(silhouette_avg)
    list_of_clusters.append(silhouette_avg)


plt.plot(range(2,20),my_list)
plt.show()


number_of_clusters = max(list_of_clusters)
number_of_clusters = list_of_clusters.index(number_of_clusters)+2

print('Number of clusters: ', number_of_clusters)
kmeans = KMeans(n_clusters = number_of_clusters, init = 'k-means++', random_state = 0)
kmeans.fit(x)

y_prediction = cv.transform(["i like to eat meat, its very tasty"])   
kmeans_prediction = kmeans.predict(y_prediction)
print("i like to eat meat, its very tasty")
print('Cluster:', kmeans_prediction[0])
print()

Answer 1

下面是一种从数据中获取最关键术语列表的方法。如果这是你的目标，你可以把这个和我的另一个答案结合起来，在散点图上打印出来：

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import scale
from sklearn.cluster import KMeans, MiniBatchKMeans
import matplotlib.pyplot as plt
from sklearn.metrics import adjusted_rand_score
from sklearn.metrics import silhouette_samples, silhouette_score
#I'm using pandas for quick row sorting by value but you can do this various ways
import pandas as pd

x = ['whats the price for this phone', 'what is the price for this cellphone', 'where can i buy this phone', 'how much this cellphone costs',
     'i do not know where can i buy this telephone', 'whats the name that store where you can find good phones', 'i love this phone, it is great', 
     'this phone is priceless', 'the best item i have ever bought', 'this store has great products', 'great item, i m going to buy it next week',
     'basketball is my favourite stport', 'i love basketball', 'basketball is borring', 'how can i learn to play basketball', 'i love sports',
     'let s play basketball', 'i love to watch basketball world cup', 'i m going to be coach when i grow up' ,'i would like to be basketball coach',
     'i want to play tennis', 'i watched tennis all day', 'tennis is my favourite sport', 'tennis is amazing sport, you should try it', 'tennis is so fun to watch',
     'Food is any substance consumed to provide nutritional support for an organism', 'I love to eat healty and tasty food', "this food is amazing",
     "The substance is ingested by an organism and assimilated by the organism's cells to provide energy, maintain life, or stimulate growth"
     "It is usually of plant or animal origin, and contains essential nutrients, such as carbohydrates, fats, proteins, vitamins, or minerals",
     "Food safety and food security are monitored by agencies", "Historically, humans secured food through two methods: hunting and gathering and agriculture",
     "Most food has its origin in plants", "Some food is obtained directly from plants", "Animals are used as food either directly or indirectly by the products they produce",
     "Even animals that are used as food sources are raised by feeding them food derived from plants", "I love eating meat, its very tasty",
     "Meat is an example of a direct product taken from an animal, which comes from muscle systems or from organs", "milk is very healty drink",
     "Food products produced by animals include milk produced by mammary glands, which in many cultures is drunk or processed into dairy products",
     "Some cultures and people do not consume meat or animal food products for cultural, dietary, health, ethical, or ideological reasons",
     "Vegetarians choose to forgo food from animal sources to varying degrees","Vegans do not consume any foods that are or contain ingredients from an animal source", 'Vegans do not eat meat',
     "A functional food is a food given an additional function by adding new ingredients or more of existing ingredients","proteins are very important",
     "A healthy diet is a diet that helps to maintain or improve overall health","A healthy diet provides the body with essential nutrition",
     "fluid, macronutrients, micronutrients, and adequate calories are important for people","if you want to live long you need to eat healty food",
     "There are specialized healthy diets, called medical nutrition therapy, for people with various diseases or conditions", "meat is food",
     "There are also prescientific ideas about such specialized diets, as in dietary therapy in traditional Chinese medicine", "fruits are food",
     "Basketball was originally played with a soccer ball","Everybody loves to play basketball because its amazing sport", "lets play basketball tomorrow",
     "basketball tournaments are held for boys and girls of all age levels","The global popularity of the sport is reflected in the nationalities represented in the NBA",
     "Women's basketball began in 1892 at Smith College", 'women can also play basketball and they are very good at it', 'you need a brain to have a good life',
     "Game of basketball is very complex and it atracts a lot of people"," Ilove to watch olympic games", 'brain is the most important organ',
     "A mobile phone or cell phone, sometimes shortened to simply cell or just phone, is a portable telephone that can make and receive calls over a radio frequency",
     "Feature phone is a term typically used as a retronym to describe mobile phones which are limited in capabilities in contrast to a modern smartphone",
     "Feature phones and basic mobile phones tend to use a proprietary, custom-designed software and user interface", "i love my new cellphone, it s soo amazing","i need to buy new phone",
     "Mobile phones communicate with cell towers that are placed to give coverage across a telephone service area which is divided up into 'cells'",         
    "A brain is an organ that serves as the center of the nervous system in all vertebrate and most invertebrate animals", 'im brainstorming',
    "brains exert centralized control over a body's other organs", "You should start using your brain if you want to be smart",
    "They act on the rest of the body both by generating patterns of muscle activity and by driving the secretion of chemicals called hormones",
    "The shape and size of the brain varies greatly between species, and identifying common features is often difficult",
    "The diversity of invertebrate body plans is matched by an equal diversity in brain structures",
    "The most obvious difference between the brains of mammals and other vertebrates is in terms of size",
    "On average, a mammal has a brain roughly twice as large as that of a bird of the same body size, and ten times as large as that of a reptile of the same body size",
    "The brain develops in an intricately orchestrated sequence of stages"]

# TfidfVectorizer combines CountVectorizer and TfidfTransformer into one, so it's preferred if you're starting with strings
cv = TfidfVectorizer(analyzer = 'word', max_features = 4000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')

#create sparse matrices from model
result = cv.fit_transform(x)

#convert sparse to dense arrays
result_dense = result.toarray()

top_terms = []
#loop dense data
for row in result_dense:
    scores = []
    #cv.vocabulary_ is a dictionary of each term, and which index in the array that term is found in
    for term, index in cv.vocabulary_.items():    
        score = row[index]
        scores.append([term, score])
    #convert term and score data to dataframe, sort by top values
    df = pd.DataFrame.from_records(scores, columns=['term', 'score']).sort_values(by='score', ascending=False).reset_index(drop=True)
    #drop all values that are zero
    df = df.loc[df['score'] > 0].reset_index(drop='True')
    #get term list
    terms = list(df['term'])
    #truncate list if longer than four words
    if len(terms) > 4:
        terms = terms[:4]
    #append to list of results
    top_terms.append(terms)

top_terms

[['whats', 'price', 'phone'],
 ['price', 'cellphone'],
 ['buy', 'phone'],
 ['costs', 'cellphone'],
 ['know', 'telephone', 'buy'],
 ['whats', 'store', 'good', 'phones'],
 ['great', 'love', 'phone'],
 ['priceless', 'phone'],
 ['bought', 'best', 'item'],
 ['store', 'great', 'products'],
 ['week', 'going', 'item', 'great'],
 ['stport', 'favourite', 'basketball'],
 ['love', 'basketball'],
 ['borring', 'basketball'],
 ['learn', 'play', 'basketball'],
 ['sports', 'love'],
 ['let', 'play', 'basketball'],
 ['world', 'cup', 'watch', 'love'],
 ['grow', 'going', 'coach'],
 ['like', 'coach', 'basketball'],
 ['want', 'tennis', 'play'],
 ['day', 'watched', 'tennis'],
 ['favourite', 'sport', 'tennis'],
 ['try', 'amazing', 'sport', 'tennis'],
 ['fun', 'watch', 'tennis'],
 ['consumed', 'nutritional', 'support', 'substance'],
 ['tasty', 'eat', 'healty', 'love'],
 ['amazing', 'food'],
 ['organism', 'fats', 'vitamins', 'carbohydrates'],
 ['food', 'agencies', 'monitored', 'security'],
 ['historically', 'humans', 'methods', 'hunting'],
 ['origin', 'plants', 'food'],
 ['obtained', 'directly', 'plants', 'food'],
 ['indirectly', 'produce', 'directly', 'used'],
 ['food', 'derived', 'feeding', 'raised'],
 ['eating', 'tasty', 'meat', 'love'],
 ['direct', 'example', 'product', 'taken'],
 ['drink', 'milk', 'healty'],
 ['produced', 'products', 'include', 'dairy'],
 ['cultural', 'ethical', 'ideological', 'reasons'],
 ['vegetarians', 'degrees', 'varying', 'forgo'],
 ['source', 'foods', 'contain', 'consume'],
 ['vegans', 'eat', 'meat'],
 ['ingredients', 'food', 'functional', 'given'],
 ['proteins', 'important'],
 ['diet', 'overall', 'improve', 'helps'],
 ['provides', 'nutrition', 'diet', 'essential'],
 ['fluid', 'macronutrients', 'adequate', 'calories'],
 ['live', 'long', 'healty', 'need'],
 ['various', 'conditions', 'diseases', 'medical'],
 ['meat', 'food'],
 ['medicine', 'chinese', 'traditional', 'ideas'],
 ['fruits', 'food'],
 ['originally', 'played', 'soccer', 'ball'],
 ['loves', 'everybody', 'sport', 'amazing'],
 ['tomorrow', 'lets', 'play', 'basketball'],
 ['tournaments', 'age', 'held', 'boys'],
 ['reflected', 'nba', 'popularity', 'global'],
 ['began', '1892', 'smith', 'college'],
 ['women', 'good', 'play', 'basketball'],
 ['life', 'need', 'good', 'brain'],
 ['lot', 'atracts', 'complex', 'game'],
 ['games', 'ilove', 'olympic', 'watch'],
 ['organ', 'important', 'brain'],
 ['phone', 'cell', 'frequency', 'shortened'],
 ['smartphone', 'limited', 'term', 'typically'],
 ['phones', 'software', 'use', 'interface'],
 ['soo', 'new', 'cellphone', 'amazing'],
 ['need', 'new', 'buy', 'phone'],
 ['service', 'divided', 'area', 'coverage'],
 ['serves', 'center', 'nervous', 'vertebrate'],
 ['im', 'brainstorming'],
 ['exert', 'control', 'centralized', 'organs'],
 ['smart', 'using', 'start', 'want'],
 ['generating', 'patterns', 'hormones', 'chemicals'],
 ['shape', 'difficult', 'features', 'common'],
 ['diversity', 'equal', 'plans', 'structures'],
 ['obvious', 'difference', 'mammals', 'vertebrates'],
 ['large', 'size', 'body', 'average'],
 ['stages', 'sequence', 'orchestrated', 'intricately']]