如何获得groupby，并为每一组Pandas获取最常用的单词和大字

Question

我正在处理这样的数据帧：

 words:                               other:   category:    
 hello, jim, you, you , jim            val1      movie
 it, seems, bye, limb, pat, paddy      val2      movie
 how, are, you, are , kim              val1      television
 ......
 ......

我试图计算出前10个最常出现的单词，以及“类别”列中每个类别的双元组。不过，我想先计算一下最常见的二元组，然后再将它们分组到各自的类别中。在

如果我最常出现的问题是第二行，那么我将从第二行中得到最常见的单词。在

二元曲线应如下所示：

^{pr2}$

然而，如果我在得到二元曲线之前进行分组，那么二元曲线将是：

 (hello, jim), (jim, you), (you, you), (you, jim), (jim, it), (it, seems), (seems,bye), (bye, limb), (limb, pat), (pat, paddy)
 (how, are), (are, you), (you, are), (are, kim)

用熊猫做这件事的最好方法是什么？在

抱歉，如果我的问题不必要的复杂，我只想包括所有的细节。有什么问题请告诉我。在

Answer 1

示例数据帧：

                                   words other    category
0             hello, jim, you, you , jim  val1       movie
1  it, seems, bye, limb, pat, hello, jim  val2       movie
2               how, are, you, are , kim  val1  television

下面是一种使用Pandas和.iterrows()计算双参数的方法：

^{pr2}$

[[('hello', 'jim'), ('jim', 'you'), ('you', 'you'), ('you', 'jim')], 
[('it', 'seems'), ('seems', 'bye'), ('bye', 'limb'), ('limb', 'pat'), ('pat', 'hello'), ('hello', 'jim')], 
[('how', 'are'), ('are', 'you'), ('you', 'are'), ('are', 'kim')]]

下面是一个使用Pandas和.apply的更有效方法：

def bigram(row):
    lst = row['words'].split(', ')
    return [(lst[x].strip(), lst[x+1].strip()) for x in range(len(lst)-1)]

bigrams = df.apply(lambda row: bigram(row), axis=1)

print(bigrams.tolist())

[[('hello', 'jim'), ('jim', 'you'), ('you', 'you'), ('you', 'jim')], 
[('it', 'seems'), ('seems', 'bye'), ('bye', 'limb'), ('limb', 'pat'), ('pat', 'hello'), ('hello', 'jim')], 
[('how', 'are'), ('are', 'you'), ('you', 'are'), ('are', 'kim')]]

然后，您可以按类别对数据进行分组，并找到前10个最常见的bigram。以下是按类别查找最常见的双元组的示例：

df['bigrams'] = bigrams
df2 = df.groupby('category').agg({'bigrams': 'sum'})

# Compute the most frequent bigrams by category
from collections import Counter
df3 = df2.bigrams.apply(lambda row: Counter(row)).to_frame()

按类别排列的双峰频率有序字典：

print(df3)

                                                      bigrams
category                                                     
movie       {('hello', 'jim'): 2, ('jim', 'you'): 1, ('you...
television  {('how', 'are'): 1, ('are', 'you'): 1, ('you',...

# Filter to just the top 3 most frequent bigrams (or 10 if you have enough data)
df3.bigrams.apply(lambda row: list(row)[0:3])

category
movie         [(hello, jim), (jim, you), (you, you)]
television      [(how, are), (are, you), (you, are)]
Name: bigrams, dtype: object