>>> df=newdata.groupBy('columnName').count()
>>> mode = df.orderBy(df['count'].desc()).collect()[0][0]
See My result
>>> newdata.groupBy('var210').count().show()
+------+-----+
|var210|count|
+------+-----+
| 3av_| 64|
| 7A3j| 509|
| g5HH| 1489|
| oT7d| 109|
| DM_V| 149|
| uKAI|44883|
+------+-----+
# store the above result in df
>>> df=newdata.groupBy('var210').count()
>>> df.orderBy(df['count'].desc()).collect()
[Row(var210='uKAI', count=44883),
Row(var210='g5HH', count=1489),
Row(var210='7A3j', count=509),
Row(var210='DM_V', count=149),
Row(var210='oT7d', count=109),
Row(var210='3av_', count=64)]
# get the first value using collect()
>>> mode = df.orderBy(df['count'].desc()).collect()[0][0]
>>> mode
'uKAI'
using groupBy() function getting count of each category in column. df is my result data frame has two columns var210,count. using orderBy() with column name 'count' in descending order give the max value in 1st row of data frame. collect()[0][0] is used to get the 1 tuple in data frame
模式的问题和中值的问题差不多。虽然计算起来很容易,但计算却相当昂贵。可以使用sort后跟本地和全局聚合,也可以只使用另一个wordcount和filter:
无论哪种方式,都可能需要对每列进行完全洗牌。
您可以使用Java代码计算列模式,如下所示:
相关问题 更多 >
编程相关推荐