我有一个示例数据帧,如下所示。但是我的真实数据是40 million rows and 5200 columns
df = pd.DataFrame({
'subject_id':[1,1,1,1,2,2,2,2,3,3,4,4,4,4,4],
'readings' : ['READ_1','READ_2','READ_1','READ_3',np.nan,'READ_5',np.nan,'READ_8','READ_10','READ_12','READ_11','READ_14','READ_09','READ_08','READ_07'],
'val' :[5,6,7,np.nan,np.nan,7,np.nan,12,13,56,32,13,45,43,46],
})
from pyspark.sql.types import *
from pyspark.sql.functions import isnan, when, count, col
mySchema = StructType([ StructField("subject_id", LongType(), True)\
,StructField("readings", StringType(), True)\
,StructField("val", FloatType(), True)])
spark_df = spark.createDataFrame(df,schema=mySchema)
spark_df.select([((count(when(isnan(c)|col(c).isNull(), c))/spark_df.count())*100).alias(c) for c in spark_df.columns]).show()
上面的代码帮助我获得percentage of nulls/nan in each column
。但是当我在实际数据上运行相同的代码时,代码已经运行了很长时间,但是还没有输出。如何优化此搜索并删除具有80%null/nan的列?下面是我的服务器配置
更新的屏幕截图
目前没有回答
相关问题 更多 >
编程相关推荐