Pyspark从datafram中的列中删除空值

2024-05-01 21:48:16 发布

您现在位置:Python中文网/ 问答频道 /正文

我的数据框如下所示

ID,FirstName,LastName

1,Navee,Srikanth

2,,Srikanth 

3,Naveen,

现在我的问题是我必须删除第2行,因为名字是空的。

我正在使用下面的pyspark脚本

join_Df1= Name.filter(Name.col(FirstName).isnotnull()).show()

我的错误是

  File "D:\0\NameValidation.py", line 13, in <module>
join_Df1= filter(Name.FirstName.isnotnull()).show()

TypeError: 'Column' object is not callable

有谁能帮我解决这个问题吗


Tags: 数据nameidshowfirstnamefilter名字pyspark
3条回答

你应该这样做

join_Df1.filter(join_Df1.FirstName.isNotNull()).show

希望这有帮助!

我想你可能需要这个。

这是您在csv文件my_test.csv中的输入:

ID,FirstName,LastName
1,Navee,Srikanth

2,,Srikanth

3,Naveen

代码:

import pandas as pd
df = pd.read_csv("my_test.csv")

print(df[df['FirstName'].notnull()])

输出:

  ID FirstName  LastName
0   1     Navee  Srikanth
2   3    Naveen       NaN

这就是你想要的!df[df['FirstName'].notnull()]

输出df['FirstName'].notnull()

0     True
1    False
2     True

这将创建一个数据帧df,其中df['FirstName'].notnull()返回True

这是怎么检查的?df['FirstName'].notnull()如果FirstName列的值不为空,则返回True;否则,如果NaN存在,则返回False

您的数据帧名似乎有空值Null。以下是一些可供尝试的选项:

df = sqlContext.createDataFrame([[1,'Navee','Srikanth'], [2,'','Srikanth'] , [3,'Naveen','']], ['ID','FirstName','LastName'])
df.show()
+---+---------+--------+
| ID|FirstName|LastName|
+---+---------+--------+
|  1|    Navee|Srikanth|
|  2|         |Srikanth|
|  3|   Naveen|        |
+---+---------+--------+

df.where(df.FirstName.isNotNull()).show() #This doen't remove null because df have empty value
+---+---------+--------+
| ID|FirstName|LastName|
+---+---------+--------+
|  1|    Navee|Srikanth|
|  2|         |Srikanth|
|  3|   Naveen|        |
+---+---------+--------+

df.where(df.FirstName != '').show()
+---+---------+--------+
| ID|FirstName|LastName|
+---+---------+--------+
|  1|    Navee|Srikanth|
|  3|   Naveen|        |
+---+---------+--------+

df.filter(df.FirstName != '').show()
+---+---------+--------+
| ID|FirstName|LastName|
+---+---------+--------+
|  1|    Navee|Srikanth|
|  3|   Naveen|        |
+---+---------+--------+

df.where("FirstName != ''").show()
+---+---------+--------+
| ID|FirstName|LastName|
+---+---------+--------+
|  1|    Navee|Srikanth|
|  3|   Naveen|        |
+---+---------+--------+

相关问题 更多 >