从spark datafram中提取Json数据

Question

+------------------------------------------------------------------+
| message                                                          |
+------------------------------------------------------------------+
|{"name":"east-desktop","viewers":447,"emptyCount":0,"version":0.3}|
|{"name":"west-desktop","viewers":111,"emptyCount":0,"version":0.6}|
|{"name":"west-desktop","viewers":115,"emptyCount":0,"version":0.1}|
+------------------------------------------------------------------+

message:string

我有一个dataframe，它在一列中包含json数据，我想将数据提取到单独的列中或作为json文件。在

我正在用pyspark在Databricks笔记本上工作。在

数据帧

^{pr2}$

或Json

{
  "name": "east-desktop",
  "viewers":  447,
  "emptyCount": 0,
  "version": 0.3,
}

Answer 1

pault是对的，这几乎是同一个问题，但您可以使用以下示例来实现数据帧输出：

df_new = spark.createDataFrame([
(str({"name":"east-desktop","viewers":447,"emptyCount":0,"version":0.3}))
],StringType())

schema = StructType(
    [
        StructField('name', StringType(), True),
        StructField('viewers', IntegerType(), True),
        StructField('emptyCount', IntegerType(), True),
        StructField('version', FloatType(), True)
   ]
)
df_new.withColumn("data", from_json("value",schema)).select("value", col('data.*')).show(truncate=False)

输出：

^{pr2}$