在pyspark中使用udf时,我有以下问题。在
只要我不使用任何udf,我的代码就可以正常工作。执行简单的操作(如选择列)或使用诸如concat之类的sql函数没有问题。当我对使用udf的DataFrame执行操作时,程序崩溃,出现以下异常:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/szymonk/Desktop/Projects/SparkTest/venv/lib/python2.7/site-packages/pyspark/jars/spark-unsafe_2.11-2.4.3.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
19/06/05 09:24:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File "/Users/szymonk/Desktop/Projects/SparkTest/Application.py", line 59, in <module>
transformations.select(udf_example(col("gender")).alias("udf_example")).show()
File "/Users/szymonk/Desktop/Projects/SparkTest/venv/lib/python2.7/site-packages/pyspark/sql/dataframe.py", line 378, in show
print(self._jdf.showString(n, 20, vertical))
File "/Users/szymonk/Desktop/Projects/SparkTest/venv/lib/python2.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/Users/szymonk/Desktop/Projects/SparkTest/venv/lib/python2.7/site-packages/pyspark/sql/utils.py", line 79, in deco
raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u'Unsupported class file major version 55'
我尝试过按照:Pyspark error - Unsupported class file major version 55中的建议更改JAVA_HOME,但没有帮助。在
我的代码没有什么特别之处。我只定义了一个简单的udf函数,它应该返回“Gender”列中的值的长度
^{pr2}$我不确定这是否重要,但我在Mac上使用了Pycharm。在
我找到了解决办法, 我不得不切换Pycharm的引导jdk(2xshift->;jdk->;select JDK1.8)
相关问题 更多 >
编程相关推荐