线程“main”org.apache.spark.SparkException中的异常与spark中的本地运行

Question

我尝试运行用main2.py编写的代码

import sys
from pyspark import SparkContext, SparkConf

sc = SparkContext()
data = ["Project",
"Gutenberg’s",
"Alice’s",
"Adventures",
"in",
"Wonderland",
"Project",
"Gutenberg’s",
"Adventures",
"in",
"Wonderland",
"Project",
"Gutenberg’s"]

rdd=sc.parallelize(data)
#map(f, preservesPartitioning=False)

rdd2=rdd.map(lambda x: (x,1))
for element in rdd2.collect():
    print(element)

我在本地运行它，使用：

./spark/bin/spark-submit --master --local[1] ./main2.py

main2.py位于根文件夹A中，spark文件夹也位于A中。 但是，当我跑步时，我有：

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/nayrow/Application/spark2/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" org.apache.spark.SparkException: Master must either be yarn or start with spark, mesos, k8s, or local
    at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:936)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:238)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我在linux上，谢谢你的回答

Answer 1

在local[1]之前不需要两个破折号：

./spark/bin/spark-submit  master local[1] ./main2.py

至于警告，似乎是一些Java版本问题。Spark仅适用于Java8或Java11，因此您应该检查您的Java版本是否正确