Pyspark：异常：Java网关进程在向驱动程序发送其端口号之前退出

Question

我想在我的MacBookAir上运行pyspark。当我尝试启动它时，我会得到错误：

Exception: Java gateway process exited before sending the driver its port number

当启动时调用sc=SparkContext（）时。我尝试运行以下命令：

./bin/pyspark
./bin/spark-shell
export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"

没有用。我也看过这里：

Spark + Python - Java gateway process exited before sending the driver its port number?

但这个问题一直没有答案。请帮忙！谢谢。

Answer 1

我的iphython笔记本（IPython 3.2.1）在Linux（ubuntu）上也有同样的问题。

我的例子中缺少的是在$PYSPARK_SUBMIT_ARGS环境中设置主URL，如下所示（假设您使用bash）：

export PYSPARK_SUBMIT_ARGS="--master spark://<host>:<port>"

例如

export PYSPARK_SUBMIT_ARGS="--master spark://192.168.2.40:7077"

你可以把它放到你的.bashrc文件中。在spark master的日志中获得正确的URL（使用/sbin/start_master.sh启动master时会报告此日志的位置）。

Answer 2

这应该对你有帮助

一种解决方案是将pyspark shell添加到shell环境变量pyspark_SUBMIT_ARGS：

export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell"

python/pyspark/java_gateway.py中有一个更改，如果用户设置了pyspark_SUBMIT_ARGS变量，则该更改要求pyspark_SUBMIT_ARGS包含pyspark shell。

Answer 3

一个可能的原因是未设置JAVA_HOME，因为未安装JAVA。

我遇到了同样的问题。上面写着

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/launcher/Main : Unsupported major.minor version 51.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/spark/python/pyspark/conf.py", line 104, in __init__
    SparkContext._ensure_initialized()
  File "/opt/spark/python/pyspark/context.py", line 243, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway()
  File "/opt/spark/python/pyspark/java_gateway.py", line 94, in launch_gateway
    raise Exception("Java gateway process exited before sending the driver its port number")
Exception: Java gateway process exited before sending the driver its port number

在sc = pyspark.SparkConf()。我通过跑步解决了这个问题

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

来自https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04