Spark:无法运行程序“jupyter”

Question

当我向Thread提交python文件时，遇到以下错误：

ERROR yarn.ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program "jupyter": error=2, No such file or directory
java.io.IOException: Cannot run program "jupyter": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:98)
    at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

本人提交人：

spark-submit --master yarn \
--deploy-mode cluster \
--executor-memory 8g \
--driver-memory 10g \
--num-executors 100 \
--executor-cores 10 \
test.py

我试图在~/.bashrc中设置export PYSPARK_PYTHON=/opt/anaconda3/bin/python，但遇到另一个错误：/opt/bin/python: No such file or directory

但我肯定我知道文件/opt/bin/python存在

.bashrc文件如下：

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/opt/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/opt/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<


# Auto deploy profile
source /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64
export PATH=$JAVA_HOME/bin:$PATH
@
"~/.bashrc" 76L, 2780C
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export SPARK_HOME=/home//spark2
export PATH=$PATH:$SPARK_HOME/bin

# hadoop
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$JAVA_HOME/lib/tools.jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$JAVA_HOME/lib/dt.jar
# JAVA
#export CLASSPATH=$(JARS=("$LIB"/*.jar); IFS=:; echo "${JARS[*]}")
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/tools.jar
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/dt.jar

# Jupyter notebook for pyspark
export PYSPARK_PYTHON=/opt/anaconda3/bin/python

我可以在本地spark上成功运行我的代码

有什么解决办法吗

Answer 1

我也遇到过类似的问题。下面的解决方案解决了我的问题

update ~/.bashrc as below

PYSPARK_PYTHON=/usr/bin/python3