正在sparksubmit上查找和加载第991行的文件“<freezed importlib.\u bootstrap>”

Question

我目前使用Python:3.7.9，spark:spark-2.4.6-bin-hadoop2.6

在这个项目中，我的设置如下：

kafka-python 2.0.2
pip 21.2.4
py4j 0.10.9
pyspark 2.4.6
setuptools 57.4.0

系统环境变量：

SPARK_HOME         C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6
PYTHONPATH         %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src.zip;%PYTHONPATH%
PATH               %SPARK_HOME%\bin;%SPARK_HOME%\python;%PATH%
JAVA_HOME          C:\Program Files\Java\jdk1.8.0_261

我正在尝试用Python提交以下代码：

import sys
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
from kafka import KafkaConsumer

kafka_consumer.py位于C:\Users\MANU\PycharmProjects\pythonProject1中

Command I ran: (venv) C:\Users\MANU\PycharmProjects\pythonProject1>spark-submit  kafka_consumer.py

我在控制台中看到：

Traceback (most recent call last):
  File "C:/Users/MANU/PycharmProjects/pythonProject1/kafka_consumer.py", line 1, in <module>
    from pyspark import SparkContext, SparkConf
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\__init__.py", line 51, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\accumulators.py", line 97, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\serializers.py", line 72, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\cloudpickle.py", line 145, in <module>
  File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
TypeError: an integer is required (got type bytes)
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.