PySpark:RDD中的音频文件

Question

我在使用Python3.6.9的spark（版本2.3.2）集群中工作。我需要在RDD中加载一些音频文件.wav并执行一些操作，比如计算频谱图。首先，我使用spark_context.binaryFiles（）加载wav文件

binary_wave_rdd = spark_context.binaryFiles(audio_dir+'*.wav')

然后我需要将RDD中的二进制文件映射到我可以使用的其他文件。我试图使用librosapython包来实现这一点，但是当我在RDD上使用collect()时，它给了我一些奇怪的错误。代码如下：

binary_wave_rdd = self.spark_context.binaryFiles(audio_dir+'*.wav')
rdd = binary_wave_rdd.map(lambda x : (x[0], librosa.load(io.BytesIO(x[1]))))
coll = self.rdd.collect()

有人知道在RDD中转换二进制数据的另一种方法吗？我基本上需要它来创建每个文件的光谱图

错误

Traceback (most recent call last):
  File "/home/user24/LSCproject/Main.py", line 45, in <module>
    wav = WAV(spark_session, spark_context)
  File "/home/user24/LSCproject/wav_manipulation/wav.py", line 29, in __init__
    self.binary_to_librosa_rdd()
  File "/home/user24/LSCproject/wav_manipulation/wav.py", line 43, in binary_to_librosa_rdd
    l = self.rdd.collect()
  File "/home/hadoop/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 814, in collect
  File "/home/hadoop/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/home/hadoop/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/home/hadoop/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input Pattern hdfs://master:9000/user/user24/Database/audio_and_txt_files/* 1.wav matches 0 files
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265)
        at org.apache.spark.input.StreamFileInputFormat.setMinPartitions(PortableDataStream.scala:51)
        at org.apache.spark.rdd.BinaryFileRDD.getPartitions(BinaryFileRDD.scala:51)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
        at org.apache.spark.api.python.PythonRDD.getPartitions(PythonRDD.scala:57)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
        at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:165)
        at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)