调用o128.resolveChoice时出错

Question

我有一个AWS胶水作业，目前每晚运行，扫描大约20 TB的原始JSON数据，并将其转换为拼花地板。我只有创建作业时生成的通用Python脚本。我遇到了一个导致作业失败并导致以下错误的问题

py4j.protocol.Py4JError: An error occurred while calling o131.resolveChoice

此作业以前已成功运行，没有任何问题。我做了一个更改并添加了分区键，在更改之后，它现在似乎失败了。这项工作几乎需要24小时才能完成，因此提出解决方案是一个缓慢的过程。我还没有找到任何与此匹配的错误，所以我很想知道这里发生了什么。有人有什么想法吗

这是Cloudwatch的回溯

Traceback (most recent call last):
  File "script_2020-09-14-02-00-49.py", line 17, in <module>
    resolvechoice = ResolveChoice.apply(frame = applymapping, choice = "make_struct", transformation_ctx = "resolvechoice")
  File "/mnt/yarn/usercache/root/appcache/application_1600045621100_0001/container_1600045621100_0001_02_000001/PyGlue.zip/awsglue/transforms/transform.py", line 24, in apply
  File "/mnt/yarn/usercache/root/appcache/application_1600045621100_0001/container_1600045621100_0001_02_000001/PyGlue.zip/awsglue/transforms/resolve_choice.py", line 17, in __call__
  File "/mnt/yarn/usercache/root/appcache/application_1600045621100_0001/container_1600045621100_0001_02_000001/PyGlue.zip/awsglue/dynamicframe.py", line 420, in resolveChoice
  File "/mnt/yarn/usercache/root/appcache/application_1600045621100_0001/container_1600045621100_0001_02_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/mnt/yarn/usercache/root/appcache/application_1600045621100_0001/container_1600045621100_0001_02_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/mnt/yarn/usercache/root/appcache/application_1600045621100_0001/container_1600045621100_0001_02_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 327, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o128.resolveChoice
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:41623)

在这方面似乎失败了

resolvechoice = ResolveChoice.apply(frame = applymapping, choice = "make_struct", transformation_ctx = "resolvechoice")

此脚本当前在我的预生产AWS环境中工作，但在我的生产环境中不工作