EMR ClassNotFoundException java中的Sparksubmit

8 月，1 周 Questions & Answers 1007

我在spring boot应用程序中创建了jar，我在其中创建spark会话来执行一些操作

然后我将这个jar发送到AWS EMR，并作为spark-submit或java -jar可执行文件运行，但该文件不起作用

这是我的jar结构，当我告诉我们jar -tf <jar name>

META-INF/
META-INF/MANIFEST.MF
org/
org/springframework/
org/springframework/boot/
org/springframework/boot/loader/
org/springframework/boot/loader/data/
org/springframework/boot/loader/data/RandomAccessDataFile$DataInputStream.class
org/springframework/boot/loader/jar/
org/springframework/boot/loader/jar/CentralDirectoryEndRecord.class
org/springframework/boot/loader/data/RandomAccessDataFile$FileAccess.class
org/springframework/boot/loader/jar/CentralDirectoryParser.class
org/springframework/boot/loader/data/RandomAccessDataFile$1.class
org/springframework/boot/loader/jar/ZipInflaterInputStream.class
org/springframework/boot/loader/data/RandomAccessDataFile.class
org/springframework/boot/loader/jar/Handler.class
org/springframework/boot/loader/data/RandomAccessData.class
org/springframework/boot/loader/archive/
org/springframework/boot/loader/archive/Archive$EntryFilter.class
org/springframework/boot/loader/jar/Bytes.class
org/springframework/boot/loader/archive/Archive.class
org/springframework/boot/loader/archive/JarFileArchive.class
org/springframework/boot/loader/PropertiesLauncher$PrefixMatchingArchiveFilter.class
org/springframework/boot/loader/PropertiesLauncher$ArchiveEntryFilter.class
org/springframework/boot/loader/PropertiesLauncher$1.class
org/springframework/boot/loader/PropertiesLauncher.class
org/springframework/boot/loader/Launcher.class
org/springframework/boot/loader/MainMethodRunner.class
org/springframework/boot/loader/ExecutableArchiveLauncher.class
org/springframework/boot/loader/WarLauncher.class
org/springframework/boot/loader/JarLauncher.class
org/springframework/boot/loader/util/
org/springframework/boot/loader/util/SystemPropertyUtils.class
org/springframework/boot/loader/LaunchedURLClassLoader$UseFastConnectionExceptionsEnumeration.class
org/springframework/boot/loader/LaunchedURLClassLoader.class
org/springframework/boot/loader/jar/JarURLConnection$1.class
org/springframework/boot/loader/archive/JarFileArchive$EntryIterator.class
org/springframework/boot/loader/jar/JarURLConnection$JarEntryName.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator$EntryComparator.class
org/springframework/boot/loader/jar/JarURLConnection.class
org/springframework/boot/loader/jar/JarFile$1.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntry.class
org/springframework/boot/loader/archive/ExplodedArchive$1.class
org/springframework/boot/loader/jar/JarFile$2.class
org/springframework/boot/loader/jar/JarFile$JarFileType.class
org/springframework/boot/loader/archive/ExplodedArchive.class
org/springframework/boot/loader/archive/JarFileArchive$JarFileEntry.class
org/springframework/boot/loader/jar/JarFile.class
org/springframework/boot/loader/jar/JarEntry.class
org/springframework/boot/loader/jar/FileHeader.class
org/springframework/boot/loader/jar/StringSequence.class
org/springframework/boot/loader/jar/AsciiBytes.class
org/springframework/boot/loader/jar/JarFileEntries$1.class
org/springframework/boot/loader/jar/JarFileEntries$EntryIterator.class
org/springframework/boot/loader/jar/JarFileEntries.class
org/springframework/boot/loader/jar/CentralDirectoryVisitor.class
org/springframework/boot/loader/jar/JarEntryFilter.class
org/springframework/boot/loader/jar/CentralDirectoryFileHeader.class
org/springframework/boot/loader/archive/Archive$Entry.class
BOOT-INF/
BOOT-INF/classes/
BOOT-INF/classes/com/
BOOT-INF/classes/com/ati/
BOOT-INF/classes/com/ati/media/
BOOT-INF/classes/com/ati/media/ref/
BOOT-INF/classes/com/ati/media/ref/scd/
BOOT-INF/classes/com/ati/media/ref/scd/config/
BOOT-INF/classes/com/ati/media/ref/scd/controllers/
BOOT-INF/classes/com/ati/media/ref/scd/error/
BOOT-INF/classes/com/ati/media/ref/scd/repositories/
BOOT-INF/classes/com/ati/media/ref/scd/services/
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/
META-INF/maven/
META-INF/maven/com.ati.media.ref/
META-INF/maven/com.ati.media.ref/ref-svc-scd/
BOOT-INF/classes/application-local.yml
BOOT-INF/classes/bootstrap.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SwaggerConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleExceptionHandler.class
BOOT-INF/classes/com/ati/media/ref/scd/ScheduleApplication.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleServiceImpl.class
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.xml
BOOT-INF/classes/bootstrap-docker.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/ScheduleConstants.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleCreationException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.xml
BOOT-INF/classes/com/ati/media/ref/scd/services/ConfigurationService.class
BOOT-INF/classes/git.properties
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.properties
BOOT-INF/classes/application.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SparkConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleEditException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/SparkAthenaRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleService.class
META-INF/build-info.properties
BOOT-INF/classes/bootstrap-local.yml
BOOT-INF/classes/com/ati/media/ref/scd/controllers/ScheduleController.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/AthenaQueryBuilder.class
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/ScheduleValidator.class
META-INF/build.properties
BOOT-INF/lib/
BOOT-INF/lib/core-tech-starter-web-base-1.0.0.jar
BOOT-INF/lib/spring-boot-starter-web-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-json-2.0.3.RELEASE.jar
BOOT-INF/lib/jackson-datatype-jdk8-2.9.6.jar
BOOT-INF/lib/jackson-datatype-jsr310-2.9.6.jar
BOOT-INF/lib/jackson-module-parameter-names-2.9.6.jar
BOOT-INF/lib/spring-boot-starter-tomcat-2.0.3.RELEASE.jar
BOOT-INF/lib/tomcat-embed-core-8.5.31.jar
BOOT-INF/lib/tomcat-embed-el-8.5.31.jar
BOOT-INF/lib/tomcat-embed-websocket-8.5.31.jar
BOOT-INF/lib/hibernate-validator-6.0.10.Final.jar
BOOT-INF/lib/spring-web-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-webmvc-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-expression-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-webflux-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-reactor-netty-2.0.3.RELEASE.jar
BOOT-INF/lib/reactor-netty-0.7.8.RELEASE.jar
BOOT-INF/lib/netty-codec-http-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-4.1.25.Final.jar
BOOT-INF/lib/netty-buffer-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-4.1.25.Final.jar
BOOT-INF/lib/netty-resolver-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-proxy-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-socks-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar
BOOT-INF/lib/netty-common-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-unix-common-4.1.25.Final.jar
BOOT-INF/lib/spring-webflux-5.0.7.RELEASE.jar
BOOT-INF/lib/reactor-core-3.1.8.RELEASE.jar
BOOT-INF/lib/nio-multipart-parser-1.1.0.jar
BOOT-INF/lib/nio-stream-storage-1.1.3.jar
BOOT-INF/lib/spring-boot-starter-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/micrometer-core-1.0.5.jar
BOOT-INF/lib/HdrHistogram-2.1.10.jar
BOOT-INF/lib/LatencyUtils-2.0.3.jar
BOOT-INF/lib/spring-boot-starter-log4j2-2.0.3.RELEASE.jar
BOOT-INF/lib/log4j-slf4j-impl-2.10.0.jar
BOOT-INF/lib/log4j-core-2.10.0.jar
BOOT-INF/lib/log4j-jul-2.10.0.jar
BOOT-INF/lib/log4j-1.2-api-2.7.jar
BOOT-INF/lib/log4j-api-2.10.0.jar
BOOT-INF/lib/lombok-1.18.0.jar
BOOT-INF/lib/micrometer-registry-prometheus-1.0.3.jar
BOOT-INF/lib/simpleclient_common-0.3.0.jar
BOOT-INF/lib/simpleclient-0.3.0.jar
BOOT-INF/lib/spring-cloud-starter-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-sleuth-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-sleuth-core-2.0.0.RELEASE.jar
BOOT-INF/lib/aspectjrt-1.8.13.jar
BOOT-INF/lib/brave-5.1.0.jar
BOOT-INF/lib/brave-context-log4j2-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-web-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-http-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-rabbit-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-kafka-clients-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpasyncclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-webmvc-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-servlet-5.1.0.jar
BOOT-INF/lib/spring-cloud-sleuth-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/zipkin-2.9.3.jar
BOOT-INF/lib/zipkin-reporter-2.7.3.jar
BOOT-INF/lib/zipkin-sender-kafka11-2.7.3.jar
BOOT-INF/lib/zipkin-sender-amqp-client-2.7.3.jar
BOOT-INF/lib/core-tech-starter-amqp-1.0.0.jar
BOOT-INF/lib/spring-cloud-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-validation-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-messaging-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-integration-core-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-integration-jmx-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-amqp-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-rabbit-2.0.4.RELEASE.jar
BOOT-INF/lib/spring-amqp-2.0.4.RELEASE.jar
BOOT-INF/lib/amqp-client-5.1.2.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-amqp-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-config-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-context-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-rsa-1.0.5.RELEASE.jar
BOOT-INF/lib/bcpkix-jdk15on-1.56.jar
BOOT-INF/lib/spring-cloud-config-client-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-aop-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-aop-5.0.7.RELEASE.jar
BOOT-INF/lib/aspectjweaver-1.8.13.jar
BOOT-INF/lib/core-tech-lib-core-1.0.0.jar
BOOT-INF/lib/spring-retry-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-openfeign-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-openfeign-core-2.0.0.RELEASE.jar
BOOT-INF/lib/feign-core-9.5.1.jar
BOOT-INF/lib/feign-slf4j-9.5.1.jar
BOOT-INF/lib/feign-hystrix-9.5.1.jar
BOOT-INF/lib/feign-java8-9.5.1.jar
BOOT-INF/lib/spring-cloud-starter-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/ribbon-2.2.5.jar
BOOT-INF/lib/ribbon-transport-2.2.5.jar
BOOT-INF/lib/rxnetty-contexts-0.4.9.jar
BOOT-INF/lib/rxnetty-servo-0.4.9.jar
BOOT-INF/lib/rxnetty-0.4.9.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final.jar
BOOT-INF/lib/ribbon-core-2.2.5.jar
BOOT-INF/lib/ribbon-httpclient-2.2.5.jar
BOOT-INF/lib/jersey-client-1.19.1.jar
BOOT-INF/lib/jersey-apache-client4-1.19.1.jar
BOOT-INF/lib/servo-core-0.12.21.jar
BOOT-INF/lib/netflix-commons-util-0.3.0.jar
BOOT-INF/lib/ribbon-loadbalancer-2.2.5.jar
BOOT-INF/lib/netflix-statistics-0.1.1.jar
BOOT-INF/lib/commons-text-1.3.jar
BOOT-INF/lib/commons-collections4-4.1.jar
BOOT-INF/lib/core-tech-starter-web-api-1.0.0.jar
BOOT-INF/lib/springfox-swagger2-2.9.2.jar
BOOT-INF/lib/swagger-annotations-1.5.20.jar
BOOT-INF/lib/swagger-models-1.5.20.jar
BOOT-INF/lib/springfox-spi-2.9.2.jar
BOOT-INF/lib/springfox-core-2.9.2.jar
BOOT-INF/lib/springfox-schema-2.9.2.jar
BOOT-INF/lib/springfox-swagger-common-2.9.2.jar
BOOT-INF/lib/springfox-spring-web-2.9.2.jar
BOOT-INF/lib/guava-20.0.jar
BOOT-INF/lib/classmate-1.3.4.jar
BOOT-INF/lib/spring-plugin-core-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-plugin-metadata-1.2.0.RELEASE.jar
BOOT-INF/lib/mapstruct-1.2.0.Final.jar
BOOT-INF/lib/springfox-swagger-ui-2.9.2.jar
BOOT-INF/lib/spring-cloud-starter-netflix-hystrix-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/archaius-core-0.7.6.jar
BOOT-INF/lib/hystrix-core-1.5.12.jar
BOOT-INF/lib/rxjava-1.3.8.jar
BOOT-INF/lib/hystrix-serialization-1.5.12.jar
BOOT-INF/lib/jackson-module-afterburner-2.9.6.jar
BOOT-INF/lib/hystrix-metrics-event-stream-1.5.12.jar
BOOT-INF/lib/hystrix-javanica-1.5.12.jar
BOOT-INF/lib/asm-5.0.4.jar
BOOT-INF/lib/rxjava-reactive-streams-1.2.1.jar
BOOT-INF/lib/reactive-streams-1.0.2.jar
BOOT-INF/lib/spring-cloud-netflix-hystrix-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-commons-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-crypto-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-stream-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/postgresql-42.2.2.jar
BOOT-INF/lib/athenaJDBC-1.1.0.jar
BOOT-INF/lib/lib-client-scd-0.0.0.jar
BOOT-INF/lib/core-tech-starter-web-client-1.0.0.jar
BOOT-INF/lib/hibernate-jpa-2.1-api-1.0.2.Final.jar
BOOT-INF/lib/gson-2.8.5.jar
BOOT-INF/lib/spring-boot-starter-data-jpa-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/javax.annotation-api-1.3.2.jar
BOOT-INF/lib/snakeyaml-1.19.jar
BOOT-INF/lib/spring-boot-starter-jdbc-2.0.3.RELEASE.jar
BOOT-INF/lib/HikariCP-2.7.9.jar
BOOT-INF/lib/spring-jdbc-5.0.7.RELEASE.jar
BOOT-INF/lib/hibernate-core-5.2.17.Final.jar
BOOT-INF/lib/jboss-logging-3.3.2.Final.jar
BOOT-INF/lib/javassist-3.22.0-GA.jar
BOOT-INF/lib/antlr-2.7.7.jar
BOOT-INF/lib/jandex-2.0.3.Final.jar
BOOT-INF/lib/dom4j-1.6.1.jar
BOOT-INF/lib/hibernate-commons-annotations-5.0.1.Final.jar
BOOT-INF/lib/javax.transaction-api-1.2.jar
BOOT-INF/lib/spring-data-jpa-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-data-commons-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-orm-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-context-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-beans-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-aspects-5.0.7.RELEASE.jar
BOOT-INF/lib/mybatis-spring-boot-starter-1.3.1.jar
BOOT-INF/lib/mybatis-spring-boot-autoconfigure-1.3.1.jar
BOOT-INF/lib/mybatis-3.4.5.jar
BOOT-INF/lib/mybatis-spring-1.3.1.jar
BOOT-INF/lib/spring-boot-starter-jooq-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-tx-5.0.7.RELEASE.jar
BOOT-INF/lib/jooq-3.10.7.jar
BOOT-INF/lib/byte-buddy-1.7.11.jar
BOOT-INF/lib/objenesis-2.6.jar
BOOT-INF/lib/spring-core-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-jcl-5.0.7.RELEASE.jar
BOOT-INF/lib/core-tech-starter-web-request-details-1.0.0.jar
BOOT-INF/lib/spark-core_2.11-2.3.0.jar
BOOT-INF/lib/avro-1.7.7.jar
BOOT-INF/lib/jackson-core-asl-1.9.13.jar
BOOT-INF/lib/paranamer-2.3.jar
BOOT-INF/lib/commons-compress-1.4.1.jar
BOOT-INF/lib/xz-1.0.jar
BOOT-INF/lib/avro-mapred-1.7.7-hadoop2.jar
BOOT-INF/lib/avro-ipc-1.7.7.jar
BOOT-INF/lib/avro-ipc-1.7.7-tests.jar
BOOT-INF/lib/chill_2.11-0.8.4.jar
BOOT-INF/lib/kryo-shaded-3.0.3.jar
BOOT-INF/lib/minlog-1.3.0.jar
BOOT-INF/lib/chill-java-0.8.4.jar
BOOT-INF/lib/xbean-asm5-shaded-4.4.jar
BOOT-INF/lib/hadoop-client-2.6.5.jar
BOOT-INF/lib/hadoop-hdfs-2.6.5.jar
BOOT-INF/lib/xercesImpl-2.9.1.jar
BOOT-INF/lib/xml-apis-1.4.01.jar
BOOT-INF/lib/htrace-core-3.0.4.jar
BOOT-INF/lib/hadoop-mapreduce-client-app-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-common-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-shuffle-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-core-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-jobclient-2.6.5.jar
BOOT-INF/lib/hadoop-annotations-2.6.5.jar
BOOT-INF/lib/spark-launcher_2.11-2.3.0.jar
BOOT-INF/lib/spark-kvstore_2.11-2.3.0.jar
BOOT-INF/lib/leveldbjni-all-1.8.jar
BOOT-INF/lib/jackson-core-2.9.6.jar
BOOT-INF/lib/spark-network-common_2.11-2.3.0.jar
BOOT-INF/lib/spark-network-shuffle_2.11-2.3.0.jar
BOOT-INF/lib/spark-unsafe_2.11-2.3.0.jar
BOOT-INF/lib/jets3t-0.9.4.jar
BOOT-INF/lib/httpcore-4.4.9.jar
BOOT-INF/lib/activation-1.1.1.jar
BOOT-INF/lib/bcprov-jdk15on-1.52.jar
BOOT-INF/lib/java-xmlbuilder-1.1.jar
BOOT-INF/lib/base64-2.3.8.jar
BOOT-INF/lib/curator-recipes-4.0.1.jar
BOOT-INF/lib/curator-framework-4.0.1.jar
BOOT-INF/lib/javax.servlet-api-3.1.0.jar
BOOT-INF/lib/commons-lang3-3.7.jar
BOOT-INF/lib/commons-math3-3.4.1.jar
BOOT-INF/lib/jsr305-1.3.9.jar
BOOT-INF/lib/slf4j-api-1.7.25.jar
BOOT-INF/lib/jul-to-slf4j-1.7.25.jar
BOOT-INF/lib/jcl-over-slf4j-1.7.25.jar
BOOT-INF/lib/compress-lzf-1.0.3.jar
BOOT-INF/lib/snappy-java-1.1.2.6.jar
BOOT-INF/lib/lz4-java-1.4.0.jar
BOOT-INF/lib/zstd-jni-1.3.2-2.jar
BOOT-INF/lib/RoaringBitmap-0.5.11.jar
BOOT-INF/lib/commons-net-2.2.jar
BOOT-INF/lib/scala-library-2.11.8.jar
BOOT-INF/lib/json4s-jackson_2.11-3.2.11.jar
BOOT-INF/lib/json4s-core_2.11-3.2.11.jar
BOOT-INF/lib/json4s-ast_2.11-3.2.11.jar
BOOT-INF/lib/scalap-2.11.0.jar
BOOT-INF/lib/scala-compiler-2.11.0.jar
BOOT-INF/lib/scala-xml_2.11-1.0.1.jar
BOOT-INF/lib/jersey-client-2.26.jar
BOOT-INF/lib/javax.ws.rs-api-2.1.jar
BOOT-INF/lib/javax.inject-2.5.0-b42.jar
BOOT-INF/lib/jersey-common-2.26.jar
BOOT-INF/lib/osgi-resource-locator-1.0.1.jar
BOOT-INF/lib/jersey-server-2.26.jar
BOOT-INF/lib/jersey-media-jaxb-2.26.jar
BOOT-INF/lib/validation-api-2.0.1.Final.jar
BOOT-INF/lib/jersey-container-servlet-2.26.jar
BOOT-INF/lib/jersey-container-servlet-core-2.26.jar
BOOT-INF/lib/netty-all-4.1.25.Final.jar
BOOT-INF/lib/netty-3.9.9.Final.jar
BOOT-INF/lib/stream-2.7.0.jar
BOOT-INF/lib/metrics-core-3.2.6.jar
BOOT-INF/lib/metrics-jvm-3.2.6.jar
BOOT-INF/lib/metrics-json-3.2.6.jar
BOOT-INF/lib/metrics-graphite-3.2.6.jar
BOOT-INF/lib/jackson-databind-2.9.6.jar
BOOT-INF/lib/jackson-module-scala_2.11-2.9.6.jar
BOOT-INF/lib/scala-reflect-2.11.11.jar
BOOT-INF/lib/jackson-module-paranamer-2.9.6.jar
BOOT-INF/lib/ivy-2.4.0.jar
BOOT-INF/lib/oro-2.0.8.jar
BOOT-INF/lib/pyrolite-4.13.jar
BOOT-INF/lib/py4j-0.10.6.jar
BOOT-INF/lib/spark-tags_2.11-2.3.0.jar
BOOT-INF/lib/commons-crypto-1.0.0.jar
BOOT-INF/lib/unused-1.0.0.jar
BOOT-INF/lib/simba-athena-jdbc-driver-2.0.2.jar
BOOT-INF/lib/spark-sql_2.11-2.3.0.jar
BOOT-INF/lib/univocity-parsers-2.5.9.jar
BOOT-INF/lib/spark-sketch_2.11-2.3.0.jar
BOOT-INF/lib/spark-catalyst_2.11-2.3.0.jar
BOOT-INF/lib/scala-parser-combinators_2.11-1.0.4.jar
BOOT-INF/lib/janino-3.0.8.jar
BOOT-INF/lib/commons-compiler-3.0.8.jar
BOOT-INF/lib/antlr4-runtime-4.7.jar
BOOT-INF/lib/orc-core-1.4.1-nohive.jar
BOOT-INF/lib/protobuf-java-2.5.0.jar
BOOT-INF/lib/commons-lang-2.6.jar
BOOT-INF/lib/aircompressor-0.8.jar
BOOT-INF/lib/orc-mapreduce-1.4.1-nohive.jar
BOOT-INF/lib/parquet-column-1.8.2.jar
BOOT-INF/lib/parquet-common-1.8.2.jar
BOOT-INF/lib/parquet-encoding-1.8.2.jar
BOOT-INF/lib/parquet-hadoop-1.8.2.jar
BOOT-INF/lib/parquet-format-2.3.1.jar
BOOT-INF/lib/parquet-jackson-1.8.2.jar
BOOT-INF/lib/arrow-vector-0.8.0.jar
BOOT-INF/lib/arrow-format-0.8.0.jar
BOOT-INF/lib/arrow-memory-0.8.0.jar
BOOT-INF/lib/hppc-0.7.2.jar
BOOT-INF/lib/flatbuffers-1.2.0-3f79e055.jar
BOOT-INF/lib/spark-yarn_2.11-2.3.0.jar
BOOT-INF/lib/hadoop-yarn-api-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-common-2.6.5.jar
BOOT-INF/lib/jaxb-api-2.3.0.jar
BOOT-INF/lib/jetty-util-6.1.26.jar
BOOT-INF/lib/jackson-jaxrs-1.9.13.jar
BOOT-INF/lib/jackson-xc-1.9.13.jar
BOOT-INF/lib/commons-cli-1.2.jar
BOOT-INF/lib/guice-servlet-3.0.jar
BOOT-INF/lib/commons-io-2.4.jar
BOOT-INF/lib/guice-3.0.jar
BOOT-INF/lib/javax.inject-1.jar
BOOT-INF/lib/aopalliance-1.0.jar
BOOT-INF/lib/log4j-1.2.17.jar
BOOT-INF/lib/hadoop-yarn-server-web-proxy-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-server-common-2.6.5.jar
BOOT-INF/lib/jetty-6.1.26.jar
BOOT-INF/lib/hadoop-yarn-client-2.6.5.jar
BOOT-INF/lib/hadoop-aws-2.7.3.jar
BOOT-INF/lib/hadoop-common-2.7.3.jar
BOOT-INF/lib/xmlenc-0.52.jar
BOOT-INF/lib/commons-collections-3.2.2.jar
BOOT-INF/lib/servlet-api-2.5.jar
BOOT-INF/lib/jsp-api-2.1.jar
BOOT-INF/lib/jersey-core-1.19.1.jar
BOOT-INF/lib/jsr311-api-1.1.1.jar
BOOT-INF/lib/jersey-json-1.9.jar
BOOT-INF/lib/jettison-1.1.jar
BOOT-INF/lib/jaxb-impl-2.2.3-1.jar
BOOT-INF/lib/jersey-server-1.19.1.jar
BOOT-INF/lib/commons-logging-1.1.3.jar
BOOT-INF/lib/commons-configuration-1.6.jar
BOOT-INF/lib/commons-digester-1.8.jar
BOOT-INF/lib/commons-beanutils-1.7.0.jar
BOOT-INF/lib/commons-beanutils-core-1.8.0.jar
BOOT-INF/lib/hadoop-auth-2.7.3.jar
BOOT-INF/lib/apacheds-kerberos-codec-2.0.0-M15.jar
BOOT-INF/lib/apacheds-i18n-2.0.0-M15.jar
BOOT-INF/lib/api-asn1-api-1.0.0-M20.jar
BOOT-INF/lib/api-util-1.0.0-M20.jar
BOOT-INF/lib/jsch-0.1.54.jar
BOOT-INF/lib/curator-client-2.7.1.jar
BOOT-INF/lib/htrace-core-3.1.0-incubating.jar
BOOT-INF/lib/zookeeper-3.4.6.jar
BOOT-INF/lib/jackson-annotations-2.9.0.jar
BOOT-INF/lib/aws-java-sdk-1.7.4.jar
BOOT-INF/lib/spark-hive_2.11-2.3.0.jar
BOOT-INF/lib/parquet-hadoop-bundle-1.6.0.jar
BOOT-INF/lib/hive-exec-1.2.1.spark2.jar
BOOT-INF/lib/javolution-5.5.1.jar
BOOT-INF/lib/apache-log4j-extras-1.2.17.jar
BOOT-INF/lib/antlr-runtime-3.4.jar
BOOT-INF/lib/stringtemplate-3.2.1.jar
BOOT-INF/lib/ST4-4.0.4.jar
BOOT-INF/lib/JavaEWAH-0.3.2.jar
BOOT-INF/lib/snappy-0.2.jar
BOOT-INF/lib/stax-api-1.0.1.jar
BOOT-INF/lib/opencsv-2.3.jar
BOOT-INF/lib/hive-metastore-1.2.1.spark2.jar
BOOT-INF/lib/bonecp-0.8.0.RELEASE.jar
BOOT-INF/lib/datanucleus-api-jdo-3.2.6.jar
BOOT-INF/lib/datanucleus-rdbms-3.2.9.jar
BOOT-INF/lib/commons-pool-1.6.jar
BOOT-INF/lib/commons-dbcp-1.4.jar
BOOT-INF/lib/jdo-api-3.0.1.jar
BOOT-INF/lib/jta-1.1.jar
BOOT-INF/lib/commons-httpclient-3.1.jar
BOOT-INF/lib/calcite-avatica-1.2.0-incubating.jar
BOOT-INF/lib/calcite-core-1.2.0-incubating.jar
BOOT-INF/lib/calcite-linq4j-1.2.0-incubating.jar
BOOT-INF/lib/eigenbase-properties-1.1.5.jar
BOOT-INF/lib/httpclient-4.5.5.jar
BOOT-INF/lib/jackson-mapper-asl-1.9.13.jar
BOOT-INF/lib/commons-codec-1.11.jar
BOOT-INF/lib/joda-time-2.9.9.jar
BOOT-INF/lib/jodd-core-3.5.2.jar
BOOT-INF/lib/datanucleus-core-3.2.10.jar
BOOT-INF/lib/libthrift-0.9.3.jar
BOOT-INF/lib/libfb303-0.9.3.jar
BOOT-INF/lib/derby-10.14.1.0.jar
BOOT-INF/lib/core-tech-starter-docker-spring-boot-1.0.0.jar

当你试着运行

spark-submit --master yarn --deploy-mode client --driver-memory 10G --executor-memory 15G --class com.ati.media.ref.scd.ScheduleApplication

我认为错误是错误

java.lang.ClassNotFoundException: com.ati.media.ref.scd.ScheduleApplication
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

当我尝试使用

java -jar <jar>应用程序启动时没有任何问题

java -cp导致与spark submit相同的错误，未找到主计算

这里出了什么问题。我在^{中有主要功能

共 (1) 个答案

# 1 楼答案
您需要指定 jars <path to your jar>或在集群级别配置extrac JAR位置

从AWS documentation开始：

A bootstrap action or custom configuration is overriding the class paths. When this happens, the class loader only picks up the JAR files that exist in the location that you specified in your configuration.

解决这一问题的方法很少：

1）指定带有jar路径的 jars：
```
spark-submit  deploy-mode client  class org.apache.spark.examples.SparkPi  master yarn spark-examples.jar 100  jars /home/hadoop/extrajars/*
```
2）将自定义JAR（包含缺少的类）的路径添加到Spark类路径

在/etc/spark/conf/spark默认值中。conf，将自定义JAR的路径附加到错误堆栈跟踪中指定的类名。在下面的示例中，/home/hadoop/extrajars/*是自定义JAR路径

例如（如何修改正在运行的集群）：
```
sudo vim /etc/spark/conf/spark-defaults.conf

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*
```
有关更多信息，或如何在新集群上执行此操作，请参阅documentation

Python中文网

有 Java 编程相关的问题?

EMR ClassNotFoundException java中的Sparksubmit

共 (1) 个答案

# 1 楼答案