java Apache Spark MySQL JavaRDD。foreachPartition为什么我得到ClassNotFoundException
我想将每个分区的数据保存到MySQL数据库。为此,我创建了实现VoidFunction<>
的类:
public class DatabaseSaveFunction implements VoidFunction<Iterator<String>> {
/**
*
*/
private static final long serialVersionUID = -7039277486852158360L;
public void call(Iterator<String> it) {
Connection connect = null;
PreparedStatement preparedStatement = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connect = DriverManager.getConnection("jdbc:mysql://"
+ "xxx.us-west-2.rds.amazonaws.com" + "/"
+ "xxx", "xxx", "xxx");
preparedStatement = connect
.prepareStatement("insert into testdatabase.test values (default, ?)");
while (it.hasNext()) {
String outputElement = it.next();
preparedStatement.setString(1, "" + outputElement.length());
preparedStatement.executeUpdate();
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
try {
connect.close();
preparedStatement.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
在我的主要方法类中,我调用:
output.foreachPartition(new DatabaseSaveFunction());
我发现以下错误:
15/05/06 15:34:00 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 4, ip-172-31-36-44.us-west-2.compute.internal): java.lang.ClassNotFoundException: DatabaseSaveFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
工作日志:
15/05/06 15:34:00 ERROR executor.Executor: Exception in task 1.0 in stage 1.0 (TID 5)
java.lang.ClassNotFoundException: DatabaseSaveFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
谁能告诉我我做错了什么吗?对此我将非常感激
# 1 楼答案
将外部类导出到jar,并添加类似于sc.addJar(“/path/x.jar”)的类,其中sc是主目录中的JavaSparkContext。那么你就不会犯这个错误了。错误是因为spark程序无法找到类。此外,在spark 1.3及更高版本中,您可以简单地使用jdbc的映射选项,然后使用load(“jdbc”,选项)创建数据帧并从任何RDBMS加载数据。它真的很方便。我不确定这种方法是否适用于将任何RDBMS连接到spark。如果你还有其他问题,请告诉我