使用MiniDFSCluster对hadoop hdfs编写进行java单元测试
我编写了一个类,用于编写hadoop HDFS。 我使用的是1.2.1版的hadoop JAR
我想考这门课。 因此,基于this one等博客,我编写了我的代码:
private void createSimulatedHdfs() {
conf = new Configuration();
// 100K blocksize
conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024 * 100);
conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 100);
conf.setInt(DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY, 1);
conf.setLong(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, DFS_REPLICATION_INTERVAL);
conf.setInt(DFSConfigKeys.DFS_NAMENODE_REPLICATION_INTERVAL_KEY, DFS_REPLICATION_INTERVAL);
try {
// simulated HDFS
cluster = new MiniDFSCluster(conf, DATANODE_COUNT, true, null);
cluster.waitActive();
simulatedHdfs = cluster.getFileSystem();
} catch (IOException e) {
Assert.fail("Could not create simulated HDFS " + e.getMessage());
}
}
但是当运行新的MiniDFSCluster时,遇到异常:
java.lang.AssertionError: Could not create simulated HDFS Cannot run program "du": CreateProcess error=2, The system cannot find the file specified
at org.junit.Assert.fail(Assert.java:88)
at com.taptica.hdfs.writer.HdfsWriterUTest.createSimulatedHdfs(HdfsWriterUTest.java:101)
at com.taptica.hdfs.writer.HdfsWriterUTest.initJunitModeTest(HdfsWriterUTest.java:42)
at com.taptica.hdfs.writer.HdfsWriterUTest.writeTest(HdfsWriterUTest.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
我没有在本地环境上安装hadoop(我也不打算这么做)。 我怎样才能克服这一点
# 1 楼答案
对于JUnit测试,可以在不安装Hadoop和第三方UTIL的情况下使用本地文件系统:
还可以查看MRunit,MR测试的helpfull实用程序:http://mrunit.apache.org/