java将数据从MapReduce导入到HBase（TableOutputFormat错误）

4 月，2 周 Questions & Answers 3912

A我试图将MapReduce作业中的数据保存到HBase中。我们制作的脚本在旧版本的Hadoop（CDH3u4）上非常有效。现在我们升级到了最新版本（CDH5.0.2），但脚本不起作用

在最新版本上运行该程序时，出现以下错误：

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:211)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:455)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
        at com.nrholding.t0_mr.main.ELogHBaseImport.main(ELogHBaseImport.java:89)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:389)
        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:366)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:247)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:188)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)
        ... 17 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:387)
        ... 22 more
Caused by: java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:195)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:479)
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:633)
        ... 27 more
Caused by: java.lang.ClassNotFoundException: org.cloudera.htrace.Trace
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 33 more

问题在于TableOutputFormat。所以我检查了一下：

正确的库在libpath中
Zookeeper仲裁和Zookeeper端口在hbase站点中设置。xml
表wp_json存在于hbase中

下面是产生问题的代码：

public static void main(String args[]) throws Exception {

    Configuration conf = new Configuration();
    conf.set("hbase.zookeeper.quorum", "zookeeper_server1,zookeeper_server2,zookeeper_server3");
    conf.set(TableOutputFormat.OUTPUT_TABLE, "wp_json");

    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    String input = otherArgs[0];
    Job job = Job.getInstance(conf, "ELogHBaseImport");
    // Input is just text files in HDFS
    FileInputFormat.addInputPath(job, new Path(input));
    job.setJarByClass(ELogHBaseImport.class);
    job.setMapperClass(Map.class);
    job.setNumReduceTasks(0);
    job.setOutputFormatClass(TableOutputFormat.class);
    job.waitForCompletion(true);
}

当我使用NullOutputFormat时，一切都很好，但没有写入hbase

TableOutputFormat中导致错误的部分如下所示：

163   /**
164    * Returns the output committer.
165    *
166    * @param context  The current context.
167    * @return The committer.
168    * @throws IOException When creating the committer fails.
169    * @throws InterruptedException When the job is aborted.
170    * @see org.apache.hadoop.mapreduce.OutputFormat#getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext)
171    */
172   @Override
173   public OutputCommitter getOutputCommitter(TaskAttemptContext context)
174   throws IOException, InterruptedException {
175     return new TableOutputCommitter();
176   }
177 
178   public Configuration getConf() {
179     return conf;
180   }
181 
182   @Override
183   public void setConf(Configuration otherConf) {
184     this.conf = HBaseConfiguration.create(otherConf);
185 
186     String tableName = this.conf.get(OUTPUT_TABLE);
187     if(tableName == null || tableName.length() <= 0) {
188       throw new IllegalArgumentException("Must specify table name");
189     }
190 
191     String address = this.conf.get(QUORUM_ADDRESS);
192     int zkClientPort = this.conf.getInt(QUORUM_PORT, 0);
193     String serverClass = this.conf.get(REGION_SERVER_CLASS);
194     String serverImpl = this.conf.get(REGION_SERVER_IMPL);
195 
196     try {
197       if (address != null) {
198         ZKUtil.applyClusterKeyToConf(this.conf, address);
199       }
200       if (serverClass != null) {
201         this.conf.set(HConstants.REGION_SERVER_IMPL, serverImpl);
202       }
203       if (zkClientPort != 0) {
204         this.conf.setInt(HConstants.ZOOKEEPER_CLIENT_PORT, zkClientPort);
205       }
206       this.table = new HTable(this.conf, tableName);
207       this.table.setAutoFlush(false, true);
208       LOG.info("Created table instance for "  + tableName);
209     } catch(IOException e) {
210       LOG.error(e);
211       throw new RuntimeException(e);
212     }
213   }

Python中文网

有 Java 编程相关的问题?

java将数据从MapReduce导入到HBase（TableOutputFormat错误）

共 (1) 个答案

# 1 楼答案