我在Apache配置单元中使用Python用户定义函数将字符从小写字符更改为大写字符。我得到的错误是“关闭运算符时配置单元运行时出错”。在
下面是我尝试的查询:
describe table1;
OK
item string
count int
city string
select * from table1;
aaa 1 tokyo
aaa 2 london
bbb 3 washington
ccc 4 moscow
ddd 5 bejing
从上表中,item和city字段应该从小写变为大写,count应该递增10。在
使用的Python脚本:
^{pr2}$将表1数据插入表2
create table table2(ITEM1 string, COUNT1 int, CITY1 string) row format delimited fields terminated by ',';
add FILE caseconvert.py
insert overwrite table table2 select TRANSFORM(item,count,city) using 'python caseconvert.py' as (ITEM1,COUNT1,CITY1) from table1;
如果我执行,我会得到以下错误。我无法追踪到这个问题。我能知道出问题了吗?在
Total MapReduce jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201508151858_0014, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201508151858_0014
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201508151858_0014
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2015-08-15 22:24:06,212 Stage-1 map = 0%, reduce = 0%
2015-08-15 22:25:01,559 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201508151858_0014 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201508151858_0014
Examining task ID: task_201508151858_0014_m_000002 (and more) from job job_201508151858_0014
Task with the most failures(4):
-----
Task ID:
task_201508151858_0014_m_000000
URL:
http://localhost.localdomain:50030/taskdetails.jsp?jobid=job_201508151858_0014&tipid=task_201508151858_0014_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Hive Runtime Error while closing operators
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:224)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:488)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:570)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:5
FAILED: Execution Error, return code 20003 from org.apache.hadoop.hive.ql.exec.MapRedTask. An error occurred when trying to close the Operator running your custom script.
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
在Python脚本的最后一行(将输出打印到STDOUT)中,在没有定义FRUIT1的情况下调用它。这应该是城市1。您还导入了字符串,但没有使用它。我会用另一种方式写剧本:
然后,我将使用CREATE TABLE作为SELECT查询(假设TABLE1和python脚本都位于HDFS中):
^{pr2}$这对我很有效。但是,有一种更简单的方法可以使用配置单元内置函数进行所需的转换:
upper(string A)>>>返回将A的所有字符转换为大写的字符串。例如,upper('fOoBaR')的结果是'fOoBaR'。在
城市(城市)当然可以。在
因此,表2可以创建如下:
^{3}$比编写自定义自定义项要容易得多。在
相关问题 更多 >
编程相关推荐