Pig Python UDF在MapReduce模式下失败
我有一个简单的Pig脚本，在里面我使用了最近推出的StreamingUDF功能来调用一个Python的用户定义函数（UDF）：
REGISTER 'process_tweet.py' USING streaming_python AS process_tweet;
REGISTER /usr/lib/hbase/lib/*.jar
tweets = LOAD 'hbase://brand_tweets' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('data:json') AS (json:chararray);
tweets = LIMIT tweets 100;
tweets = foreach tweets generate flatten(process_tweet.extract(json)) as (userid:long, text: chararray);
dump tweets;
这个Python函数process_tweet.extract其实就是把一个JSON对象（推文）反序列化，然后返回其中的一些值。
from pig_util import outputSchema
import json
@outputSchema("(userid:long, text:chararray)")
def extract(tweet):
content = json.loads(tweet)
return long(content['user']['id']), content['text']
当在本地模式下执行这个脚本（pig -x local）时，它运行得很顺利，没有错误，并且返回了预期的结果。然而，在MapReduce模式下，这个任务运行得非常慢，最后失败并出现了以下错误信息：
Backend error message
---------------------
AttemptID:attempt_1394221905204_0172_r_000000_1 Info:Error: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.impl.builtin.StreamingUDF)[tuple] - scope-7 Operator Key: scope-7) children: null at []]: java.lang.NullPointerException
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:464)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:412)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:645)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:405)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.NullPointerException
at org.apache.pig.impl.builtin.StreamingUDF.getControllerPath(StreamingUDF.java:268)
at org.apache.pig.impl.builtin.StreamingUDF.constructCommand(StreamingUDF.java:199)
at org.apache.pig.impl.builtin.StreamingUDF.startUdfController(StreamingUDF.java:163)
at org.apache.pig.impl.builtin.StreamingUDF.initialize(StreamingUDF.java:156)
at org.apache.pig.impl.builtin.StreamingUDF.exec(StreamingUDF.java:146)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:330)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextTuple(POUserFunc.java:369)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:333)
... 14 more