遍历列表中的每个元素，并将其传递给函数

Question

我想迭代列表中的每个元素并将其传递给函数。 这是我尝试过的，但得到以下错误。你知道吗

import call_functions
newObject = call_functions.call_functions_class()
size_DF = newObject.descr_size(sc, dataBase)
size_RDD = sc.parallelize(size_DF).map(lambda x : x[0])

def full_item_new(sc, dataBase, length, end_date):
    newObject.full_item(sc, dataBase, length, end_date)
size_RDD.map(lambda x : full_item_new(sc, dataBase, x[0], end_date)).collect()

def full_item(sc, dataBase, length, end_date):
    sqlContext = SQLContext(sc)
    insertDF = sqlContext.sql("insert into -----")
    return insertDF

错误：

"It appears that you are attempting to reference SparkContext from a broadcast " Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Answer 1

在没有看到函数full_item_new的情况下，很难回答这个问题，但是只要看看参数，就可以提供sc，这显然是您的spark上下文变量。因此，这意味着您试图在rdd上执行转换时使用sc执行操作或转换。这根本不可能。你知道吗

full_item_new在worker上执行，但sc只能在驱动程序上使用。你知道吗