气流计划程序为DAG抛出错误，计划间隔为None

Question

我对气流有意见。有一个客户生成器脚本，它接受来自yaml文件的输入并加载DAG。当所有DAG yaml文件的计划间隔都为非“无”时，它可以正常工作。有许多DAG的计划_间隔为零，很少有@one

YAML文件示例为-

cluster:
  nodes: 10
  subnet: "subnet-A"
  instance: "m4.2xlarge"
  configbucket: "bucketabc"
  jar: "s3://xxxxx.jar"
  conf: "app.conf"

schedule:
  state: "unpause"
  concurrency: 10
  startdate: "2050-08-05 00:00"
  cron: "None"

生成器脚本包含以下内容-

            if "schedule" in project_settings:
                schedule_settings = project_settings["schedule"]
                concurrency = schedule_settings["concurrency"]
                cron =  schedule_settings["cron"]
                startdate =  datetime.strptime(schedule_settings["startdate"], "%Y-%m-%d %H:%M")

            #print "my projectname is: " + project

            dag = DAG(
                dag_id = project,
                default_args=args,
                user_defined_macros=user_macros,
                schedule_interval=cron,
                concurrency=concurrency,
                start_date=startdate
            )

当存在多个DAG且schedule_interval=None时出现的错误

INFO - [2020-04-08 12:30:45,529] {dagbag.py:302} ERROR - Failed to bag_dag: /home/deploy/airflow/dags/genertor.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 296, in process_file
    croniter(dag._schedule_interval)
  File "/usr/local/lib/python3.6/site-packages/croniter/croniter.py", line 91, in __init__
    self.expanded, self.nth_weekday_of_month = self.expand(expr_format)
  File "/usr/local/lib/python3.6/site-packages/croniter/croniter.py", line 468, in expand
    raise CroniterBadCronError(cls.bad_length)
croniter.croniter.CroniterBadCronError: Exactly 5 or 6 columns has to be specified for iteratorexpression.

有人面对过这个问题吗

Answer 1

气流DAGschedule_interval可以是作为string的cron表达，也可以是None（注意，不是string{}）

在您的设置中，您有：

cron: "None"

这是Python中的一个字符串。如果无法将该YAML文件更改为：

cron: None

您仍然可以在DAG本身中检查该字符串：

schedule_interval = None if cron == "None" else cron