data-depgraph 0.4.4

重要零件

Dataset定义单个数据产品，由 文件名，name。可以按顺序提供其他关键字参数 以促进构建过程。

数据集的祖先可以使用Dataset.parents(n)检索， 其中n是要包含的生成数。n=0表示包含 只有直系父母，而n=1包括祖父母。n=-1 包括每个祖先。Dataset.roots()返回顶级 祖先，即没有额外父母的人。

类似地，Dataset.children(n)产生数据集的后代， 如果有的话

关系用Dataset.dependson(obj)定义，其中obj 是另一个Dataset实例。可以定义关系 以编程方式构造大型依赖关系图。

用户定义的build(dataset, reason)函数（名称不重要） 获取数据集并基于其祖先和任何其他 Dataset的属性。原因是一个 指定生成步骤的动机。

depgraph.buildall()函数或Dataset.buildnext()方法 可用于获取要馈送到 build()函数。或者，build()函数可以是 用buildmanagerdecorator装饰，它创建一个函数 通过组合数据集的依赖项自动构造数据集的 按顺序（参见下面的示例）。

使用^{tt20}可以可视化复杂的依赖关系图$ 函数，返回DOT language字符串 对可视图形进行编码。

示例

声明一组类似下图的依赖项：

R0      R1      R2      R3         [raw data]
  \     /       |       |
    DA0         DA1    /
        \      /  \   /
           DB0     DB1
            \     / |  \
             \   /  |   \
              DC0  DC1  DC2        [products]

fromdepgraphimportDataset,buildmanager# Define Datasets# Use an optional keyword `tool` to provide a key instructing our build tool# how to assemble this product. Here we've used strings, but another pattern# would be to provide a callback functionR0=Dataset("data/raw0",tool="read_csv")R1=Dataset("data/raw1",tool="read_csv")R2=Dataset("data/raw2",tool="database_query")R3=Dataset("data/raw3",tool="read_hdf")DA0=Dataset("step1/da0",tool="merge_fish_counts")DA1=Dataset("step1/da1",tool="process_filter")DB0=Dataset("step2/db0",tool="join_counts")DB1=Dataset("step2/db1",tool="join_by_date")DC0=Dataset("results/dc0",tool="merge_model_obs")DC1=Dataset("results/dc1",tool="compute_uncertainty")DC2=Dataset("results/dc2",tool="make_plots")# Declare dependency relationships so that depgraph and determine the order of# the buildDA0.dependson(R0,R1)DA1.dependson(R2)DB0.dependson(DA0,DA1)DB1.dependson(DA1,R3)DC0.dependson(DB0,DB1)DC1.dependson(DB1)DC2.dependson(DB1)# Option 1:# Define a function that builds individual dependencies. The *buildmanager*# decorator transforms it into a loop that builds all dependencies above a# target@buildmanagerdefbatchbuilder(dependency,reason):# [....]returnexitcodebatchbuilder(DC1)# Option 2:# Implement the build loop manuallyfromdepgraphimportbuildalldefbuild(dependency,reason):# This may have the same logic as `batchbuilder` above, but we# will call it directly rather than wrapping it in @buildmanager# [....]returnexitcodeforstageinbuildall(DC1):# A build stage is a list of dependencies whose own dependencies are met and# that are independent, i.e. they can be built in parallelfordep,reasoninstage:# Each target is a dataset with a 'name' attribute and whatever# additional keyword arguments where defined with it.# The 'reason' is a depgraph.Reason object that codifies why a# particular target is necessary (e.g. it's out of date, it's missing# and required by a subsequent target, etc.)print("Building {0} with {1} because {2}".format(dep.name,dep.tool,reason))# Call a function or start a subprocess that will result in the# target being built and saved to a filereturn_val=build(dep,reason)# Perform logging, clean-up, or error handling operations# [....]

更改