kfx 0.1.0a7

kfx公司

kfx是一个具有名称空间kfx的python包。目前，它提供 以下子包

• 在

  kfx.lib.dsl-kubeflow管道dsl的扩展。在

  在
• 在

  kfx.lib.vis-数据模型和助手，帮助生成在kubeflow管道UI中呈现可视化所需的mlpipeline-metrics.json和{}。另请参见https://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/和https://www.kubeflow.org/docs/pipelines/sdk/output-viewer/

  在

• Documentation: https://kfx.readthedocs.io.
• Repo: https://github.com/e2fyi/kfx

NOTE this is currently alpha

There will likely to have breaking changes, and feel free to do a feature request

Known issues

• kfx.vis.vega.vega_web_app and KfpArtifact does not work well together (see example) because of CORs - the web app is hosted inside an iFrame which prevents it from accessing the ml-pipeline-ui API server.
• kfx.vis.vega.vega_web_app is only supported in the latest kubeflow pipeline UI (as inline is only supported after 0.2.5)

变更日志

请参阅CHANGELOG.md。在

快速入门

安装

pip install kfx

使用

示例：使用ContainerOpTransform配置内部k8s属性 kubeflow管道任务。在

kfx.dsl.ContainerOpTransform is a helper to modify the interal k8s properties (e.g. resources, environment variables, etc) of kubeflow pipeline tasks.

^{pr2}$

示例：使用ArtifactLocationHelper和KfpArtifact来确定 由kubeflow管道任务生成的数据工件的uri。在

kfx.dsl.ArtifactLocationHelper is a helper to modify the kubeflow pipeline task so that you can use kfx.dsl.KfpArtifact to represent the artifact generated inside the task.

importkfp.componentsimportkfp.dslimportkfx.dsl# creates the helper that has the argo configs (tells you how artifacts will be stored)# see https://github.com/argoproj/argo/blob/master/docs/workflow-controller-configmap.yamlhelper=kfx.dsl.ArtifactLocationHelper(scheme="minio",bucket="mlpipeline",key_prefix="artifacts/")@kfp.components.func_to_container_opdeftest_op(mlpipeline_ui_metadata:OutputTextFile(str),markdown_data_file:OutputTextFile(str)):"A test kubeflow pipeline task."importjsonimportkfx.dslimportkfx.visimportkfx.vis.vega# `KfpArtifact` provides the reference to data artifact created# inside this taskspec={"$schema":"https://vega.github.io/schema/vega-lite/v4.json","description":"A simple bar chart","data":{"values":[{"a":"A","b":28},{"a":"B","b":55},{"a":"C","b":43},{"a":"D","b":91},{"a":"E","b":81},{"a":"F","b":53},{"a":"G","b":19},{"a":"H","b":87},{"a":"I","b":52},]},"mark":"bar","encoding":{"x":{"field":"a","type":"ordinal"},"y":{"field":"b","type":"quantitative"},},}# write the markdown to the `markdown-data` artifactmarkdown_data_file.write("### hello world")# creates an ui metadata objectui_metadata=kfx.vis.kfp_ui_metadata(# Describes the vis to generate in the kubeflow pipeline UI.[# markdown vis from a markdown artifact.# `KfpArtifact` provides the reference to data artifact created# inside this taskkfx.vis.markdown(kfx.dsl.KfpArtifact("markdown_data_file")),# a vega web app from the vega data artifact.kfx.vis.vega.vega_web_app(spec),])# writes the ui metadata object as the `mlpipeline-ui-metadata` artifactmlpipeline_ui_metadata.write(kfx.vis.asjson(ui_metadata))# prints the uri to the markdown artifactprint(ui_metadata.outputs[0].source)@kfp.dsl.pipeline()deftest_pipeline():"A test kubeflow pipeline"op:kfp.dsl.ContainerOp=test_op()# modify kfp operator with artifact location metadata through env varsop.apply(helper.set_envs())

示例：使用pydantic数据模型生成mlpipeline-metrics.json和 mlpipeline-ui-metadata.json。在

（另见https://www.kubeflow.org/docs/pipelines/sdk/output-viewer/和 https://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/）。在

kfx.vis has helper functions (with corresponding hints) to describe and create mlpipeline-metrics.json and mlpipeline-ui-metadata.json files (required by kubeflow pipeline UI to render any metrics or visualizations).

importfunctoolsimportkfp.components# install kfxkfx_component=functools.partial(kfp.components.func_to_container_op,packages_to_install=["kfx"])@kfx_componentdefsome_op(# mlpipeline_metrics is a path - i.e. open(mlpipeline_metrics, "w")mlpipeline_metrics:kfp.components.OutputPath(str),# mlpipeline_ui_metadata is a FileLike obj - i.e. mlpipeline_ui_metadata.write("something")mlpipeline_ui_metadata:kfp.components.OutputTextFile(str),):"kfp operator that provides metrics and metadata for visualizations."# import inside kfp taskimportkfx.vis# output metrics to mlpipeline_metrics pathkfx.vis.kfp_metrics([# render as percentkfx.vis.kfp_metric("recall-score",0.9,percent=true),# override metric format with custom valuekfx.vis.kfp_metric(name="percision-score",value=0.8,metric_format="PERCENTAGE"),# render raw scorekfx.vis.kfp_metric("raw-score",123.45),]).write_to(mlpipeline_metrics)# output visualization metadata to mlpipeline_ui_metadata objkfx.vis.kfp_ui_metadata([# creates a confusion matrix viskfx.vis.confusion_matrix(source="gs://your_project/your_bucket/your_cm_file",labels=["True","False"],),# creates a markdown with inline sourcekfx.vis.markdown("# Inline Markdown: [A link](https://www.kubeflow.org/)",storage="inline",),# creates a markdown with a remote sourcekfx.vis.markdown("gs://your_project/your_bucket/your_markdown_file",),# creates a ROC curve with a remote sourcekfx.vis.roc("gs://your_project/your_bucket/your_roc_file",),# creates a Table with a remote sourcekfx.vis.table("gs://your_project/your_bucket/your_csv_file",header=["col1","col2"],),# creates a tensorboard viewerkfx.vis.tensorboard("gs://your_project/your_bucket/logs/*",),# creates a custom web app from a remote html filekfx.vis.web_app("gs://your_project/your_bucket/your_html_file",),# creates a Vega-Lite vis as a web appkfx.vis.vega_web_app(spec={"$schema":"https://vega.github.io/schema/vega-lite/v4.json","description":"A simple bar chart with embedded data.","data":{"values":[{"a":"A","b":28},{"a":"B","b":55},{"a":"C","b":43},{"a":"D","b":91},{"a":"E","b":81},{"a":"F","b":53},{"a":"G","b":19},{"a":"H","b":87},{"a":"I","b":52}]},"mark":"bar","encoding":{"x":{"field":"a","type":"ordinal"},"y":{"field":"b","type":"quantitative"}}})]).write_to(mlpipeline_ui_metadata)

开发人员指南

本项目使用：

• isort：管理进口订单
• pylint：管理通用编码最佳实践
• flake8：管理代码复杂性和编码最佳实践
• 黑色：管理格式和样式
• pydocstyle：管理docstr样式/格式
• pytest/coverage：管理单元测试和代码覆盖率
• 班迪特：找出常见的安全问题
• python-dev-3版本（python-dev-manage）
• pipenv：管理dev-env:python包

单元测试的惯例是以_test作为后缀，并与实际的 python模块-即<module_name>_test.py。在

软件包的版本是从version.txt-即，请更新 适当的语义版本（主要->突破性更改、次要->新功能、修补程序->错误修复、后缀->发布前/发布后）。在

Makefile:

# autoformat codes with docformatter, isort, and black
make format

# check style, formats, and code complexity
make check

# check style, formats, code complexity, and run unit tests
make test# test everything including building the package and check the sdist
make test-all

# run unit test only
make test-only

# generate and update the requirements.txt and requirements-dev.txt
make requirements

# generate the docs with sphinx and autoapi extension
make docs

# generate distributions
make dists

# publish to pypi with twine (twine must be configured)
make publish

变更日志

v0.1.0.a7

New features

• kfx.dsl.ContainerOpTransform helps modify internal k8s properties of a kubeflow pipeline task.

标签：

• 项目

欢迎加入QQ群-->： 979659372

推荐PyPI第三方库

• epages-client

  epages rest api的python客户端。

• ztfy.sendit

  用于共享文件的ztfy应用程序

• Impostor

  员工可以作为不同的用户登录。

• letsencrypt-proxmox

  用于加密客户端的proxmox ve插件

• split-quer

  用于分解查询以进行缓存、处理等的工具。

• bioma

  使用图形数据库和web语义的生物相关数据管理。

• static-frame

  带标记轴的一维和二维计算的不可变结构

• ice_latency_metrics_collector

  零延迟度量收集器

• astmonke

  astmonkey是一组使用python ast的工具。

• PascalsTriangle

  计算给定深度的帕斯卡三角形

• toastedmarshmallow-enum

  Toasted Marshmallow Enum包使基于Toasted Marshmallow提供的超快速序列化来转储和加载枚举值成为可能。

• mp3-tagger

  ID3数据编辑器

• pycast

  预测与平滑库

• Lunas

  一种数据处理流水线和迭代器，具有最小的机器学习依赖性。

• vcomp

  视频压缩程序（VCOMP）