用于jupyter笔记本的azure数据湖管理魔法
adlmagics的Python项目详细描述
azure数据服务笔记本(alpha)
azure数据服务笔记本是一组用于使用Jupyter Notebook使用azure数据服务(例如azure data lake、hdisight、cosmosdb、azure sql和azure data warehouse等)的扩展。
警告:此sdk/cli目前处于非常早期的开发阶段。它可以而且将以向后不兼容的方式改变。
最新版本:0.0.1a0
功能
azure数据服务笔记本目前提供了一组Jupyter Magic Functions供用户访问azure数据湖。下表显示了可用的魔法。请单击命令名以查看语法引用。
Command | Function |
---|---|
%adl login | Line magic* to log in to Azure Data Lake. |
%adl listaccounts | Line magic to list the Azure Data Lake Analytic accounts for current user. |
%adl listjobs | Line magic to list the Azure Data Lake jobs for a given account. |
%%adl submitjob | Cell magic* to submit a USQL job to Azure Data Lake cluster. |
%adl viewjob | Line magic to view detailed job info. |
%adl liststoreaccounts | Line magic to list the Azure Data Lake Store accounts. |
%adl liststorefolders | Line magic to list the folders under a given directory. |
%adl liststorefiles | Line magic to list the files under a given directory. |
%adl sample | Line magic to sample a given file, return results as Pandas DataFrame. |
%adl logout | Line magic to log out. |
*有关Line magic
和Cell magics
的详细定义,请检查Magic Functions 。
安装
- 下载并安装python 3.6+
- 安装jupyter:
pip install jupyter
- 安装Adlmagic扩展:
pip install --no-cache-dir adlmagics
示例
- adlmagics_demo.ipynb,用于azure数据湖作业控制和数据探索的
adlmgics
函数的演示文件。 - usql_samples.ipynb,常见u-sql场景的示例代码,例如在脚本中查询tsv文件、创建数据库、填充表、查询表和创建行集。
反馈
- 您可以在本回购协议中直接提交bug report或feature request。我们的团队将积极地解决问题。
参考
%adl login
Line magic to login to Azure Data Lake service.
^{pr 1}$Input Parameters
^{tb 2}$%adl listaccounts
Line magic to enumerate the Azure Data Lake Analytic accounts for current user. The account list will be returned as Pandas DataFrame, you can call Pandas funtions directly afterward.
^{pr 2}$Input Parameters
^{tb 3}$%adl listjobs
Line magic to enumerate the Azure Data Lake jobs for a given account. The job list will be returned as Pandas DataFrame, you can call Pandas funtions directly afterward.
^{pr 3}$Input Parameters
^{tb 4}$%%adl submitjob
Cell magic to submit a U-SQL job to Azure Data Lake cluster.
^{pr 4}$Input Parameters
^{tb 5}$%adl viewjob
Line magic to view detailed job info.
^{pr 5}$Input Parameters
^{tb 6}$%adl liststoreacconts
Line magic to list the Azure Data Lake Store accounts.
^{pr 6}$%adl liststorefolders
Line magic to list the folders under a given directory.
^{pr 7}$Input Parameters
^{tb 7}$%adl liststorefiles
Line magic to list the files under a given directory.
^{pr 8}$Input Parameters
^{tb 7}$%adl sample
Line magic to sample a given file, return results as Pandas DataFrame.
^{pr 9}$Input Parameters
^{tb 9}$%adl logout
Line magic to log out.
^{pr 10}$Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com。
当您提交拉取请求时,cla bot将自动确定您是否需要提供 a cla并适当地装饰pr(例如,标签、注释)。只需按照说明操作 由机器人提供。您只需要使用我们的CLA在所有回购协议中执行一次。
这个项目采用了Microsoft Open Source Code of Conduct。 有关详细信息,请参见Code of Conduct FAQ或 如有任何其他问题或意见,请与opencode@microsoft.com联系。