strpipe 0.5.0

安装

pip install strpipe

用法

importstrpipeasspp=sp.Pipe()p.add_step_by_op_name('ZhCharTokenizer')p.add_step_by_op_name('AddSosEos')p.add_checkpoint()p.add_step_by_op_name('Pad')p.add_step_by_op_name('TokenToIndex')data=['你好啊','早安','你早上好',]p.fit(data)result,tx_info,intermediates=p.transform(data)# convention: tx => tranformback_data=p.inverse_transform(result,tx_info)

# Save itp.save_json('/path/of/pipe')# Load itp=sp.Pipe.restore_from_json('/path/of/pipe')result,meta=p.transform(['你好'])

测试

$ make test

文档

$ make docs

Docs will be built in the `docs/build/html` folder. (Note: this also reinstalls the package because we
need Cython code to be rebuilt.)

扩展操作

1. 使用BaseOp

   扩展新操作

2. 定义input_type，output_type

3. 执行操作创建

4. 实现拟合、变换、逆变换。如果操作是无状态的， fit方法不应返回任何值。

   Note: It is expected that an ops’s functionality will often be able to be decomposed into several functions. These functions should be written into (or imported from) the toolkit package for easy reuse. Ops in the ops package will, for the most part, be wrappers for functions in toolkit.

5. 编写测试

6. 注册到op_factory