用于将数据加载到PostgreSQL的singer.io目标-管道兼容
pipelinewise-target-postgres的Python项目详细描述
管道式目标postgres
Singer在Singer spec之后将数据加载到雪花中的目标。
这是一个PipelineWise兼容的目标连接器。
如何使用
运行此目标的建议方法是从PipelineWise中使用它。当从pipelinewise运行它时，您不需要用json文件配置这个tap，而且大多数事情都是自动化的。请查看Target Postgres上的相关文档
如果要独立运行此Singer Target，请进一步阅读。
安装
首先，确保python 3安装在您的系统上，或者遵循以下步骤 Mac或的安装说明 Ubuntu。
建议使用virtualenv:
python3 -m venv venv pip install pipelinewise-target-postgres
或
python3 -m venv venv . venv/bin/activate pip install --upgrade pip pip install .
运行
像其他任何一个目标一样，遵循歌手的具体要求：
some-singer-tap | target-postgres --config [config.json]
它从stdin读取传入消息，并使用config.json中的属性将数据上载到postgres。
注意：要避免版本冲突，请在单独的虚拟环境中运行tap和targets。
配置设置
运行目标连接器需要一个config.json文件。最小设置的示例：
{"host":"localhost","port":5432,"user":"my_user","password":"secret","dbname":"my_db_name","default_target_schema":"my_target_schema"}
config.json：
| Property | Type | Required? | Description |
|---|---|---|---|
| host | String | Yes | PostgreSQL host |
| port | Integer | Yes | PostgreSQL port |
| user | String | Yes | PostgreSQL user |
| password | String | Yes | PostgreSQL password |
| dbname | String | Yes | PostgreSQL database name |
| batch_size | Integer | (Default: 100000) Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Snowflake. | |
| default_target_schema | String | Name of the schema where the tables will be created. If ^{ | |
| default_target_schema_select_permission | String | Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created | |
| schema_mapping | Object | Useful if you want to load multiple streams from one tap to multiple Snowflake schemas. If the tap sends the ^{ Note: This is an experimental feature and recommended to use via PipelineWise YAML files that will generate the object mapping in the right JSON format. For further info check a PipelineWise YAML Example. | |
| add_metadata_columns | Boolean | (Default: False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix ^{ | |
| hard_delete | Boolean | (Default: False) When ^{ |
运行测试：
- 定义需要运行测试的环境变量
export TARGET_POSTGRES_HOST=<postgres-host>
export TARGET_POSTGRES_PORT=<postgres-port>
export TARGET_POSTGRES_USER=<postgres-password>
export TARGET_POSTGRES_PASSWORD=<postgres-password>
export TARGET_POSTGRES_DBNAME=<postgres-dbname>
export TARGET_POSTGRES_SCHEMA=<postgres-schema>
- 在虚拟环境中安装python依赖项并运行nose单元和集成测试
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
pip install nose
- 运行单元测试：
nosetests --where=tests/unit
- 要运行集成测试：
nosetests --where=tests/integration
运行pylint:
- 安装python依赖项并运行python linter
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
pip install pylint
pylint target_postgres -d C,W,unexpected-keyword-arg,duplicate-code
许可证
apache许可证2.0版
请参见LICENSE以查看全文。
欢迎加入QQ群-->： 979659372
