此工具允许您在日志数据平台上获取所有冷存储存档的本地副本。
ldp-archive-mirror的Python项目详细描述
此工具允许您在日志数据平台上获取所有冷存储存档的本地副本:
[2018-12-14 17:20:00,200 INFO LDP Mirror] Create local DB if not exists [2018-12-14 17:20:00,204 INFO LDP Mirror] Looking for new archives of stream e9397556-31d1-4d4d-b2bd-e5367b522cc8 [2018-12-14 17:20:05,190 INFO LDP Mirror] Service ldp-jb-52560 found for stream e9397556-31d1-4d4d-b2bd-e5367b522cc8 [2018-12-14 17:20:05,579 INFO LDP Mirror] Archive 9fb75957-2cde-435e-bdd9-6dfd33663f2c added to cache [2018-12-14 17:20:05,850 INFO LDP Mirror] Archive 11d8630a-7b38-4fa4-9d7c-dfd17b0b00f6 added to cache ... [2018-12-14 17:20:08,392 INFO LDP Mirror] Directory mirror/e9397556-31d1-4d4d-b2bd-e5367b522cc8 created [2018-12-14 17:20:08,535 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst missing, download scheduled [2018-12-14 17:20:08,635 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst missing, download scheduled ... [2018-12-14 17:20:09,535 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst after 596 seconds [2018-12-14 17:20:09,745 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst after 598 seconds ... [2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst downloaded [2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst [2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst downloaded [2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst
快速启动
首先,使用pip:
pip install -U ldp-archive-mirror
然后可以使用二进制ldp-mirror
:
usage: ldp-mirror [-h] [--app-key KEY] [--app-secret SECRET] [--consumer-key KEY] [--ovh-region REGION] [--db DIR] [--mirror DIR] [--ldp-host HOST] [--ldp-token TOKEN] STREAM_ID [STREAM_ID ...] LDP archive Mirror CLI - 0.1.0 positional arguments: STREAM_ID LDP Stream UUIDs optional arguments: -h, --help show this help message and exit --app-key KEY OVH application key (default: None) --app-secret SECRET OVH application secret (default: None) --consumer-key KEY OVH consumer key (default: None) --ovh-region REGION OVH region (default: ovh-eu) --db DIR Where to place the local sqlite database (default: db) --mirror DIR Where to place your archives (default: mirror) --ldp-host HOST If set, push logs of the current application to given LDP hostname --ldp-token TOKEN If set, push logs of the current application to associated LDP stream token
设置
一。创建应用程序
要与api交互,应用程序需要使用
application_key
和一个application_secret
。为了得到他们,你需要
注册你的申请。根据您计划使用的API,请访问:
- OVH Europe
- OVH US
- OVH North-America
- So you Start Europe
- So you Start North America
- Kimsufi Europe
- Kimsufi North America
在限制步骤中,我们邀请您设置以下访问规则:
curl -XPOST -H"X-Ovh-Application: OVH_API_AK" -H "Content-type: application/json" \ https://eu.api.ovh.com/1.0/auth/credential -d '{ "accessRules": [ { "method": "GET", "path": "/dbaas/logs" }, { "method": "GET", "path": "/dbaas/logs/*/output/graylog/stream" }, { "method": "GET", "path": "/dbaas/logs/*/output/graylog/stream/*/archive*" }, { "method": "POST", "path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/url" } ], "redirection":"https://www.mywebsite.com/" }'
创建后,您将获得一个application key(ovh_api_ak)和一个application 秘密(ovh_api_as)。
2.环境变量
可以使用环境设置默认的cli值:
Cli parameter | Name | About |
---|---|---|
–app-key | OVH_API_AK | OVH application key |
–app-secret | OVH_API_AS | OVH application secret |
–consumer-key | OVH_API_CK | OVH customer key |
–ovh-region | OVH_API_REGION | OVH api location. Default: ovh-eu |
–db | DB_DIRECTORY | Where to place the local sqlite database. Default: db |
–mirror | MIRROR_DIRECTORY | Where to place your archives. Default: mirror |
–ldp-host | LDP_HOST | If set, push logs of the current application to given LDP hostname |
–ldp-token | LDP_TOKEN | If set, push logs of the current application to associated LDP stream token |
三。启动
设置完所有必需的环境变量后,启动如下过程:
$ ldp-mirror STREAM_ID [STREAM_ID ...]
这将:
- 查找与给定stream\u id关联的ldp服务(s)
- 使用API上找到的所有存档填充本地缓存
- 为每个用户请求一个临时下载URL
- 到达解封时间后下载文件
- 每小时询问API是否有新的存档可用
从源创建Docker图像
由于这个应用程序应该无限期地保持活动状态,从Docker守护进程启动它看起来很明显。
要从源构建图像,请使用给定的Makefile
:
$ git clone https://github.com/ovh/ldp-archive-mirror $ cd ldp-archive-mirror $ make build-docker
运行它:
$ docker run -v /my_backup/mirror/:/data/mirror -v /my_backup/db:/data/db \ -e OVH_API_AK=MY_OVH_AK -e OVH_API_AS=MY_OVH_AS -e OVH_API_CK=MY_OVH_CK \ -t MY_LDP_STREAM_ID_1 MY_LDP_STREAM_ID_2
要求
- python>;=3.3