终极尾巴
tailchaser的Python项目详细描述
终极旋转和窗口友好的木尾加上更多…
- 回填旋转的文件
- 处理gz和bz2文件
- 处理多行记录
- 不会中断Windows循环日志处理程序
- 易于扩展
- 免费软件：BSD许可证
- 除了六个纯python stdlib。
安装
pip install tailchaser
thsi将在您的站点包中安装tailchaser库，并将脚本tailchase添加到脚本目录中。
用法
$ tailchase /where/my/logs/*.log
$ tailchase -h
usage: tailchase [-h] [--only-backfill] [--dont-backfill]
[--clear-checkpoint] [--read-period READ_PERIOD]
[--read-pause READ_PAUSE] [--reading-from {unix,win}]
[--temp-dir TEMP_DIR]
[--logging {DEBUG,INFO,WARN,ERROR,CRITICAL}]
file-pattern
positional arguments:
file-pattern The file pattern to tail, such as /var/log/access.*
optional arguments:
-h, --help show this help message and exit
--only-backfill dont't tail, default: False
--dont-backfill basically only tail, default: False
--clear-checkpoint start form the begining, default: False
--read-period READ_PERIOD
how long you read before you pause. If zero you don't
pause, default: 1
--read-pause READ_PAUSE
how long you pause between reads, default: 0
--reading-from PLATFORM
sets how long you read and then pause can be one of {unix,win}, default: win
--temp-dir TEMP_DIR on back fill files are copied to a temp directory.Use
this to set this directory, default: None
--logging LEVEL
logging level it can be one of DEBUG,INFO,WARN,ERROR,CRITICAL, default: ERROR
以最简单的形式
在最简单的形式下，tailchase的工作方式类似于cat和tail-f的组合，只是它将从最旧的旋转文件开始。如果你有
/var/log/opendirectoryd.log /var/log/opendirectoryd.log.0 /var/log/opendirectoryd.log.1 /var/log/opendirectoryd.log.2 /var/log/opendirectoryd.log.3 /var/log/opendirectoryd.log.4
它首先输出/var/log/opendirectoryd.log.4的文本，然后输出/var/log/opendirectoryd.log.3等，直到它到达最新的文件，然后当它到达/var/log/opendirectoryd.log的末尾时，恢复为“tail-f”行为。
所以如果你要做
tailchase 'tests/logs/opendirectoryd.*'
“注意：在bash中，需要引用通配符，否则通配符将扩展到scripts argv数组中。“
您将得到类似于
2015-12-24 15:39:56.733754 EST - AID: 0x0000000000000000 - Registered node with name '/Contacts' 2015-12-24 15:39:56.733933 EST - AID: 0x0000000000000000 - Registered node with name '/LDAPv3' as hidden 2015-12-24 15:39:56.736154 EST - AID: 0x0000000000000000 - Registered node with name '/Local' as hidden 2015-12-24 15:39:56.736868 EST - AID: 0x0000000000000000 - Registered node with name '/NIS' as hidden 2015-12-24 15:39:56.737134 EST - AID: 0x0000000000000000 - Discovered configuration for node name '/Search' at path ' 2015-12-24 15:39:56.737151 EST - AID: 0x0000000000000000 - Registered node with name '/Search' 2015-12-24 15:39:56.738794 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules 2015-12-24 15:39:56.740509 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules/
使用追尾器编码
在项目中使用tailchaser库可能是最好的例子。
示例1-追尾服务。
#
# Example 1 - Tail to Elastic
#
import requests
import tailchaser
class TailToElastic(tailchaser.Tailer):
def handoff(self, file_tailed, checkpoint, record):
""" Expect a record like:
20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties]
20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties]
20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties]
"""
date, time, level, source, _, message = record.split(5)
result = requests.json("http://someelacticserver.com:9200/myindex/log", json={
'timestamp': '{}T{}'.format(date, time)
'level': level,
'source': source,
'message': message
})
return result.status_code == requests.codes.ok
示例2-追尾至卡夫卡
#
# Example 2 - Tail to Kafka - shows how to add your own arguments and then send messages to kafka.
#
from kafka import KafkaProducer
rom tailchaser.tailer import Tailer
class TailToKafka(Tailer):
def add_arguments(cls, parser=None):
parser = super(TailToKafka, cls).add_arguments(parser)
HOSTS = 'localhost:1234'
TOPIC = b'log'
def startup(self):
self.kafka_producer = KafkaProducer(bootstrap_servers=self.HOSTS,value_serializer=msgpack.dumps)
def handoff(self, file_tailed, checkpoint, record):
""" Expect a record like:
20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties]
20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties]
20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties]
"""
self.kafka_producer.send(self.TOPIC, record).get(timeout=10)
return True
示例3-保存和加载来自zookeeper的最后偏移量
#
# Example 3 - Saving and Loading last Offset from Zookeeper
#
import platform
import pickle
import six
from kazoo.client import KazooClient
from tailchaser.tailer import Tailer
class TailToElastic(Tailer):
def start(self):
self.zk = KazooClient(hosts=self.config.zookeeper_host)
self.zk.start()
def stop():
self.zk.stop()
def load_checkpoint(self):
zk_path = self.config.checkpoint_filename
try:
checkpoint = pickle.loads(self.zk.get(zk_path))
log.debug('loaded: %s %s', zk_path, checkpoint)
return checkpoint
except (IOError, EOFError):
log.debug('failed to load: %s', zk_path)
return self.INIT_CHECKPOINT
def save_checkpoint(self, checkpoint):
log.debug('dumping %s %s', self.config.checkpoint_filename, checkpoint)
return self.zk.set(self.config.checkpoint_filename, six.b(pickle.dumps(checkpoint)))
@staticmethod
def make_checkpoint_filename(source_pattern, path=None):
zk_path = '/'.join(['tailchase', platform.node(), slugify(source_pattern)]
self.zk.ensure_path(zk_path)
return zk_path
开发
要运行所有测试，请运行：
tox
注意，要合并来自所有tox环境的覆盖率数据，请运行：
| Windows | ^{pr 10}$ |
|---|---|
| Other | ^{pr 11}$ |
更改日志
0.2.7（2016-07-24）
- 添加–过滤Re以过滤出15行
0.2.1（2016-04-30）
- 修正了12，其中切换正在获取tmp文件名…
- 移交后删除的临时文件14
0.2.0（2016-04-30）
- 添加了来自ILABS代码库的管道代码。
- 添加了远程存储检查点的示例8
- 增加了休息配置9
- 将init_checkpoint class属性添加到tailer 11
0.1.3（2016-04-12）
- 增加了断尾测试和从中驱动的修复程序-6
- 修正了应用程序描述！-#7
0.1.2（2016-04-05）
- 添加多行记录支持-3
- 添加–多行记录的记录重新开始选项。-#4
- 将argparser定义从cli移动到tailer类。-#5
0.1.0（2016-02-21）
- pypi上的第一个版本。
欢迎加入QQ群-->： 979659372
