为什么在Windows上保存到文件时python多处理日志会丢失一些日志?

2024-05-16 14:31:00 发布

您现在位置:Python中文网/ 问答频道 /正文

我尝试在使用的多处理时创建简单的日志记录多处理管道。其思想是通过每个正在运行的进程将记录的消息放入队列中,并仅通过父进程将这些消息从队列中取出,然后保存到文件中。这样,只有父进程才能访问该文件。虽然这在Ubuntu上正常工作,但在windows10上保存到文件时会丢失一些日志。将日志打印到控制台工作正常,没有日志丢失。我使用多处理日志库。Anaconda上的Python版本3.6.8。你知道吗

我在追踪问题的源头文件处理程序.emit()似乎针对每个日志执行。问题似乎发生在多处理库中。你知道吗

import logging
import multiprocessing
from multiprocessing import freeze_support

import attr

import multiprocessing as mp
import multiprocessing_logging

FORMAT = '[%(asctime)s] [%(levelname)s] ' \
         '[%(process)s] [%(process_)s] --- %(message)s'
formatter = logging.Formatter(FORMAT)

fh = logging.FileHandler("test.log")
fh.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(fh)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
logger.addHandler(sh)
logger.setLevel(logging.DEBUG)

multiprocessing_logging.install_mp_handler(logger)


@attr.s
class Message:
    id = attr.ib()
    value = attr.ib(repr=False)


def test_func1(name, in_stream, out_stream):
    for a in range(1, 101):
        m = Message(a, a)
        logging.info(m, extra={'process_': name})
        out_stream.send(m)


def test_func2(name, in_stream, out_stream):
    while 1:
        if in_stream.poll(0.1):
            m: Message = in_stream.recv()
            m.value = m.value**2
            logging.warning(m, extra={'process_': name})
            out_stream.send(m)


def test_func3(name, in_stream, out_stream):
    while 1:
        if in_stream.poll(0.1):
            m: Message = in_stream.recv()
            m.value = m.value*2
            logging.error(m, extra={'process_': name})


services = []
@attr.s
class Service:
    name = attr.ib()
    in_stream = attr.ib()
    out_stream = attr.ib()
    func = attr.ib()

    def run(self):
        worker = mp.Process(
            target=self.func,
            args=(
                self.name,
                self.in_stream,
                self.out_stream,
            ),
        )
        worker.start()


if __name__ == '__main__':
    freeze_support()

    prev_reader = None
    next_reader, next_writer = mp.Pipe()
    s1 = Service('f1', prev_reader, next_writer, test_func1)
    services.append(s1)
    prev_reader = next_reader
    next_reader, next_writer = mp.Pipe()
    s2 = Service('f22', prev_reader, next_writer, test_func2)
    services.append(s2)
    prev_reader = next_reader
    next_reader, next_writer = mp.Pipe()
    s3 = Service('f333', prev_reader, next_writer, test_func3)
    services.append(s3)

    for s in services:
        s.run()

部分日志保存到文件中:

[2019-08-12 10:09:30,317] [ERROR] [10748] [f333] --- Message(id=31)
[2019-08-12 10:09:30,320] [WARNING] [14060] [f22] --- Message(id=40)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=53)
2)
[2019-08-12 10:09:30,320] [WARNING] [14060] [f22] --- Message(id=41)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=54)
[2019-08-12 10:09:30,320] [WARNING] [14060] [f22] --- Message(id=42)
[2019-08-12 10:09:30,318] [ERROR] [10748] [f333] --- Message(id=33)

控制台输出中日志的相同部分:

[2019-08-12 10:09:30,317] [ERROR] [10748] [f333] --- Message(id=31)
[2019-08-12 10:09:30,318] [WARNING] [14060] [f22] --- Message(id=33)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=52)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=53)
[2019-08-12 10:09:30,317] [ERROR] [10748] [f333] --- Message(id=32)
[2019-08-12 10:09:30,318] [WARNING] [14060] [f22] --- Message(id=34)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=54)
[2019-08-12 10:09:30,318] [ERROR] [10748] [f333] --- Message(id=33)

请告诉我这里有什么问题,是否可以在Windows上使用这种方法,或者我是否应该改变它。你知道吗


Tags: nameintestimportidmessagestreamlogging