如何从日志文件中提取xml以在python中解析

Question

我有一个日志文件，里面包含了两种类型的xml数据（请求和响应）。我需要做的是解析这个文件，提取出xml内容，并把它们放进两个数组里，分别作为字符串（第一个数组存放请求，第二个数组存放响应），这样我就可以后续处理它们了。

有没有什么办法可以用python来实现这个呢？

这是要解析的日志文件的一部分（日志内容包含）：

2014-10-31 12:27:33,600 INFO  Recharger_MTelemedia2Channel [mbpa.module.mgw.mtelemedia.mtbilling.MTSender][] Sending BILL request
2014-10-31 12:27:33,601 INFO  Recharger_MTelemedia2Channel [mbpa.module.mgw.mtelemedia.mtbilling.MTSender][] <?xml version="1.0" encoding="UTF-8"?>
<request xmlns="XXX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <transactionheader>
            <username>XXX</username>
            <password>XXX</password>
            <time>31/10/2014 12:27:33</time>
            <clientreferencenumber>123</clientreferencenumber>
            <numberrequests>3</numberrequests>
            <information>Description</information>
            <postbackurl>http://localhost/status</postbackurl>
    </transactionheader>
    <transactiondetails>
            <items>
                    <item id="1" client="XXX1" keyword="test"/>
                    <item id="2" client="XXX2" keyword="test"/>
                    <item id="3" client="XXX3" keyword="test"/>
            </items>
    </transactiondetails>
</request>
2014-10-31 12:27:34,487 INFO  Recharger_MTelemedia2Channel [mbpa.module.mgw.mtelemedia.mtbilling.MTSender][] Response code 200 for bill request 
2014-10-31 12:27:34,489 INFO  Recharger_MTelemedia2Channel [mbpa.module.mgw.mtelemedia.mtbilling.MTSender][] <?xml version="1.0" encoding="UTF-8"?>

<response xmlns="XXX" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <serverreferencenumber>XXX123XXX</serverreferencenumber>
    <clientreferencenumber>123</clientreferencenumber>
    <information>Queued for Processing</information>
    <status>OK</status>
</response>

非常感谢你的回复！

祝好，
罗伯特

Answer 1

正如@Paco和@Lord_Gestalter所建议的，你可以使用 xml.etree 模块，并从你的文件中替换掉那些不是XML的部分，像这样：

# I use re to substitute non-XML elements
import re
# then use xml module as a parser
import xml.etree.ElementTree as ET

# read your file and store in string 's'
with open('yourfilehere','r') as f:
    s = f.read()
# then remove non-XML element with re
# I also remove <?xml ...?> part as your file consists of multiple xml logs
s = re.sub(r'<\?xml.*?>', '', ''.join(re.findall(r'<.*>', s)))
# wrap your s with a root element
s = '<root>'+s+'</root>'
# parse s with ElementTree
tree = ET.fromstring(s)

tree
<Element 'root' at 0x7f2ab877e190>

---

如果你不在乎XML解析器，只想要 'request' 和 'response' 的字符串，可以使用 re.search。

with open('yourfilehere','r') as f:
    s = f.read()    
# put the string of both request and response into 'req' and 'res'
# or you need to construct a better re.search if you have multiple requests, responses
req = [re.search(r'<request.*\/request>', s).group()]
res = [re.search(r'<response.*\/response>', s).group()]

req
['<request xmlns="XXX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><transactionheader><username>XXX</username><password>XXX</password><time>31/10/2014 12:27:33</time><clientreferencenumber>123</clientreferencenumber><numberrequests>3</numberrequests><information>Description</information><postbackurl>http://localhost/status</postbackurl></transactionheader><transactiondetails><items><item id="1" client="XXX1" keyword="test"/><item id="2" client="XXX2" keyword="test"/><item id="3" client="XXX3" keyword="test"/></items></transactiondetails></request>']

res
['<response xmlns="XXX" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><serverreferencenumber>XXX123XXX</serverreferencenumber><clientreferencenumber>123</clientreferencenumber><information>Queued for Processing</information><status>OK</status></response>']