使用Python在MongoDB中查询变量键的数据显示结果

Question

我现在在做一个使用SaltStack的项目，里面用到了MongoDB来存储结果，我想查询这些结果以便做报告。不过，返回的数据写入数据库的方式看起来是这样的：

db.hostname21.find({"20140421035007474763" : {$exists : 1}}, {_id: 0}).pretty()

{
     "fun" : "state.sls",
     "20140421035007474763" : {
            "file_|-Get-Logins-Stats_|-/scripts/server_logins_stats_|-managed" : {
                    "comment" : "File /scripts/server_logins_stats is in the correct state",
                    "__run_num__" : 2,
                    "changes" : {

                    },
                    "name" : "/scripts/server_logins_stats",
                    "result" : true
            },
            "service_|-Logger_|-rsyslog_|-running" : {
                    "comment" : "Service rsyslog is already enabled, and is in the desired state",
                    "__run_num__" : 1,
                    "changes" : {

                    },
                    "name" : "rsyslog",
                    "result" : true
            },
            "cmd_|-Run_Script_|-/scripts/server_logins_stats_|-run" : {
                    "comment" : "Command \"/scripts/server_logins_stats\" run",
                    "__run_num__" : 4,
                    "changes" : {
                            "pid" : 20899,
                            "retcode" : 0,
                            "stderr" : "",
                            "stdout" : "0"
                    },
                    "name" : "/scripts/server_logins_stats",
                    "result" : true
            },
            "cron_|-Schedule_Run_|-/scripts/server_logins_stats_|-present" : {
                    "comment" : "Cron /scripts/server_logins_stats already present",
                    "__run_num__" : 3,
                    "changes" : {

                    },
                    "name" : "/scripts/server_logins_stats",
                    "result" : true
            },
            "pkg_|-nc_|-nc_|-installed" : {
                    "comment" : "Package nc is already installed",
                    "__run_num__" : 0,
                    "changes" : {

                    },
                    "name" : "nc",
                    "result" : true
            }
    }
}

从这里可以看到，数据的键（也就是标识符）总是在变化，而不是有一个固定的键来标识每个脚本。不过，我发现失败的记录格式比较一致，只是没有任何键来表明这是一个失败的记录，只有一串字符串数组：

       "20140421041507478163" : [
            "Pillar failed to render with the following messages:",
            "Specified SLS 'globals' in environment 'Production' is not available on the salt master"
       ],
       "fun" : "state.sls"

所以我最终想做的是能够针对每个失败的记录进行报告，识别出作业、主机和失败的性质，以及一段时间内成功和失败的总数。如果你注意到，每个主机（也叫从机）都有自己的数据集合。因此，我写了一个Python脚本，可以遍历这些集合，以确定作业是否在该主机上实际执行过：

import datetime
import pymongo


#hosts = ["mongoDBStaging", "mongoDBUAT"]
hosts = ["mongodbuat"]

for host in hosts:
    conn = pymongo.Connection(host)
    mdb = conn['salt']
    collections = set(mdb.collection_names())


    hosts = []
    jids = []

    # for every collection, identify whether it is a host or a job
    for c in collections:   
        # if the collection is a host add it to the host array
        if not (len(c) == 20 and int(c)):
            #print "{0} is a host".format(c)
            hosts.append(c)
        # other  wise add it to the job array
        else:
            #print "{0} is a jid".format(c)
            jids.append(c)


    for h in hosts:
        # for every job in a host connect to that collection 
        # and search for the job id to see if it exists 
        # and what its return was so we can report on that info
        for j in jids:
            coll = conn['salt'][h]
            #print "%s collection, %s jid" % (coll, j)
            for doc in coll.find({j: {'$exists': True}, "fun": "state.sls"}, {"_id": 0}):
                print "{0}".format(coll)
                print "{0} is a doc".format(doc)

但我在查询结果时遇到困难，无法确定作业是否成功。我想从返回的文档中提取元素，以便查看每个文档的结果。

如果有人有建议，能让我在键不断变化的情况下，稳定地查询到结果，那将非常有帮助。

Answer 1

如果有人在想这个问题，我自己解决了。使用了下面的Python代码。虽然这不是性能最好的方法，也没有充分利用MongoDB，但确实能工作。我可能会建议更新盐返回器，让它更好地使用MongoDB，因为在命令行中查询的功能比较有限。

import datetime
import pymongo
import json
import re

hosts = ["mongodbuat"]

# initialize failures and successes
failures = 0
successes = 0

for host in hosts:
    conn = pymongo.Connection(host)
    mdb = conn['salt']
    collections = set(mdb.collection_names())

    hosts = []
    jids = []

    # for every collection, identify whether it is a host or a job
    for c in collections:   
        # if the collection is a host add it to the host array
        if not (len(c) == 20 and int(c)):
            hosts.append(c)
        # otherwise add it to the job array
        else:
            jids.append(c)


    for h in hosts:
        # for every job in a host connect to that collection 
        # and search for the job id to see if it exists 
        # and what its return was so we can report on that info
        for j in jids:
            coll = conn['salt'][h]

            # search for the json documents returned from mongodb
            # if the jobid exists in that host
            for doc in coll.find({j: {'$exists': True}, "fun": "state.sls"}, {"_id": 0}):

                # if the jobid exists find the host name in a readable format
                c = str(coll)
                thishost = ''
                match = re.search('(\w+)\.spottrading\.com',c)
                if match:
                    thishost = match.group(1)

                # search the document returned in the form of a dictionary for
                # the states you want to report on
                for jobid, states in doc.iteritems():
                    if re.search('\d+', jobid):
                        print '\njob id =', jobid
                        if isinstance(states, list):
                            print states
                            failures += 1
                        elif isinstance(states, dict):
                            for job, data in states.iteritems():
                                print '\tjob: {0}, result: {1}'.format(job, data[u'result'])
                            successes += 1

print "{0} successes, {1} failures".format(successes, failures)