文本差异比较JSON
在我的发布流程中，我需要比较一些应用程序使用的JSON配置数据。最开始，我只是把JSON格式化得好看一些，然后用工具（比如kdiff3或者diff）来比较它们。
不过，随着数据量的增加，kdiff3在输出中把不同的部分搞混了，让新增的内容看起来像是大规模的修改，还有一些奇怪的删除。这让我们很难搞清楚到底有什么不同。我也试过其他的比较工具（比如meld、kompare、diff，还有一些其他的），但它们都遇到了同样的问题。
尽管我尽了最大努力，但我似乎无法把JSON格式化成diff工具能理解的样子。
示例数据：
[
{
"name": "date",
"type": "date",
"nullable": true,
"state": "enabled"
},
{
"name": "owner",
"type": "string",
"nullable": false,
"state": "enabled",
}
...lots more...
]
上面的数据可能不会引起问题（问题出现在行数达到几百的时候），但这就是我在比较的内容的大致情况。
这只是一个样本；完整的对象有4到5个属性，有些属性里面还有4到5个属性。属性名称比较统一，但它们的值却各不相同。
总的来说，所有的比较工具似乎都把结束的“}”和下一个对象的结束“}”搞混了。我似乎无法让它们改掉这个习惯。
我尝试过添加空格、改变缩进，并在各自的对象前后加上“BEGIN”和“END”这样的字符串，但工具还是会搞混。
6 个回答
2
你可以看看substack的difflet。它既是一个node.js模块，也是一个命令行工具，正好可以完成这个功能：
4
我之前写了一个工具，可以对JSON文件进行统一的差异比较，可能会对你有点帮助。
https://github.com/jclulow/jsondiff
这个工具的输入和输出示例可以在GitHub页面上找到。
22
如果你的工具有这个选项的话，Patience Diff 可能会对你更有效。我会试着找一个有这个功能的工具（除了 Git 和 Bazaar），然后再告诉你结果。
补充：看起来Bazaar 中的实现可以稍微改动一下就作为独立工具使用。
补充2：哎，为什么不把你让我修改的新酷 diff 脚本的源代码贴出来呢？在这里，没什么版权问题，这只是Bram/Canonical的代码重新整理了一下。
#!/usr/bin/env python
# Copyright (C) 2005, 2006, 2007 Canonical Ltd
# Copyright (C) 2005 Bram Cohen, Copyright (C) 2005, 2006 Canonical Ltd
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
import os
import sys
import time
import difflib
from bisect import bisect
__all__ = ['PatienceSequenceMatcher', 'unified_diff', 'unified_diff_files']
py3k = False
try:
xrange
except NameError:
py3k = True
xrange = range
# This is a version of unified_diff which only adds a factory parameter
# so that you can override the default SequenceMatcher
# this has been submitted as a patch to python
def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
tofiledate='', n=3, lineterm='\n',
sequencematcher=None):
r"""
Compare two sequences of lines; generate the delta as a unified diff.
Unified diffs are a compact way of showing line changes and a few
lines of context. The number of context lines is set by 'n' which
defaults to three.
By default, the diff control lines (those with ---, +++, or @@) are
created with a trailing newline. This is helpful so that inputs
created from file.readlines() result in diffs that are suitable for
file.writelines() since both the inputs and outputs have trailing
newlines.
For inputs that do not have trailing newlines, set the lineterm
argument to "" so that the output will be uniformly newline free.
The unidiff format normally has a header for filenames and modification
times. Any or all of these may be specified using strings for
'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'. The modification
times are normally expressed in the format returned by time.ctime().
Example:
>>> for line in unified_diff('one two three four'.split(),
... 'zero one tree four'.split(), 'Original', 'Current',
... 'Sat Jan 26 23:30:50 1991', 'Fri Jun 06 10:20:52 2003',
... lineterm=''):
... print line
--- Original Sat Jan 26 23:30:50 1991
+++ Current Fri Jun 06 10:20:52 2003
@@ -1,4 +1,4 @@
+zero
one
-two
-three
+tree
four
"""
if sequencematcher is None:
import difflib
sequencematcher = difflib.SequenceMatcher
if fromfiledate:
fromfiledate = '\t' + str(fromfiledate)
if tofiledate:
tofiledate = '\t' + str(tofiledate)
started = False
for group in sequencematcher(None,a,b).get_grouped_opcodes(n):
if not started:
yield '--- %s%s%s' % (fromfile, fromfiledate, lineterm)
yield '+++ %s%s%s' % (tofile, tofiledate, lineterm)
started = True
i1, i2, j1, j2 = group[0][3], group[-1][4], group[0][5], group[-1][6]
yield "@@ -%d,%d +%d,%d @@%s" % (i1+1, i2-i1, j1+1, j2-j1, lineterm)
for tag, i1, i2, j1, j2 in group:
if tag == 'equal':
for line in a[i1:i2]:
yield ' ' + line
continue
if tag == 'replace' or tag == 'delete':
for line in a[i1:i2]:
yield '-' + line
if tag == 'replace' or tag == 'insert':
for line in b[j1:j2]:
yield '+' + line
def unified_diff_files(a, b, sequencematcher=None):
"""Generate the diff for two files.
"""
mode = 'rb'
if py3k: mode = 'r'
# Should this actually be an error?
if a == b:
return []
if a == '-':
file_a = sys.stdin
time_a = time.time()
else:
file_a = open(a, mode)
time_a = os.stat(a).st_mtime
if b == '-':
file_b = sys.stdin
time_b = time.time()
else:
file_b = open(b, mode)
time_b = os.stat(b).st_mtime
# TODO: Include fromfiledate and tofiledate
return unified_diff(file_a.readlines(), file_b.readlines(),
fromfile=a, tofile=b,
sequencematcher=sequencematcher)
def unique_lcs_py(a, b):
"""Find the longest common subset for unique lines.
:param a: An indexable object (such as string or list of strings)
:param b: Another indexable object (such as string or list of strings)
:return: A list of tuples, one for each line which is matched.
[(line_in_a, line_in_b), ...]
This only matches lines which are unique on both sides.
This helps prevent common lines from over influencing match
results.
The longest common subset uses the Patience Sorting algorithm:
http://en.wikipedia.org/wiki/Patience_sorting
"""
# set index[line in a] = position of line in a unless
# a is a duplicate, in which case it's set to None
index = {}
for i in xrange(len(a)):
line = a[i]
if line in index:
index[line] = None
else:
index[line]= i
# make btoa[i] = position of line i in a, unless
# that line doesn't occur exactly once in both,
# in which case it's set to None
btoa = [None] * len(b)
index2 = {}
for pos, line in enumerate(b):
next = index.get(line)
if next is not None:
if line in index2:
# unset the previous mapping, which we now know to
# be invalid because the line isn't unique
btoa[index2[line]] = None
del index[line]
else:
index2[line] = pos
btoa[pos] = next
# this is the Patience sorting algorithm
# see http://en.wikipedia.org/wiki/Patience_sorting
backpointers = [None] * len(b)
stacks = []
lasts = []
k = 0
for bpos, apos in enumerate(btoa):
if apos is None:
continue
# as an optimization, check if the next line comes at the end,
# because it usually does
if stacks and stacks[-1] < apos:
k = len(stacks)
# as an optimization, check if the next line comes right after
# the previous line, because usually it does
elif stacks and stacks[k] < apos and (k == len(stacks) - 1 or
stacks[k+1] > apos):
k += 1
else:
k = bisect(stacks, apos)
if k > 0:
backpointers[bpos] = lasts[k-1]
if k < len(stacks):
stacks[k] = apos
lasts[k] = bpos
else:
stacks.append(apos)
lasts.append(bpos)
if len(lasts) == 0:
return []
result = []
k = lasts[-1]
while k is not None:
result.append((btoa[k], k))
k = backpointers[k]
result.reverse()
return result
def recurse_matches_py(a, b, alo, blo, ahi, bhi, answer, maxrecursion):
"""Find all of the matching text in the lines of a and b.
:param a: A sequence
:param b: Another sequence
:param alo: The start location of a to check, typically 0
:param ahi: The start location of b to check, typically 0
:param ahi: The maximum length of a to check, typically len(a)
:param bhi: The maximum length of b to check, typically len(b)
:param answer: The return array. Will be filled with tuples
indicating [(line_in_a, line_in_b)]
:param maxrecursion: The maximum depth to recurse.
Must be a positive integer.
:return: None, the return value is in the parameter answer, which
should be a list
"""
if maxrecursion < 0:
print('max recursion depth reached')
# this will never happen normally, this check is to prevent DOS attacks
return
oldlength = len(answer)
if alo == ahi or blo == bhi:
return
last_a_pos = alo-1
last_b_pos = blo-1
for apos, bpos in unique_lcs_py(a[alo:ahi], b[blo:bhi]):
# recurse between lines which are unique in each file and match
apos += alo
bpos += blo
# Most of the time, you will have a sequence of similar entries
if last_a_pos+1 != apos or last_b_pos+1 != bpos:
recurse_matches_py(a, b, last_a_pos+1, last_b_pos+1,
apos, bpos, answer, maxrecursion - 1)
last_a_pos = apos
last_b_pos = bpos
answer.append((apos, bpos))
if len(answer) > oldlength:
# find matches between the last match and the end
recurse_matches_py(a, b, last_a_pos+1, last_b_pos+1,
ahi, bhi, answer, maxrecursion - 1)
elif a[alo] == b[blo]:
# find matching lines at the very beginning
while alo < ahi and blo < bhi and a[alo] == b[blo]:
answer.append((alo, blo))
alo += 1
blo += 1
recurse_matches_py(a, b, alo, blo,
ahi, bhi, answer, maxrecursion - 1)
elif a[ahi - 1] == b[bhi - 1]:
# find matching lines at the very end
nahi = ahi - 1
nbhi = bhi - 1
while nahi > alo and nbhi > blo and a[nahi - 1] == b[nbhi - 1]:
nahi -= 1
nbhi -= 1
recurse_matches_py(a, b, last_a_pos+1, last_b_pos+1,
nahi, nbhi, answer, maxrecursion - 1)
for i in xrange(ahi - nahi):
answer.append((nahi + i, nbhi + i))
def _collapse_sequences(matches):
"""Find sequences of lines.
Given a sequence of [(line_in_a, line_in_b),]
find regions where they both increment at the same time
"""
answer = []
start_a = start_b = None
length = 0
for i_a, i_b in matches:
if (start_a is not None
and (i_a == start_a + length)
and (i_b == start_b + length)):
length += 1
else:
if start_a is not None:
answer.append((start_a, start_b, length))
start_a = i_a
start_b = i_b
length = 1
if length != 0:
answer.append((start_a, start_b, length))
return answer
def _check_consistency(answer):
# For consistency sake, make sure all matches are only increasing
next_a = -1
next_b = -1
for (a, b, match_len) in answer:
if a < next_a:
raise ValueError('Non increasing matches for a')
if b < next_b:
raise ValueError('Non increasing matches for b')
next_a = a + match_len
next_b = b + match_len
class PatienceSequenceMatcher_py(difflib.SequenceMatcher):
"""Compare a pair of sequences using longest common subset."""
_do_check_consistency = True
def __init__(self, isjunk=None, a='', b=''):
if isjunk is not None:
raise NotImplementedError('Currently we do not support'
' isjunk for sequence matching')
difflib.SequenceMatcher.__init__(self, isjunk, a, b)
def get_matching_blocks(self):
"""Return list of triples describing matching subsequences.
Each triple is of the form (i, j, n), and means that
a[i:i+n] == b[j:j+n]. The triples are monotonically increasing in
i and in j.
The last triple is a dummy, (len(a), len(b), 0), and is the only
triple with n==0.
>>> s = PatienceSequenceMatcher(None, "abxcd", "abcd")
>>> s.get_matching_blocks()
[(0, 0, 2), (3, 2, 2), (5, 4, 0)]
"""
# jam 20060525 This is the python 2.4.1 difflib get_matching_blocks
# implementation which uses __helper. 2.4.3 got rid of helper for
# doing it inline with a queue.
# We should consider doing the same for recurse_matches
if self.matching_blocks is not None:
return self.matching_blocks
matches = []
recurse_matches_py(self.a, self.b, 0, 0,
len(self.a), len(self.b), matches, 10)
# Matches now has individual line pairs of
# line A matches line B, at the given offsets
self.matching_blocks = _collapse_sequences(matches)
self.matching_blocks.append( (len(self.a), len(self.b), 0) )
if PatienceSequenceMatcher_py._do_check_consistency:
if __debug__:
_check_consistency(self.matching_blocks)
return self.matching_blocks
unique_lcs = unique_lcs_py
recurse_matches = recurse_matches_py
PatienceSequenceMatcher = PatienceSequenceMatcher_py
def main(args):
import optparse
p = optparse.OptionParser(usage='%prog [options] file_a file_b'
'\nFiles can be "-" to read from stdin')
p.add_option('--patience', dest='matcher', action='store_const', const='patience',
default='patience', help='Use the patience difference algorithm')
p.add_option('--difflib', dest='matcher', action='store_const', const='difflib',
default='patience', help='Use python\'s difflib algorithm')
algorithms = {'patience':PatienceSequenceMatcher, 'difflib':difflib.SequenceMatcher}
(opts, args) = p.parse_args(args)
matcher = algorithms[opts.matcher]
if len(args) != 2:
print('You must supply 2 filenames to diff')
return -1
for line in unified_diff_files(args[0], args[1], sequencematcher=matcher):
sys.stdout.write(line)
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
补充3：我还做了一个最简独立版本，这是Neil Fraser的Diff Match and Patch，我很想知道你用这个的结果和我的比较。再次声明，我没有版权要求。
补充4：我刚发现了DataDiff，这可能是另一个可以尝试的工具。
DataDiff 是一个库，可以提供 人类可读的 Python 数据 结构的差异对比。它可以处理序列 类型（列表、元组等）、集合和 字典。
字典和序列会在适用的情况下 递归地进行差异对比。