带相邻权重/惩罚的莱文斯坦距离
我正在使用字符串编辑距离（也叫Levenshtein距离）来比较眼动实验中的扫描路径。（现在我在R语言中使用的是stringdist这个包）
简单来说，这些字符串中的字母代表的是一个6x4的矩阵中的（注视）位置。这个矩阵的配置如下：
[,1] [,2] [,3] [,4]
[1,] 'a' 'g' 'm' 's'
[2,] 'b' 'h' 'n' 't'
[3,] 'c' 'i' 'o' 'u'
[4,] 'd' 'j' 'p' 'v'
[5,] 'e' 'k' 'q' 'w'
[6,] 'f' 'l' 'r' 'x'
如果我用基本的Levenshtein距离来比较字符串，那么在字符串中比较a和g的结果和比较a和x的结果是一样的。
比如：
'abc' compared to 'agc' -> 1
'abc' compared to 'axc' -> 1
这意味着这些字符串在相似度上是一样的。
我希望能够在字符串比较中加入权重，以便考虑矩阵中的相邻关系。比如，a和x之间的距离应该被认为比a和g之间的距离更大。
一种方法是计算从一个字母到另一个字母在矩阵中的“步数”（水平和垂直的移动），然后除以最大“步数”（也就是从a到x的步数）。比如，从a到g的“步数”是1，而从a到x的“步数”是8，这样就得到了权重1/8和1。
有没有办法在R或者Python中实现这个呢？
4 个回答
0
还有一个处理权重的选项（适用于Python 3.5），我和它没有任何关系，链接是 https://github.com/luozhouyang/python-string-similarity
pip install strsim
3
看看这个库：https://github.com/infoscout/weighted-levenshtein（声明：我是这个库的作者）。它支持加权的Levenshtein距离、加权的最佳字符串对齐和加权的Damerau-Levenshtein距离。这个库是用Cython写的，性能非常好，可以通过 pip install weighted-levenshtein 很方便地安装。欢迎大家反馈和提交改进建议。
示例用法：
import numpy as np
from weighted_levenshtein import lev
insert_costs = np.ones(128, dtype=np.float64) # make an array of all 1's of size 128, the number of ASCII characters
insert_costs[ord('D')] = 1.5 # make inserting the character 'D' have cost 1.5 (instead of 1)
# you can just specify the insertion costs
# delete_costs and substitute_costs default to 1 for all characters if unspecified
print lev('BANANAS', 'BANDANAS', insert_costs=insert_costs) # prints '1.5'
5
如果有人遇到同样的“问题”，这里有我的解决办法。我为Kyle Gorman写的Wagner-Fischer算法的Python实现做了一个扩展。
这个扩展是一个权重函数，以及在_dist函数中实现了这个权重函数。
#!/usr/bin/env python
# wagnerfischer.py: Dynamic programming Levensthein distance function
# Kyle Gorman <gormanky@ohsu.edu>
#
# Based on:
#
# Robert A. Wagner and Michael J. Fischer (1974). The string-to-string
# correction problem. Journal of the ACM 21(1):168-173.
#
# The thresholding function was inspired by BSD-licensed code from
# Babushka, a Ruby tool by Ben Hoskings and others.
#
# Unlike many other Levenshtein distance functions out there, this works
# on arbitrary comparable Python objects, not just strings.
try: # use numpy arrays if possible...
from numpy import zeros
def _zeros(*shape):
""" like this syntax better...a la MATLAB """
return zeros(shape)
except ImportError: # otherwise do this cute solution
def _zeros(*shape):
if len(shape) == 0:
return 0
car = shape[0]
cdr = shape[1:]
return [_zeros(*cdr) for i in range(car)]
def weight(A,B, weights):
if weights == True:
from numpy import matrix
from numpy import where
# cost_weight defines the matrix structure of the AOI-placement
cost_weight = matrix([["a","b","c","d","e","f"],["g","h","i","j","k","l"],
["m","n","o","p","q","r"],["s","t","u","v","w","x"]])
max_walk = 8.00 # defined as the maximum posible distance between letters in
# the cost_weight matrix
indexA = where(cost_weight==A)
indexB = where(cost_weight==B)
walk = abs(indexA[0][0]-indexB[0][0])+abs(indexA[1][0]-indexB[1][0])
w = walk/max_walk
return w
else:
return 1
def _dist(A, B, insertion, deletion, substitution, weights=True):
D = _zeros(len(A) + 1, len(B) + 1)
for i in xrange(len(A)):
D[i + 1][0] = D[i][0] + deletion * weight(A[i],B[0], weights)
for j in xrange(len(B)):
D[0][j + 1] = D[0][j] + insertion * weight(A[0],B[j], weights)
for i in xrange(len(A)): # fill out middle of matrix
for j in xrange(len(B)):
if A[i] == B[j]:
D[i + 1][j + 1] = D[i][j] # aka, it's free.
else:
D[i + 1][j + 1] = min(D[i + 1][j] + insertion * weight(A[i],B[j], weights),
D[i][j + 1] + deletion * weight(A[i],B[j], weights),
D[i][j] + substitution * weight(A[i],B[j], weights))
return D
def _dist_thresh(A, B, thresh, insertion, deletion, substitution):
D = _zeros(len(A) + 1, len(B) + 1)
for i in xrange(len(A)):
D[i + 1][0] = D[i][0] + deletion
for j in xrange(len(B)):
D[0][j + 1] = D[0][j] + insertion
for i in xrange(len(A)): # fill out middle of matrix
for j in xrange(len(B)):
if A[i] == B[j]:
D[i + 1][j + 1] = D[i][j] # aka, it's free.
else:
D[i + 1][j + 1] = min(D[i + 1][j] + insertion,
D[i][j + 1] + deletion,
D[i][j] + substitution)
if min(D[i + 1]) >= thresh:
return
return D
def _levenshtein(A, B, insertion, deletion, substitution):
return _dist(A, B, insertion, deletion, substitution)[len(A)][len(B)]
def _levenshtein_ids(A, B, insertion, deletion, substitution):
"""
Perform a backtrace to determine the optimal path. This was hard.
"""
D = _dist(A, B, insertion, deletion, substitution)
i = len(A)
j = len(B)
ins_c = 0
del_c = 0
sub_c = 0
while True:
if i > 0:
if j > 0:
if D[i - 1][j] <= D[i][j - 1]: # if ins < del
if D[i - 1][j] < D[i - 1][j - 1]: # if ins < m/s
ins_c += 1
else:
if D[i][j] != D[i - 1][j - 1]: # if not m
sub_c += 1
j -= 1
i -= 1
else:
if D[i][j - 1] <= D[i - 1][j - 1]: # if del < m/s
del_c += 1
else:
if D[i][j] != D[i - 1][j - 1]: # if not m
sub_c += 1
i -= 1
j -= 1
else: # only insert
ins_c += 1
i -= 1
elif j > 0: # only delete
del_c += 1
j -= 1
else:
return (ins_c, del_c, sub_c)
def _levenshtein_thresh(A, B, thresh, insertion, deletion, substitution):
D = _dist_thresh(A, B, thresh, insertion, deletion, substitution)
if D != None:
return D[len(A)][len(B)]
def levenshtein(A, B, thresh=None, insertion=1, deletion=1, substitution=1):
"""
Compute levenshtein distance between iterables A and B
"""
# basic checks
if len(A) == len(B) and A == B:
return 0
if len(B) > len(A):
(A, B) = (B, A)
if len(A) == 0:
return len(B)
if thresh:
if len(A) - len(B) > thresh:
return
return _levenshtein_thresh(A, B, thresh, insertion, deletion,
substitution)
else:
return _levenshtein(A, B, insertion, deletion, substitution)
def levenshtein_ids(A, B, insertion=1, deletion=1, substitution=1):
"""
Compute number of insertions deletions, and substitutions for an
optimal alignment.
There may be more than one, in which case we disfavor substitution.
"""
# basic checks
if len(A) == len(B) and A == B:
return (0, 0, 0)
if len(B) > len(A):
(A, B) = (B, A)
if len(A) == 0:
return len(B)
else:
return _levenshtein_ids(A, B, insertion, deletion, substitution)
6
你需要一个版本的Wagner-Fisher算法，这个版本在内部循环中使用的不是固定的成本。也就是说，通常算法中加的+1，你要改成+del_cost(a[i])，等等。同时，你需要定义del_cost、ins_cost和sub_cost这几个函数，它们可以接收一个或两个符号（可能只是查表而已）。