在Python中使用Whoosh进行模糊字符串搜索

12 投票
5 回答
6745 浏览
提问于 2025-04-16 21:37

我在MongoDB里建立了一个很大的银行数据库。我可以很轻松地用这些信息在whoosh中创建索引。比如,我想把银行名称'Eagle Bank & Trust Co of Missouri'和'Eagle Bank and Trust Company of Missouri'进行匹配。下面的代码可以处理简单的模糊搜索,但无法匹配上面这两个名称:

from whoosh.index import create_in
from whoosh.fields import *

schema = Schema(name=TEXT(stored=True))
ix = create_in("indexdir", schema)
writer = ix.writer()

test_items = [u"Eagle Bank and Trust Company of Missouri"]

writer.add_document(name=item)
writer.commit()

from whoosh.qparser import QueryParser
from whoosh.query import FuzzyTerm

with ix.searcher() as s:
    qp = QueryParser("name", schema=ix.schema, termclass=FuzzyTerm)
    q = qp.parse(u"Eagle Bank & Trust Co of Missouri")
    results = s.search(q)
    print results

这段代码给我的结果是:

<Top 0 Results for And([FuzzyTerm('name', u'eagle', boost=1.000000, minsimilarity=0.500000, prefixlength=1), FuzzyTerm('name', u'bank', boost=1.000000, minsimilarity=0.500000, prefixlength=1), FuzzyTerm('name', u'trust', boost=1.000000, minsimilarity=0.500000, prefixlength=1), FuzzyTerm('name', u'co', boost=1.000000, minsimilarity=0.500000, prefixlength=1), FuzzyTerm('name', u'missouri', boost=1.000000, minsimilarity=0.500000, prefixlength=1)]) runtime=0.00166392326355>

我想知道,使用Whoosh能实现我想要的匹配吗?如果不能,还有什么其他基于Python的解决方案可以选择?

5 个回答

1

也许这些东西对你有帮助(这是由seatgeek团队开源的字符串匹配工具):

https://github.com/seatgeek/fuzzywuzzy

3

为了将来参考,可能还有更好的方法来做到这一点,不过这是我想到的办法。

# -*- coding: utf-8 -*-
import whoosh
from whoosh.index import create_in
from whoosh.fields import *
from whoosh.query import *
from whoosh.qparser import QueryParser

schema = Schema(name=TEXT(stored=True))
idx = create_in("C:\\idx_name\\", schema, "idx_name")

writer = idx.writer()

writer.add_document(name=u"This is craaazy shit")
writer.add_document(name=u"This is craaazy beer")
writer.add_document(name=u"Raphaël rocks")
writer.add_document(name=u"Rockies are mountains")

writer.commit()

s = idx.searcher()
print "Fields: ", list(s.lexicon("name"))
qp = QueryParser("name", schema=schema, termclass=FuzzyTerm)

for i in range(1,40):
    res = s.search(FuzzyTerm("name", "just rocks", maxdist=i, prefixlength=0))
    if len(res) > 0:
        for r in res:
            print "Potential match ( %s ): [  %s  ]" % ( i, r["name"] )
        break
    else:
        print "Pass: %s" % i

s.close()
11

你可以用Whoosh的模糊搜索把CoCompany匹配起来,但不建议这样做,因为CoCompany之间的差别很大。就像CoCompany的关系,BeBeast也有相似之处,而nyCompany的关系也是如此。你可以想象,这样的搜索结果会有多糟糕和多混乱。

不过,如果你想把CompancompaniCompaneeCompany匹配起来,你可以使用FuzzyTerm的个性化类,设置默认的maxdist为2或更高:

maxdist – 指的是给定文本的最大编辑距离。

class MyFuzzyTerm(FuzzyTerm):
     def __init__(self, fieldname, text, boost=1.0, maxdist=2, prefixlength=1, constantscore=True):
         super(MyFuzzyTerm, self).__init__(fieldname, text, boost, maxdist, prefixlength, constantscore)

然后:

 qp = QueryParser("name", schema=ix.schema, termclass=MyFuzzyTerm)

你可以把CoCompany匹配起来,方法是把maxdist设置为5,但正如我所说,这样会得到很糟糕的搜索结果。我建议把maxdist保持在13之间。

如果你想匹配一个单词的语言变体,最好使用whoosh.query.Variations

注意:旧版本的Whoosh使用minsimilarity来代替maxdist

撰写回答