手动标记的Span中具有ENT_类型的图案不工作

Question

作为完成此任务的替代方案：Patterns with multi-terms entries in the IN attribute

我编写了以下代码来匹配短语、标记它们，然后在EntityRuler模式中使用它们：

# %%
import spacy
from spacy.matcher import PhraseMatcher
from spacy.pipeline import EntityRuler
from spacy.tokens import Span

class PhraseRuler(object):
    name = 'phrase_ruler'

    def __init__(self, nlp, terms, label):
        patterns = [nlp(term) for term in terms]
        self.matcher = PhraseMatcher(nlp.vocab)
        self.matcher.add(label, None, *patterns)

    def __call__(self, doc):
        matches = self.matcher(doc)
        spans = []
        for label, start, end in matches:
            span = Span(doc, start, end, label=label)
            spans.append(span)
        doc.ents = spans
        return doc

nlp = spacy.load("en_core_web_lg")

entity_matcher = PhraseRuler(nlp, ["Best Wishes", "Warm Welcome"], "GREETING")
nlp.add_pipe(entity_matcher, before="ner")


ruler = EntityRuler(nlp)
patterns = [{"label": "SUPER_GREETING", "pattern": [{"LOWER": "super"}, {"ENT_TYPE": "GREETING"}]}]
ruler.add_patterns(patterns)
#ruler.to_disk("./data/patterns.jsonl")
nlp.add_pipe(ruler)

print(nlp.pipe_names) 

doc = nlp("Mary said Best Wishes and I said super Warm Welcome.")
print(doc.to_json())

不幸的是，这不起作用，因为它不返回mySUPER_GREETING：

'ents': [
   {'start': 0, 'end': 4, 'label': 'PERSON'}, 
   {'start': 10, 'end': 21, 'label': 'GREETING'}, 
   {'start': 39, 'end': 51, 'label': 'GREETING'}
]

我做错了什么？我怎么修理它

Answer 1

您的想法是正确的，但这里的问题是spaCy中固有的设计选择，即任何令牌只能是一个命名实体的一部分。所以你不能让“热烈欢迎”既是“问候”又是“超级问候”的一部分

解决这个问题的一种方法是使用custom extensions。例如，一种解决方案是将问候语位存储在令牌级别：

Token.set_extension("mylabel", default="")

然后我们调整PhraseRuler.__call__，使其不写入doc.ents，而是执行以下操作：

for token in span:
    token._.mylabel = "MY_GREETING"

现在，我们可以将SUPER_问候语模式改写为：

patterns = [{"label": "SUPER_GREETING", "pattern": [{"LOWER": "super"}, {"_": {"mylabel": "MY_GREETING"}, "OP": "+"}]}]

它将匹配“super”，后跟一个或多个“MY_问候”标记。它将贪婪地匹配，并在命中时输出“超级热烈欢迎”

下面是生成的代码片段，从您的代码开始，按照所述进行调整：

    Token.set_extension("mylabel", default="")

    class PhraseRuler(object):
        name = 'phrase_ruler'

        def __init__(self, nlp, terms, label):
            patterns = [nlp(term) for term in terms]
            self.matcher = PhraseMatcher(nlp.vocab)
            self.matcher.add(label, None, *patterns)

        def __call__(self, doc):
            matches = self.matcher(doc)
            for label, start, end in matches:
                span = Span(doc, start, end, label=label)
                for token in span:
                    token._.mylabel = "MY_GREETING"
            return doc

    nlp = spacy.load("en_core_web_lg")

    entity_matcher = PhraseRuler(nlp, ["Best Wishes", "Warm Welcome"], "GREETING")
    nlp.add_pipe(entity_matcher, name="entity_matcher", before="ner")

    ruler = EntityRuler(nlp)
    patterns = [{"label": "SUPER_GREETING", "pattern": [{"LOWER": "super"}, {"_": {"mylabel": "MY_GREETING"}, "OP": "+"}]}]
    ruler.add_patterns(patterns)
    nlp.add_pipe(ruler, after="entity_matcher")

    print(nlp.pipe_names)

    doc = nlp("Mary said Best Wishes and I said super Warm Welcome.")
    print("TOKENS:")
    for token in doc:
        print(token.text, token._.mylabel)
    print()

    print("ENTITIES:")
    for ent in doc.ents:
        print(ent.text, ent.label_)

哪个输出

TOKENS:
Mary 
said 
Best MY_GREETING
Wishes MY_GREETING
and 
I 
said 
super 
Warm MY_GREETING
Welcome MY_GREETING
. 

ENTITIES:
Mary PERSON
super Warm Welcome SUPER_GREETING

这可能并不完全是您需要/想要的-但我希望它能帮助您为您的特定用例提供替代解决方案。如果您确实希望在最后的doc.ents中使用正常的“问候语”跨度，那么您可以在运行EntityRuler之后在后处理中重新组合它们，例如，如果自定义属性不重叠，则将其移动到doc.ents，或者将spans的缓存保留在某个位置