如何为句子列表创建窗口/区块?

2024-03-29 11:01:09 发布

您现在位置:Python中文网/ 问答频道 /正文

我有一个句子列表,我想创建skipgram(window size = 3),但我不想让计数器跨越句子,因为它们都是不相关的。你知道吗

所以,如果我有以下句子:

[["my name is John"] , ["This PC is black"]]

三胞胎是:

[my name is]
[name is john]
[this PC is]
[PC is black]

最好的方法是什么?你知道吗


Tags: name列表sizeismy计数器windowthis
3条回答

下面是一个简单的函数。你知道吗

def skipgram(corpus, window_size = 3):
    sg = []
    for sent in corpus:
        sent = sent[0].split()
        if len(sent) <= window_size:
            sg.append(sent)
        else:
            for i in range(0, len(sent)-window_size+1):
                sg.append(sent[i: i+window_size])
    return sg

corpus = [["my name is John"] , ["This PC is black"]]
skipgram(corups)

你并不真的想要一个skipgram本身,但是你想要一个按大小划分的块,试试这个:

from lazyme import per_chunk

tokens = "my name is John".split()
list(per_chunk(tokens, 2))

[输出]:

[('my', 'name'), ('is', 'John')]

如果需要滚动窗口,即ngrams

from lazyme import per_window

tokens = "my name is John".split()
list(per_window(tokens, 2))

[输出]:

[('my', 'name'), ('name', 'is'), ('is', 'John')]

与ngrams的NLTK类似:

from nltk import ngrams

tokens = "my name is John".split()
list(ngrams(tokens, 2))

[输出]:

[('my', 'name'), ('name', 'is'), ('is', 'John')]

如果你想要实际的技巧图,How to compute skipgrams in python?

from nltk import skipgrams

tokens = "my name is John".split()
list(skipgrams(tokens, n=2, k=1))

[输出]:

[('my', 'name'),
 ('my', 'is'),
 ('name', 'is'),
 ('name', 'John'),
 ('is', 'John')]

试试这个!你知道吗

from nltk import ngrams

def generate_ngrams(sentences,window_size =3):
    for sentence in sentences:
        yield from ngrams(sentence[0].split(), window_size)

sentences= [["my name is John"] , ["This PC is black"]]

for c in generate_ngrams(sentences,3):
    print (c)

#output:
('my', 'name', 'is')
('name', 'is', 'John')
('This', 'PC', 'is')
('PC', 'is', 'black')

相关问题 更多 >