我正在尝试标记和解析文本,这些文本已经被拆分成句子,并且已经被标记化。例如:
sents = [['I', 'like', 'cookies', '.'], ['Do', 'you', '?']]
处理批量文本的最快方法是.pipe()
。但是,我不清楚如何将其用于预标记化和预分段文本。在这里,性能是关键。我尝试了下面的方法,但出现了一个错误
跟踪:
Traceback (most recent call last):
File "C:\Python\Python37\Lib\multiprocessing\pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "C:\Python\projects\PreDicT\predicting-wte\build_id_dictionary.py", line 204, in process_batch
self.nlp.tagger(docs)
File "pipes.pyx", line 377, in spacy.pipeline.pipes.Tagger.__call__
File "pipes.pyx", line 396, in spacy.pipeline.pipes.Tagger.predict
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\model.py", line 169, in __call__
return self.predict(x)
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\feed_forward.py", line 40, in predict
X = layer(X)
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\model.py", line 169, in __call__
return self.predict(x)
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\model.py", line 133, in predict
y, _ = self.begin_update(X, drop=None)
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\feature_extracter.py", line 14, in begin_update
features = [self._get_feats(doc) for doc in docs]
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\feature_extracter.py", line 14, in <listcomp>
features = [self._get_feats(doc) for doc in docs]
File "C:\Users\bmvroy\.virtualenvs\predicting-wte-YKqW76ba\lib\site-packages\thinc\neural\_classes\feature_extracter.py", line 21, in _get_feats
arr = doc.doc.to_array(self.attrs)[doc.start : doc.end]
AttributeError: 'list' object has no attribute 'doc'
只需将管道中的默认标记器替换为
nlp.tokenizer.tokens_from_list
,而不是单独调用它:输出:
^{pr2}$相关问题 更多 >
编程相关推荐