我有一个包含40列的数据集,正在其中5列上使用.apply(word_tokenize)
,比如so:df['token_column'] = df.column.apply(word_tokenize)
。
我只得到其中一列的TypeError,我们将此问题称为列
TypeError: expected string or bytes-like object
下面是完整的错误(去掉df和列名,以及pii),我对Python还不太熟悉,仍在试图找出错误消息的哪些部分是相关的:
TypeError Traceback (most recent call last)
<ipython-input-51-22429aec3622> in <module>()
----> 1 df['token_column'] = df.problem_column.apply(word_tokenize)
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds)
2353 else:
2354 values = self.asobject
-> 2355 mapped = lib.map_infer(values, f, convert=convert_dtype)
2356
2357 if len(mapped) and isinstance(mapped[0], Series):
pandas\_libs\src\inference.pyx in pandas._libs.lib.map_infer (pandas\_libs\lib.c:66440)()
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\__init__.py in word_tokenize(text, language, preserve_line)
128 :type preserver_line: bool
129 """
--> 130 sentences = [text] if preserve_line else sent_tokenize(text, language)
131 return [token for sent in sentences
132 for token in _treebank_word_tokenizer.tokenize(sent)]
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\__init__.py in sent_tokenize(text, language)
95 """
96 tokenizer = load('tokenizers/punkt/{0}.pickle'.format(language))
---> 97 return tokenizer.tokenize(text)
98
99 # Standard word tokenizer.
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in tokenize(self, text, realign_boundaries)
1233 Given a text, returns a list of the sentences in that text.
1234 """
-> 1235 return list(self.sentences_from_text(text, realign_boundaries))
1236
1237 def debug_decisions(self, text):
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in sentences_from_text(self, text, realign_boundaries)
1281 follows the period.
1282 """
-> 1283 return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
1284
1285 def _slices_from_text(self, text):
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in span_tokenize(self, text, realign_boundaries)
1272 if realign_boundaries:
1273 slices = self._realign_boundaries(text, slices)
-> 1274 return [(sl.start, sl.stop) for sl in slices]
1275
1276 def sentences_from_text(self, text, realign_boundaries=True):
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in <listcomp>(.0)
1272 if realign_boundaries:
1273 slices = self._realign_boundaries(text, slices)
-> 1274 return [(sl.start, sl.stop) for sl in slices]
1275
1276 def sentences_from_text(self, text, realign_boundaries=True):
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in _realign_boundaries(self, text, slices)
1312 """
1313 realign = 0
-> 1314 for sl1, sl2 in _pair_iter(slices):
1315 sl1 = slice(sl1.start + realign, sl1.stop)
1316 if not sl2:
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in _pair_iter(it)
310 """
311 it = iter(it)
--> 312 prev = next(it)
313 for el in it:
314 yield (prev, el)
C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py in _slices_from_text(self, text)
1285 def _slices_from_text(self, text):
1286 last_break = 0
-> 1287 for match in self._lang_vars.period_context_re().finditer(text):
1288 context = match.group() + match.group('after_tok')
1289 if self.text_contains_sentbreak(context):
TypeError: expected string or bytes-like object
这5列都是字符/字符串(在SQL Server、SAS中进行了验证,并使用.select_dtypes(include=[object]))
)。
为了更好的度量,我使用了.to_string()
来确保问题列除了字符串之外真的没有任何东西,但是我仍然得到错误。如果我单独处理列good_column1-good_column4继续工作,problem_column仍将生成错误。
我到处搜索,除了从集合中删除任何数字(我做不到,因为那些是有意义的),我没有找到任何额外的修复。
问题是DF中没有(NA)类型。试试这个:
试试看
这就是我得到想要结果的原因。
相关问题 更多 >
编程相关推荐