读取分块的大制表符分隔文件时出现异常

Question

我有一个350MB的标签分隔文本文件。如果我试图把它读入内存，就会出现内存不足的异常。所以我试着沿着这些思路（即，只在几列专栏中阅读）：

import pandas as pd

input_file_and_path = r'C:\Christian\ModellingData\X.txt'

column_names = [
    'X1'
    # , 'X2
]
raw_data = pd.DataFrame()
for chunk in pd.read_csv(input_file_and_path, names=column_names, chunksize=1000, sep='\t'):
    raw_data = pd.concat([raw_data, chunk], ignore_index=True)

print(raw_data.head())

不幸的是，我知道：

^{pr2}$

在处理上述异常时，发生了另一个异常：

Traceback (most recent call last):
  File "C:/xxxx/EdaDataPrepRange1.py", line 17, in <module>
    for chunk in pd.read_csv(input_file_and_path, header=None, names=column_names, chunksize=1000, sep='\t'):
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1007, in __next__
    return self.get_chunk()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1070, in get_chunk
    return self.read(nrows=size)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1036, in read
    ret = self._engine.read(nrows)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1848, in read
    data = self._reader.read(nrows)
  File "pandas\_libs\parsers.pyx", line 876, in pandas._libs.parsers.TextReader.read
  File "pandas\_libs\parsers.pyx", line 903, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas\_libs\parsers.pyx", line 968, in pandas._libs.parsers.TextReader._read_rows
  File "pandas\_libs\parsers.pyx", line 1094, in pandas._libs.parsers.TextReader._convert_column_data
  File "pandas\_libs\parsers.pyx", line 1141, in pandas._libs.parsers.TextReader._convert_tokens
  File "pandas\_libs\parsers.pyx", line 1240, in pandas._libs.parsers.TextReader._convert_with_dtype
  File "pandas\_libs\parsers.pyx", line 1256, in pandas._libs.parsers.TextReader._string_convert
  File "pandas\_libs\parsers.pyx", line 1494, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 5: invalid start byte

有什么想法吗。顺便说一句，我一般如何处理大文件和插补，例如缺失的变量？最后，我需要阅读所有的东西来确定，例如，要插补的中值。在

Answer 1

对于大文件问题，只需使用文件处理程序和上下文管理器：

with open("your_file.txt") as fileObject:
    for line in fileObject:
        do_something_with(line)

## No need to close file as 'with' automatically does that

这不会将整个文件加载到内存中。相反，它将一次加载一行，并且将“忘记”以前的行，除非您存储引用。在

另外，关于编码问题，只需在使用pd.read_csv时使用encoding="utf-8"。在