pd.read_csv一个750 MB的文本文件

Question

我使用pd.read_csv读取了一个750MB的文本文件，文本中前5行的字段比其余行的字段少。我所做的是：

ICP_CTCT_TRADER_MEP=pd.read_csv('EDA_CTCT_TRADER_MEP.txt', sep=',', header=None, skiprows=5, encoding='ISO-8859-1',engine='python')

t前5行：

0   1   2   3   4   5   6   7   8   9   ... 12  13  14  15  16  17  18  19  20  21
DET 0000000001CP630 METERSUMMARY    MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN NGCM    1   ... N   Y   2   N   NaN None    None    None    None    None
METERINSTALL    MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   1   ... HHR F   21/10/2011  21/10/2026  N   X   03/05/2023  N145361 90  NaN
METERCOMP   MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   211313791   ... Y   1   1   NGCS    NaN None    None    None    None    None
METERCHANNEL    MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   211313791   ... UN  24  kWh X   C   Y   NaN None    None    None
17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   211313791   ... CN  17  kWh X   C   Y   NaN None    None    None

自动标题来自第六列，前5列没有标题。我的解决方案是将其导出到.csv，然后将其重新导入，以获得每个列的正确标题，如下所示

ICP_CTCT_TRADER_MEP=ICP_CTCT_TRADER_MEP.dropna(thresh=200000, axis='columns') # to remove the NaN Columns
ICP_CTCT_TRADER_MEP.to_csv('ICP_CTCT_TRADER_MEP.csv') # to export it to a .csv file
ICP_CTCT_TRADER_MEP_1=pd.read_csv('ICP_CTCT_TRADER_MEP.csv') #to import it back again

我的问题是：有没有更好的方法？我上面的方法需要很长时间，有时不起作用，因为内存错误而失败