有没有更快的方法将Pandas数据帧处理成大的csv？

Question

我想让下面的代码更快地导出到csv（平均文件大小800MB）包含100+列。 ................................................................... ,................................................................. ................................................................. 你知道吗

            INPUT:

            DATE_TIME;          DATA1;        DATA2         
            12.18.2018 00:00:00;XXXXXXXXXXXX;YYYYYYYY   
            12.18.2018 00:00:00;XXXXXXXXXXXX;YYYYYYYY   
            12.18.2018 00:00:00;XXXXXXXXXXXX;YYYYYYYY   
            12.18.2018 01:00:00;XXXXXXXXXXXX;YYYYYYYY   

            OUTPUT will be no header:

            DATE,        TIME,   DATA1,      DATA2      
            2018-12-18,00:00:00,XXXXXXXXXXXX,YYYYYYYY   
            2018-12-18,00:00:00,XXXXXXXXXXXX,YYYYYYYY   
            2018-12-18,00:00:00,XXXXXXXXXXXX,YYYYYYYY   
            2018-12-18,01:00:00,XXXXXXXXXXXX,YYYYYYYY 

CODE

import pandas
import glob

dfraw = []  

rawCSV = glob.glob('C:\RAW\*.csv')

rawCSV

for filename in rawCSV:
    data = pandas.read_csv(filename, delimiter=';')
    dfraw.append(data)

totalFile = len(dfraw)
x=0
dfclean=[]
for x in range(totalFile):
    tempdf =dfraw[x]["DATE_TIME"].str.split(" ", n = 1, expand = True)
    tempdf[0] = tempdf[0].str.replace('.','-')
    dfraw[x].drop(columns =["DATE_TIME"], inplace = True)
    dfraw[x].insert(loc=0, column='DATE_ONLY', value=tempdf[0])
    dfraw[x].insert(loc=1, column='TIME_ONLY', value=tempdf[1])
    dfraw[x]['DATE_TIME'] = dfraw[x]['DATE_TIME'].astype('datetime64[ns]')
    dfclean.append(dfraw[x])
concatdf=pandas.concat(dfclean, axis=0)

#dfclean.columns = dfclean.iloc[0]
#dfclean = dfclean[1:]
concatdf.to_csv('C:\CLEAN\__result.csv', index=False , header=False)

Answer 1

这可能是你能得到的最快的了；我认为它应该有用。它在读取文件的同时将文件写出来，而不是将所有内容都堆在内存中直到最后再执行concat（这有点慢）

import pandas
import glob

rawCSV = glob.glob('C:\RAW\*.csv')

for filename in rawCSV:
    data = pandas.read_csv(filename, delimiter=';')
    date_time = data['DATE_TIME'].str.split(" ", n = 1, expand = True)
    data.drop(columns =["DATE_TIME"], inplace = True)
    data.insert(loc=0, column='DATE_ONLY', value=date_time[0].str.replace('.','-'))
    data.insert(loc=1, column='TIME_ONLY', value=date_time[1])
    with open('C:\CLEAN\__result.csv', 'a') as fh:
        data.to_csv(fh, index=False , header=False)

可能值得做一些检查，以确保您没有附加到已经存在的文件等等。你知道吗