使用pandas read\u cs时跳过0xff字节

Question

我试图从我的锅炉中读取一些日志文件，但它们的格式相当糟糕。在

当我试图用

import pandas

print(pandas.read_csv('./data/CM120102.CSV', delimiter=';'))

我明白了

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 49: invalid start byte

由于某种原因，csv头以空字节结尾。在

https://gist.github.com/Ession/6e5bf67392276048c7bd

http://mathiasjost.com/CM120102.CSV<；==这个应该有效（或者说不起作用）

有没有办法不先修复就用熊猫读这些文件？在

Answer 1

我会把它读成一串。然后用python咀嚼一下，然后把它传递给熊猫.read_csv. 下面是示例代码。在

# get the data as a python string
with open ("CM120102.CSV", "r") as myfile:
    data=myfile.read()

# munge in python - get rid of the garbage in the input (lots of xff bytes)
import re
data = re.sub(r'[^a-zA-Z0-9_\.;:\n]', '', data) # get rid of the rubbish
data = data + '\n' # the very last one is missing?
data = re.sub(r';\n', r'\n', data) # last ; separator on line is problematic

# now let's suck into a pandas DataFrame
from StringIO import StringIO
import pandas as pd
df = pd.read_csv(StringIO(data), index_col=None, header=0,
    skipinitialspace=True, sep=';', parse_dates=True)