Pandas测向at有些台词非常慢

Question

我有一个.txt日志文件与IMU传感器测量需要解析成.CSV文件。加速度计、陀螺仪有500Hz的ODR（输出数据率）、100Hz的磁强计、gps 1Hz和baro 1Hz。Wi-fi、BLE、pressure、light等也有记录，但大多数都不需要。智能手机应用程序不会按顺序保存所有测量值。你知道吗

将一个包含20多万行的文件解析为一个数据帧需要1000多秒，根据时间戳对数据帧进行排序并将其保存为csv文件。你知道吗

当在数据帧中的坐标（行=时间戳，列=传感器测量）上分配传感器测量值时，有些需要约40%的运行时间，而有些则需要+-0.1%的运行时间。你知道吗

为什么会这样？ 不需要1000多秒。。

---

日志文件中的内容：

ACCE;AppTimestamp(s);SensorTimestamp(s);Acc_X(m/s^2);Acc_Y(m/s^2);Acc_Z(m/s^2);Accuracy(integer)
GYRO;AppTimestamp(s);SensorTimestamp(s);Gyr_X(rad/s);Gyr_Y(rad/s);Gyr_Z(rad/s);Accuracy(integer)
MAGN;AppTimestamp(s);SensorTimestamp(s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Accuracy(integer)
MAGN;AppTimestamp(s);SensorTimestamp(s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Accuracy(integer)
PRES;AppTimestamp(s);SensorTimestamp(s);Pres(mbar);Accuracy(integer)
LIGH;AppTimestamp(s);SensorTimestamp(s);Light(lux);Accuracy(integer)
PROX;AppTimestamp(s);SensorTimestamp(s);prox(?);Accuracy(integer)
HUMI;AppTimestamp(s);SensorTimestamp(s);humi(Percentage);Accuracy(integer)
TEMP;AppTimestamp(s);SensorTimestamp(s);temp(Celsius);Accuracy(integer)
AHRS;AppTimestamp(s);SensorTimestamp(s);PitchX(deg);RollY(deg);YawZ(deg);RotVecX();RotVecY();RotVecZ();Accuracy(int)
GNSS;AppTimestamp(s);SensorTimeStamp(s);Latit(deg);Long(deg);Altitude(m);Bearing(deg);Accuracy(m);Speed(m/s);SatInView;SatInUse
WIFI;AppTimestamp(s);SensorTimeStamp(s);Name_SSID;MAC_BSSID;RSS(dBm);
BLUE;AppTimestamp(s);Name;MAC_Address;RSS(dBm);
BLE4;AppTimestamp(s);MajorID;MinorID;RSS(dBm);
SOUN;AppTimestamp(s);RMS;Pressure(Pa);SPL(dB);
RFID;AppTimestamp(s);ReaderNumber(int);TagID(int);RSS_A(dBm);RSS_B(dBm);
IMUX;AppTimestamp(s);SensorTimestamp(s);Counter;Acc_X(m/s^2);Acc_Y(m/s^2);Acc_Z(m/s^2);Gyr_X(rad/s);Gyr_Y(rad/s);Gyr_Z(rad/s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Roll(deg);Pitch(deg);Yaw(deg);Quat(1);Quat(2);Quat(3);Quat(4);Pressure(mbar);Temp(Celsius)
IMUL;AppTimestamp(s);SensorTimestamp(s);Counter;Acc_X(m/s^2);Acc_Y(m/s^2);Acc_Z(m/s^2);Gyr_X(rad/s);Gyr_Y(rad/s);Gyr_Z(rad/s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Roll(deg);Pitch(deg);Yaw(deg);Quat(1);Quat(2);Quat(3);Quat(4);Pressure(mbar);Temp(Celsius)
POSI;Timestamp(s);Counter;Latitude(degrees); Longitude(degrees);floor ID(0,1,2..4);Building ID(0,1,2..3)

原始.txt日志文件的一部分：

MAGN;1.249;343268.933;2.64000;-97.50000;-69.06000;0
GYRO;1.249;343268.934;0.02153;0.06943;0.09880;3
ACCE;1.249;343268.934;-0.24900;0.53871;9.59625;3 GNSS;1.250;1570711878.000;52.225976;5.174543;58.066;175.336;3.0;0.0;23;20
ACCE;1.253;343268.936;-0.26576;0.52674;9.58428;3
GYRO;1.253;343268.936;0.00809;0.06515;0.10002;3
ACCE;1.253;343268.938;-0.29450;0.49561;9.57710;3
GYRO;1.253;343268.938;0.00015;0.06088;0.10613;3
PRES;1.253;343268.929;1011.8713;3
GNSS;1.254;1570711878.000;52.225976;5.174543;58.066;175.336;3.0;0.0;23;20
ACCE;1.255;343268.940;-0.29450;0.49801;9.57710;3
GYRO;1.255;343268.940;-0.00596;0.05843;0.10979;3
ACCE;1.260;343268.942;-0.30647;0.50280;9.55795;3
GYRO;1.261;343268.942;-0.01818;0.05721;0.11529;3
MAGN;1.262;343268.943;2.94000;-97.74000;-68.88000;0

---

fileContent是txt文件的字符串，如上所示。你知道吗

一段代码：

def parseValues(line):
    valArr = []
    valArr = np.fromstring(line[5:], dtype=float, sep=";")

    return (valArr)

i = 0
while i < len(fileContent):
    if (fileContent[i][:4] == "ACCE"):
        vals = parseValues(fileContent[i])
        idx = vals[1] - initialSensTS
        df.at[idx, 'ax'] = vals[2]
        df.at[idx, 'ay'] = vals[3]
        df.at[idx, 'az'] = vals[4]
        df.at[idx, 'accStat'] = vals[5]
        i += 1

---

代码可以工作，但在某些方面非常慢测向at[idx，'xx']行。

见第28行。你知道吗

测线轮廓仪输出：

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
22         1          1.0      1.0      0.0      i = 0
23    232250     542594.0      2.3      0.0      while i < len(fileContent):
24    232249  294337000.0   1267.3     23.8          update_progress(i / len(fileContent))
25    232249     918442.0      4.0      0.1          if (fileContent[i][:4] == "ACCE"):
26     54602    1584625.0     29.0      0.1              vals = parseValues(fileContent[i])
27     54602     316968.0      5.8      0.0              idx = vals[1] - initialSensTS
28     54602  504189480.0   9233.9     40.8              df.at[idx, 'ax'] = vals[2]
29     54602    8311109.0    152.2      0.7              df.at[idx, 'ay'] = vals[3]
30     54602    4901983.0     89.8      0.4              df.at[idx, 'az'] = vals[4]
31     54602    4428239.0     81.1      0.4              df.at[idx, 'accStat'] = vals[5]
32     54602     132590.0      2.4      0.0              i += 1

Answer 1

这并没有解决有关排序时间戳等问题的部分，但应该是'ACCE'解析代码的有效替代。你知道吗

import pandas as pd
import collections as colls

logs_file_path = '../resources/imu_logs_raw.txt'

msmt_type_dict = colls.defaultdict(list)

with open(logs_file_path, 'r') as file_1:
    for line in file_1:
        curr_measure_type, *rest_str = line.split(';')
        rest_str[-1] = rest_str[-1].strip()
        msmt_type_dict[curr_measure_type].append(rest_str)

acce_df = pd.DataFrame(data=msmt_type_dict['ACCE'], columns=['app_timestamp', 'sensor_timestamp', 'acc_x', 'acc_y', 'acc_z', 'accuracy'])

如果您能提供更多的信息/上下文，我想看看时间戳排序方面。你知道吗