大数据集计算的优化算法

Question

我又一次发现自己被熊猫困住了，以及如何最好地执行“向量操作”。我的代码可以工作，但是遍历所有内容需要很长时间。你知道吗

代码试图做的是循环遍历shapes.cv，确定哪个shape_pt_sequence是stop_id，然后将stop_lat和stop_lon分配给shape_pt_lat和shape_pt_lon，同时将shape_pt_sequence标记为is_stop。你知道吗

吉斯特

stop_times.csvLINK

trips.csvLINK

shapes.csvLINK

这是我的密码：

import pandas as pd
from haversine import *

'''
iterate through shapes and match stops along a shape_pt_sequence within
x amount of distance. for shape_pt_sequence that is closest, replace the stop
lat/lon to the shape_pt_lat/shape_pt_lon, and mark is_stop column with 1.
'''

# readability assignments for shapes.csv
shapes = pd.read_csv('csv/shapes.csv')
shapes_index = list(set(shapes['shape_id']))
shapes_index.sort(key=int)
shapes.set_index(['shape_id', 'shape_pt_sequence'], inplace=True)

# readability assignments for trips.csv
trips = pd.read_csv('csv/trips.csv')
trips_index = list(set(trips['trip_id']))
trips.set_index(['trip_id'], inplace=True)

# readability assignments for stops_times.csv
stop_times = pd.read_csv('csv/stop_times.csv')
stop_times.set_index(['trip_id','stop_sequence'], inplace=True)
print(len(stop_times.loc[1423492]))

# readability assginments for stops.csv
stops = pd.read_csv('csv/stops.csv')
stops.set_index(['stop_id'], inplace=True)

# for each trip_id
for i in trips_index:
    print('******NEW TRIP_ID******')
    print(i)
    i = i.astype(int)

    # for each stop_sequence in stop_times
    for x in range(len(stop_times.loc[i])):
        stop_lat = stop_times.loc[i,['stop_lat','stop_lon']].iloc[x,[0,1]][0]
        stop_lon = stop_times.loc[i,['stop_lat','stop_lon']].iloc[x,[0,1]][1]
        stop_coordinate = (stop_lat, stop_lon)
        print(stop_coordinate)

        # shape_id that matches trip_id
        print('**SHAPE_ID**')
        trips_shape_id = trips.loc[i,['shape_id']].iloc[0]
        trips_shape_id = int(trips_shape_id)
        print(trips_shape_id)

        smallest = 0

        for y in range(len(shapes.loc[trips_shape_id])):
            shape_lat = shapes.loc[trips_shape_id].iloc[y,[0,1]][0]
            shape_lon = shapes.loc[trips_shape_id].iloc[y,[0,1]][1]

            shape_coordinate = (shape_lat, shape_lon)

            haversined = haversine_mi(stop_coordinate, shape_coordinate)

            if smallest == 0 or haversined < smallest:
                smallest = haversined
                smallest_shape_pt_indexer = y
            else:
                pass

            print(haversined)
            print('{0:.20f}'.format(smallest))

        print('{0:.20f}'.format(smallest))
        print(smallest_shape_pt_indexer)

        # mark is_stop as 1
        shapes.iloc[smallest_shape_pt_indexer,[2]] = 1

        # replace coordinate value
        shapes.loc[trips_shape_id].iloc[y,[0,1]][0] = stop_lat
        shapes.loc[trips_shape_id].iloc[y,[0,1]][1] = stop_lon

shapes.to_csv('csv/shapes.csv', index=False)

Answer 1

需要考虑的一点是，一组行程通常具有相同的停车顺序和相同的形状数据（行程之间的唯一区别是计时）。因此，在（stop\u id，shape\u id）的shape操作中缓存find closest point可能是有意义的。我打赌这会减少你的运行时间一个数量级。你知道吗

Answer 2

要优化这段代码，您可以使用一些线程/工作线程来代替这些线程/工作线程。你知道吗

我建议使用Pool of Workes，因为它使用起来非常简单。你知道吗

在：

for i in trips_index:

你可以用这样的方法：

from multiprocessing import Pool

pool = Pool(processes=4)
result = pool.apply_async(func, trips_index)

方法func如下：

def func(i):
   #code here

您可以简单地将整个for循环放在这个方法中。 在本例中，它可以与4个子进程一起工作，git是一个很好的改进。你知道吗