Pandas只删除连续的重复行，忽略特定列

Question

我在下面有一个数据框

df = pd.DataFrame({
    'ID': ['James', 'James', 'James', 'James',
           'Max', 'Max', 'Max', 'Max', 'Max',
           'Park', 'Park','Park', 'Park',
           'Tom', 'Tom', 'Tom', 'Tom'],
    'From_num': [578, 420, 420, 'Started', 298, 78, 36, 298, 'Started', 28, 28, 311, 'Started', 60, 520, 99, 'Started'],
    'To_num': [96, 578, 578, 420, 36, 298, 78, 36, 298, 112, 112, 28, 311, 150, 60, 520, 99],
    'Date': ['2020-05-12', '2020-02-02', '2020-02-01', '2019-06-18',
             '2019-08-26', '2019-06-20', '2019-01-30', '2018-10-23',
             '2018-08-29', '2020-05-21', '2020-05-20', '2019-11-22',
             '2019-04-12', '2019-10-16', '2019-08-26', '2018-12-11', '2018-10-09']})

它是这样的：

       ID From_num  To_num        Date
0   James      578      96  2020-05-12
1   James      420     578  2020-02-02
2   James      420     578  2020-02-01 # Drop the this duplicated row (ignore date)
3   James  Started     420  2019-06-18
4     Max      298      36  2019-08-26
5     Max       78     298  2019-06-20
6     Max       36      78  2019-01-30
7     Max      298      36  2018-10-23
8     Max  Started     298  2018-08-29
9    Park       28     112  2020-05-21
10   Park       28     112  2020-05-20 # Drop this duplicate row (ignore date)
11   Park      311      28  2019-11-22
12   Park  Started     311  2019-04-12
13    Tom       60     150  2019-10-16
14    Tom      520      60  2019-08-26
15    Tom       99     520  2018-12-11
16    Tom  Started      99  2018-10-09

每个“ID”（名称）中都有一些连续的重复值（忽略日期值），例如，对于James，第1行和第2行的From_num都是420，与第9行和第10行相同，我希望删除第2个重复行并保留第一个重复行。我写了循环条件，但它是非常冗余和缓慢的，我想可能有更简单的方法来做到这一点，所以请帮助，如果你有想法。非常感谢。预期结果如下：

       ID  From_num  To_num    Date
0   James      578      96  2020-05-12
1   James      420     578  2020-02-02
2   James  Started     420  2019-06-18
3     Max      298      36  2019-08-26
4     Max       78     298  2019-06-20
5     Max       36      78  2019-01-30
6     Max      298      36  2018-10-23
7     Max  Started     298  2018-08-29
8    Park       28     112  2020-05-21
9    Park      311      28  2019-11-22
10   Park  Started     311  2019-04-12
11    Tom       60     150  2019-10-16
12    Tom      520      60  2019-08-26
13    Tom       99     520  2018-12-11
14    Tom  Started      99  2018-10-09

Answer 1

有点晚了，但这是你想要的吗？这将删除连续的重复项，忽略“日期”

t = df[['ID', 'From_num', 'To_num']]     
df[(t.ne(t.shift())).any(axis=1)]

       ID From_num  To_num        Date
0   James      578      96  2020-05-12
1   James      420     578  2020-02-02
3   James  Started     420  2019-06-18
4     Max      298      36  2019-08-26
5     Max       78     298  2019-06-20
6     Max       36      78  2019-01-30
7     Max      298      36  2018-10-23
8     Max  Started     298  2018-08-29
9    Park       28     112  2020-05-21
11   Park      311      28  2019-11-22
12   Park  Started     311  2019-04-12
13    Tom       60     150  2019-10-16
14    Tom      520      60  2019-08-26
15    Tom       99     520  2018-12-11
16    Tom  Started      99  2018-10-09

这将删除索引值为2和10的行

Answer 2

在我看来，这正是DataFrame.drop_duplicates所做的，默认情况下，它保留第一次出现，并删除其余的

unique_df = df.drop_duplicates(['ID', 'From_num', 'To_num'])

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html

编辑

正如问题中提到的，只应处理连续的行，为此，我建议先标记它们，然后在标记行的子集上运行drop_duplicates（我不确定这是否是最佳解决方案）

df['original_index'] = null
indices = df.index[1:]
for i in range(1, indices):
    # if current row equals the previous one
    if df.loc[indices[i - 1], 'ID'] == df.loc[indices[i], 'ID'] and df.loc[indices[i -1], 'From_num'] == df.loc[indices[i], 'From_num'] and df.loc[indices[i -1], 'To_num'] == df.loc[indices[i], 'To_num']:
        # get the original index if it has been already set on row index -1
        if df.loc[indices[i - 1], 'original_index'] not null:
            df.loc[indices[i], 'original_index'] = df.loc[indices[i - 1], 'original_index']
        else:
            # else set it to be current index for both rows
            df.loc[indices[i - 1], 'original_index'] = indices[i - 1]
            df.loc[indices[i], 'original_index'] = indices[i - 1]

现在我们添加“原始索引”列以删除重复项

unique_df = df.drop_duplicates(['ID', 'From_num', 'To_num', 'original_index'])

Answer 3

将下面的行与上面的行进行比较，反转布尔值以获得结果：

cond1 = df.ID.eq(df.ID.shift())
cond2 = df.From_num.eq(df.From_num.shift())
cond = cond1 & cond2

df.loc[~cond].reset_index(drop=True)

备选方案：较长路线：

(
    df.assign(
        temp=df.groupby(["ID", "From_num"]).From_num.transform("size"),
        check=lambda x: (x.From_num.eq(x.From_num.shift())) &
                        (x.temp.eq(x.temp.shift())),
    )
    .query("check == 0")
    .drop(["temp", "check"], axis=1)
)