如何在测试和训练中分别使用时间分割数据

Question

如何使用基于时间的拆分将数据拆分成列车并进行测试。

我知道火车测试是随机分开的，如何根据时间来分开。

  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42) 
  # this splits the data randomly as 67% test and 33% train

如何根据67%列车和33%试验的时间分割相同的数据集？

数据集具有列时间戳。

我试着寻找类似的问题，但不确定方法。

有人能简单解释一下吗

Answer 1

如果您有一个简单的数据集，其中每一行都是一个观测值（例如，分类问题的非时间序列数据集），并且您希望将其拆分为train和test，则此函数将基于一列日期拆分为train和test：

import pandas as pd
import numpy as np
from math import ceil

def train_test_split_sorted(X, y, test_size, dates):
"""Splits X and y into train and test sets, with test set separated by most recent dates.

    Example:
    --------
    >>> from sklearn import datasets

    # Fake dataset:
    >>> gen_data = datasets.make_classification(n_samples=10000, n_features=5)
    >>> dates = np.array(pd.date_range('2016-01-01', periods=10000, freq='5min'))
    >>> np.random.shuffle(dates)
    >>> df = pd.DataFrame(gen_data[0])
    >>> df['date'] = dates
    >>> df['target'] = gen_data[1]

    # Separate:
    >>> X_train, X_test, y_train, y_test = train_test_split_sorted(df.drop('target', axis=1), df['target'], 0.33, df['date'])

    >>> print('Length train set: {}'.format(len(y_train)))
    Length train set: 8000
    >>> print('Length test set: {}'.format(len(y_test)))
    Length test set: 2000
    >>> print('Last date in train set: {}'.format(X_train['date'].max()))
    Last date in train set: 2016-01-28 18:35:00
    >>> print('First date in test set: {}'.format(X_test['date'].min()))
    First date in test set: 2016-01-28 18:40:00
    """

    n_test = ceil(test_size * len(X))

    sorted_index = [x for _, x in sorted(zip(np.array(dates), np.arange(0, len(dates))), key=lambda pair: pair[0])]
    train_idx = sorted_index[:-n_test]
    test_idx = sorted_index[-n_test:]

    if isinstance(X, (pd.Series, pd.DataFrame)):
        X_train = X.iloc[train_idx]
        X_test = X.iloc[test_idx]
    else:
        X_train = X[train_idx]
        X_test = X[test_idx]
    if isinstance(y, (pd.Series, pd.DataFrame)):
        y_train = y.iloc[train_idx]
        y_test = y.iloc[test_idx]
    else:
        y_train = y[train_idx]
        y_test = y[test_idx]

    return X_train, X_test, y_train, y_test

参数dates实际上可以是任何类型的数组或序列，您可以使用它们对数据进行排序。

在您的例子中，您应该调用：X_train, X_test, y_train, y_test = train_test_split_sorted(X, y, 0.333, TimeStamp)，其中TimeStamp是数组或列，您可以在其中获得有关每个观察的时间戳的信息。