Seq2Seq LSTM学习不当

Question

我正在用Pytorch解决一个序列到序列的问题，使用的是LSTM。具体来说，我是用5个元素的序列来预测接下来的5个元素。我的困扰主要是关于数据的转换。我有一些张量，大小是[bs, seq_length, features]，其中seq_length = 10，features = 1。每个特征都是一个介于0到3之间的整数（包括0和3）。

我原以为输入数据需要用MinMaxScaler转换成浮点数范围[0, 1]，这样可以让LSTM的学习过程更顺利。之后，我会应用一个线性层，把隐藏状态转换成对应的输出，输出的大小是features。这是我在Pytorch中定义的LSTM网络：

class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, num_layers, dropout_prob):
        super(LSTM, self).__init__()
        self.lstm_layer = nn.LSTM(input_dim, hidden_dim, num_layers, dropout=dropout_prob)
        self.output_layer = nn.Linear(hidden_dim, output_dim)
    ...
    def forward(self, X):
        out, (hidden, cell) = self.lstm_layer(X)
        out = self.output_layer(out)
        return out

我用来进行训练循环的代码如下：

def train_loop(t, checkpoint_epoch, dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, X in enumerate(dataloader):
        X = X[0].type(torch.float).to(device)
        # X = torch.Size([batch_size, 10, input_dim])
        # Split sequences into input and target
        inputs = transform(X[:, :5, :]) # inputs = [batch_size, 5, input_dim]
        targets = transform(X[:, 5:, :]) # targets = [batch_size, 5, input_dim]
        # predictions (forward pass)
        with autocast():
            pred = model(inputs)  # pred = [batch_size, 5, input_dim]
            loss = loss_fn(pred, targets)
        # backprop
        optimizer.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            #print(f"Current loss: {loss:>7f}, [{current:>5d}/{size:>5d}]")
        # Delete variables and empty cache
        del X, inputs, targets, pred
        torch.cuda.empty_cache()
    return loss

我用来预处理数据的代码是：

def main():
    num_agents = 2
    # Open the HDF5 file
    with h5py.File('dataset_' + str(num_agents) + 'UAV.hdf5', 'r') as f:
        # Access the dataset
        data = f['data'][:]
        # Convert to PyTorch tensor
        data_tensor = torch.tensor(data)
        size = data_tensor.size()
        seq_length = 10
        reshaped = data_tensor.view(-1, size[2], size[3])
        r_size = reshaped.size()
        reshaped = reshaped[:, :, 1:]
        reshaped_v2 = reshaped.view(r_size[0], -1)
        dataset = create_dataset(reshaped_v2.numpy(), seq_length)
        f.close()
    dataset = TensorDataset(dataset)
    # Split the dataset into training and validation sets
    train_size = int(0.8 * len(dataset))  # 80% for training
    val_size = len(dataset) - train_size  # 20% for validation
    train_dataset, val_dataset = random_split(dataset, [train_size, val_size])
    train_dataloader = DataLoader(train_dataset, batch_size=params['batch_size'], shuffle=True, pin_memory=True)
    val_dataloader = DataLoader(val_dataset, batch_size=params['batch_size'], shuffle=False, pin_memory=True)

尝试这样做后，模型的学习效果不太好，所以我在想，可能直接计算targets（范围在[0, 1]的浮点值）和pred（我认为是因为LSTM层的tanh激活函数而导致的范围在[-1, 1]的浮点值）之间的损失，可能会因为尺度不同而出问题。于是，我尝试在前向传播中在线性层后面加一个sigmoid激活函数，但效果也不好。我尝试了很多超参数组合，但都没有得到“正常”的训练曲线。我还附上了一张5000个训练周期的截图，以展示训练过程：

我有以下几个问题：

我的训练过程有什么问题吗？
我说的有什么地方理解错了吗？

数据预处理 pytorch lstm 超参数调优 seq2seq 激活函数训练过程线性层

Seq2Seq LSTM学习不当

1 个回答

撰写回答