如何正確使用RNN中的Dropout

Question

用神经网络来改善我的神经功能。 不过，我还是按照下面的指南来实施的。所以我不确定我是否犯了什么错误。但是如果我用10个时代来训练我的模型，在验证之后我得到的结果几乎相同。这就是为什么我不确定是否正确地使用了dropout函数。在下面的代码中这是正确的实现还是我做错了什么？如果我什么都做对了为什么我得到的结果几乎一样呢？在

hm_epochs = 10
n_classes = 2
batch_size = 128
chunk_size = 341
n_chunks = 5
rnn_size = 32

dropout_prop = 0.5 # Dropout, probability to drop a unit

batch_size_validation = 65536
x = tf.placeholder('float', [None, n_chunks, chunk_size])
y = tf.placeholder('float')

def recurrent_neural_network(x):
     layer = {'weights':tf.Variable(tf.random_normal([rnn_size, n_classes])),
              'biases':tf.Variable(tf.random_normal([n_classes]))}

     x = tf.transpose(x, [1,0,2])
     x = tf.reshape(x, [-1, chunk_size])
     x = tf.split(x, n_chunks, 0)

     lstm_cell = rnn.BasicLSTMCell(rnn_size) 
     outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)

     output = tf.matmul(outputs[-1], layer['weights']) + layer['biases']

     #DROPOUT Implementation -> is this code really working?
     #The result is nearly the same after 20 epochs...
     output_layer = tf.layers.dropout(output, rate=dropout_prop)

     return output

def train_neural_network(x):
     prediction = recurrent_neural_network(x)
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=prediction,labels=y))
     optimizer = tf.train.AdamOptimizer().minimize(cost)

     with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())

         for epoch in range(1,hm_epochs+1):
             epoch_loss = 0

             for i in range(0, training_data.shape[0], batch_size):
                 epoch_x = np.array(training_data[i:i+batch_size, :, :], dtype='float')
                 epoch_y = np.array(training_labels[i:i+batch_size, :], dtype='float')

                 if len(epoch_x) != batch_size:
                     epoch_x = epoch_x.reshape((len(epoch_x), n_chunks, chunk_size))
                 else:
                     epoch_x = epoch_x.reshape((batch_size, n_chunks, chunk_size))

                 _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y}) 
                 epoch_loss += c

 train_neural_network(x)
 print("rnn - finished!")

Answer 1

在其最基本的形式中，脱落应该发生在单元内部，并应用于权重。你只是事后才用的。This article用一些很好的可视化和很少的变化很好地解释了它。在

要在代码中使用它，您可以

1. 实现自己的RNN单元，其中keep probability是初始化单元的参数，或者是每次调用时传入的参数。

2. 使用rnn dropout包装here。