内存问题

Question

我试图建立一个具有张量流的高斯RBM模型。但是程序会占用太多内存。在

高斯分布_百万卢比在

import tensorflow as tf
import math
import input_data
import numpy as np

def sample_prob(probs):
    return tf.nn.relu(
        tf.sign(
            probs - tf.random_uniform(tf.shape(probs))))

class RBM(object):
    """ represents a sigmoidal rbm """

    def __init__(self, name, input_size, output_size, gaussian_std_val=0.1):
        with tf.name_scope("rbm_" + name):
            self.weights = tf.Variable(
                tf.truncated_normal([input_size, output_size],
                    stddev=1.0 / math.sqrt(float(input_size))), name="weights")
            self.v_bias = tf.Variable(tf.zeros([input_size]), name="v_bias")
            self.h_bias = tf.Variable(tf.zeros([output_size]), name="h_bias")
            self.input = tf.placeholder("float", shape=[None, 784])

            #Gaussian
            def_a = 1/(np.sqrt(2)*gaussian_std_val)
            def_a = tf.constant(def_a, dtype=tf.float32)
            self.a = tf.Variable( tf.ones(shape=[input_size]) * def_a,
                                  name="a")


    def propup(self, visible):
        """ P(h|v) """
        return tf.nn.sigmoid(tf.matmul(visible, self.weights) + self.h_bias)

    def propdown(self, hidden):
        """ P(v|h) """
        # return tf.nn.sigmoid(tf.matmul(hidden, tf.transpose(self.weights)) + self.v_bias)
        return ( tf.matmul(hidden, tf.transpose(self.weights)) + self.v_bias ) / (2 * (self.a * self.a))

    def sample_h_given_v(self, v_sample):
        """ Generate a sample from the hidden layer """
        return sample_prob(self.propup(v_sample))

    def sample_v_given_h(self, h_sample):
        """ Generate a sample from the visible layer """
        return self.sample_gaussian(self.propdown(h_sample))

    def gibbs_hvh(self, h0_sample):
        """ A gibbs step starting from the hidden layer """
        v_sample = self.sample_v_given_h(h0_sample)
        h_sample = self.sample_h_given_v(v_sample)
        return [v_sample, h_sample]

    def gibbs_vhv(self, v0_sample):
        """ A gibbs step starting from the visible layer """
        h_sample = self.sample_h_given_v(v0_sample)
        v_sample = self.sample_v_given_h(h_sample)
        return  [h_sample, v_sample]

    def sample_gaussian(self, mean_field):
        return tf.random_normal(shape=tf.shape(mean_field),
                                mean=mean_field,
                                stddev=1.0 / (np.sqrt(2) * self.a))

    def cd1(self, learning_rate=0.1):
        " One step of contrastive divergence, with Rao-Blackwellization "
        h_start = self.sample_h_given_v(self.input)
        v_end = self.sample_v_given_h(h_start)
        h_end = self.sample_h_given_v(v_end)
        w_positive_grad = tf.matmul(tf.transpose(self.input), h_start)
        w_negative_grad = tf.matmul(tf.transpose(v_end), h_end)

        update_w = self.weights + (learning_rate * (w_positive_grad - w_negative_grad) / tf.to_float(tf.shape(self.input)[0]))

        update_vb = self.v_bias + (learning_rate * tf.reduce_mean(self.input - v_end, 0))

        update_hb = self.h_bias + (learning_rate * tf.reduce_mean(h_start - h_end, 0))

        return [update_w, update_vb, update_hb]

    def cal_err(self):
        err = self.input - self.gibbs_vhv(self.input)[1]
        return tf.reduce_mean(err * err)

试验_mnist.py在

^{pr2}$

输出是

run test_mnist.py Extracting MNIST_data/train-images-idx3-ubyte.gz Extracting MNIST_data/train-labels-idx1-ubyte.gz Extracting MNIST_data/t10k-images-idx3-ubyte.gz Extracting MNIST_data/t10k-labels-idx1-ubyte.gz I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: name: GeForce GTX 560 major: 2 minor: 1 memoryClockRate (GHz) 1.62 pciBusID 0000:01:00.0 Total memory: 1018.69MiB Free memory: 916.73MiB I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:684] Ignoring gpu device (device: 0, name: GeForce GTX 560, pci bus id: 0000:01:00.0) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.5. step: 0 0.0911714 0.0781856 0.0773076 0.0770751 0.0776582 0.0764748 0.0755164 0.0741131 0.0726497 0.0712237 0.0701839 0.0686315 0.0664856 0.0658309 0.0646239 0.0626652 0.0616178 0.0610061 0.0598332 0.0588843 0.0587477 0.0572056 0.0561556 0.0554848 Killed

有什么方法可以监视内存吗？ 有人能帮我吗？在

Answer 1

答案似乎是正确的，（计算能力不足，无法运行最新版本的CUDA/Tensorflow

然而，最低要求似乎是“计算能力=3.0”，因为我的GTX U 770M能够运行Tensorflow 1.0/CUDA8.0（见下文）

和/或尝试从源代码重新编译tensorflow，并在生成过程中包含2.0目标（默认情况下建议3.5-5.5）

祝你今天愉快！！在

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.51                 Driver Version: 375.51                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 770M    Off  | 0000:01:00.0     N/A |                  N/A |
|100%   48C    P0    N/A /  N/A |   2819MiB /  3017MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+

Answer 2

训练循环可能有问题，导致计算机内存不足。在

对于循环的每次迭代，您将调用：

sess.run(rbm_modle.cd1(), feed_dict={rbm_modle.input : trX[start : end]})

在这个rbm_modle.cd1()函数中，您正在创建几个新操作，例如tf.matmul()，因此每次调用rbm_modle.cd1()时，都会创建新的操作，这将导致每次迭代后使用的内存增加。在

您应该在循环之前定义所有操作，然后在使用sess.run()运行操作期间定义所有操作，而不创建新操作。在

Answer 3

您可以使用命令nvidia-smi监视GPU内存

看起来你的GPU不支持运行tensorflow所需的CUDA的更新版本。您可以检查CUDA-Enabled GeForce Products

从你的输出看来，tensorflow足够聪明，不能使用GPU，所以要么你的型号/批次太大，要么内存泄漏。在

尝试使用log_device_placement=True运行运行会话，以查看tensorflow正在逐步执行的操作，同时运行“top”来监视内存？在

    with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess: