Theano中的熵和概率

Question

我写了一个简单的Python代码来计算一个集合的熵，现在我想用Theano来实现同样的功能。

import math

# this computes the probabilities of each element in the set
def prob(values):
    return [float(values.count(v))/len(values) for v in values]

# this computes the entropy
def entropy(values):
    p = prob(values)
    return -sum([v*math.log(v) for v in p])

我正在尝试在Theano中写出等效的代码，但我不太确定该怎么做：

import theano
import theano.tensor as T

v = T.vector('v') # I create a symbolic vector to represent my initial values
p = T.vector('p') # The same for the probabilities 

# this is my attempt to compute the probabilities which would feed vector p
theano.scan(fn=prob,outputs_info=p,non_sequences=v,n_steps=len(values))

# considering the previous step would work, the entropy is just
e = -T.sum(p*T.log(p))
entropy = theano.function([values],e)

不过，扫描的部分不太对，我遇到了很多错误。我不确定有没有简单的方法来计算一个向量的熵，还是说我需要在扫描函数上花更多的功夫。有什么想法吗？

Answer 1

除了nouiz提到的观点，P不应该被声明为T.vector，因为它是你计算值向量的结果。

另外，要计算像熵这样的东西，你不需要使用Scan（因为Scan会增加计算的负担，所以只有在没有其他方法或者为了减少内存使用时才应该使用）；你可以采用这样的方式：

values = T.vector('values')
nb_values = values.shape[0]

# For every element in 'values', obtain the total number of times
# its value occurs in 'values'.
# NOTE : I've done the broadcasting a bit more explicitly than
# needed, for clarity.
freqs = T.eq(values[:,None], values[None, :]).sum(0).astype("float32")

# Compute a vector containing, for every value in 'values', the
# probability of that value in the vector 'values'.
# NOTE : these probabilities do *not* sum to 1 because they do not
# correspond to the probability of every element in the vector 'values
# but to the probability of every value in 'values'. For instance, if
# 'values' is [1, 1, 0] then 'probs' will be [2/3, 2/3, 1/3] because the
# value 1 has probability 2/3 and the value 0 has probability 1/3 in
# values'.
probs = freqs / nb_values

entropy = -T.sum(T.log2(probs) / nb_values)
fct = theano.function([values], entropy)

# Will output 0.918296...
print fct([0, 1, 1])